455 commits

Author	SHA1	Message	Date
Starbeamrainbowlabs	ad156a9a00	ai dataset: centre crop the water data to 75% original size ... this should both help the model and reduce memory usage	2022-09-02 18:05:32 +01:00
Starbeamrainbowlabs	efe41b96ec	fixup	2022-09-02 17:58:02 +01:00
Starbeamrainbowlabs	f8ee0afca1	ai: fix arch_name plumbing	2022-09-02 17:57:07 +01:00
Starbeamrainbowlabs	3d44831080	Add NO_PREFETCH env var	2022-09-02 17:55:04 +01:00
Starbeamrainbowlabs	3e0ca6a315	ai: fix summary file writing; make water encoder smaller	2022-09-02 17:51:45 +01:00
Starbeamrainbowlabs	389216b391	reordering	2022-09-02 17:31:19 +01:00
Starbeamrainbowlabs	b9d01ddadc	summary logger → summarywriter	2022-09-02 17:28:00 +01:00
Starbeamrainbowlabs	9f7f4af784	dataset: properly resize	2022-09-02 17:09:38 +01:00
Starbeamrainbowlabs	c78b6562ef	debug adjustment	2022-09-02 17:00:21 +01:00
Starbeamrainbowlabs	c89677abd7	dataset: explicit reshape	2022-09-02 16:57:59 +01:00
Starbeamrainbowlabs	c066ea331b	dataset: namespace → dict ... Python is so annoying	2022-09-02 16:07:44 +01:00
Starbeamrainbowlabs	dbff15d4a9	add bug comment	2022-09-01 19:05:50 +01:00
Starbeamrainbowlabs	3e4128c0a8	resize rainfall to be 1/2 size of current	2022-09-01 18:47:07 +01:00
Starbeamrainbowlabs	8a86728b54	ai: how did I miss that	2022-09-01 18:07:53 +01:00
Starbeamrainbowlabs	b2f1ba29bb	debug a	2022-09-01 18:05:33 +01:00
Starbeamrainbowlabs	e2f621e212	even moar debug logging	2022-09-01 17:49:51 +01:00
Starbeamrainbowlabs	2663123407	moar logging	2022-09-01 17:40:23 +01:00
Starbeamrainbowlabs	c2fcb3b954	ai: channels_first → channels_last	2022-09-01 17:06:18 +01:00
Starbeamrainbowlabs	f1d7973f22	ai: add dummy label	2022-09-01 17:01:00 +01:00
Starbeamrainbowlabs	17d42fe899	ai: json.dumps	2022-09-01 16:21:52 +01:00
Starbeamrainbowlabs	940f7aa1b5	ai: set self.model	2022-09-01 16:20:23 +01:00
Starbeamrainbowlabs	f1be5fe2bd	ai: add missing arguments to LossContrastive	2022-09-01 16:14:00 +01:00
Starbeamrainbowlabs	ddb375e906	ai: another typo	2022-09-01 16:12:05 +01:00
Starbeamrainbowlabs	cb3e1d3a23	ai: fixup	2022-09-01 16:10:50 +01:00
Starbeamrainbowlabs	e4c95bc7e3	typo	2022-09-01 16:09:24 +01:00
Starbeamrainbowlabs	cfbbe8e8cf	ai: global? really?	2022-09-01 16:06:24 +01:00
Starbeamrainbowlabs	4952ead094	ai: try the nonlocal keyword? I'm unsure what's going on here ... ....clearly I need to read up on scoping in python.	2022-09-01 16:02:37 +01:00
Starbeamrainbowlabs	1d9d0fbb73	requirements.txt:a dd tensorflow_addons	2022-09-01 15:56:02 +01:00
Starbeamrainbowlabs	8bdded23eb	ai: fix 'nother crash' name ConvNeXt submodels	2022-08-31 18:57:27 +01:00
Starbeamrainbowlabs	b2a320134e	ai: typo	2022-08-31 18:54:03 +01:00
Starbeamrainbowlabs	e4edc68df5	ai: add missing gamma layer	2022-08-31 18:52:35 +01:00
Starbeamrainbowlabs	51cf08a386	ResNetRSV2 → ConvNeXt ... ironically this makes the model simpler o/	2022-08-31 18:51:01 +01:00
Starbeamrainbowlabs	3d614d105b	channels first	2022-08-31 18:06:59 +01:00
Starbeamrainbowlabs	654eefd9ca	properly handle water dimensions; add log files to .gitignore ... TODO: add heightmap	2022-08-31 18:03:39 +01:00
Starbeamrainbowlabs	5846828f9e	debug logging	2022-08-31 17:48:09 +01:00
Starbeamrainbowlabs	12c77e128d	handle feature_dim properly	2022-08-31 17:41:51 +01:00
Starbeamrainbowlabs	c52a9f961c	and another	2022-08-31 17:37:28 +01:00
Starbeamrainbowlabs	58dbabd561	fix another crash	2022-08-31 17:33:07 +01:00
Starbeamrainbowlabs	5fc0725917	slurm: chmod +x	2022-08-31 16:33:07 +01:00
Starbeamrainbowlabs	dbf929325a	typo; add pretrain slurm job file	2022-08-31 16:32:17 +01:00
Starbeamrainbowlabs	e0162bc70b	requirements.txt: add missing dependencies	2022-08-31 16:25:47 +01:00
Starbeamrainbowlabs	f2312c1184	fix crash	2022-08-31 16:25:27 +01:00
Starbeamrainbowlabs	15a3519107	ai: the best thing about implementing a model is that you don't have to test it on the same day :P	2022-08-11 18:26:28 +01:00
Starbeamrainbowlabs	c0a9cb12d8	ai: start creating initial model implementation. ... it's not hooked up to the CLI yet though. Focus is still on ensuring the dataset is in the right format though	2022-08-10 19:03:25 +01:00
Starbeamrainbowlabs	b52c7f89a7	Move dataset parsing function to the right place	2022-08-10 17:24:55 +01:00
Starbeamrainbowlabs	222a6146ec	write glue for .jsonl.gz → .tfrecord.gz converter	2022-08-08 15:33:59 +01:00
Starbeamrainbowlabs	28a3f578d5	update .gitignore	2022-08-04 16:49:53 +01:00
Starbeamrainbowlabs	323d708692	dataset: add todo ... just why, Tensorflow?! tf.data.TextLineDataset looks almost too good to be true..... and it is, as despite supporting decompressing via gzip(!) it doesn't look like we can convince it to parse JSON :-/	2022-07-26 19:53:18 +01:00
Starbeamrainbowlabs	b53c77a2cb	index.py: call static function name run	2022-07-26 19:51:28 +01:00
Starbeamrainbowlabs	a7ed58fc03	ai: move requirements.txt to the right place	2022-07-26 19:25:11 +01:00
Starbeamrainbowlabs	e93a95f1b3	ai dataset: add if main == main	2022-07-26 19:24:40 +01:00
Starbeamrainbowlabs	de4c3dab17	typo	2022-07-26 19:14:55 +01:00
Starbeamrainbowlabs	18a7d3674b	ai: create (untested) dataset	2022-07-26 19:14:10 +01:00
Starbeamrainbowlabs	dac6919fcd	ai: start creating initial scaffolding	2022-07-25 19:01:10 +01:00
Starbeamrainbowlabs	8a9cd6c1c0	Lay out some basic scaffolding ... I *really* hope this works. This is the 3rd major revision of this model. I've learnt a ton of stuff between now and my last attempt, so here's hoping that all goes well :D The basic idea behind this attempt is *Contrastive Learning*. If we don't get anything useful with this approach, then we can assume that it's not really possible / feasible. Something we need to watch out for is the variance (or rather lack thereof) in the dataset. We have 1.5M timesteps, but not a whole lot will be happening in most of those.... We may need to analyse the variance of the water depth data and extract a subsample that's more balanced.	2022-05-13 19:06:15 +01:00