Starbeamrainbowlabs
e4c95bc7e3
typo
2022-09-01 16:09:24 +01:00
Starbeamrainbowlabs
cfbbe8e8cf
ai: global? really?
2022-09-01 16:06:24 +01:00
Starbeamrainbowlabs
4952ead094
ai: try the nonlocal keyword? I'm unsure what's going on here
...
....clearly I need to read up on scoping in python.
2022-09-01 16:02:37 +01:00
Starbeamrainbowlabs
1d9d0fbb73
requirements.txt:a dd tensorflow_addons
2022-09-01 15:56:02 +01:00
Starbeamrainbowlabs
8bdded23eb
ai: fix 'nother crash' name ConvNeXt submodels
2022-08-31 18:57:27 +01:00
Starbeamrainbowlabs
b2a320134e
ai: typo
2022-08-31 18:54:03 +01:00
Starbeamrainbowlabs
e4edc68df5
ai: add missing gamma layer
2022-08-31 18:52:35 +01:00
Starbeamrainbowlabs
51cf08a386
ResNetRSV2 → ConvNeXt
...
ironically this makes the model simpler o/
2022-08-31 18:51:01 +01:00
Starbeamrainbowlabs
3d614d105b
channels first
2022-08-31 18:06:59 +01:00
Starbeamrainbowlabs
654eefd9ca
properly handle water dimensions; add log files to .gitignore
...
TODO: add heightmap
2022-08-31 18:03:39 +01:00
Starbeamrainbowlabs
5846828f9e
debug logging
2022-08-31 17:48:09 +01:00
Starbeamrainbowlabs
12c77e128d
handle feature_dim properly
2022-08-31 17:41:51 +01:00
Starbeamrainbowlabs
c52a9f961c
and another
2022-08-31 17:37:28 +01:00
Starbeamrainbowlabs
58dbabd561
fix another crash
2022-08-31 17:33:07 +01:00
Starbeamrainbowlabs
5fc0725917
slurm: chmod +x
2022-08-31 16:33:07 +01:00
Starbeamrainbowlabs
dbf929325a
typo; add pretrain slurm job file
2022-08-31 16:32:17 +01:00
Starbeamrainbowlabs
e0162bc70b
requirements.txt: add missing dependencies
2022-08-31 16:25:47 +01:00
Starbeamrainbowlabs
f2312c1184
fix crash
2022-08-31 16:25:27 +01:00
Starbeamrainbowlabs
fe7a8b3fc0
Add 9 simple steps to use the rainfall radar model..... oh boy.
2022-08-31 16:18:28 +01:00
Starbeamrainbowlabs
15a3519107
ai: the best thing about implementing a model is that you don't have to test it on the same day :P
2022-08-11 18:26:28 +01:00
Starbeamrainbowlabs
28bcdf2192
vscode settings file
2022-08-11 16:38:25 +01:00
Starbeamrainbowlabs
c0a9cb12d8
ai: start creating initial model implementation.
...
it's not hooked up to the CLI yet though.
Focus is still on ensuring the dataset is in the right format though
2022-08-10 19:03:25 +01:00
Starbeamrainbowlabs
6cdf2b2389
wrangler python child: explicitly close stdout+stderr.
...
Hopefully this will avoid any more hanging issues.
2022-08-10 18:51:30 +01:00
Starbeamrainbowlabs
5880bf9020
wrangler: add current date to process indicator.
...
There's a bug that causes it to hang, but we don't know why
2022-08-10 18:50:57 +01:00
Starbeamrainbowlabs
231c832888
wrangler bugfix: crashes; logging output
2022-08-10 17:33:10 +01:00
Starbeamrainbowlabs
b52c7f89a7
Move dataset parsing function to the right place
2022-08-10 17:24:55 +01:00
Starbeamrainbowlabs
50f214450f
wrangler: fix crash
2022-08-10 17:05:01 +01:00
Starbeamrainbowlabs
0bac8c8c0c
fixup
2022-08-08 17:23:24 +01:00
Starbeamrainbowlabs
405f1a0bb0
fixup
2022-08-08 17:22:31 +01:00
Starbeamrainbowlabs
5e1356513c
slurm: use compute, because 28 tf processes in parallel is too much for the GPU memory
2022-08-08 17:22:18 +01:00
Starbeamrainbowlabs
133ef59af3
fixup
2022-08-08 16:33:05 +01:00
Starbeamrainbowlabs
80e1a33ee2
slurm-jsonl2tfrecord.job: auto install dependencies
2022-08-08 16:31:49 +01:00
Starbeamrainbowlabs
1442d20524
slurm: request gpu
2022-08-08 15:56:46 +01:00
Starbeamrainbowlabs
f6f2e3694c
json2tfrecord: write slurm job file
2022-08-08 15:53:32 +01:00
Starbeamrainbowlabs
222a6146ec
write glue for .jsonl.gz → .tfrecord.gz converter
2022-08-08 15:33:59 +01:00
Starbeamrainbowlabs
f3652edf82
fixup
2022-08-05 19:10:40 +01:00
Starbeamrainbowlabs
9399d1d8f5
Create (untested) JS interface to Python jsonl→tfrecord converter
...
also test Python .jsonl.gz → .tfrecord.gz
2022-08-05 19:10:28 +01:00
Starbeamrainbowlabs
a02c3436ab
get python bridge working t convert .jsonl.gz → .tfrecord.gz
2022-08-05 18:07:04 +01:00
Starbeamrainbowlabs
28a3f578d5
update .gitignore
2022-08-04 16:49:53 +01:00
Starbeamrainbowlabs
2ccc1be414
json2tfrecord: write (untested python to convert .jsonl → .tfrecord
2022-07-28 19:48:25 +01:00
Starbeamrainbowlabs
323d708692
dataset: add todo
...
just why, Tensorflow?!
tf.data.TextLineDataset looks almost too good to be true..... and it is, as despite supporting decompressing via gzip(!) it doesn't look like we can convince it to parse JSON :-/
2022-07-26 19:53:18 +01:00
Starbeamrainbowlabs
b53c77a2cb
index.py: call static function name run
2022-07-26 19:51:28 +01:00
Starbeamrainbowlabs
a7ed58fc03
ai: move requirements.txt to the right place
2022-07-26 19:25:11 +01:00
Starbeamrainbowlabs
e93a95f1b3
ai dataset: add if main == main
2022-07-26 19:24:40 +01:00
Starbeamrainbowlabs
de4c3dab17
typo
2022-07-26 19:14:55 +01:00
Starbeamrainbowlabs
18a7d3674b
ai: create (untested) dataset
2022-07-26 19:14:10 +01:00
Starbeamrainbowlabs
dac6919fcd
ai: start creating initial scaffolding
2022-07-25 19:01:10 +01:00
Starbeamrainbowlabs
1ec502daea
Remove rogue package*.json files
2022-07-25 19:00:21 +01:00
Starbeamrainbowlabs
927c30e189
recompress files in the right order
2022-07-25 18:44:23 +01:00
Starbeamrainbowlabs
3332fa598a
Add new recompress subcommand
...
also fix typos, CLI definitions
2022-07-25 17:54:23 +01:00