Starbeamrainbowlabs
13a8f3f511
pretrain-predict: only queue pretrain-plot if I we output jsonl
2022-10-10 17:11:10 +01:00
Starbeamrainbowlabs
ffcb2e3735
pretrain-predict: queue for the actual input
2022-10-10 16:53:28 +01:00
Starbeamrainbowlabs
f883986eaa
Bugfix: modeset to enable TFRecordWriter instead of bare handle
2022-10-06 20:07:59 +01:00
Starbeamrainbowlabs
e9a8e2eb57
fixup
2022-10-06 19:23:31 +01:00
Starbeamrainbowlabs
9f3ae96894
finish wiring for --water-size
2022-10-06 19:21:50 +01:00
Starbeamrainbowlabs
5dac70aa08
typo
2022-10-06 19:17:03 +01:00
Starbeamrainbowlabs
2960d3b645
exception → warning
2022-10-06 18:26:40 +01:00
Starbeamrainbowlabs
0ee6703c1e
Add todo and comment
2022-10-03 19:06:56 +01:00
Starbeamrainbowlabs
2b182214ea
typo
2022-10-03 17:53:10 +01:00
Starbeamrainbowlabs
92c380bff5
fiddle with Conv2DTranspose
...
you need to set the `stride` argument to actually get it to upscale..... :P
2022-10-03 17:51:41 +01:00
Starbeamrainbowlabs
d544553800
fixup
2022-10-03 17:33:06 +01:00
Starbeamrainbowlabs
058e3b6248
model_segmentation: cast float → int
2022-10-03 17:31:36 +01:00
Starbeamrainbowlabs
04e5ae0c45
model_segmentation: redo reshape
...
much cheese was applied :P
2022-10-03 17:27:52 +01:00
Starbeamrainbowlabs
deffe69202
typo
2022-10-03 16:59:36 +01:00
Starbeamrainbowlabs
fc6d2dabc9
Upscale first, THEN convnext...
2022-10-03 16:38:43 +01:00
Starbeamrainbowlabs
6a0790ff50
convnext_inverse: add returns; change ordering
2022-10-03 16:32:09 +01:00
Starbeamrainbowlabs
fe813cb46d
slurm predict: fix plotting subcall
2022-10-03 16:03:26 +01:00
Starbeamrainbowlabs
e51087d0a9
add reshape layer
2022-09-28 18:22:48 +01:00
Starbeamrainbowlabs
a336cdee90
and continues
2022-09-28 18:18:10 +01:00
Starbeamrainbowlabs
de47a883d9
missing units
2022-09-28 18:17:22 +01:00
Starbeamrainbowlabs
b5e08f92fe
the long night continues
2022-09-28 18:14:09 +01:00
Starbeamrainbowlabs
dc159ecfdb
and again
2022-09-28 18:11:46 +01:00
Starbeamrainbowlabs
4cf0485e32
fixup... again
2022-09-28 18:10:11 +01:00
Starbeamrainbowlabs
030d8710b6
fixup
2022-09-28 18:08:31 +01:00
Starbeamrainbowlabs
4ee7f2a0d6
add water thresholding
2022-09-28 18:07:26 +01:00
Starbeamrainbowlabs
404dc30f08
and again
2022-09-28 17:39:09 +01:00
Starbeamrainbowlabs
4cd8fc6ded
segmentation: param name fix
2022-09-28 17:37:42 +01:00
Starbeamrainbowlabs
41ba980d69
segmentationP implement dataset parser
2022-09-28 17:19:21 +01:00
Starbeamrainbowlabs
d618e6f8d7
pretrain-predict: params.json → metadata.jsonl
2022-09-28 16:35:22 +01:00
Starbeamrainbowlabs
e9e6139c7a
typo
2022-09-28 16:28:18 +01:00
Starbeamrainbowlabs
3dee3d8908
update cli help
2022-09-28 16:23:47 +01:00
Starbeamrainbowlabs
b836f7f70c
again
2022-09-27 19:06:41 +01:00
Starbeamrainbowlabs
52dff130dd
slurm pretraian predict: moar logging
2022-09-27 19:05:54 +01:00
Starbeamrainbowlabs
2d24174e0a
slurm pretrain predict: add $OUTPUT
2022-09-27 18:58:02 +01:00
Starbeamrainbowlabs
7d0e3913ae
fix logging
2022-09-27 18:51:58 +01:00
Starbeamrainbowlabs
d765b3b14e
fix crash
2022-09-27 18:43:43 +01:00
Starbeamrainbowlabs
2cd59a01a5
slurm-pretrain-plot: add ARGS
2022-09-27 18:41:35 +01:00
Starbeamrainbowlabs
f4d1d1d77e
just wh
2022-09-27 18:25:45 +01:00
Starbeamrainbowlabs
4c24d69ae6
$d → +d
2022-09-27 18:17:07 +01:00
Starbeamrainbowlabs
cdb19b4d9f
fixup
2022-09-27 18:13:21 +01:00
Starbeamrainbowlabs
c4d3c16873
add some logging
2022-09-27 18:10:58 +01:00
Starbeamrainbowlabs
3772c3227e
fixup
2022-09-27 17:57:21 +01:00
Starbeamrainbowlabs
dbfa45a016
write params.json properly
2022-09-27 17:49:54 +01:00
Starbeamrainbowlabs
a5455dc22a
fix import
2022-09-27 17:41:24 +01:00
Starbeamrainbowlabs
d6ff3fb2ce
pretrain_predict fix write mode
2022-09-27 17:38:12 +01:00
Starbeamrainbowlabs
f95fd8f9e4
pretrain-predict: add .tfrecord output function
2022-09-27 16:59:31 +01:00
Starbeamrainbowlabs
30b8dd063e
fixup
2022-09-27 15:54:37 +01:00
Starbeamrainbowlabs
3cf99587e4
Contraster: add gamma layer to load_model
2022-09-27 15:53:52 +01:00
Starbeamrainbowlabs
9e9852d066
UMAP: 100k random
...
no labels.
2022-09-27 15:52:45 +01:00
Starbeamrainbowlabs
58c65bdc86
slurm: allow predictions on gpu
2022-09-23 19:21:57 +01:00
Starbeamrainbowlabs
d59de41ebb
embeddings: change title rendering; make even moar widererer
...
We need to see that parallel coordinates plot in detail
2022-09-23 18:56:39 +01:00
Starbeamrainbowlabs
b4ddb24589
slurm plot: compute → highmem
2022-09-22 18:27:58 +01:00
Starbeamrainbowlabs
df12470e78
flip conda and time
...
hopefully we can capture the exit code this way
2022-09-21 14:40:28 +01:00
Starbeamrainbowlabs
5252a81238
vis: don't call add_subplot
2022-09-20 19:06:21 +01:00
Starbeamrainbowlabs
32bb55652b
slurm predict: autoqueue UMAP plot
2022-09-16 19:36:57 +01:00
Starbeamrainbowlabs
24c5263bf0
slurm predict-plot: fixup
2022-09-16 19:27:26 +01:00
Starbeamrainbowlabs
7e9c119b04
slurm: fix job name
2022-09-16 19:20:59 +01:00
Starbeamrainbowlabs
1574529704
slurm pretrain-predict: move to compute; make exclusive just in case
...
also shortent o 3 days
2022-09-16 19:17:42 +01:00
Starbeamrainbowlabs
d7f5958af0
slurm: write new job files
2022-09-16 19:00:43 +01:00
Starbeamrainbowlabs
a552cc4dad
ai vis: make parallel coordinates wider
2022-09-16 18:51:49 +01:00
Starbeamrainbowlabs
a70794e661
umap: no min_dist
2022-09-16 17:09:09 +01:00
Starbeamrainbowlabs
5778fc51f7
embeddings: fix title; remove colourmap
2022-09-16 17:08:04 +01:00
Starbeamrainbowlabs
4fd852d782
fixup
2022-09-16 16:44:35 +01:00
Starbeamrainbowlabs
fcab227f6a
cheese: set label for everything to 1
2022-09-16 16:42:05 +01:00
Starbeamrainbowlabs
b31645bd5d
pretrain-plot: fix crash; remove water code
...
the model doesn't save the water encoder at this time
2022-09-16 16:24:07 +01:00
Starbeamrainbowlabs
a5f03390ef
pretrain-plot: handle open w -> r
2022-09-16 16:14:30 +01:00
Starbeamrainbowlabs
1103ae5561
ai: tweak progress bar
2022-09-16 16:07:16 +01:00
Starbeamrainbowlabs
1e35802d2b
ai: fix embed i/o
2022-09-16 16:02:27 +01:00
Starbeamrainbowlabs
ed94da7492
fixup
2022-09-16 15:51:26 +01:00
Starbeamrainbowlabs
366db658a8
ds predict: fix filenames in
2022-09-16 15:45:22 +01:00
Starbeamrainbowlabs
e333dcba9c
tweak projection head
2022-09-16 15:36:01 +01:00
Starbeamrainbowlabs
6defd24000
bugfix: too many values to unpack
2022-09-15 19:56:17 +01:00
Starbeamrainbowlabs
e3c8277255
ai: tweak the segmentation model structure
2022-09-15 19:54:50 +01:00
Starbeamrainbowlabs
1bc8a5bf13
ai: fix crash
2022-09-15 19:37:06 +01:00
Starbeamrainbowlabs
bd64986332
ai: implement batched_iterator to replace .batch()
...
...apparently .batch() means you get a BatchedDataset or whatever when you iterate it like a tf.function instead of the actual tensor :-/
2022-09-15 19:16:38 +01:00
Starbeamrainbowlabs
ccd256c00a
embed rainfall radar, not both
2022-09-15 17:37:04 +01:00
Starbeamrainbowlabs
2c74676902
predict → predict_on_batch
2022-09-15 17:31:50 +01:00
Starbeamrainbowlabs
f036e79098
fixup
2022-09-15 17:09:26 +01:00
Starbeamrainbowlabs
d5f1a26ba3
disable prefetching when predicting a thing
2022-09-15 17:09:09 +01:00
Starbeamrainbowlabs
8770638022
ai: call .predict(), not the model itself
2022-09-14 17:41:01 +01:00
Starbeamrainbowlabs
a96cefde62
ai: predict oops
2022-09-14 17:37:48 +01:00
Starbeamrainbowlabs
fa3165a5b2
dataset: simplify dataset_predict
2022-09-14 17:33:17 +01:00
Starbeamrainbowlabs
279e27c898
fixup
2022-09-14 17:16:49 +01:00
Starbeamrainbowlabs
fad3313ede
fixup
2022-09-14 17:14:04 +01:00
Starbeamrainbowlabs
1e682661db
ai: kwargs in from_checkpoint
2022-09-14 17:11:06 +01:00
Starbeamrainbowlabs
6bda24d4da
ai: how did I miss that?!
...
bugfix ah/c
2022-09-14 16:53:43 +01:00
Starbeamrainbowlabs
decdd434d8
ai from_checkpoint: bugfix
2022-09-14 16:49:01 +01:00
Starbeamrainbowlabs
c9e00ea485
pretrain_predict: fix import
2022-09-14 16:02:36 +01:00
Starbeamrainbowlabs
f568d8d19f
io.open → handle_open
...
this was we get transparent .gz support
2022-09-14 16:01:22 +01:00
Starbeamrainbowlabs
9b25186541
fix --only-gpu
2022-09-14 15:55:21 +01:00
Starbeamrainbowlabs
a9c9c70d13
typo
2022-09-14 15:17:59 +01:00
Starbeamrainbowlabs
fb8f884487
add umap dependencies
2022-09-14 15:16:45 +01:00
Starbeamrainbowlabs
1876a8883c
ai pretrain-predict: fix - → _ in cli parsing
2022-09-14 15:12:07 +01:00
Starbeamrainbowlabs
f97b771922
make --help display help
2022-09-14 15:03:07 +01:00
Starbeamrainbowlabs
3c1aef5913
update help
2022-09-13 19:36:56 +01:00
Starbeamrainbowlabs
206257f9f5
ai pretrain_predict: no need to plot here anymore
2022-09-13 19:35:44 +01:00
Starbeamrainbowlabs
7685ec3e8b
implement ability to embed & plot pretrained embeddings
2022-09-13 19:18:59 +01:00
Starbeamrainbowlabs
7130c4fdf8
start implementing core image segmentation model
2022-09-07 17:45:38 +01:00
Starbeamrainbowlabs
22620a1854
ai: implement saving only the rainfall encoder
2022-09-06 19:48:46 +01:00
Starbeamrainbowlabs
4c4358c3e5
whitespace
2022-09-06 16:24:11 +01:00
Starbeamrainbowlabs
4202821d98
typo
2022-09-06 15:37:36 +01:00
Starbeamrainbowlabs
3e13ad12c8
ai Bugfix LayerContrastiveEncoder: channels → input_channels
...
for consistency
2022-09-05 23:53:16 +01:00
Starbeamrainbowlabs
ead8009425
pretrain: add CLI arg for size of watch prediction width/height
2022-09-05 15:36:40 +01:00
Starbeamrainbowlabs
9d39215dd5
dataset: drop incomplete batches
2022-09-05 15:36:10 +01:00
Starbeamrainbowlabs
c94f5d042e
ai: slurm fixup again
2022-09-02 19:13:56 +01:00
Starbeamrainbowlabs
7917820e59
ai: slurm fixup
2022-09-02 19:11:54 +01:00
Starbeamrainbowlabs
cd104190e8
slurm: no need to be exclusive anymore
2022-09-02 19:09:45 +01:00
Starbeamrainbowlabs
a9dede7bfe
ai: n=10 slurm
2022-09-02 19:09:07 +01:00
Starbeamrainbowlabs
42de502f99
slurm-pretrain: set name
2022-09-02 19:08:16 +01:00
Starbeamrainbowlabs
457c2eef0d
ai: fixup
2022-09-02 19:06:54 +01:00
Starbeamrainbowlabs
a3f03b6d8d
slurm-pretrain: prepare
2022-09-02 19:05:18 +01:00
Starbeamrainbowlabs
1d1533d160
ai: how did things get this confusing
2022-09-02 18:51:46 +01:00
Starbeamrainbowlabs
1c5defdcd6
ai: cats
2022-09-02 18:45:23 +01:00
Starbeamrainbowlabs
6135bcd0cd
fixup
2022-09-02 18:41:31 +01:00
Starbeamrainbowlabs
c33c0a0899
ai: these shapes are so annoying
2022-09-02 18:39:24 +01:00
Starbeamrainbowlabs
88acd54a97
ai: off-by-one
2022-09-02 18:13:48 +01:00
Starbeamrainbowlabs
23bcd294c4
AI: 0.75 → 0.5?
2022-09-02 18:12:51 +01:00
Starbeamrainbowlabs
b0e1aeac35
ai: knew it
2022-09-02 18:10:29 +01:00
Starbeamrainbowlabs
ef0f15960d
send correct water shape to
2022-09-02 18:09:36 +01:00
Starbeamrainbowlabs
ad156a9a00
ai dataset: centre crop the water data to 75% original size
...
this should both help the model and reduce memory usage
2022-09-02 18:05:32 +01:00
Starbeamrainbowlabs
efe41b96ec
fixup
2022-09-02 17:58:02 +01:00
Starbeamrainbowlabs
f8ee0afca1
ai: fix arch_name plumbing
2022-09-02 17:57:07 +01:00
Starbeamrainbowlabs
3d44831080
Add NO_PREFETCH env var
2022-09-02 17:55:04 +01:00
Starbeamrainbowlabs
3e0ca6a315
ai: fix summary file writing; make water encoder smaller
2022-09-02 17:51:45 +01:00
Starbeamrainbowlabs
389216b391
reordering
2022-09-02 17:31:19 +01:00
Starbeamrainbowlabs
b9d01ddadc
summary logger → summarywriter
2022-09-02 17:28:00 +01:00
Starbeamrainbowlabs
9f7f4af784
dataset: properly resize
2022-09-02 17:09:38 +01:00
Starbeamrainbowlabs
c78b6562ef
debug adjustment
2022-09-02 17:00:21 +01:00
Starbeamrainbowlabs
c89677abd7
dataset: explicit reshape
2022-09-02 16:57:59 +01:00
Starbeamrainbowlabs
c066ea331b
dataset: namespace → dict
...
Python is so annoying
2022-09-02 16:07:44 +01:00
Starbeamrainbowlabs
dbff15d4a9
add bug comment
2022-09-01 19:05:50 +01:00
Starbeamrainbowlabs
3e4128c0a8
resize rainfall to be 1/2 size of current
2022-09-01 18:47:07 +01:00
Starbeamrainbowlabs
8a86728b54
ai: how did I miss that
2022-09-01 18:07:53 +01:00
Starbeamrainbowlabs
b2f1ba29bb
debug a
2022-09-01 18:05:33 +01:00
Starbeamrainbowlabs
e2f621e212
even moar debug logging
2022-09-01 17:49:51 +01:00
Starbeamrainbowlabs
2663123407
moar logging
2022-09-01 17:40:23 +01:00
Starbeamrainbowlabs
c2fcb3b954
ai: channels_first → channels_last
2022-09-01 17:06:18 +01:00
Starbeamrainbowlabs
f1d7973f22
ai: add dummy label
2022-09-01 17:01:00 +01:00
Starbeamrainbowlabs
17d42fe899
ai: json.dumps
2022-09-01 16:21:52 +01:00
Starbeamrainbowlabs
940f7aa1b5
ai: set self.model
2022-09-01 16:20:23 +01:00
Starbeamrainbowlabs
f1be5fe2bd
ai: add missing arguments to LossContrastive
2022-09-01 16:14:00 +01:00
Starbeamrainbowlabs
ddb375e906
ai: another typo
2022-09-01 16:12:05 +01:00
Starbeamrainbowlabs
cb3e1d3a23
ai: fixup
2022-09-01 16:10:50 +01:00
Starbeamrainbowlabs
e4c95bc7e3
typo
2022-09-01 16:09:24 +01:00
Starbeamrainbowlabs
cfbbe8e8cf
ai: global? really?
2022-09-01 16:06:24 +01:00
Starbeamrainbowlabs
4952ead094
ai: try the nonlocal keyword? I'm unsure what's going on here
...
....clearly I need to read up on scoping in python.
2022-09-01 16:02:37 +01:00
Starbeamrainbowlabs
1d9d0fbb73
requirements.txt:a dd tensorflow_addons
2022-09-01 15:56:02 +01:00
Starbeamrainbowlabs
8bdded23eb
ai: fix 'nother crash' name ConvNeXt submodels
2022-08-31 18:57:27 +01:00
Starbeamrainbowlabs
b2a320134e
ai: typo
2022-08-31 18:54:03 +01:00
Starbeamrainbowlabs
e4edc68df5
ai: add missing gamma layer
2022-08-31 18:52:35 +01:00
Starbeamrainbowlabs
51cf08a386
ResNetRSV2 → ConvNeXt
...
ironically this makes the model simpler o/
2022-08-31 18:51:01 +01:00
Starbeamrainbowlabs
3d614d105b
channels first
2022-08-31 18:06:59 +01:00
Starbeamrainbowlabs
654eefd9ca
properly handle water dimensions; add log files to .gitignore
...
TODO: add heightmap
2022-08-31 18:03:39 +01:00
Starbeamrainbowlabs
5846828f9e
debug logging
2022-08-31 17:48:09 +01:00
Starbeamrainbowlabs
12c77e128d
handle feature_dim properly
2022-08-31 17:41:51 +01:00
Starbeamrainbowlabs
c52a9f961c
and another
2022-08-31 17:37:28 +01:00
Starbeamrainbowlabs
58dbabd561
fix another crash
2022-08-31 17:33:07 +01:00
Starbeamrainbowlabs
5fc0725917
slurm: chmod +x
2022-08-31 16:33:07 +01:00
Starbeamrainbowlabs
dbf929325a
typo; add pretrain slurm job file
2022-08-31 16:32:17 +01:00
Starbeamrainbowlabs
e0162bc70b
requirements.txt: add missing dependencies
2022-08-31 16:25:47 +01:00
Starbeamrainbowlabs
f2312c1184
fix crash
2022-08-31 16:25:27 +01:00
Starbeamrainbowlabs
15a3519107
ai: the best thing about implementing a model is that you don't have to test it on the same day :P
2022-08-11 18:26:28 +01:00
Starbeamrainbowlabs
c0a9cb12d8
ai: start creating initial model implementation.
...
it's not hooked up to the CLI yet though.
Focus is still on ensuring the dataset is in the right format though
2022-08-10 19:03:25 +01:00
Starbeamrainbowlabs
b52c7f89a7
Move dataset parsing function to the right place
2022-08-10 17:24:55 +01:00
Starbeamrainbowlabs
222a6146ec
write glue for .jsonl.gz → .tfrecord.gz converter
2022-08-08 15:33:59 +01:00
Starbeamrainbowlabs
28a3f578d5
update .gitignore
2022-08-04 16:49:53 +01:00
Starbeamrainbowlabs
323d708692
dataset: add todo
...
just why, Tensorflow?!
tf.data.TextLineDataset looks almost too good to be true..... and it is, as despite supporting decompressing via gzip(!) it doesn't look like we can convince it to parse JSON :-/
2022-07-26 19:53:18 +01:00
Starbeamrainbowlabs
b53c77a2cb
index.py: call static function name run
2022-07-26 19:51:28 +01:00
Starbeamrainbowlabs
a7ed58fc03
ai: move requirements.txt to the right place
2022-07-26 19:25:11 +01:00
Starbeamrainbowlabs
e93a95f1b3
ai dataset: add if main == main
2022-07-26 19:24:40 +01:00
Starbeamrainbowlabs
de4c3dab17
typo
2022-07-26 19:14:55 +01:00
Starbeamrainbowlabs
18a7d3674b
ai: create (untested) dataset
2022-07-26 19:14:10 +01:00
Starbeamrainbowlabs
dac6919fcd
ai: start creating initial scaffolding
2022-07-25 19:01:10 +01:00
Starbeamrainbowlabs
8a9cd6c1c0
Lay out some basic scaffolding
...
I *really* hope this works. This is the 3rd major revision of this
model. I've learnt a ton of stuff between now and my last attempt, so
here's hoping that all goes well :D
The basic idea behind this attempt is *Contrastive Learning*. If we
don't get anything useful with this approach, then we can assume that
it's not really possible / feasible.
Something we need to watch out for is the variance (or rather lack
thereof) in the dataset. We have 1.5M timesteps, but not a whole lot
will be happening in most of those....
We may need to analyse the variance of the water depth data and extract
a subsample that's more balanced.
2022-05-13 19:06:15 +01:00