Commit graph

517 commits

Author SHA1 Message Date
Starbeamrainbowlabs cad82cd1bc
CBAM: unsure if it's 1 ro 3 dense ayers in the shared mlp 2022-12-02 17:21:13 +00:00
Starbeamrainbowlabs 62f6a993bb
implement CBAM, but it's UNTESTED
Convolutional Block Attention Module.
2022-12-02 17:17:45 +00:00
Starbeamrainbowlabs 9d666c3b38
train mono: type=int → float 2022-12-01 15:39:44 +00:00
Starbeamrainbowlabs 53dfa32685
model_mono: log learning rate 2022-12-01 15:10:51 +00:00
Starbeamrainbowlabs c384d55dff
add arg to adjust learning rate 2022-11-29 20:55:00 +00:00
Starbeamrainbowlabs 8e23e9d341
model_segmenter: we're no longer using sparse 2022-11-29 19:28:27 +00:00
Starbeamrainbowlabs 9a2b4c6838
dsseg: fix reshape/onehot ordering 2022-11-29 19:28:13 +00:00
Starbeamrainbowlabs df774146d9
dataset_segmenter: reshape, not squeeze 2022-11-29 19:24:54 +00:00
Starbeamrainbowlabs 77b8a1a8db
dataset_segmenter: squeeze 2022-11-29 19:16:15 +00:00
Starbeamrainbowlabs 2258b5a229
slurm-train: reduce RAM required by 10GB 2022-11-29 19:15:34 +00:00
Starbeamrainbowlabs 01101ad30b
losscrossentropy: return the reduced value * facepalm * 2022-11-29 19:07:08 +00:00
Starbeamrainbowlabs ff65393e78
log file naming update 2022-11-29 18:41:14 +00:00
Starbeamrainbowlabs 37f196a785
LossCrossentropy: add kwargs 2022-11-29 15:40:35 +00:00
Starbeamrainbowlabs 838ff56a3b
mono: fix loading checkpoint 2022-11-29 15:25:11 +00:00
Starbeamrainbowlabs dba6cbffcd
WHY. * facepalms * 2022-11-28 19:33:42 +00:00
Starbeamrainbowlabs 57b8eb93fb
fixup 2022-11-28 19:09:35 +00:00
Starbeamrainbowlabs 6640a41bb7
almost got it....? it's not what I expected....! 2022-11-28 19:08:50 +00:00
Starbeamrainbowlabs f48473b703
fixup 2022-11-28 19:00:11 +00:00
Starbeamrainbowlabs f6feb125e3
this iss ome serious debugging.
This commit will produce an extremely large volume of output.
2022-11-28 18:57:41 +00:00
Starbeamrainbowlabs 09f81b0746
train_mono: debug
this commit will generate a large amount of debug output.
2022-11-28 16:46:17 +00:00
Starbeamrainbowlabs f39e4ade70
LayerConvNextGamma: fix config serialisation bug
.....this is unlikely to be the problem as this bug is in an unused code path.
2022-11-25 21:16:31 +00:00
Starbeamrainbowlabs e7410fb480
train_mono_predict: limit label size to 64x64
that's the size the model predicts
2022-11-25 17:47:17 +00:00
Starbeamrainbowlabs 51dd484d13
fixup 2022-11-25 16:55:45 +00:00
Starbeamrainbowlabs 884c4eb150
rainfall_stats: formatting again 2022-11-24 19:08:07 +00:00
Starbeamrainbowlabs bfe038086c
rainfall_stats: formatting 2022-11-24 19:07:44 +00:00
Starbeamrainbowlabs 7dba03200f
fixup 2022-11-24 19:06:48 +00:00
Starbeamrainbowlabs e5258b9c66
typo 2022-11-24 19:06:13 +00:00
Starbeamrainbowlabs 64d646bb13
rainfall_stats: formatting 2022-11-24 19:05:35 +00:00
Starbeamrainbowlabs 675c7a7448
fixup 2022-11-24 19:03:28 +00:00
Starbeamrainbowlabs afc1cdcf02
fixup 2022-11-24 19:02:58 +00:00
Starbeamrainbowlabs e4bea89c89
typo 2022-11-24 19:01:52 +00:00
Starbeamrainbowlabs a40cbe8705
rainfall_stats: remove unused imports 2022-11-24 19:01:18 +00:00
Starbeamrainbowlabs fe57d6aab2
rainfall_stats: initial implementation
this might reveal why we are having problems. If most/all the rainfall radar
data is v small numbers, normalising
might help.
2022-11-24 18:58:16 +00:00
Starbeamrainbowlabs 3131b4f7b3
debug2 2022-11-24 18:25:32 +00:00
Starbeamrainbowlabs d55a13f536
debug 2022-11-24 18:24:03 +00:00
Starbeamrainbowlabs 1f60f2a580
do_argmax 2022-11-24 18:11:03 +00:00
Starbeamrainbowlabs 6c09d5254d
fixup 2022-11-24 17:57:48 +00:00
Starbeamrainbowlabs 54a841efe9
train_mono_predict: convert to correct format 2022-11-24 17:56:07 +00:00
Starbeamrainbowlabs 105dc5bc56
missing kwargs 2022-11-24 17:51:29 +00:00
Starbeamrainbowlabs 1e1d6dd273
fixup 2022-11-24 17:48:19 +00:00
Starbeamrainbowlabs 011e0aef78
update cli docs 2022-11-24 16:38:07 +00:00
Starbeamrainbowlabs 773944f9fa
train_mono_predict: initial implementation 2022-11-24 16:33:50 +00:00
Starbeamrainbowlabs d31326cb30
slurm train mono: fix partition name 2022-11-22 17:02:02 +00:00
Starbeamrainbowlabs ce28ac4013
slurm: add job for train_mono 2022-11-22 16:58:46 +00:00
Starbeamrainbowlabs 3a0356929c
mono: drop the sparse 2022-11-22 16:20:56 +00:00
Starbeamrainbowlabs 7e8f63f8ba
fixup 2022-11-21 19:38:24 +00:00
Starbeamrainbowlabs ace4c8b246
dataset_mono: debug 2022-11-21 18:46:21 +00:00
Starbeamrainbowlabs 527b34942d
convnext_inverse: kernel_size 4→2 2022-11-11 19:29:37 +00:00
Starbeamrainbowlabs 0662d0854b
model_mono: fix bottleneck 2022-11-11 19:11:40 +00:00
Starbeamrainbowlabs 73acda6d9a
fix debug logging 2022-11-11 19:08:38 +00:00
Starbeamrainbowlabs 9da059d738
model shape logging 2022-11-11 19:03:37 +00:00
Starbeamrainbowlabs 00917b2698
dataset_mono: log shapes 2022-11-11 19:02:43 +00:00
Starbeamrainbowlabs 54ae88b1b4
in this entire blasted project I have yet to get the rotation of anything correct....! 2022-11-11 18:58:45 +00:00
Starbeamrainbowlabs a7a475dcd1
debug 2 2022-11-11 18:38:07 +00:00
Starbeamrainbowlabs bf2f6e9b64
debug logging
it begins again
2022-11-11 18:31:40 +00:00
Starbeamrainbowlabs 481eeb3759
mono: fix dataset preprocessing
rogue dimension
2022-11-11 18:31:27 +00:00
Starbeamrainbowlabs 9035450213
mono: instantiate right model 2022-11-11 18:28:29 +00:00
Starbeamrainbowlabs 69a2d0cf04
fixup 2022-11-11 18:27:01 +00:00
Starbeamrainbowlabs 65e801cf28
train_mono: fix crash 2022-11-11 18:26:25 +00:00
Starbeamrainbowlabs 8ac5159adc
dataset_mono: simplify param passing, onehot+threshold water depth data 2022-11-11 18:23:50 +00:00
Starbeamrainbowlabs 3a3f7e85da
typo 2022-11-11 18:03:09 +00:00
Starbeamrainbowlabs 3313f77c88
Add (untested) mono rainfall → water depth model
* sighs *
Unfortunately I can't seem to get contrastive learning to work.....
2022-11-10 22:36:11 +00:00
Starbeamrainbowlabs ce194d9227
slurm: customise log file names 2022-11-10 21:09:34 +00:00
Starbeamrainbowlabs 9384b89165
model_segmentation: spare → normal crossentropy, activation functions at end 2022-11-10 20:53:37 +00:00
Starbeamrainbowlabs b6676e7361
switch from sparse to normal crossentropy 2022-11-10 20:50:56 +00:00
Starbeamrainbowlabs d8be26d476
Merge branch 'main' of git.starbeamrainbowlabs.com:sbrl/PhD-Rainfall-Radar 2022-11-10 20:49:01 +00:00
Starbeamrainbowlabs b03388de60
dataset_segmenter: DEBUG: fix water shape 2022-11-10 20:48:21 +00:00
Starbeamrainbowlabs daf691bf43
typo 2022-11-10 19:55:00 +00:00
Starbeamrainbowlabs 0aa2ce19f5
read_metadata: support file inputs as well as dirs 2022-11-10 19:53:30 +00:00
Starbeamrainbowlabs aa7d9b8cf6
fixup 2022-11-10 19:46:09 +00:00
Starbeamrainbowlabs 0894bd09e8
train_predict: add error message for parrams.json not found 2022-11-10 19:45:41 +00:00
Starbeamrainbowlabs 0353072d15
allow pretrain to run on gpu
we've slashed the size of the 2nd encoder, so ti should fit naow?
2022-11-04 17:02:07 +00:00
Starbeamrainbowlabs 44ad51f483
CallbackNBatchCsv: bugfix .sort() → sorted() 2022-11-04 16:40:21 +00:00
Starbeamrainbowlabs 4dddcfcb42
pretrain_predict: missing \n 2022-11-04 16:01:28 +00:00
Starbeamrainbowlabs 1375201c5f
CallbackNBatchCsv: open_handle mode 2022-11-03 18:29:00 +00:00
Starbeamrainbowlabs 3206d6b7e7
slurm: rename segmenter job name 2022-11-03 17:12:27 +00:00
Starbeamrainbowlabs f2ae74ce7b
how could I be so stupid..... round 2 2022-11-02 17:38:26 +00:00
Starbeamrainbowlabs 5f8d6dc6ea
Add metrics every 64 batches
this is important, because with large batches it can be difficult to tell what's happening inside each epoch.
2022-10-31 19:26:10 +00:00
Starbeamrainbowlabs cf872ef739
how could I be so *stupid*...... 2022-10-31 18:40:58 +00:00
Starbeamrainbowlabs da32d75778
make_callbacks: display steps, not samples 2022-10-31 18:36:28 +00:00
Starbeamrainbowlabs dfef7db421
moar debugging 2022-10-31 18:26:34 +00:00
Starbeamrainbowlabs 172cf9d8ce
tweak 2022-10-31 18:19:43 +00:00
Starbeamrainbowlabs dbe35ee943
loss: comment l2 norm 2022-10-31 18:09:03 +00:00
Starbeamrainbowlabs 5e60319024
fixup 2022-10-31 17:56:49 +00:00
Starbeamrainbowlabs b986b069e2
debug party time 2022-10-31 17:50:29 +00:00
Starbeamrainbowlabs 458faa96d2
loss: fixup 2022-10-31 17:18:21 +00:00
Starbeamrainbowlabs 55dc05e8ce
contrastive: comment weights that aren't needed 2022-10-31 16:26:48 +00:00
Starbeamrainbowlabs 33391eaf16
train_predict/jsonl: don't argmax
I'm interested inthe raw values
2022-10-26 17:21:19 +01:00
Starbeamrainbowlabs 74f2cdb900
train_predict: .list() → .tolist() 2022-10-26 17:12:36 +01:00
Starbeamrainbowlabs 4f9d543695
train_predict: don't pass model_code
it's redundant
2022-10-26 17:11:36 +01:00
Starbeamrainbowlabs 1b489518d0
segmenter: add LayerStack2Image to custom_objects 2022-10-26 17:05:50 +01:00
Starbeamrainbowlabs 48ae8a5c20
LossContrastive: normalise features as per the paper 2022-10-26 16:52:56 +01:00
Starbeamrainbowlabs 843cc8dc7b
contrastive: rewrite the loss function.
The CLIP paper *does* kinda make sense I think
2022-10-26 16:45:45 +01:00
Starbeamrainbowlabs fad1399c2d
convnext: whitespace 2022-10-26 16:45:20 +01:00
Starbeamrainbowlabs 1d872cb962
contrastive: fix initial temperature value
It should be 1/0.07, but we had it set to 0.07......
2022-10-26 16:45:01 +01:00
Starbeamrainbowlabs f994d449f1
Layer2Image: fix 2022-10-25 21:32:17 +01:00
Starbeamrainbowlabs 6a29105f56
model_segmentation: stack not reshape 2022-10-25 21:25:15 +01:00
Starbeamrainbowlabs 98417a3e06
prepare for NCE loss
.....but Tensorflow's implementation looks to be for supervised models :-(
2022-10-25 21:15:05 +01:00
Starbeamrainbowlabs bb0679a509
model_segmentation: don't softmax twice 2022-10-25 21:11:48 +01:00
Starbeamrainbowlabs f2e2ca1484
model_contrastive: make water encoder significantly shallower 2022-10-24 20:52:31 +01:00
Starbeamrainbowlabs a6b07a49cb
count water/nowater pixels in Jupyter Notebook 2022-10-24 18:05:34 +01:00
Starbeamrainbowlabs a8b101bdae
dataset_predict: add shape_water_desired 2022-10-24 18:05:13 +01:00
Starbeamrainbowlabs 587c1dfafa
train_predict: revamp jsonl handling 2022-10-21 16:53:08 +01:00
Starbeamrainbowlabs 8195318a42
SparseCategoricalAccuracy: losses → metrics 2022-10-21 16:51:20 +01:00
Starbeamrainbowlabs 612735aaae
rename shuffle arg 2022-10-21 16:35:45 +01:00
Starbeamrainbowlabs c98d8d05dd
segmentation: use the right accuracy 2022-10-21 16:17:05 +01:00
Starbeamrainbowlabs bb0258f5cd
flip squeeze operator ordering 2022-10-21 15:38:57 +01:00
Starbeamrainbowlabs af26964c6a
batched_iterator: reset i_item after every time 2022-10-21 15:35:43 +01:00
Starbeamrainbowlabs c5b1501dba
train-predict fixup 2022-10-21 15:27:39 +01:00
Starbeamrainbowlabs 42aea7a0cc
plt.close() fixup 2022-10-21 15:23:54 +01:00
Starbeamrainbowlabs 12dad3bc87
vis/segmentation: fix titles 2022-10-21 15:22:35 +01:00
Starbeamrainbowlabs 0cb2de5d06
train-preedict: close matplotlib after we've finished
they act like file handles
2022-10-21 15:19:31 +01:00
Starbeamrainbowlabs 81e53efd9c
PNG: create output dir if doesn't exist 2022-10-21 15:17:39 +01:00
Starbeamrainbowlabs 3f7db6fa78
fix embedding confusion 2022-10-21 15:15:59 +01:00
Starbeamrainbowlabs 847cd97ec4
fixup 2022-10-21 14:26:58 +01:00
Starbeamrainbowlabs 0e814b7e98
Contraster → Segmenter 2022-10-21 14:25:43 +01:00
Starbeamrainbowlabs 1b658a1b7c
train-predict: can't destructure array when iterating generator
....it seems to lead to undefined behaviour or something
2022-10-20 19:34:04 +01:00
Starbeamrainbowlabs aed2348a95
train_predict: fixup 2022-10-20 15:42:33 +01:00
Starbeamrainbowlabs cc6679c609
batch data; use generator 2022-10-20 15:22:29 +01:00
Starbeamrainbowlabs d306853c42
use right daataset 2022-10-20 15:16:24 +01:00
Starbeamrainbowlabs 59cfa4a89a
basename paths 2022-10-20 15:11:14 +01:00
Starbeamrainbowlabs 4d8ae21a45
update cli help text 2022-10-19 17:31:42 +01:00
Starbeamrainbowlabs 200076596b
finish train_predict 2022-10-19 17:26:40 +01:00
Starbeamrainbowlabs 488f78fca5
pretrain_predict: default to parallel_reads=0 2022-10-19 16:59:45 +01:00
Starbeamrainbowlabs 63e909d9fc
datasets: add shuffle=True/False to get_filepaths.
This is important because otherwise it SCAMBLES the filenames, which is a disaster for making predictions in the right order....!
2022-10-19 16:52:07 +01:00
Starbeamrainbowlabs fe43ddfbf9
start implementing driver for train_predict, but not finished yet 2022-10-18 19:37:55 +01:00
Starbeamrainbowlabs b3ea189d37
segmentation: softmax the output 2022-10-13 21:02:57 +01:00
Starbeamrainbowlabs f121bfb981
fixup summaryfile 2022-10-13 17:54:42 +01:00
Starbeamrainbowlabs 5c35c0cee4
model_segmentation: document; remove unused args 2022-10-13 17:50:16 +01:00
Starbeamrainbowlabs f12e6ab905
No need for a CLI arg for feature_dim_in - metadata should contain this 2022-10-13 17:37:16 +01:00
Starbeamrainbowlabs e201372252
write quick Jupyter notebook to test data
....I'm paranoid
2022-10-13 17:27:17 +01:00
Starbeamrainbowlabs ae53130e66
layout 2022-10-13 14:54:20 +01:00
Starbeamrainbowlabs 7933564c66
typo 2022-10-12 17:33:54 +01:00
Starbeamrainbowlabs dbe4fb0eab
train: add slurm job file 2022-10-12 17:27:10 +01:00
Starbeamrainbowlabs 6423bf6702
LayerConvNeXtGamma: avoid adding an EagerTensor to config
Very weird how this is a problem when it wasn't before..
2022-10-12 17:12:07 +01:00
Starbeamrainbowlabs 32f5200d3b
pass model_arch properly 2022-10-12 16:50:06 +01:00
Starbeamrainbowlabs 5933fb1061
fixup 2022-10-11 19:23:41 +01:00
Starbeamrainbowlabs c45b90764e
segmentation: adds xxtiny, but unsure if it's small enough 2022-10-11 19:22:37 +01:00
Starbeamrainbowlabs f4a2c742d9
typo 2022-10-11 19:19:23 +01:00
Starbeamrainbowlabs 11f91a7cf4
train: add --arch; default to convnext_i_xtiny 2022-10-11 19:18:01 +01:00
Starbeamrainbowlabs 5666c5a0d9
typo 2022-10-10 18:12:51 +01:00
Starbeamrainbowlabs 131c0a0a5b
pretrain-predict: create dir if not exists 2022-10-10 18:00:55 +01:00
Starbeamrainbowlabs deede32241
slurm-pretrain: limit memory usage 2022-10-10 17:45:29 +01:00
Starbeamrainbowlabs 13a8f3f511
pretrain-predict: only queue pretrain-plot if I we output jsonl 2022-10-10 17:11:10 +01:00
Starbeamrainbowlabs ffcb2e3735
pretrain-predict: queue for the actual input 2022-10-10 16:53:28 +01:00
Starbeamrainbowlabs f883986eaa
Bugfix: modeset to enable TFRecordWriter instead of bare handle 2022-10-06 20:07:59 +01:00
Starbeamrainbowlabs e9a8e2eb57
fixup 2022-10-06 19:23:31 +01:00
Starbeamrainbowlabs 9f3ae96894
finish wiring for --water-size 2022-10-06 19:21:50 +01:00
Starbeamrainbowlabs 5dac70aa08
typo 2022-10-06 19:17:03 +01:00
Starbeamrainbowlabs 2960d3b645
exception → warning 2022-10-06 18:26:40 +01:00
Starbeamrainbowlabs 0ee6703c1e
Add todo and comment 2022-10-03 19:06:56 +01:00
Starbeamrainbowlabs 2b182214ea
typo 2022-10-03 17:53:10 +01:00
Starbeamrainbowlabs 92c380bff5
fiddle with Conv2DTranspose
you need to set the `stride` argument to actually get it to upscale..... :P
2022-10-03 17:51:41 +01:00
Starbeamrainbowlabs d544553800
fixup 2022-10-03 17:33:06 +01:00
Starbeamrainbowlabs 058e3b6248
model_segmentation: cast float → int 2022-10-03 17:31:36 +01:00
Starbeamrainbowlabs 04e5ae0c45
model_segmentation: redo reshape
much cheese was applied :P
2022-10-03 17:27:52 +01:00
Starbeamrainbowlabs deffe69202
typo 2022-10-03 16:59:36 +01:00
Starbeamrainbowlabs fc6d2dabc9
Upscale first, THEN convnext... 2022-10-03 16:38:43 +01:00
Starbeamrainbowlabs 6a0790ff50
convnext_inverse: add returns; change ordering 2022-10-03 16:32:09 +01:00
Starbeamrainbowlabs fe813cb46d
slurm predict: fix plotting subcall 2022-10-03 16:03:26 +01:00
Starbeamrainbowlabs e51087d0a9
add reshape layer 2022-09-28 18:22:48 +01:00
Starbeamrainbowlabs a336cdee90
and continues 2022-09-28 18:18:10 +01:00
Starbeamrainbowlabs de47a883d9
missing units 2022-09-28 18:17:22 +01:00
Starbeamrainbowlabs b5e08f92fe
the long night continues 2022-09-28 18:14:09 +01:00
Starbeamrainbowlabs dc159ecfdb
and again 2022-09-28 18:11:46 +01:00
Starbeamrainbowlabs 4cf0485e32
fixup... again 2022-09-28 18:10:11 +01:00
Starbeamrainbowlabs 030d8710b6
fixup 2022-09-28 18:08:31 +01:00
Starbeamrainbowlabs 4ee7f2a0d6
add water thresholding 2022-09-28 18:07:26 +01:00
Starbeamrainbowlabs 404dc30f08
and again 2022-09-28 17:39:09 +01:00
Starbeamrainbowlabs 4cd8fc6ded
segmentation: param name fix 2022-09-28 17:37:42 +01:00
Starbeamrainbowlabs 41ba980d69
segmentationP implement dataset parser 2022-09-28 17:19:21 +01:00
Starbeamrainbowlabs d618e6f8d7
pretrain-predict: params.json → metadata.jsonl 2022-09-28 16:35:22 +01:00
Starbeamrainbowlabs e9e6139c7a
typo 2022-09-28 16:28:18 +01:00
Starbeamrainbowlabs 3dee3d8908
update cli help 2022-09-28 16:23:47 +01:00
Starbeamrainbowlabs b836f7f70c
again 2022-09-27 19:06:41 +01:00
Starbeamrainbowlabs 52dff130dd
slurm pretraian predict: moar logging 2022-09-27 19:05:54 +01:00
Starbeamrainbowlabs 2d24174e0a
slurm pretrain predict: add $OUTPUT 2022-09-27 18:58:02 +01:00
Starbeamrainbowlabs 7d0e3913ae
fix logging 2022-09-27 18:51:58 +01:00
Starbeamrainbowlabs d765b3b14e
fix crash 2022-09-27 18:43:43 +01:00
Starbeamrainbowlabs 2cd59a01a5
slurm-pretrain-plot: add ARGS 2022-09-27 18:41:35 +01:00
Starbeamrainbowlabs f4d1d1d77e
just wh 2022-09-27 18:25:45 +01:00
Starbeamrainbowlabs 4c24d69ae6
$d → +d 2022-09-27 18:17:07 +01:00
Starbeamrainbowlabs cdb19b4d9f
fixup 2022-09-27 18:13:21 +01:00
Starbeamrainbowlabs c4d3c16873
add some logging 2022-09-27 18:10:58 +01:00
Starbeamrainbowlabs 3772c3227e
fixup 2022-09-27 17:57:21 +01:00
Starbeamrainbowlabs dbfa45a016
write params.json properly 2022-09-27 17:49:54 +01:00
Starbeamrainbowlabs a5455dc22a
fix import 2022-09-27 17:41:24 +01:00
Starbeamrainbowlabs d6ff3fb2ce
pretrain_predict fix write mode 2022-09-27 17:38:12 +01:00
Starbeamrainbowlabs f95fd8f9e4
pretrain-predict: add .tfrecord output function 2022-09-27 16:59:31 +01:00
Starbeamrainbowlabs 30b8dd063e
fixup 2022-09-27 15:54:37 +01:00
Starbeamrainbowlabs 3cf99587e4
Contraster: add gamma layer to load_model 2022-09-27 15:53:52 +01:00
Starbeamrainbowlabs 9e9852d066
UMAP: 100k random
no labels.
2022-09-27 15:52:45 +01:00
Starbeamrainbowlabs 58c65bdc86
slurm: allow predictions on gpu 2022-09-23 19:21:57 +01:00
Starbeamrainbowlabs d59de41ebb
embeddings: change title rendering; make even moar widererer
We need to see that parallel coordinates  plot in detail
2022-09-23 18:56:39 +01:00
Starbeamrainbowlabs b4ddb24589
slurm plot: compute → highmem 2022-09-22 18:27:58 +01:00
Starbeamrainbowlabs df12470e78
flip conda and time
hopefully we can capture the exit code this way
2022-09-21 14:40:28 +01:00
Starbeamrainbowlabs 5252a81238
vis: don't call add_subplot 2022-09-20 19:06:21 +01:00
Starbeamrainbowlabs 32bb55652b
slurm predict: autoqueue UMAP plot 2022-09-16 19:36:57 +01:00
Starbeamrainbowlabs 24c5263bf0
slurm predict-plot: fixup 2022-09-16 19:27:26 +01:00
Starbeamrainbowlabs 7e9c119b04
slurm: fix job name 2022-09-16 19:20:59 +01:00
Starbeamrainbowlabs 1574529704
slurm pretrain-predict: move to compute; make exclusive just in case
also shortent o 3 days
2022-09-16 19:17:42 +01:00
Starbeamrainbowlabs d7f5958af0
slurm: write new job files 2022-09-16 19:00:43 +01:00
Starbeamrainbowlabs a552cc4dad
ai vis: make parallel coordinates wider 2022-09-16 18:51:49 +01:00
Starbeamrainbowlabs a70794e661
umap: no min_dist 2022-09-16 17:09:09 +01:00
Starbeamrainbowlabs 5778fc51f7
embeddings: fix title; remove colourmap 2022-09-16 17:08:04 +01:00
Starbeamrainbowlabs 4fd852d782
fixup 2022-09-16 16:44:35 +01:00
Starbeamrainbowlabs fcab227f6a
cheese: set label for everything to 1 2022-09-16 16:42:05 +01:00
Starbeamrainbowlabs b31645bd5d
pretrain-plot: fix crash; remove water code
the model doesn't save the water encoder at this time
2022-09-16 16:24:07 +01:00
Starbeamrainbowlabs a5f03390ef
pretrain-plot: handle open w -> r 2022-09-16 16:14:30 +01:00
Starbeamrainbowlabs 1103ae5561
ai: tweak progress bar 2022-09-16 16:07:16 +01:00
Starbeamrainbowlabs 1e35802d2b
ai: fix embed i/o 2022-09-16 16:02:27 +01:00
Starbeamrainbowlabs ed94da7492
fixup 2022-09-16 15:51:26 +01:00
Starbeamrainbowlabs 366db658a8
ds predict: fix filenames in 2022-09-16 15:45:22 +01:00
Starbeamrainbowlabs e333dcba9c
tweak projection head 2022-09-16 15:36:01 +01:00
Starbeamrainbowlabs 6defd24000
bugfix: too many values to unpack 2022-09-15 19:56:17 +01:00
Starbeamrainbowlabs e3c8277255
ai: tweak the segmentation model structure 2022-09-15 19:54:50 +01:00
Starbeamrainbowlabs 1bc8a5bf13
ai: fix crash 2022-09-15 19:37:06 +01:00
Starbeamrainbowlabs bd64986332
ai: implement batched_iterator to replace .batch()
...apparently .batch() means you get a BatchedDataset or whatever when you iterate it like a tf.function instead of the actual tensor :-/
2022-09-15 19:16:38 +01:00
Starbeamrainbowlabs ccd256c00a
embed rainfall radar, not both 2022-09-15 17:37:04 +01:00
Starbeamrainbowlabs 2c74676902
predict → predict_on_batch 2022-09-15 17:31:50 +01:00
Starbeamrainbowlabs f036e79098
fixup 2022-09-15 17:09:26 +01:00
Starbeamrainbowlabs d5f1a26ba3
disable prefetching when predicting a thing 2022-09-15 17:09:09 +01:00
Starbeamrainbowlabs 8770638022
ai: call .predict(), not the model itself 2022-09-14 17:41:01 +01:00
Starbeamrainbowlabs a96cefde62
ai: predict oops 2022-09-14 17:37:48 +01:00
Starbeamrainbowlabs fa3165a5b2
dataset: simplify dataset_predict 2022-09-14 17:33:17 +01:00
Starbeamrainbowlabs 279e27c898
fixup 2022-09-14 17:16:49 +01:00
Starbeamrainbowlabs fad3313ede
fixup 2022-09-14 17:14:04 +01:00
Starbeamrainbowlabs 1e682661db
ai: kwargs in from_checkpoint 2022-09-14 17:11:06 +01:00
Starbeamrainbowlabs 6bda24d4da
ai: how did I miss that?!
bugfix ah/c
2022-09-14 16:53:43 +01:00
Starbeamrainbowlabs decdd434d8
ai from_checkpoint: bugfix 2022-09-14 16:49:01 +01:00
Starbeamrainbowlabs c9e00ea485
pretrain_predict: fix import 2022-09-14 16:02:36 +01:00
Starbeamrainbowlabs f568d8d19f
io.open → handle_open
this was we get transparent .gz support
2022-09-14 16:01:22 +01:00
Starbeamrainbowlabs 9b25186541
fix --only-gpu 2022-09-14 15:55:21 +01:00
Starbeamrainbowlabs a9c9c70d13
typo 2022-09-14 15:17:59 +01:00
Starbeamrainbowlabs fb8f884487
add umap dependencies 2022-09-14 15:16:45 +01:00
Starbeamrainbowlabs 1876a8883c
ai pretrain-predict: fix - → _ in cli parsing 2022-09-14 15:12:07 +01:00
Starbeamrainbowlabs f97b771922
make --help display help 2022-09-14 15:03:07 +01:00
Starbeamrainbowlabs 3c1aef5913
update help 2022-09-13 19:36:56 +01:00
Starbeamrainbowlabs 206257f9f5
ai pretrain_predict: no need to plot here anymore 2022-09-13 19:35:44 +01:00
Starbeamrainbowlabs 7685ec3e8b
implement ability to embed & plot pretrained embeddings 2022-09-13 19:18:59 +01:00
Starbeamrainbowlabs 7130c4fdf8
start implementing core image segmentation model 2022-09-07 17:45:38 +01:00
Starbeamrainbowlabs 22620a1854
ai: implement saving only the rainfall encoder 2022-09-06 19:48:46 +01:00
Starbeamrainbowlabs 4c4358c3e5
whitespace 2022-09-06 16:24:11 +01:00
Starbeamrainbowlabs 4202821d98
typo 2022-09-06 15:37:36 +01:00
Starbeamrainbowlabs 3e13ad12c8
ai Bugfix LayerContrastiveEncoder: channels → input_channels
for consistency
2022-09-05 23:53:16 +01:00
Starbeamrainbowlabs ead8009425
pretrain: add CLI arg for size of watch prediction width/height 2022-09-05 15:36:40 +01:00
Starbeamrainbowlabs 9d39215dd5
dataset: drop incomplete batches 2022-09-05 15:36:10 +01:00
Starbeamrainbowlabs c94f5d042e
ai: slurm fixup again 2022-09-02 19:13:56 +01:00
Starbeamrainbowlabs 7917820e59
ai: slurm fixup 2022-09-02 19:11:54 +01:00
Starbeamrainbowlabs cd104190e8
slurm: no need to be exclusive anymore 2022-09-02 19:09:45 +01:00