Commit graph

457 commits

Author SHA1 Message Date
Starbeamrainbowlabs 4fd9feba4f
dlr eo: tidyup 2023-03-01 16:47:36 +00:00
Starbeamrainbowlabs 69b5ae8838
dlr eo: this should fix it 2023-02-23 17:24:30 +00:00
Starbeamrainbowlabs 9f1cee2927
dlr eo: cheese it by upsampling and then downsampling again 2023-02-23 16:47:00 +00:00
Starbeamrainbowlabs 96b94ec55b
upsampling test 2023-02-23 16:19:44 +00:00
Starbeamrainbowlabs 747ddfd41b
weird, XLA_FLAGS cuda data dir wasn't needed before
libdevice not found at ./libdevice.10.bc
2023-02-10 13:28:34 +00:00
Starbeamrainbowlabs e43274cd91
dlr eo: add VAL_STEPS_PER_EPOCH 2023-02-03 16:41:30 +00:00
Starbeamrainbowlabs 8446a842d1
typo 2023-02-03 16:01:54 +00:00
Starbeamrainbowlabs 1a8f10339a
LayerConvNeXtGamma: fix for mixed precision mode 2023-02-02 16:22:08 +00:00
Starbeamrainbowlabs a630db2c49
dlr eo: fixup 2023-02-02 16:17:52 +00:00
Starbeamrainbowlabs 2bf1872aca
dlr eo: add JIT_COMPILE and MIXED_PRECISION 2023-02-02 16:14:09 +00:00
Starbeamrainbowlabs 71088b8c0b
typo 2023-02-02 15:48:49 +00:00
Starbeamrainbowlabs f7666865a0
dlr eo: add STEPS_PER_EXECUTION 2023-02-02 15:47:08 +00:00
Starbeamrainbowlabs f8202851a1
dlr eo: add LEARNING_RATE 2023-01-27 16:51:13 +00:00
Starbeamrainbowlabs fb898ea72b
slurm eo: seriously....? 2023-01-26 17:02:33 +00:00
Starbeamrainbowlabs be946091b1
slurm eo: DIR_OUTPUT → DIRPATH_OUTPUT 2023-01-26 16:52:14 +00:00
Starbeamrainbowlabs 4703bdbea1
SLURM: add job file for encoderonly
It's pretty much bugfixed, but illykin doesn't have enough RAM to support it at the moment :-(
2023-01-20 20:32:35 +00:00
Starbeamrainbowlabs 818d77c733
Make dirpath_rainfallwater consistent with other experiments 2023-01-20 20:31:26 +00:00
Starbeamrainbowlabs e72d3991b8
switch to a smaller ConvNeXt 2023-01-20 19:14:38 +00:00
Starbeamrainbowlabs e1ad16a213
debug A 2023-01-20 18:58:45 +00:00
Starbeamrainbowlabs 65a2e16a4c
ds_eo: lower memory usage 2023-01-20 18:55:52 +00:00
Starbeamrainbowlabs b5e68fc1a3
eo: don't downsample ConvNeXt at beginning 2023-01-20 18:49:46 +00:00
Starbeamrainbowlabs d5fdab50ed
dlreo: missing import 2023-01-20 18:40:35 +00:00
Starbeamrainbowlabs 4514086dc6
make_encoderonly: kwargs 2023-01-20 18:39:35 +00:00
Starbeamrainbowlabs 35dbd3f8bc
ds eo: scale up rainfall data
It's taken most fo the afternoon to spot this one 🤦
2023-01-20 18:37:08 +00:00
Starbeamrainbowlabs 5b54ceec48
ds eo: debug 2023-01-20 18:36:14 +00:00
Starbeamrainbowlabs a3787f0647
debug 2023-01-20 18:34:56 +00:00
Starbeamrainbowlabs 9bd5e0f7a3
ds_eo: debug
there's something fishy going on here.
2023-01-20 17:25:22 +00:00
Starbeamrainbowlabs aa3831bc4f
ds_eo: pass water to reshape 2023-01-17 19:07:16 +00:00
Starbeamrainbowlabs a01c49414f
ds_eo: typo 2023-01-17 19:06:29 +00:00
Starbeamrainbowlabs b9bea26d26
ds_eo: rogue variables 2023-01-17 19:03:14 +00:00
Starbeamrainbowlabs 64c57bbc21
dlr: add no-requeue
Ref https://support.hull.ac.uk/tas/public/ssp/content/detail/incident?unid=652db7ac6e73485c9f7658db78b2b628
2023-01-17 18:20:26 +00:00
Starbeamrainbowlabs a28bbb9cf7
encoderonly: make executable 2023-01-17 17:55:10 +00:00
Starbeamrainbowlabs 835b376c72
slurm dlr: log exit code 2023-01-17 15:18:26 +00:00
Starbeamrainbowlabs 40a550f155
slurm dlr: fixup 2023-01-16 18:45:08 +00:00
Starbeamrainbowlabs 7bcf13f8d8
dlr: typo 2023-01-16 18:02:09 +00:00
Starbeamrainbowlabs 6ff2864d23
slurm dlr: shell out in conda; redirect stderr & stdout to disk inside the experiments folder
Also, if the job restarts, we still save the previous run's results because we append rather than overwrite
2023-01-16 17:32:22 +00:00
Starbeamrainbowlabs 0b31c9e700
log start time, end time, and elapsed
just in case...!
2023-01-16 17:30:20 +00:00
Starbeamrainbowlabs 1a4ac3ed66
encoderonly: add graph plotting 2023-01-13 19:08:38 +00:00
Starbeamrainbowlabs 82e01da70b
fixup.... again
oops
2023-01-13 18:47:29 +00:00
Starbeamrainbowlabs 7b10f5c5fe
dlr: add learning_rate env var 2023-01-13 18:29:39 +00:00
Starbeamrainbowlabs 2f0ce0aa13
again 2023-01-13 18:27:03 +00:00
Starbeamrainbowlabs e04d6ab1b6
fixup again 2023-01-13 18:23:32 +00:00
Starbeamrainbowlabs 37d1598b0b
loss cel+dice: fixup 2023-01-13 18:21:11 +00:00
Starbeamrainbowlabs 0f0b691b5d
loss cel + dice: fixup 2023-01-13 18:09:31 +00:00
Starbeamrainbowlabs be77f035c8
dlr: add cross-entropy + dice loss fn option 2023-01-13 17:58:00 +00:00
Starbeamrainbowlabs b2a5acaf4e
dlr dsm: clash 2023-01-13 17:26:38 +00:00
Starbeamrainbowlabs f7672db599
annoying 2023-01-13 17:00:47 +00:00
Starbeamrainbowlabs 3c4d1c5140
dlr: Add support for stripping isolated water pixels
That is, water pixels that have no other water pixels immediately adjacent thereto (diagonals count).
2023-01-13 16:57:26 +00:00
Starbeamrainbowlabs ce1467461d
fixup 2023-01-13 16:47:52 +00:00
Starbeamrainbowlabs 0b676fa391
move shuffle to subdir 2023-01-13 16:47:35 +00:00