Commit graph

569 commits

Author SHA1 Message Date
fe2f8b3821
dlr: implement stage 2 WIP of opt regression support via rmse
....this is so stupid. There's a reason why diffusion models are a thing and we don't use image segmentation models for this!!
2024-12-20 19:36:10 +00:00
58a7e22a4d
scripts/crossval-stbl: add TODOs 2024-12-20 18:38:21 +00:00
5a412ddc26
scripts/crossval-stbl: finish off the script
TODO switch ou  median absolute distance for something else when Nina replies
2024-12-20 18:37:18 +00:00
fda40b05c9
scripts/stbl-crossval: initial WIP draft 2024-12-20 15:11:41 +00:00
f8a1e1b594
Fix lots of ruff linter warnings 2024-12-19 15:39:09 +00:00
0260e626db
dataset_mono: address ruff linting errors 2024-12-19 15:22:59 +00:00
edfc2721a8
Merge branch 'main' of github.com:sbrl/research-rainfallradar 2024-12-19 15:21:51 +00:00
8c3ddbd86f
dlr: address some ruff linting warnings 2024-12-19 15:21:36 +00:00
efc5d2c253
dlr: remove todo that has already been done 2024-12-12 23:37:18 +00:00
4dd66a9bdf
dataset_mono: add water_threshold=None support
This is for the stupid pointless regression thing
Like just let me get on with sample weighting and accounting for extreme event bias already!
2024-12-12 23:36:54 +00:00
7e00ede747
Merge branch 'main' of github.com:sbrl/research-rainfallradar 2024-12-12 19:36:09 +00:00
c7a3a2d08a
dlr: fix crash ref CallbackExtraValidation 2024-12-12 19:33:18 +00:00
fe374560a1
I *hate* Tensorflow SO MUCH...... 2024-11-14 22:38:27 +00:00
7c4f3d325d
slurm/dlr: fix workaround logic 2024-11-14 22:28:39 +00:00
17d2d2bcaf
slurm/dlr: tensorflow is dumb
Workaround for this crash on Tensorflow 2.13:

Could not load library libcublasLt.so.12. Error: libcublasLt.so.12: cannot open shared object file: No such file or directory
2024-11-14 22:26:16 +00:00
52af6f00ec
dlr/dataset_mono: fix crash in new ssplit3 setup 2024-11-14 22:03:22 +00:00
159f8a4679
slurm/dlr: DIR_RAINFALLWATER default → ~/data/.... 2024-11-14 21:59:32 +00:00
090ab78457
dlr: fix (another) crash 2024-11-14 21:53:59 +00:00
7be2222207
dlr: import ...as env 2024-11-14 21:52:41 +00:00
a7ab5ee341
slurm/dlr: cpu cores 14→9 2024-11-14 21:49:54 +00:00
e83d901779
env.py: fix crash 2024-11-14 21:49:03 +00:00
2b69d2c4f2
slurm/dlr: correct logging msg 2024-11-14 21:37:11 +00:00
bd2c6b1c3f
slurm/dlr: don't runmodule load .... on csgpu cluster 2024-11-14 21:34:04 +00:00
4ac7082754
write commit info to file in DIR_OUTPUT 2024-11-14 21:17:53 +00:00
8befef5fc1
slurm/dlr: logging
it Should™ work now?
TODO test this!
2024-11-14 19:55:59 +00:00
c82e4bf5ac
Merge branch 'main' of github.com:sbrl/research-rainfallradar 2024-11-14 19:09:39 +00:00
daa923a88b
pyproject.toml: typo 2024-11-14 19:09:26 +00:00
04ea305b70
dlr/slurm: implement USE_CONDA, module command opt-support 2024-11-08 21:47:55 +00:00
f94f777f0a
dlr: make cv2 import optional 2024-11-01 20:40:55 +00:00
e908e4b693
dlr: Note about RANDSEED 2024-08-30 18:50:26 +01:00
0761651ccf
tvt: implement CallbackExtraValidation, which allows for a third split
it should tie into Tensorflow's logging  just fine so long as it's the first callback in the queue.

***** TEST SCRIPT *****

model = tf.keras.Sequential([
    tf.keras.layers.Dense(64, activation='relu', input_shape=(10,)),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1)
])

model.compile(optimizer='adam', loss='mse', metrics=['mae'])
X = np.random.random((100, 10))
y = np.random.random((100, 1))

split = 80
X_train, X_val = X[:split], X[split:]
y_train, y_val = y[:split], y[split:]

train_dataset = tf.data.Dataset.from_tensor_slices((X_train, y_train)).batch(10)
val_dataset = tf.data.Dataset.from_tensor_slices((X_val, y_val)).batch(10)

history = model.fit(train_dataset,
	epochs=10,
	validation_data=val_dataset,
	callbacks=[
		CallbackExtraValidation({
			"test": val_dataset
		}, verbose=0),
		tf.keras.callbacks.CSVLogger("/dev/stdout", separator="\t")
	],
	verbose=0
)

print(f"DEBUG history {history}")
2024-08-30 18:07:17 +01:00
e5f6e6394f
Implement initial UNTESTED support for split_validation and split_test 2024-08-29 19:33:40 +01:00
b2b96ab636
replace tensorflow_addons with keras_cv 2024-08-29 19:32:54 +01:00
2d73d84e20
uv pip sync doesn't install dependencies of packages 2024-08-29 19:31:42 +01:00
b31485f7a1
add pyproject.toml 2024-08-29 18:58:57 +01:00
61ee2326cf
licence change 2024-08-29 18:36:11 +01:00
c98c42fa7e
dataset_mono: Implement validate_percentage + test_percentage support
This removes the train_percentage argument

TODO: map this forwards to enable support in deeplabv3_plus_test_rainfall

...thinking about it, it's really not a test now, is it? Updating the filename would be such a /hassle/ though....
2024-08-29 16:44:19 +01:00
5d62e3cee8
Implement env from PhD-smflooding-scene 2024-08-29 16:43:29 +01:00
a75d4f5d79
dataset_mono: add todo 2024-08-02 19:48:29 +01:00
0f9f185983
dlr: add PARALLEL_READS env var, update docs 2023-11-30 16:33:22 +00:00
e14fa275ab
rr de: rerun to recaalculate
these values are accurate to our dataset now
2023-11-23 18:35:39 +00:00
76b2956703
rr de: update to dataset_mono_predict 2023-11-03 15:48:53 +00:00
e44b5533b1
plot_metrics_multi: add option to plot train/val separately 2023-07-20 15:49:09 +01:00
b5b26d980b
dlr: print filepaths
we need to know which is which with  this seed so we can visualise for the paper
2023-07-13 19:58:38 +01:00
636b316bfc
plot_multi: actually fix the plots 2023-07-05 16:35:26 +01:00
9efc72db73
dlr ds/mono: just why 2023-06-16 18:36:18 +01:00
a4e80229fb
dlr ds/mono: fixup 2023-06-16 18:35:02 +01:00
7869505cfb
dlr: add PREDICT_AS_ONE 2023-06-16 18:23:40 +01:00
4bbc4c29c4
plot_metrics_multi: add RESOLUTION env var 2023-06-14 15:50:37 +01:00
18db54f0a7
plot_metrics_multi: FINALLY write a script to generate charts for multiple models at the same time 2023-06-08 18:13:20 +01:00