Commit graph

540 commits

Author SHA1 Message Date
e908e4b693
dlr: Note about RANDSEED 2024-08-30 18:50:26 +01:00
0761651ccf
tvt: implement CallbackExtraValidation, which allows for a third split
it should tie into Tensorflow's logging  just fine so long as it's the first callback in the queue.

***** TEST SCRIPT *****

model = tf.keras.Sequential([
    tf.keras.layers.Dense(64, activation='relu', input_shape=(10,)),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1)
])

model.compile(optimizer='adam', loss='mse', metrics=['mae'])
X = np.random.random((100, 10))
y = np.random.random((100, 1))

split = 80
X_train, X_val = X[:split], X[split:]
y_train, y_val = y[:split], y[split:]

train_dataset = tf.data.Dataset.from_tensor_slices((X_train, y_train)).batch(10)
val_dataset = tf.data.Dataset.from_tensor_slices((X_val, y_val)).batch(10)

history = model.fit(train_dataset,
	epochs=10,
	validation_data=val_dataset,
	callbacks=[
		CallbackExtraValidation({
			"test": val_dataset
		}, verbose=0),
		tf.keras.callbacks.CSVLogger("/dev/stdout", separator="\t")
	],
	verbose=0
)

print(f"DEBUG history {history}")
2024-08-30 18:07:17 +01:00
e5f6e6394f
Implement initial UNTESTED support for split_validation and split_test 2024-08-29 19:33:40 +01:00
b2b96ab636
replace tensorflow_addons with keras_cv 2024-08-29 19:32:54 +01:00
2d73d84e20
uv pip sync doesn't install dependencies of packages 2024-08-29 19:31:42 +01:00
b31485f7a1
add pyproject.toml 2024-08-29 18:58:57 +01:00
61ee2326cf
licence change 2024-08-29 18:36:11 +01:00
c98c42fa7e
dataset_mono: Implement validate_percentage + test_percentage support
This removes the train_percentage argument

TODO: map this forwards to enable support in deeplabv3_plus_test_rainfall

...thinking about it, it's really not a test now, is it? Updating the filename would be such a /hassle/ though....
2024-08-29 16:44:19 +01:00
5d62e3cee8
Implement env from PhD-smflooding-scene 2024-08-29 16:43:29 +01:00
a75d4f5d79
dataset_mono: add todo 2024-08-02 19:48:29 +01:00
0f9f185983
dlr: add PARALLEL_READS env var, update docs 2023-11-30 16:33:22 +00:00
e14fa275ab
rr de: rerun to recaalculate
these values are accurate to our dataset now
2023-11-23 18:35:39 +00:00
76b2956703
rr de: update to dataset_mono_predict 2023-11-03 15:48:53 +00:00
e44b5533b1
plot_metrics_multi: add option to plot train/val separately 2023-07-20 15:49:09 +01:00
b5b26d980b
dlr: print filepaths
we need to know which is which with  this seed so we can visualise for the paper
2023-07-13 19:58:38 +01:00
636b316bfc
plot_multi: actually fix the plots 2023-07-05 16:35:26 +01:00
9efc72db73
dlr ds/mono: just why 2023-06-16 18:36:18 +01:00
a4e80229fb
dlr ds/mono: fixup 2023-06-16 18:35:02 +01:00
7869505cfb
dlr: add PREDICT_AS_ONE 2023-06-16 18:23:40 +01:00
4bbc4c29c4
plot_metrics_multi: add RESOLUTION env var 2023-06-14 15:50:37 +01:00
18db54f0a7
plot_metrics_multi: FINALLY write a script to generate charts for multiple models at the same time 2023-06-08 18:13:20 +01:00
fa37185950
plot_metrics: improve CLI 2023-06-08 15:20:32 +01:00
3be38823db
dlr: fix another crash 2023-05-19 22:00:23 +01:00
3e42972fb0
dlr: add comment 2023-05-11 17:45:52 +01:00
fe05dd33fc
dlr: missing quotes 2023-05-11 16:23:36 +01:00
5a73388a80
dlr: add RANDSEED to slurm 2023-05-11 16:02:13 +01:00
ae0c672f60
Merge branch 'main' of git.starbeamrainbowlabs.com:sbrl/PhD-Rainfall-Radar 2023-05-11 15:59:08 +01:00
20092c6829
shuffle: add random seed env var 2023-05-11 15:59:01 +01:00
535bd6f579
dlr: fix crash 2023-05-07 19:00:02 +01:00
ac040717e6
autoset the value of factor 2023-05-04 19:57:02 +01:00
214afcc914
dlr: steps per execution is ugh 2023-05-04 19:54:51 +01:00
f0a2e7c450
dlr: print values of new env vars 2023-05-04 19:42:40 +01:00
8593999eb6
dlr: add JIT_COMPILE 2023-05-04 18:22:18 +01:00
dddc08c663
dlr: set steps_per_execution to 16 by default 2023-05-04 18:13:08 +01:00
e2e6a56b40
dlr: add UPSAMPLE env var
...AND actually add the functionality this time!
2023-05-04 17:40:16 +01:00
31687da931
celldice: actually do log(cosh())
....just wow.
2023-05-03 15:00:10 +01:00
4a4df380e3
if name == main 2023-03-23 18:09:52 +00:00
81cad8e6b4
newline 2023-03-23 18:01:21 +00:00
1bd59dc038
finish script to plot generic metrics 2023-03-23 18:00:00 +00:00
698bbe2ffb
start working on plotting script, but it isn't finished yet 2023-03-22 17:45:06 +00:00
6b17d45aad
dlr: plot all metrics 2023-03-22 17:41:34 +00:00
e565c36149
deeplabv3+: prepare for ConvNeXt 2023-03-14 21:51:41 +00:00
779b546897
dlr: log new env vars 2023-03-14 20:28:22 +00:00
623208ba6d
dlr: add env var for water thresholding 2023-03-14 20:18:39 +00:00
c5fc62c411
dlr CHANGE: Add optional log(cosh(dice_loss))
Ref https://doi.org/10.1109/cibcb48159.2020.9277638
2023-03-10 20:24:13 +00:00
f25d1b5b1a
dlr CHANGE: properly normalise the heightmap 2023-03-10 19:13:32 +00:00
8e76ac2864
dlr: typo 2023-03-10 17:40:16 +00:00
5e79825292
dlr: tf.Tensor → numpy array → list → json → disk 2023-03-10 17:34:49 +00:00
b287970032
dlr: missing import 2023-03-10 17:31:03 +00:00
e6cd0aa1b9
dlr: fix crash 2023-03-10 17:14:06 +00:00