sbrl-github/research-rainfallradar
1
0
Fork 0
mirror of https://github.com/sbrl/research-rainfallradar synced 2024-12-22 06:05:01 +00:00
Code Issues Projects Releases Packages Wiki Activity
653 commits 1 branch 0 tags 2.5 MiB
Commit graph

653 commits

This branch
This branch
All branches
Author SHA1 Message Date
Starbeamrainbowlabs
4dd66a9bdf
dataset_mono: add water_threshold=None support 
This is for the stupid pointless regression thing
Like just let me get on with sample weighting and accounting for extreme event bias already!
 2024-12-12 23:36:54 +00:00
Starbeamrainbowlabs
7e00ede747
Merge branch 'main' of github.com:sbrl/research-rainfallradar 2024-12-12 19:36:09 +00:00
Starbeamrainbowlabs
c7a3a2d08a
dlr: fix crash ref CallbackExtraValidation 2024-12-12 19:33:18 +00:00
Starbeamrainbowlabs
fe374560a1
I *hate* Tensorflow SO MUCH...... 2024-11-14 22:38:27 +00:00
Starbeamrainbowlabs
7c4f3d325d
slurm/dlr: fix workaround logic 2024-11-14 22:28:39 +00:00
Starbeamrainbowlabs
17d2d2bcaf
slurm/dlr: tensorflow is dumb 
Workaround for this crash on Tensorflow 2.13:

Could not load library libcublasLt.so.12. Error: libcublasLt.so.12: cannot open shared object file: No such file or directory
 2024-11-14 22:26:16 +00:00
Starbeamrainbowlabs
52af6f00ec
dlr/dataset_mono: fix crash in new ssplit3 setup 2024-11-14 22:03:22 +00:00
Starbeamrainbowlabs
159f8a4679
slurm/dlr: DIR_RAINFALLWATER default → ~/data/.... 2024-11-14 21:59:32 +00:00
Starbeamrainbowlabs
090ab78457
dlr: fix (another) crash 2024-11-14 21:53:59 +00:00
Starbeamrainbowlabs
7be2222207
dlr: import ...as env 2024-11-14 21:52:41 +00:00
Starbeamrainbowlabs
a7ab5ee341
slurm/dlr: cpu cores 14→9 2024-11-14 21:49:54 +00:00
Starbeamrainbowlabs
e83d901779
env.py: fix crash 2024-11-14 21:49:03 +00:00
Starbeamrainbowlabs
2b69d2c4f2
slurm/dlr: correct logging msg 2024-11-14 21:37:11 +00:00
Starbeamrainbowlabs
bd2c6b1c3f
slurm/dlr: don't runmodule load .... on csgpu cluster 2024-11-14 21:34:04 +00:00
Starbeamrainbowlabs
4ac7082754
write commit info to file in DIR_OUTPUT 2024-11-14 21:17:53 +00:00
Starbeamrainbowlabs
8befef5fc1
slurm/dlr: logging 
it Should™ work now?
TODO test this!
 2024-11-14 19:55:59 +00:00
Starbeamrainbowlabs
c82e4bf5ac
Merge branch 'main' of github.com:sbrl/research-rainfallradar 2024-11-14 19:09:39 +00:00
Starbeamrainbowlabs
daa923a88b
pyproject.toml: typo 2024-11-14 19:09:26 +00:00
Starbeamrainbowlabs
d0dc1b4280
.gitignore: ignore some more backup files: draw.io, .bak.png 2024-11-08 22:05:13 +00:00
Starbeamrainbowlabs
04ea305b70
dlr/slurm: implement USE_CONDA, module command opt-support 2024-11-08 21:47:55 +00:00
Starbeamrainbowlabs
f94f777f0a
dlr: make cv2 import optional 2024-11-01 20:40:55 +00:00
Starbeamrainbowlabs
05fd955067
README: note about RANDSEED 2024-08-30 19:02:18 +01:00
Starbeamrainbowlabs
e908e4b693
dlr: Note about RANDSEED 2024-08-30 18:50:26 +01:00
Starbeamrainbowlabs
f46a86c9a4
README: Document SPLIT_VALIDATE, SPLIT_TEST 2024-08-30 18:49:38 +01:00
Starbeamrainbowlabs
0761651ccf
tvt: implement CallbackExtraValidation, which allows for a third split 
it should tie into Tensorflow's logging  just fine so long as it's the first callback in the queue.

***** TEST SCRIPT *****

model = tf.keras.Sequential([
    tf.keras.layers.Dense(64, activation='relu', input_shape=(10,)),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1)
])

model.compile(optimizer='adam', loss='mse', metrics=['mae'])
X = np.random.random((100, 10))
y = np.random.random((100, 1))

split = 80
X_train, X_val = X[:split], X[split:]
y_train, y_val = y[:split], y[split:]

train_dataset = tf.data.Dataset.from_tensor_slices((X_train, y_train)).batch(10)
val_dataset = tf.data.Dataset.from_tensor_slices((X_val, y_val)).batch(10)

history = model.fit(train_dataset,
	epochs=10,
	validation_data=val_dataset,
	callbacks=[
		CallbackExtraValidation({
			"test": val_dataset
		}, verbose=0),
		tf.keras.callbacks.CSVLogger("/dev/stdout", separator="\t")
	],
	verbose=0
)

print(f"DEBUG history {history}")
 2024-08-30 18:07:17 +01:00
Starbeamrainbowlabs
b5310304bd
wrangler: update dependencies 2024-08-29 19:38:02 +01:00
Starbeamrainbowlabs
e5f6e6394f
Implement initial UNTESTED support for split_validation and split_test 2024-08-29 19:33:40 +01:00
Starbeamrainbowlabs
b2b96ab636
replace tensorflow_addons with keras_cv 2024-08-29 19:32:54 +01:00
Starbeamrainbowlabs
2d73d84e20
uv pip sync doesn't install dependencies of packages 2024-08-29 19:31:42 +01:00
Starbeamrainbowlabs
b31485f7a1
add pyproject.toml 2024-08-29 18:58:57 +01:00
Starbeamrainbowlabs
61ee2326cf
licence change 2024-08-29 18:36:11 +01:00
Starbeamrainbowlabs
18fd4b5d4f
Correct github language bar; Jupyter notebook are not part of the main project 2024-08-29 18:32:28 +01:00
Starbeamrainbowlabs
c98c42fa7e
dataset_mono: Implement validate_percentage + test_percentage support 
This removes the train_percentage argument

TODO: map this forwards to enable support in deeplabv3_plus_test_rainfall

...thinking about it, it's really not a test now, is it? Updating the filename would be such a /hassle/ though....
 2024-08-29 16:44:19 +01:00
Starbeamrainbowlabs
5d62e3cee8
Implement env from PhD-smflooding-scene 2024-08-29 16:43:29 +01:00
Starbeamrainbowlabs
a75d4f5d79
dataset_mono: add todo 2024-08-02 19:48:29 +01:00
Starbeamrainbowlabs
e15f6cd954
Update README w/proper paper link 2024-06-21 16:03:58 +01:00
Starbeamrainbowlabs
d22dd6ff18
README: more tidy up 2023-11-30 17:08:50 +00:00
Starbeamrainbowlabs
1cbb241786
README: tidy up 2023-11-30 17:02:08 +00:00
Starbeamrainbowlabs
844d8e6dd4
rw: update dependencies 2023-11-30 17:01:13 +00:00
Starbeamrainbowlabs
c5be527ffb
rw/README: fill out 2023-11-30 16:59:52 +00:00
Starbeamrainbowlabs
4626549352
rw/recompress: clarify subcommand description 2023-11-30 16:59:45 +00:00
Starbeamrainbowlabs
18195c2d15
rw: mark as 'private' in package.json to avoid accidental publication 2023-11-30 16:59:31 +00:00
Starbeamrainbowlabs
002203d20b
vscodium/settings: spellings 2023-11-30 16:59:06 +00:00
Starbeamrainbowlabs
294fde7c82
rw; add requirements.txt 2023-11-30 16:45:04 +00:00
Starbeamrainbowlabs
92fc34ebb1
rw/child_process:json2tfrecord: Add RAINFALL_MAX_NUMBER env var 
use use you'd need to export RAINFALL_MAX_NUMBER=some_integer_value, since it is used by a subprocess rather than the main Node.js process itself
 2023-11-30 16:44:43 +00:00
Starbeamrainbowlabs
0f9f185983
dlr: add PARALLEL_READS env var, update docs 2023-11-30 16:33:22 +00:00
Starbeamrainbowlabs
60674fb6b3
README: finish filling out 2023-11-30 16:32:46 +00:00
Starbeamrainbowlabs
13ae4df725
vscodium/settings: spellings 2023-11-30 16:32:17 +00:00
Starbeamrainbowlabs
df621fd7d2
README: Continue filling out, but we're not there yet. 2023-11-29 17:31:47 +00:00
Starbeamrainbowlabs
3eec01a28c
README: update image 2023-11-29 15:16:01 +00:00