359 commits

Author	SHA1	Message	Date
Starbeamrainbowlabs	77b8a1a8db	dataset_segmenter: squeeze	2022-11-29 19:16:15 +00:00
Starbeamrainbowlabs	2258b5a229	slurm-train: reduce RAM required by 10GB	2022-11-29 19:15:34 +00:00
Starbeamrainbowlabs	01101ad30b	losscrossentropy: return the reduced value * facepalm *	2022-11-29 19:07:08 +00:00
Starbeamrainbowlabs	ff65393e78	log file naming update	2022-11-29 18:41:14 +00:00
Starbeamrainbowlabs	37f196a785	LossCrossentropy: add kwargs	2022-11-29 15:40:35 +00:00
Starbeamrainbowlabs	838ff56a3b	mono: fix loading checkpoint	2022-11-29 15:25:11 +00:00
Starbeamrainbowlabs	dba6cbffcd	WHY. * facepalms *	2022-11-28 19:33:42 +00:00
Starbeamrainbowlabs	57b8eb93fb	fixup	2022-11-28 19:09:35 +00:00
Starbeamrainbowlabs	6640a41bb7	almost got it....? it's not what I expected....!	2022-11-28 19:08:50 +00:00
Starbeamrainbowlabs	f48473b703	fixup	2022-11-28 19:00:11 +00:00
Starbeamrainbowlabs	f6feb125e3	this iss ome serious debugging. ... This commit will produce an extremely large volume of output.	2022-11-28 18:57:41 +00:00
Starbeamrainbowlabs	09f81b0746	train_mono: debug ... this commit will generate a large amount of debug output.	2022-11-28 16:46:17 +00:00
Starbeamrainbowlabs	f39e4ade70	LayerConvNextGamma: fix config serialisation bug ... .....this is unlikely to be the problem as this bug is in an unused code path.	2022-11-25 21:16:31 +00:00
Starbeamrainbowlabs	e7410fb480	train_mono_predict: limit label size to 64x64 ... that's the size the model predicts	2022-11-25 17:47:17 +00:00
Starbeamrainbowlabs	51dd484d13	fixup	2022-11-25 16:55:45 +00:00
Starbeamrainbowlabs	884c4eb150	rainfall_stats: formatting again	2022-11-24 19:08:07 +00:00
Starbeamrainbowlabs	bfe038086c	rainfall_stats: formatting	2022-11-24 19:07:44 +00:00
Starbeamrainbowlabs	7dba03200f	fixup	2022-11-24 19:06:48 +00:00
Starbeamrainbowlabs	e5258b9c66	typo	2022-11-24 19:06:13 +00:00
Starbeamrainbowlabs	64d646bb13	rainfall_stats: formatting	2022-11-24 19:05:35 +00:00
Starbeamrainbowlabs	675c7a7448	fixup	2022-11-24 19:03:28 +00:00
Starbeamrainbowlabs	afc1cdcf02	fixup	2022-11-24 19:02:58 +00:00
Starbeamrainbowlabs	e4bea89c89	typo	2022-11-24 19:01:52 +00:00
Starbeamrainbowlabs	a40cbe8705	rainfall_stats: remove unused imports	2022-11-24 19:01:18 +00:00
Starbeamrainbowlabs	fe57d6aab2	rainfall_stats: initial implementation ... this might reveal why we are having problems. If most/all the rainfall radar data is v small numbers, normalising might help.	2022-11-24 18:58:16 +00:00
Starbeamrainbowlabs	3131b4f7b3	debug2	2022-11-24 18:25:32 +00:00
Starbeamrainbowlabs	d55a13f536	debug	2022-11-24 18:24:03 +00:00
Starbeamrainbowlabs	1f60f2a580	do_argmax	2022-11-24 18:11:03 +00:00
Starbeamrainbowlabs	6c09d5254d	fixup	2022-11-24 17:57:48 +00:00
Starbeamrainbowlabs	54a841efe9	train_mono_predict: convert to correct format	2022-11-24 17:56:07 +00:00
Starbeamrainbowlabs	105dc5bc56	missing kwargs	2022-11-24 17:51:29 +00:00
Starbeamrainbowlabs	1e1d6dd273	fixup	2022-11-24 17:48:19 +00:00
Starbeamrainbowlabs	011e0aef78	update cli docs	2022-11-24 16:38:07 +00:00
Starbeamrainbowlabs	773944f9fa	train_mono_predict: initial implementation	2022-11-24 16:33:50 +00:00
Starbeamrainbowlabs	d31326cb30	slurm train mono: fix partition name	2022-11-22 17:02:02 +00:00
Starbeamrainbowlabs	ce28ac4013	slurm: add job for train_mono	2022-11-22 16:58:46 +00:00
Starbeamrainbowlabs	3a0356929c	mono: drop the sparse	2022-11-22 16:20:56 +00:00
Starbeamrainbowlabs	7e8f63f8ba	fixup	2022-11-21 19:38:24 +00:00
Starbeamrainbowlabs	ace4c8b246	dataset_mono: debug	2022-11-21 18:46:21 +00:00
Starbeamrainbowlabs	527b34942d	convnext_inverse: kernel_size 4→2	2022-11-11 19:29:37 +00:00
Starbeamrainbowlabs	0662d0854b	model_mono: fix bottleneck	2022-11-11 19:11:40 +00:00
Starbeamrainbowlabs	73acda6d9a	fix debug logging	2022-11-11 19:08:38 +00:00
Starbeamrainbowlabs	9da059d738	model shape logging	2022-11-11 19:03:37 +00:00
Starbeamrainbowlabs	00917b2698	dataset_mono: log shapes	2022-11-11 19:02:43 +00:00
Starbeamrainbowlabs	54ae88b1b4	in this entire blasted project I have yet to get the rotation of anything correct....!	2022-11-11 18:58:45 +00:00
Starbeamrainbowlabs	a7a475dcd1	debug 2	2022-11-11 18:38:07 +00:00
Starbeamrainbowlabs	bf2f6e9b64	debug logging ... it begins again	2022-11-11 18:31:40 +00:00
Starbeamrainbowlabs	481eeb3759	mono: fix dataset preprocessing ... rogue dimension	2022-11-11 18:31:27 +00:00
Starbeamrainbowlabs	9035450213	mono: instantiate right model	2022-11-11 18:28:29 +00:00
Starbeamrainbowlabs	69a2d0cf04	fixup	2022-11-11 18:27:01 +00:00
Starbeamrainbowlabs	65e801cf28	train_mono: fix crash	2022-11-11 18:26:25 +00:00
Starbeamrainbowlabs	8ac5159adc	dataset_mono: simplify param passing, onehot+threshold water depth data	2022-11-11 18:23:50 +00:00
Starbeamrainbowlabs	3a3f7e85da	typo	2022-11-11 18:03:09 +00:00
Starbeamrainbowlabs	3313f77c88	Add (untested) mono rainfall → water depth model ... * sighs * Unfortunately I can't seem to get contrastive learning to work.....	2022-11-10 22:36:11 +00:00
Starbeamrainbowlabs	ce194d9227	slurm: customise log file names	2022-11-10 21:09:34 +00:00
Starbeamrainbowlabs	9384b89165	model_segmentation: spare → normal crossentropy, activation functions at end	2022-11-10 20:53:37 +00:00
Starbeamrainbowlabs	b6676e7361	switch from sparse to normal crossentropy	2022-11-10 20:50:56 +00:00
Starbeamrainbowlabs	d8be26d476	Merge branch 'main' of git.starbeamrainbowlabs.com:sbrl/PhD-Rainfall-Radar	2022-11-10 20:49:01 +00:00
Starbeamrainbowlabs	b03388de60	dataset_segmenter: DEBUG: fix water shape	2022-11-10 20:48:21 +00:00
Starbeamrainbowlabs	daf691bf43	typo	2022-11-10 19:55:00 +00:00
Starbeamrainbowlabs	0aa2ce19f5	read_metadata: support file inputs as well as dirs	2022-11-10 19:53:30 +00:00
Starbeamrainbowlabs	aa7d9b8cf6	fixup	2022-11-10 19:46:09 +00:00
Starbeamrainbowlabs	0894bd09e8	train_predict: add error message for parrams.json not found	2022-11-10 19:45:41 +00:00
Starbeamrainbowlabs	0353072d15	allow pretrain to run on gpu ... we've slashed the size of the 2nd encoder, so ti should fit naow?	2022-11-04 17:02:07 +00:00
Starbeamrainbowlabs	44ad51f483	CallbackNBatchCsv: bugfix .sort() → sorted()	2022-11-04 16:40:21 +00:00
Starbeamrainbowlabs	4dddcfcb42	pretrain_predict: missing \n	2022-11-04 16:01:28 +00:00
Starbeamrainbowlabs	1375201c5f	CallbackNBatchCsv: open_handle mode	2022-11-03 18:29:00 +00:00
Starbeamrainbowlabs	3206d6b7e7	slurm: rename segmenter job name	2022-11-03 17:12:27 +00:00
Starbeamrainbowlabs	f2ae74ce7b	how could I be so stupid..... round 2	2022-11-02 17:38:26 +00:00
Starbeamrainbowlabs	5f8d6dc6ea	Add metrics every 64 batches ... this is important, because with large batches it can be difficult to tell what's happening inside each epoch.	2022-10-31 19:26:10 +00:00
Starbeamrainbowlabs	cf872ef739	how could I be so *stupid*......	2022-10-31 18:40:58 +00:00
Starbeamrainbowlabs	da32d75778	make_callbacks: display steps, not samples	2022-10-31 18:36:28 +00:00
Starbeamrainbowlabs	dfef7db421	moar debugging	2022-10-31 18:26:34 +00:00
Starbeamrainbowlabs	172cf9d8ce	tweak	2022-10-31 18:19:43 +00:00
Starbeamrainbowlabs	dbe35ee943	loss: comment l2 norm	2022-10-31 18:09:03 +00:00
Starbeamrainbowlabs	5e60319024	fixup	2022-10-31 17:56:49 +00:00
Starbeamrainbowlabs	b986b069e2	debug party time	2022-10-31 17:50:29 +00:00
Starbeamrainbowlabs	458faa96d2	loss: fixup	2022-10-31 17:18:21 +00:00
Starbeamrainbowlabs	55dc05e8ce	contrastive: comment weights that aren't needed	2022-10-31 16:26:48 +00:00
Starbeamrainbowlabs	33391eaf16	train_predict/jsonl: don't argmax ... I'm interested inthe raw values	2022-10-26 17:21:19 +01:00
Starbeamrainbowlabs	74f2cdb900	train_predict: .list() → .tolist()	2022-10-26 17:12:36 +01:00
Starbeamrainbowlabs	4f9d543695	train_predict: don't pass model_code ... it's redundant	2022-10-26 17:11:36 +01:00
Starbeamrainbowlabs	1b489518d0	segmenter: add LayerStack2Image to custom_objects	2022-10-26 17:05:50 +01:00
Starbeamrainbowlabs	48ae8a5c20	LossContrastive: normalise features as per the paper	2022-10-26 16:52:56 +01:00
Starbeamrainbowlabs	843cc8dc7b	contrastive: rewrite the loss function. ... The CLIP paper *does* kinda make sense I think	2022-10-26 16:45:45 +01:00
Starbeamrainbowlabs	fad1399c2d	convnext: whitespace	2022-10-26 16:45:20 +01:00
Starbeamrainbowlabs	1d872cb962	contrastive: fix initial temperature value ... It should be 1/0.07, but we had it set to 0.07......	2022-10-26 16:45:01 +01:00
Starbeamrainbowlabs	f994d449f1	Layer2Image: fix	2022-10-25 21:32:17 +01:00
Starbeamrainbowlabs	6a29105f56	model_segmentation: stack not reshape	2022-10-25 21:25:15 +01:00
Starbeamrainbowlabs	98417a3e06	prepare for NCE loss ... .....but Tensorflow's implementation looks to be for supervised models :-(	2022-10-25 21:15:05 +01:00
Starbeamrainbowlabs	bb0679a509	model_segmentation: don't softmax twice	2022-10-25 21:11:48 +01:00
Starbeamrainbowlabs	f2e2ca1484	model_contrastive: make water encoder significantly shallower	2022-10-24 20:52:31 +01:00
Starbeamrainbowlabs	a6b07a49cb	count water/nowater pixels in Jupyter Notebook	2022-10-24 18:05:34 +01:00
Starbeamrainbowlabs	a8b101bdae	dataset_predict: add shape_water_desired	2022-10-24 18:05:13 +01:00
Starbeamrainbowlabs	587c1dfafa	train_predict: revamp jsonl handling	2022-10-21 16:53:08 +01:00
Starbeamrainbowlabs	8195318a42	SparseCategoricalAccuracy: losses → metrics	2022-10-21 16:51:20 +01:00
Starbeamrainbowlabs	612735aaae	rename shuffle arg	2022-10-21 16:35:45 +01:00
Starbeamrainbowlabs	c98d8d05dd	segmentation: use the right accuracy	2022-10-21 16:17:05 +01:00
Starbeamrainbowlabs	bb0258f5cd	flip squeeze operator ordering	2022-10-21 15:38:57 +01:00
Starbeamrainbowlabs	af26964c6a	batched_iterator: reset i_item after every time	2022-10-21 15:35:43 +01:00