445 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Starbeamrainbowlabs	f3652edf82	fixup	2022-08-05 19:10:40 +01:00
Starbeamrainbowlabs	9399d1d8f5	Create (untested) JS interface to Python jsonl→tfrecord converter ... also test Python .jsonl.gz → .tfrecord.gz	2022-08-05 19:10:28 +01:00
Starbeamrainbowlabs	a02c3436ab	get python bridge working t convert .jsonl.gz → .tfrecord.gz	2022-08-05 18:07:04 +01:00
Starbeamrainbowlabs	28a3f578d5	update .gitignore	2022-08-04 16:49:53 +01:00
Starbeamrainbowlabs	2ccc1be414	json2tfrecord: write (untested python to convert .jsonl → .tfrecord	2022-07-28 19:48:25 +01:00
Starbeamrainbowlabs	323d708692	dataset: add todo ... just why, Tensorflow?! tf.data.TextLineDataset looks almost too good to be true..... and it is, as despite supporting decompressing via gzip(!) it doesn't look like we can convince it to parse JSON :-/	2022-07-26 19:53:18 +01:00
Starbeamrainbowlabs	b53c77a2cb	index.py: call static function name run	2022-07-26 19:51:28 +01:00
Starbeamrainbowlabs	a7ed58fc03	ai: move requirements.txt to the right place	2022-07-26 19:25:11 +01:00
Starbeamrainbowlabs	e93a95f1b3	ai dataset: add if main == main	2022-07-26 19:24:40 +01:00
Starbeamrainbowlabs	de4c3dab17	typo	2022-07-26 19:14:55 +01:00
Starbeamrainbowlabs	18a7d3674b	ai: create (untested) dataset	2022-07-26 19:14:10 +01:00
Starbeamrainbowlabs	dac6919fcd	ai: start creating initial scaffolding	2022-07-25 19:01:10 +01:00
Starbeamrainbowlabs	1ec502daea	Remove rogue package*.json files	2022-07-25 19:00:21 +01:00
Starbeamrainbowlabs	927c30e189	recompress files in the right order	2022-07-25 18:44:23 +01:00
Starbeamrainbowlabs	3332fa598a	Add new recompress subcommand ... also fix typos, CLI definitions	2022-07-25 17:54:23 +01:00
Starbeamrainbowlabs	d9b9a4f9fc	note tos elf	2022-07-22 19:04:41 +01:00
Starbeamrainbowlabs	593dc2d5ce	fixup	2022-07-22 18:51:29 +01:00
Starbeamrainbowlabs	a593077d46	add slurm job file for uniq	2022-07-22 18:46:05 +01:00
Starbeamrainbowlabs	03e398504a	Bugfix: fix crash when target dir isn't specified	2022-07-22 18:36:00 +01:00
Starbeamrainbowlabs	82e826fd69	Fix bugs in remainder of rainfallwrangler:uniq :D	2022-07-22 18:05:03 +01:00
Starbeamrainbowlabs	31bd7899b6	Merge branch 'main' of git.starbeamrainbowlabs.com:sbrl/PhD-Rainfall-Radar	2022-07-22 17:10:52 +01:00
Starbeamrainbowlabs	ce303814d6	Bugfix: don't make 1 group for each duplicate....	2022-07-22 17:06:02 +01:00
Starbeamrainbowlabs	38a0bd0942	uniq: bugfix a lot, but it's not working right just yet ... There's still a bug in the file line deletor	2022-07-09 00:31:32 +01:00
Starbeamrainbowlabs	a966cdff35	uniq: bugfix a lot, but it's not working right just yet ... There's still a bug in the file line deletor	2022-07-08 19:54:24 +01:00
Starbeamrainbowlabs	3b2715c6cd	recordify: fix process exiting and imcomplete files issues ... • Node.js not exiting at all • Node.js exiting on end_safe ing stream.Writable (?????) • Incomplete files - "unexpected end of file" errors and invalid JSON	2022-07-08 18:54:00 +01:00
Starbeamrainbowlabs	cb922ae8c8	fixup	2022-07-08 16:52:19 +01:00
Starbeamrainbowlabs	b9a018f9a9	properly close all teh streams	2022-07-08 16:51:17 +01:00
Starbeamrainbowlabs	1a657bd653	add new uniq subcommand ... It deduplicates lines in the files, with the potential to add the ability to filter on a specific property later. The reasoningf or this is thus: 1. There will naturally be periods of time where nothing happens 2. Too many duplicates will interfere and confuse with the contrastive learning algorithm, as in each batch it will have less variance in samples This is especially important because contrastive learning causes it to compare every item in each batch with every othear item in the batch.	2022-07-04 19:46:06 +01:00
Starbeamrainbowlabs	234e2b7978	Write \n end of line character ... we actually forgot this, wow....	2022-07-04 17:05:05 +01:00
Starbeamrainbowlabs	920cc3feaf	Properly close last writer ... otherwise Node.js doesn't quit	2022-07-04 17:04:11 +01:00
Starbeamrainbowlabs	588ee87b83	Bugfix: fix end-of-file	2022-07-01 19:34:26 +01:00
Starbeamrainbowlabs	5b2d71f41f	it works ... .....I think	2022-07-01 19:08:36 +01:00
Starbeamrainbowlabs	1297f41105	.tfrecord files are too much hassle ... let's go with a standard of .jsonl.gz instead	2022-07-01 18:28:39 +01:00
Starbeamrainbowlabs	f5f267c6b6	Update dependencies	2022-07-01 16:56:51 +01:00
Starbeamrainbowlabs	ba258fbba0	Remove debug loogging	2022-05-19 19:25:44 +01:00
Starbeamrainbowlabs	e030e6c2d5	Fix remaining(?) crashes= in our code	2022-05-19 19:13:28 +01:00
Starbeamrainbowlabs	3cb7e42505	it doesn't crash as much now, but it still isn't behaving.	2022-05-19 18:52:15 +01:00
Starbeamrainbowlabs	bb018c53f6	Fix many bugs ... Many bugs remain though	2022-05-19 17:54:14 +01:00
Starbeamrainbowlabs	cc5efbae8a	Implement tfrecodify subcommand. ... It's all still untested, but that's the next step	2022-05-19 17:15:15 +01:00
Starbeamrainbowlabs	0fa7ae9d6a	Imnplement plumbing, but it's all untested	2022-05-18 17:47:02 +01:00
Starbeamrainbowlabs	bf4866bdbc	Add data readers	2022-05-18 17:04:11 +01:00
Starbeamrainbowlabs	5829db062b	Merge branch 'main' of git.starbeamrainbowlabs.com:sbrl/PhD-Rainfall-Radar	2022-05-13 19:09:03 +01:00
Starbeamrainbowlabs	58dfaf2ece	Initial commit	2022-05-13 19:08:14 +01:00
Starbeamrainbowlabs	9411ad3218	tweak licence	2022-05-13 19:08:04 +01:00
Starbeamrainbowlabs	8a9cd6c1c0	Lay out some basic scaffolding ... I *really* hope this works. This is the 3rd major revision of this model. I've learnt a ton of stuff between now and my last attempt, so here's hoping that all goes well :D The basic idea behind this attempt is *Contrastive Learning*. If we don't get anything useful with this approach, then we can assume that it's not really possible / feasible. Something we need to watch out for is the variance (or rather lack thereof) in the dataset. We have 1.5M timesteps, but not a whole lot will be happening in most of those.... We may need to analyse the variance of the water depth data and extract a subsample that's more balanced.	2022-05-13 19:06:15 +01:00