64 commits

Author	SHA1	Message	Date
Starbeamrainbowlabs	4b2e418ddc	finish comment ... ...oops	2022-11-11 18:55:35 +00:00
Starbeamrainbowlabs	ce194d9227	slurm: customise log file names	2022-11-10 21:09:34 +00:00
Starbeamrainbowlabs	e519b0adb3	GzipChildProcess: spawn-stream is buggy IIRC	2022-11-09 16:43:05 +00:00
Starbeamrainbowlabs	ddbf2cb734	slurm-process: -n28 → --exclusive	2022-11-04 17:42:50 +00:00
Starbeamrainbowlabs	a762664063	slurm-process: -n28 fo uniq call	2022-11-04 17:23:20 +00:00
Starbeamrainbowlabs	0166b4d09e	slurm-process: change log file names	2022-11-04 17:11:10 +00:00
Starbeamrainbowlabs	441ad92b12	slurm: fixup	2022-11-01 19:57:15 +00:00
Starbeamrainbowlabs	bc0e5f05a8	slurm: fixup	2022-11-01 19:55:04 +00:00
Starbeamrainbowlabs	784b8ed35c	recordify: catch NaN --count-file	2022-11-01 19:53:21 +00:00
Starbeamrainbowlabs	c17a4ca05a	slurm: fix sanity logic	2022-11-01 19:38:04 +00:00
Starbeamrainbowlabs	79b231198f	slurm-process: check input files are readable	2022-11-01 19:03:37 +00:00
Starbeamrainbowlabs	a69fa9f0f3	slurm: rename	2022-11-01 18:59:55 +00:00
Starbeamrainbowlabs	f8341e7d89	slurm: add .log	2022-11-01 18:59:15 +00:00
Starbeamrainbowlabs	fecc63b6a2	wrangler: write high-level job file	2022-11-01 18:56:27 +00:00
Starbeamrainbowlabs	91152ebb1c	wrangler:recordify update cli help ... we only output .jsonl.gz to a DIRECTORY, so update cli help to reflect this	2022-11-01 18:29:47 +00:00
Starbeamrainbowlabs	0c11ddca4b	rainfallwrangler does NOT mess up the ordering of the data	2022-10-18 19:07:14 +01:00
Starbeamrainbowlabs	9edda1f397	rainfallwrangler json2tfrecord.py: normalise data	2022-09-01 19:03:15 +01:00
Starbeamrainbowlabs	3e4128c0a8	resize rainfall to be 1/2 size of current	2022-09-01 18:47:07 +01:00
Starbeamrainbowlabs	6cdf2b2389	wrangler python child: explicitly close stdout+stderr. ... Hopefully this will avoid any more hanging issues.	2022-08-10 18:51:30 +01:00
Starbeamrainbowlabs	5880bf9020	wrangler: add current date to process indicator. ... There's a bug that causes it to hang, but we don't know why	2022-08-10 18:50:57 +01:00
Starbeamrainbowlabs	231c832888	wrangler bugfix: crashes; logging output	2022-08-10 17:33:10 +01:00
Starbeamrainbowlabs	b52c7f89a7	Move dataset parsing function to the right place	2022-08-10 17:24:55 +01:00
Starbeamrainbowlabs	50f214450f	wrangler: fix crash	2022-08-10 17:05:01 +01:00
Starbeamrainbowlabs	0bac8c8c0c	fixup	2022-08-08 17:23:24 +01:00
Starbeamrainbowlabs	405f1a0bb0	fixup	2022-08-08 17:22:31 +01:00
Starbeamrainbowlabs	5e1356513c	slurm: use compute, because 28 tf processes in parallel is too much for the GPU memory	2022-08-08 17:22:18 +01:00
Starbeamrainbowlabs	133ef59af3	fixup	2022-08-08 16:33:05 +01:00
Starbeamrainbowlabs	80e1a33ee2	slurm-jsonl2tfrecord.job: auto install dependencies	2022-08-08 16:31:49 +01:00
Starbeamrainbowlabs	1442d20524	slurm: request gpu	2022-08-08 15:56:46 +01:00
Starbeamrainbowlabs	f6f2e3694c	json2tfrecord: write slurm job file	2022-08-08 15:53:32 +01:00
Starbeamrainbowlabs	222a6146ec	write glue for .jsonl.gz → .tfrecord.gz converter	2022-08-08 15:33:59 +01:00
Starbeamrainbowlabs	f3652edf82	fixup	2022-08-05 19:10:40 +01:00
Starbeamrainbowlabs	9399d1d8f5	Create (untested) JS interface to Python jsonl→tfrecord converter ... also test Python .jsonl.gz → .tfrecord.gz	2022-08-05 19:10:28 +01:00
Starbeamrainbowlabs	a02c3436ab	get python bridge working t convert .jsonl.gz → .tfrecord.gz	2022-08-05 18:07:04 +01:00
Starbeamrainbowlabs	2ccc1be414	json2tfrecord: write (untested python to convert .jsonl → .tfrecord	2022-07-28 19:48:25 +01:00
Starbeamrainbowlabs	927c30e189	recompress files in the right order	2022-07-25 18:44:23 +01:00
Starbeamrainbowlabs	3332fa598a	Add new recompress subcommand ... also fix typos, CLI definitions	2022-07-25 17:54:23 +01:00
Starbeamrainbowlabs	593dc2d5ce	fixup	2022-07-22 18:51:29 +01:00
Starbeamrainbowlabs	a593077d46	add slurm job file for uniq	2022-07-22 18:46:05 +01:00
Starbeamrainbowlabs	03e398504a	Bugfix: fix crash when target dir isn't specified	2022-07-22 18:36:00 +01:00
Starbeamrainbowlabs	82e826fd69	Fix bugs in remainder of rainfallwrangler:uniq :D	2022-07-22 18:05:03 +01:00
Starbeamrainbowlabs	31bd7899b6	Merge branch 'main' of git.starbeamrainbowlabs.com:sbrl/PhD-Rainfall-Radar	2022-07-22 17:10:52 +01:00
Starbeamrainbowlabs	ce303814d6	Bugfix: don't make 1 group for each duplicate....	2022-07-22 17:06:02 +01:00
Starbeamrainbowlabs	38a0bd0942	uniq: bugfix a lot, but it's not working right just yet ... There's still a bug in the file line deletor	2022-07-09 00:31:32 +01:00
Starbeamrainbowlabs	a966cdff35	uniq: bugfix a lot, but it's not working right just yet ... There's still a bug in the file line deletor	2022-07-08 19:54:24 +01:00
Starbeamrainbowlabs	3b2715c6cd	recordify: fix process exiting and imcomplete files issues ... • Node.js not exiting at all • Node.js exiting on end_safe ing stream.Writable (?????) • Incomplete files - "unexpected end of file" errors and invalid JSON	2022-07-08 18:54:00 +01:00
Starbeamrainbowlabs	cb922ae8c8	fixup	2022-07-08 16:52:19 +01:00
Starbeamrainbowlabs	b9a018f9a9	properly close all teh streams	2022-07-08 16:51:17 +01:00
Starbeamrainbowlabs	1a657bd653	add new uniq subcommand ... It deduplicates lines in the files, with the potential to add the ability to filter on a specific property later. The reasoningf or this is thus: 1. There will naturally be periods of time where nothing happens 2. Too many duplicates will interfere and confuse with the contrastive learning algorithm, as in each batch it will have less variance in samples This is especially important because contrastive learning causes it to compare every item in each batch with every othear item in the batch.	2022-07-04 19:46:06 +01:00
Starbeamrainbowlabs	234e2b7978	Write \n end of line character ... we actually forgot this, wow....	2022-07-04 17:05:05 +01:00