sbrl-github/research-rainfallradar
1
0
Fork 0
mirror of https://github.com/sbrl/research-rainfallradar synced 2024-06-30 18:44:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
456 commits 1 branch 0 tags 2.1 MiB
Commit graph

64 commits

Author SHA1 Message Date
Starbeamrainbowlabs 4b2e418ddc
finish comment 
...oops
 2022-11-11 18:55:35 +00:00
Starbeamrainbowlabs ce194d9227
slurm: customise log file names 2022-11-10 21:09:34 +00:00
Starbeamrainbowlabs e519b0adb3
GzipChildProcess: spawn-stream is buggy IIRC 2022-11-09 16:43:05 +00:00
Starbeamrainbowlabs ddbf2cb734
slurm-process: -n28 → --exclusive 2022-11-04 17:42:50 +00:00
Starbeamrainbowlabs a762664063
slurm-process: -n28 fo uniq call 2022-11-04 17:23:20 +00:00
Starbeamrainbowlabs 0166b4d09e
slurm-process: change log file names 2022-11-04 17:11:10 +00:00
Starbeamrainbowlabs 441ad92b12
slurm: fixup 2022-11-01 19:57:15 +00:00
Starbeamrainbowlabs bc0e5f05a8
slurm: fixup 2022-11-01 19:55:04 +00:00
Starbeamrainbowlabs 784b8ed35c
recordify: catch NaN --count-file 2022-11-01 19:53:21 +00:00
Starbeamrainbowlabs c17a4ca05a
slurm: fix sanity logic 2022-11-01 19:38:04 +00:00
Starbeamrainbowlabs 79b231198f
slurm-process: check input files are readable 2022-11-01 19:03:37 +00:00
Starbeamrainbowlabs a69fa9f0f3
slurm: rename 2022-11-01 18:59:55 +00:00
Starbeamrainbowlabs f8341e7d89
slurm: add .log 2022-11-01 18:59:15 +00:00
Starbeamrainbowlabs fecc63b6a2
wrangler: write high-level job file 2022-11-01 18:56:27 +00:00
Starbeamrainbowlabs 91152ebb1c
wrangler:recordify update cli help 
we only output .jsonl.gz to a DIRECTORY, so update cli help to reflect this
 2022-11-01 18:29:47 +00:00
Starbeamrainbowlabs 0c11ddca4b
rainfallwrangler does NOT mess up the ordering of the data 2022-10-18 19:07:14 +01:00
Starbeamrainbowlabs 9edda1f397
rainfallwrangler json2tfrecord.py: normalise data 2022-09-01 19:03:15 +01:00
Starbeamrainbowlabs 3e4128c0a8
resize rainfall to be 1/2 size of current 2022-09-01 18:47:07 +01:00
Starbeamrainbowlabs 6cdf2b2389
wrangler python child: explicitly close stdout+stderr. 
Hopefully this will avoid any more hanging issues.
 2022-08-10 18:51:30 +01:00
Starbeamrainbowlabs 5880bf9020
wrangler: add current date to process indicator. 
There's a bug that causes it to hang, but we don't know why
 2022-08-10 18:50:57 +01:00
Starbeamrainbowlabs 231c832888
wrangler bugfix: crashes; logging output 2022-08-10 17:33:10 +01:00
Starbeamrainbowlabs b52c7f89a7
Move dataset parsing function to the right place 2022-08-10 17:24:55 +01:00
Starbeamrainbowlabs 50f214450f
wrangler: fix crash 2022-08-10 17:05:01 +01:00
Starbeamrainbowlabs 0bac8c8c0c
fixup 2022-08-08 17:23:24 +01:00
Starbeamrainbowlabs 405f1a0bb0
fixup 2022-08-08 17:22:31 +01:00
Starbeamrainbowlabs 5e1356513c
slurm: use compute, because 28 tf processes in parallel is too much for the GPU memory 2022-08-08 17:22:18 +01:00
Starbeamrainbowlabs 133ef59af3
fixup 2022-08-08 16:33:05 +01:00
Starbeamrainbowlabs 80e1a33ee2
slurm-jsonl2tfrecord.job: auto install dependencies 2022-08-08 16:31:49 +01:00
Starbeamrainbowlabs 1442d20524
slurm: request gpu 2022-08-08 15:56:46 +01:00
Starbeamrainbowlabs f6f2e3694c
json2tfrecord: write slurm job file 2022-08-08 15:53:32 +01:00
Starbeamrainbowlabs 222a6146ec
write glue for .jsonl.gz → .tfrecord.gz converter 2022-08-08 15:33:59 +01:00
Starbeamrainbowlabs f3652edf82
fixup 2022-08-05 19:10:40 +01:00
Starbeamrainbowlabs 9399d1d8f5
Create (untested) JS interface to Python jsonl→tfrecord converter 
also test Python .jsonl.gz → .tfrecord.gz
 2022-08-05 19:10:28 +01:00
Starbeamrainbowlabs a02c3436ab
get python bridge working t convert .jsonl.gz → .tfrecord.gz 2022-08-05 18:07:04 +01:00
Starbeamrainbowlabs 2ccc1be414
json2tfrecord: write (untested python to convert .jsonl → .tfrecord 2022-07-28 19:48:25 +01:00
Starbeamrainbowlabs 927c30e189
recompress files in the right order 2022-07-25 18:44:23 +01:00
Starbeamrainbowlabs 3332fa598a
Add new recompress subcommand 
also fix typos, CLI definitions
 2022-07-25 17:54:23 +01:00
Starbeamrainbowlabs 593dc2d5ce
fixup 2022-07-22 18:51:29 +01:00
Starbeamrainbowlabs a593077d46
add slurm job file for uniq 2022-07-22 18:46:05 +01:00
Starbeamrainbowlabs 03e398504a
Bugfix: fix crash when target dir isn't specified 2022-07-22 18:36:00 +01:00
Starbeamrainbowlabs 82e826fd69
Fix bugs in remainder of rainfallwrangler:uniq :D 2022-07-22 18:05:03 +01:00
Starbeamrainbowlabs 31bd7899b6
Merge branch 'main' of git.starbeamrainbowlabs.com:sbrl/PhD-Rainfall-Radar 2022-07-22 17:10:52 +01:00
Starbeamrainbowlabs ce303814d6
Bugfix: don't make 1 group for each duplicate.... 2022-07-22 17:06:02 +01:00
Starbeamrainbowlabs 38a0bd0942
uniq: bugfix a lot, but it's not working right just yet 
There's still a bug in the file line deletor
 2022-07-09 00:31:32 +01:00
Starbeamrainbowlabs a966cdff35
uniq: bugfix a lot, but it's not working right just yet 
There's still a bug in the file line deletor
 2022-07-08 19:54:24 +01:00
Starbeamrainbowlabs 3b2715c6cd
recordify: fix process exiting and imcomplete files issues 
• Node.js not exiting at all
 • Node.js exiting on end_safe ing stream.Writable (?????)
 • Incomplete files - "unexpected end of file" errors and invalid JSON
 2022-07-08 18:54:00 +01:00
Starbeamrainbowlabs cb922ae8c8
fixup 2022-07-08 16:52:19 +01:00
Starbeamrainbowlabs b9a018f9a9
properly close all teh streams 2022-07-08 16:51:17 +01:00
Starbeamrainbowlabs 1a657bd653
add new uniq subcommand 
It deduplicates lines in the files, with the potential to add the ability to filter on a specific property later.
The reasoningf or this is thus:
1. There will naturally be periods of time where nothing happens
2. Too many duplicates will interfere and confuse with the contrastive learning algorithm, as in each batch it will have less variance in samples

This is especially important because contrastive learning causes it to compare every item in each batch with every othear item in the batch.
 2022-07-04 19:46:06 +01:00
Starbeamrainbowlabs 234e2b7978
Write \n end of line character 
we actually forgot this, wow....
 2022-07-04 17:05:05 +01:00