Commit graph

72 commits

Author SHA1 Message Date
844d8e6dd4
rw: update dependencies 2023-11-30 17:01:13 +00:00
c5be527ffb
rw/README: fill out 2023-11-30 16:59:52 +00:00
4626549352
rw/recompress: clarify subcommand description 2023-11-30 16:59:45 +00:00
18195c2d15
rw: mark as 'private' in package.json to avoid accidental publication 2023-11-30 16:59:31 +00:00
294fde7c82
rw; add requirements.txt 2023-11-30 16:45:04 +00:00
92fc34ebb1
rw/child_process:json2tfrecord: Add RAINFALL_MAX_NUMBER env var
use use you'd need to export RAINFALL_MAX_NUMBER=some_integer_value, since it is used by a subprocess rather than the main Node.js process itself
2023-11-30 16:44:43 +00:00
df621fd7d2
README: Continue filling out, but we're not there yet. 2023-11-29 17:31:47 +00:00
d93365a191
Update license to AGPL. 2023-07-06 15:16:11 +01:00
4b2e418ddc
finish comment
...oops
2022-11-11 18:55:35 +00:00
ce194d9227
slurm: customise log file names 2022-11-10 21:09:34 +00:00
e519b0adb3
GzipChildProcess: spawn-stream is buggy IIRC 2022-11-09 16:43:05 +00:00
ddbf2cb734
slurm-process: -n28 → --exclusive 2022-11-04 17:42:50 +00:00
a762664063
slurm-process: -n28 fo uniq call 2022-11-04 17:23:20 +00:00
0166b4d09e
slurm-process: change log file names 2022-11-04 17:11:10 +00:00
441ad92b12
slurm: fixup 2022-11-01 19:57:15 +00:00
bc0e5f05a8
slurm: fixup 2022-11-01 19:55:04 +00:00
784b8ed35c
recordify: catch NaN --count-file 2022-11-01 19:53:21 +00:00
c17a4ca05a
slurm: fix sanity logic 2022-11-01 19:38:04 +00:00
79b231198f
slurm-process: check input files are readable 2022-11-01 19:03:37 +00:00
a69fa9f0f3
slurm: rename 2022-11-01 18:59:55 +00:00
f8341e7d89
slurm: add .log 2022-11-01 18:59:15 +00:00
fecc63b6a2
wrangler: write high-level job file 2022-11-01 18:56:27 +00:00
91152ebb1c
wrangler:recordify update cli help
we only output .jsonl.gz to a DIRECTORY, so update cli help to reflect this
2022-11-01 18:29:47 +00:00
0c11ddca4b
rainfallwrangler does NOT mess up the ordering of the data 2022-10-18 19:07:14 +01:00
9edda1f397
rainfallwrangler json2tfrecord.py: normalise data 2022-09-01 19:03:15 +01:00
3e4128c0a8
resize rainfall to be 1/2 size of current 2022-09-01 18:47:07 +01:00
6cdf2b2389
wrangler python child: explicitly close stdout+stderr.
Hopefully this will avoid any more hanging issues.
2022-08-10 18:51:30 +01:00
5880bf9020
wrangler: add current date to process indicator.
There's a bug that causes it to hang, but we don't know why
2022-08-10 18:50:57 +01:00
231c832888
wrangler bugfix: crashes; logging output 2022-08-10 17:33:10 +01:00
b52c7f89a7
Move dataset parsing function to the right place 2022-08-10 17:24:55 +01:00
50f214450f
wrangler: fix crash 2022-08-10 17:05:01 +01:00
0bac8c8c0c
fixup 2022-08-08 17:23:24 +01:00
405f1a0bb0
fixup 2022-08-08 17:22:31 +01:00
5e1356513c
slurm: use compute, because 28 tf processes in parallel is too much for the GPU memory 2022-08-08 17:22:18 +01:00
133ef59af3
fixup 2022-08-08 16:33:05 +01:00
80e1a33ee2
slurm-jsonl2tfrecord.job: auto install dependencies 2022-08-08 16:31:49 +01:00
1442d20524
slurm: request gpu 2022-08-08 15:56:46 +01:00
f6f2e3694c
json2tfrecord: write slurm job file 2022-08-08 15:53:32 +01:00
222a6146ec
write glue for .jsonl.gz → .tfrecord.gz converter 2022-08-08 15:33:59 +01:00
f3652edf82
fixup 2022-08-05 19:10:40 +01:00
9399d1d8f5
Create (untested) JS interface to Python jsonl→tfrecord converter
also test Python .jsonl.gz → .tfrecord.gz
2022-08-05 19:10:28 +01:00
a02c3436ab
get python bridge working t convert .jsonl.gz → .tfrecord.gz 2022-08-05 18:07:04 +01:00
2ccc1be414
json2tfrecord: write (untested python to convert .jsonl → .tfrecord 2022-07-28 19:48:25 +01:00
927c30e189
recompress files in the right order 2022-07-25 18:44:23 +01:00
3332fa598a
Add new recompress subcommand
also fix typos, CLI definitions
2022-07-25 17:54:23 +01:00
593dc2d5ce
fixup 2022-07-22 18:51:29 +01:00
a593077d46
add slurm job file for uniq 2022-07-22 18:46:05 +01:00
03e398504a
Bugfix: fix crash when target dir isn't specified 2022-07-22 18:36:00 +01:00
82e826fd69
Fix bugs in remainder of rainfallwrangler:uniq :D 2022-07-22 18:05:03 +01:00
31bd7899b6
Merge branch 'main' of git.starbeamrainbowlabs.com:sbrl/PhD-Rainfall-Radar 2022-07-22 17:10:52 +01:00