sbrl-github/research-rainfallradar
1
0
Fork 0
mirror of https://github.com/sbrl/research-rainfallradar synced 2024-06-28 17:44:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
586 commits 1 branch 0 tags 2.1 MiB
Commit graph

586 commits

This branch
This branch
All branches
Author SHA1 Message Date
Starbeamrainbowlabs de4c3dab17
typo 2022-07-26 19:14:55 +01:00
Starbeamrainbowlabs 18a7d3674b
ai: create (untested) dataset 2022-07-26 19:14:10 +01:00
Starbeamrainbowlabs dac6919fcd
ai: start creating initial scaffolding 2022-07-25 19:01:10 +01:00
Starbeamrainbowlabs 1ec502daea
Remove rogue package*.json files 2022-07-25 19:00:21 +01:00
Starbeamrainbowlabs 927c30e189
recompress files in the right order 2022-07-25 18:44:23 +01:00
Starbeamrainbowlabs 3332fa598a
Add new recompress subcommand 
also fix typos, CLI definitions
 2022-07-25 17:54:23 +01:00
Starbeamrainbowlabs d9b9a4f9fc
note tos elf 2022-07-22 19:04:41 +01:00
Starbeamrainbowlabs 593dc2d5ce
fixup 2022-07-22 18:51:29 +01:00
Starbeamrainbowlabs a593077d46
add slurm job file for uniq 2022-07-22 18:46:05 +01:00
Starbeamrainbowlabs 03e398504a
Bugfix: fix crash when target dir isn't specified 2022-07-22 18:36:00 +01:00
Starbeamrainbowlabs 82e826fd69
Fix bugs in remainder of rainfallwrangler:uniq :D 2022-07-22 18:05:03 +01:00
Starbeamrainbowlabs 31bd7899b6
Merge branch 'main' of git.starbeamrainbowlabs.com:sbrl/PhD-Rainfall-Radar 2022-07-22 17:10:52 +01:00
Starbeamrainbowlabs ce303814d6
Bugfix: don't make 1 group for each duplicate.... 2022-07-22 17:06:02 +01:00
Starbeamrainbowlabs 38a0bd0942
uniq: bugfix a lot, but it's not working right just yet 
There's still a bug in the file line deletor
 2022-07-09 00:31:32 +01:00
Starbeamrainbowlabs a966cdff35
uniq: bugfix a lot, but it's not working right just yet 
There's still a bug in the file line deletor
 2022-07-08 19:54:24 +01:00
Starbeamrainbowlabs 3b2715c6cd
recordify: fix process exiting and imcomplete files issues 
• Node.js not exiting at all
 • Node.js exiting on end_safe ing stream.Writable (?????)
 • Incomplete files - "unexpected end of file" errors and invalid JSON
 2022-07-08 18:54:00 +01:00
Starbeamrainbowlabs cb922ae8c8
fixup 2022-07-08 16:52:19 +01:00
Starbeamrainbowlabs b9a018f9a9
properly close all teh streams 2022-07-08 16:51:17 +01:00
Starbeamrainbowlabs 1a657bd653
add new uniq subcommand 
It deduplicates lines in the files, with the potential to add the ability to filter on a specific property later.
The reasoningf or this is thus:
1. There will naturally be periods of time where nothing happens
2. Too many duplicates will interfere and confuse with the contrastive learning algorithm, as in each batch it will have less variance in samples

This is especially important because contrastive learning causes it to compare every item in each batch with every othear item in the batch.
 2022-07-04 19:46:06 +01:00
Starbeamrainbowlabs 234e2b7978
Write \n end of line character 
we actually forgot this, wow....
 2022-07-04 17:05:05 +01:00
Starbeamrainbowlabs 920cc3feaf
Properly close last writer 
otherwise Node.js doesn't quit
 2022-07-04 17:04:11 +01:00
Starbeamrainbowlabs 588ee87b83
Bugfix: fix end-of-file 2022-07-01 19:34:26 +01:00
Starbeamrainbowlabs 5b2d71f41f
it works 
.....I think
 2022-07-01 19:08:36 +01:00
Starbeamrainbowlabs 1297f41105
.tfrecord files are too much hassle 
let's go with a standard of .jsonl.gz instead
 2022-07-01 18:28:39 +01:00
Starbeamrainbowlabs f5f267c6b6
Update dependencies 2022-07-01 16:56:51 +01:00
Starbeamrainbowlabs ba258fbba0
Remove debug loogging 2022-05-19 19:25:44 +01:00
Starbeamrainbowlabs e030e6c2d5
Fix remaining(?) crashes= in our code 2022-05-19 19:13:28 +01:00
Starbeamrainbowlabs 3cb7e42505
it doesn't crash as much now, but it still isn't behaving. 2022-05-19 18:52:15 +01:00
Starbeamrainbowlabs bb018c53f6
Fix many bugs 
Many bugs remain though
 2022-05-19 17:54:14 +01:00
Starbeamrainbowlabs cc5efbae8a
Implement tfrecodify subcommand. 
It's all still untested, but that's the next step
 2022-05-19 17:15:15 +01:00
Starbeamrainbowlabs 0fa7ae9d6a
Imnplement plumbing, but it's all untested 2022-05-18 17:47:02 +01:00
Starbeamrainbowlabs bf4866bdbc
Add data readers 2022-05-18 17:04:11 +01:00
Starbeamrainbowlabs 5829db062b
Merge branch 'main' of git.starbeamrainbowlabs.com:sbrl/PhD-Rainfall-Radar 2022-05-13 19:09:03 +01:00
Starbeamrainbowlabs 58dfaf2ece Initial commit 2022-05-13 19:08:14 +01:00
Starbeamrainbowlabs 9411ad3218
tweak licence 2022-05-13 19:08:04 +01:00
Starbeamrainbowlabs 8a9cd6c1c0
Lay out some basic scaffolding 
I *really* hope this works. This is the 3rd major revision of this
model. I've learnt a ton of stuff between now and my last attempt, so
here's hoping that all goes well :D

The basic idea behind this attempt is *Contrastive Learning*. If we
don't get anything useful with this approach, then we can assume that
it's not really possible / feasible.

Something we need to watch out for is the variance (or rather lack
thereof) in the dataset. We have 1.5M timesteps, but not a whole lot
will be happening in most of those....

We may need to analyse the variance of the water depth data and extract
a subsample that's more balanced.
 2022-05-13 19:06:15 +01:00