Commit graph

Select branches

Hide pull requests

main

#1

Mono Color

• 5e1356513c
  
  slurm: use compute, because 28 tf processes in parallel is too much for the GPU memory Starbeamrainbowlabs 2022-08-08 17:22:18 +0100
• 133ef59af3
  
  fixup Starbeamrainbowlabs 2022-08-08 16:33:05 +0100
• 80e1a33ee2
  
  slurm-jsonl2tfrecord.job: auto install dependencies Starbeamrainbowlabs 2022-08-08 16:31:49 +0100
• 1442d20524
  
  slurm: request gpu Starbeamrainbowlabs 2022-08-08 15:56:46 +0100
• f6f2e3694c
  
  json2tfrecord: write slurm job file Starbeamrainbowlabs 2022-08-08 15:53:32 +0100
• 222a6146ec
  
  write glue for .jsonl.gz → .tfrecord.gz converter Starbeamrainbowlabs 2022-08-08 15:33:59 +0100
• f3652edf82
  
  fixup Starbeamrainbowlabs 2022-08-05 19:10:40 +0100
• 9399d1d8f5
  
  Create (untested) JS interface to Python jsonl→tfrecord converter also test Python .jsonl.gz → .tfrecord.gz Starbeamrainbowlabs 2022-08-05 19:10:28 +0100
• a02c3436ab
  
  get python bridge working t convert .jsonl.gz → .tfrecord.gz Starbeamrainbowlabs 2022-08-05 18:07:04 +0100
• 28a3f578d5
  
  update .gitignore Starbeamrainbowlabs 2022-08-04 16:49:53 +0100
• 2ccc1be414
  
  json2tfrecord: write (untested python to convert .jsonl → .tfrecord Starbeamrainbowlabs 2022-07-28 19:48:25 +0100
• 323d708692
  
  dataset: add todo just why, Tensorflow?! tf.data.TextLineDataset looks almost too good to be true..... and it is, as despite supporting decompressing via gzip(!) it doesn't look like we can convince it to parse JSON :-/ Starbeamrainbowlabs 2022-07-26 19:53:18 +0100
• b53c77a2cb
  
  index.py: call static function name run Starbeamrainbowlabs 2022-07-26 19:51:28 +0100
• a7ed58fc03
  
  ai: move requirements.txt to the right place Starbeamrainbowlabs 2022-07-26 19:25:11 +0100
• e93a95f1b3
  
  ai dataset: add if main == main Starbeamrainbowlabs 2022-07-26 19:24:40 +0100
• de4c3dab17
  
  typo Starbeamrainbowlabs 2022-07-26 19:14:55 +0100
• 18a7d3674b
  
  ai: create (untested) dataset Starbeamrainbowlabs 2022-07-26 19:14:10 +0100
• dac6919fcd
  
  ai: start creating initial scaffolding Starbeamrainbowlabs 2022-07-25 19:00:53 +0100
• 1ec502daea
  
  Remove rogue package*.json files Starbeamrainbowlabs 2022-07-25 19:00:21 +0100
• 927c30e189
  
  recompress files in the right order Starbeamrainbowlabs 2022-07-25 18:44:23 +0100
• 3332fa598a
  
  Add new recompress subcommand also fix typos, CLI definitions Starbeamrainbowlabs 2022-07-25 17:54:23 +0100
• d9b9a4f9fc
  
  note tos elf Starbeamrainbowlabs 2022-07-22 19:04:41 +0100
• 593dc2d5ce
  
  fixup Starbeamrainbowlabs 2022-07-22 18:51:29 +0100
• a593077d46
  
  add slurm job file for uniq Starbeamrainbowlabs 2022-07-22 18:46:05 +0100
• 03e398504a
  
  Bugfix: fix crash when target dir isn't specified Starbeamrainbowlabs 2022-07-22 18:36:00 +0100
• 82e826fd69
  
  Fix bugs in remainder of rainfallwrangler:uniq :D Starbeamrainbowlabs 2022-07-22 18:05:03 +0100
• 31bd7899b6
  
  Merge branch 'main' of git.starbeamrainbowlabs.com:sbrl/PhD-Rainfall-Radar Starbeamrainbowlabs 2022-07-22 17:10:52 +0100
• ce303814d6
  
  Bugfix: don't make 1 group for each duplicate.... Starbeamrainbowlabs 2022-07-22 17:06:02 +0100
• 38a0bd0942
  
  uniq: bugfix a lot, but it's not working right just yet Starbeamrainbowlabs 2022-07-08 19:54:24 +0100
• a966cdff35
  
  uniq: bugfix a lot, but it's not working right just yet Starbeamrainbowlabs 2022-07-08 19:54:24 +0100
• 3b2715c6cd
  
  recordify: fix process exiting and imcomplete files issues Starbeamrainbowlabs 2022-07-08 18:54:00 +0100
• cb922ae8c8
  
  fixup Starbeamrainbowlabs 2022-07-08 16:52:19 +0100
• b9a018f9a9
  
  properly close all teh streams Starbeamrainbowlabs 2022-07-08 16:51:17 +0100
• 1a657bd653
  
  add new uniq subcommand It deduplicates lines in the files, with the potential to add the ability to filter on a specific property later. The reasoningf or this is thus: 1. There will naturally be periods of time where nothing happens 2. Too many duplicates will interfere and confuse with the contrastive learning algorithm, as in each batch it will have less variance in samples Starbeamrainbowlabs 2022-07-04 19:46:06 +0100
• 234e2b7978
  
  Write \n end of line character we actually forgot this, wow.... Starbeamrainbowlabs 2022-07-04 17:05:05 +0100
• 920cc3feaf
  
  Properly close last writer Starbeamrainbowlabs 2022-07-04 17:04:11 +0100
• 588ee87b83
  
  Bugfix: fix end-of-file Starbeamrainbowlabs 2022-07-01 19:34:26 +0100
• 5b2d71f41f
  
  it works .....I think Starbeamrainbowlabs 2022-07-01 19:08:36 +0100
• 1297f41105
  
  .tfrecord files are too much hassle let's go with a standard of .jsonl.gz instead Starbeamrainbowlabs 2022-07-01 18:28:39 +0100
• f5f267c6b6
  
  Update dependencies Starbeamrainbowlabs 2022-07-01 16:56:51 +0100
• ba258fbba0
  
  Remove debug loogging Starbeamrainbowlabs 2022-05-19 19:25:44 +0100
• e030e6c2d5
  
  Fix remaining(?) crashes= in our code Starbeamrainbowlabs 2022-05-19 19:13:28 +0100
• 3cb7e42505
  
  it doesn't crash as much now, but it still isn't behaving. Starbeamrainbowlabs 2022-05-19 18:52:15 +0100
• bb018c53f6
  
  Fix many bugs Starbeamrainbowlabs 2022-05-19 17:54:14 +0100
• cc5efbae8a
  
  Implement tfrecodify subcommand. It's all still untested, but that's the next step Starbeamrainbowlabs 2022-05-19 17:15:15 +0100
• 0fa7ae9d6a
  
  Imnplement plumbing, but it's all untested Starbeamrainbowlabs 2022-05-18 17:47:02 +0100
• bf4866bdbc
  
  Add data readers Starbeamrainbowlabs 2022-05-18 17:04:11 +0100
• 5829db062b
  
  Merge branch 'main' of git.starbeamrainbowlabs.com:sbrl/PhD-Rainfall-Radar Starbeamrainbowlabs 2022-05-13 19:09:03 +0100
• 58dfaf2ece Initial commit Starbeamrainbowlabs 2022-05-13 19:08:14 +0100
• 9411ad3218
  
  tweak licence Starbeamrainbowlabs 2022-05-13 19:08:04 +0100
• 8a9cd6c1c0
  
  Lay out some basic scaffolding Starbeamrainbowlabs 2022-05-13 19:06:15 +0100