sbrl-github/research-rainfallradar
1
0
Fork 0
mirror of https://github.com/sbrl/research-rainfallradar synced 2024-11-04 17:13:02 +00:00
Code Issues Projects Releases Packages Wiki Activity
research-rainfallradar/README.md

53 lines
3.1 KiB
Markdown
Raw Normal View History

Lay out some basic scaffolding I *really* hope this works. This is the 3rd major revision of this model. I've learnt a ton of stuff between now and my last attempt, so here's hoping that all goes well :D The basic idea behind this attempt is *Contrastive Learning*. If we don't get anything useful with this approach, then we can assume that it's not really possible / feasible. Something we need to watch out for is the variance (or rather lack thereof) in the dataset. We have 1.5M timesteps, but not a whole lot will be happening in most of those.... We may need to analyse the variance of the water depth data and extract a subsample that's more balanced.
2022-05-13 18:06:15 +00:00
# Rainfall Radar
> A model to predict water depth data from rainfall radar information.
This is the 3rd major version of this model.
fix crash
2022-08-31 15:25:27 +00:00
Unfortunately using this model is rather complicated and involves a large number of steps. There is no way around this. This README (will) explain it the best I can though.
Lay out some basic scaffolding I *really* hope this works. This is the 3rd major revision of this model. I've learnt a ton of stuff between now and my last attempt, so here's hoping that all goes well :D The basic idea behind this attempt is *Contrastive Learning*. If we don't get anything useful with this approach, then we can assume that it's not really possible / feasible. Something we need to watch out for is the variance (or rather lack thereof) in the dataset. We have 1.5M timesteps, but not a whole lot will be happening in most of those.... We may need to analyse the variance of the water depth data and extract a subsample that's more balanced.
2022-05-13 18:06:15 +00:00
Add 9 simple steps to use the rainfall radar model..... oh boy.
2022-08-31 15:18:28 +00:00
## System Requirements
- Linux (Windows *may* work but is untested. You will probably have a bad day if you use Windows)
- Node.js (a *recent* version - i.e. v16+ - the version in the default Ubuntu repositories is too old)
- Python 3.8+
- Nvidia GPU (16GiB RAM+ is **strongly recommended**) + CUDA and CuDNN (see [this table](https://www.tensorflow.org/install/source#gpu) for which versions you need)
- Experience with the command line
- 1TiB disk space free
- Lots of time and patience
Lay out some basic scaffolding I *really* hope this works. This is the 3rd major revision of this model. I've learnt a ton of stuff between now and my last attempt, so here's hoping that all goes well :D The basic idea behind this attempt is *Contrastive Learning*. If we don't get anything useful with this approach, then we can assume that it's not really possible / feasible. Something we need to watch out for is the variance (or rather lack thereof) in the dataset. We have 1.5M timesteps, but not a whole lot will be happening in most of those.... We may need to analyse the variance of the water depth data and extract a subsample that's more balanced.
2022-05-13 18:06:15 +00:00
Add 9 simple steps to use the rainfall radar model..... oh boy.
2022-08-31 15:18:28 +00:00
## Overview
The process of using this model is as follows.
1. Apply for access to [CEDA's 1km rainfall radar dataset](https://catalogue.ceda.ac.uk/uuid/27dd6ffba67f667a18c62de5c3456350)
2. Obtain rainfall radar data (use [`nimrod-data-downloader`](https://www.npmjs.com/package/nimrod-data-downloader))
3. Obtain a heightmap (or *Digital Elevation Model*, as it's sometimes known) from the Ordnance Survey (can't remember the link, please PR to add this)
4. Use [`terrain50-cli`](https://www.npmjs.com/package/terrain50-cli) to slice the the output from steps #2 and #3 to be exactly the same size [TODO: Preprocess to extract just a single river basin from the data]
5. Push through [HAIL-CAESAR](*https://github.com/sbrl/HAIL-CAESAR) (this fork has the ability to handle streams of .asc files rather than each time step having it's own filename)
fix crash
2022-08-31 15:25:27 +00:00
6. Use `rainfallwrangler` in this repository (finally!) to convert the output to .json.gz then .tfrecord files
Add 9 simple steps to use the rainfall radar model..... oh boy.
2022-08-31 15:18:28 +00:00
7. Pretrain a contrastive learning model
8. Encode the rainfall radar data with the contrastive learning model you pretrained
9. Train the *actual* model to predict water depth
Only steps #6 to #9 actually use code in this repository.
Lay out some basic scaffolding I *really* hope this works. This is the 3rd major revision of this model. I've learnt a ton of stuff between now and my last attempt, so here's hoping that all goes well :D The basic idea behind this attempt is *Contrastive Learning*. If we don't get anything useful with this approach, then we can assume that it's not really possible / feasible. Something we need to watch out for is the variance (or rather lack thereof) in the dataset. We have 1.5M timesteps, but not a whole lot will be happening in most of those.... We may need to analyse the variance of the water depth data and extract a subsample that's more balanced.
2022-05-13 18:06:15 +00:00
## rainfallwrangler
`rainfallwrangler` is a Node.js application to wrangle the dataset into something more appropriate for training an AI efficiently. The rainfall radar and water depth data are considered temporally to be regular time steps. Here's a diagram explaining the terminology:
```
NOW
│ │ │Water depth
│▼ Rainfall Radar Data ▼│[Offset] │▼
├─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┼─┬─┬─┬─┬─┼─┐
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
└─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┼─┴─┴─┴─┴─┴─┘
│
◄────────── Timesteps ─────────────►
```
note tos elf
2022-07-22 18:04:41 +00:00
rainfallwrangler does NOT mess up the ordering of the data
2022-10-18 18:07:14 +00:00
Note to self: 150.12 hashes/sec on i7-4770 4c8t, ???.?? hashes/sec on Viper compute
After double checking, rainfallwrangler does NOT mess with the ordering of the data.
Copy permalink