Fill out the README with instructions.

HARDWARE.md

@@ -0,0 +1,16 @@
+# Hardware
+
+TODO: Fill this out with a tutorial
+
+## Parts list
+ - [RFM95 w/ breakout](https://onecall.farnell.com/seeed-studio/114990615/868mhz-dragino-lora-shield-for/dp/SC14518) (alternatively [this](https://shop.pimoroni.com/products/adafruit-rfm95w-lora-radio-transceiver-breakout?variant=19595325639))
+ - [GPS](https://www.hobbytronics.co.uk/ublox-gps-neo6m) (or [this](https://thepihut.com/products/gps-module-with-enclosure?variant=27741075537))
+ - [microSD breakout](https://www.hobbytronics.co.uk/microsd-card-regulated-v2)
+ - DC in + USB-a power regulator (or maybe [this](https://www.adafruit.com/product/1833))
+ - [microSD Card](https://smile.amazon.co.uk/SanDisk-microSDHC-Memory-Adapter-Performance/dp/B073K14CVB/) - any card that's 1GiB+ will do.
+ - [Arduino Uno](https://smile.amazon.co.uk/Elegoo-Board-ATmega328P-ATMEGA16U2-Arduino/dp/B01EWOE0UU/)
+ - [Breadboard](https://shop.pimoroni.com/products/solderless-breadboard-400-point)
+ - Jumper cables (e.g. [a](https://thepihut.com/products/adafruit-premium-female-male-extension-jumper-wires-20-x-3), [b](https://thepihut.com/products/adafruit-premium-male-male-jumper-wires-20-x-6-150mm), and [c](https://thepihut.com/products/adafruit-premium-female-female-jumper-wires-20-x-6-150mm))
+ - [Battery](https://smile.amazon.co.uk/Anker-PowerCore-Aluminum-Portable-Lipstick-Sized-Black/dp/B005QI1A8C/) - Size currently unknown - I'll need to run some tests to see how much power it uses to figure out the right battery size.
+ - [Project box](https://smile.amazon.co.uk/waterproof-plastic-Enclosure-Power-junction-Grey/dp/B01JHBHNMA/) - if we can't 3D print one
+ - [TPL-5111](https://www.adafruit.com/product/3573) - also available from Pimoroni

README.md

@@ -2,35 +2,148 @@
 
 > My Msc Summer Project
 
-## Hardware Used
- - [RFM95 w/ breakout](https://onecall.farnell.com/seeed-studio/114990615/868mhz-dragino-lora-shield-for/dp/SC14518) (alternatively [this](https://shop.pimoroni.com/products/adafruit-rfm95w-lora-radio-transceiver-breakout?variant=19595325639))
- - [GPS](https://www.hobbytronics.co.uk/ublox-gps-neo6m) (or [this](https://thepihut.com/products/gps-module-with-enclosure?variant=27741075537))
- - [microSD breakout](https://www.hobbytronics.co.uk/microsd-card-regulated-v2)
- - DC in + USB-a power regulator (or maybe [this](https://www.adafruit.com/product/1833))
- - [microSD Card](https://smile.amazon.co.uk/SanDisk-microSDHC-Memory-Adapter-Performance/dp/B073K14CVB/) - any card that's 1GiB+ will do.
- - [Arduino Uno](https://smile.amazon.co.uk/Elegoo-Board-ATmega328P-ATMEGA16U2-Arduino/dp/B01EWOE0UU/)
- - [Breadboard](https://shop.pimoroni.com/products/solderless-breadboard-400-point)
- - Jumper cables (e.g. [a](https://thepihut.com/products/adafruit-premium-female-male-extension-jumper-wires-20-x-3), [b](https://thepihut.com/products/adafruit-premium-male-male-jumper-wires-20-x-6-150mm), and [c](https://thepihut.com/products/adafruit-premium-female-female-jumper-wires-20-x-6-150mm))
- - [Battery](https://smile.amazon.co.uk/Anker-PowerCore-Aluminum-Portable-Lipstick-Sized-Black/dp/B005QI1A8C/) - Size currently unknown - I'll need to run some tests to see how much power it uses to figure out the right battery size.
- - [Project box](https://smile.amazon.co.uk/waterproof-plastic-Enclosure-Power-junction-Grey/dp/B01JHBHNMA/) - if we can't 3D print one
- - [TPL-5111](https://www.adafruit.com/product/3573) - also available from Pimoroni
+This repository contains my Masters-level summer project. The title is _LoRaWAN Signal Mapping_. The software contained here provides a complete system for mapping and visualising the signal coverage of [The Things Network](https://thethingsnetwork.org/).
+
+## Structure
+The system is split up into 2 primary parts:
+
+1. An Arduino program that collects the data.
+2. A Node.js program that stores and analyses the data
+
+It's best explained with the aid of a diagram or two:
+
+![](images/Manual Diagrams-Workflow.png)
+
+The above flowchart describes the workflow when using the system:
+
+1. First, the IoT device is turned on and the TTN Listener is launched.
+2. Then, the IoT device is taken around the target area that needs mapping. This can be done by anyone - the device does not require operation beyond turning it on and off once provisioned.
+3. Once the device has been carried around the target area, the `DATA.TSV` file on the IoT device's microSD card is copied off and placed on the server.
+4. The server data processor is then run to fold the data into the database.
+5. The AI trainer is now run on the collected data
+6. The data is displayed in a web browser, with the help of a web server
+
+The flow of data during use can be shown using a diagram:
+
+![](images/Manual Diagrams-Data Flow.png)
+
+## System requirements
+This software has a number of requirements in order to function properly:
+
+ - A Linux-based server (_Ubuntu Server LTS_ is recommended), with the following installed:
+	 + Node.js v10+ with npm 6+ - see [this website](https://github.com/nodesource/distributions) for instructions on how to install a recent version of Node.js (for the various application server stuff)
+	 + Git (for cloning the code repository)
+	 + `awk` (for text processing by the build script)
+	 + Bash v4+ (for the build script)
+	 + PHP (for the temporary test web server; not otherwise used any static web server will do)
+ - Arduino IDE (for programming the IoT device)
+ - Git (for cloning the code repository)
+ - 
 
 ## Getting Started
-Run `bash ./build setup` from the command line at the root of this repository.
 
-### IoT Device
- - Copy `settings.custom.cpp.example` to `settings.custom.cpp` and fill in the fields there
-     - This Bash one-liner can be used to generate a new encryption key in the right formats: `head -c16 /dev/urandom | od -tx1 | awk '{ gsub(/^0+\s+/, "", $0); toml=$0; gsub(/\s+/, "", toml); print("settings.toml format: " toml); arduino=toupper($0); gsub(/\s+/, ", 0x", arduino); print("settings.custom.cpp format: 0x" arduino); exit }'`
- - Review `settings.h` to make sure it matches your setup
- - Copy the folders in `iot/libraries` to your Arduino IDE libraries folder
+### Step 0: Initial setup
+Before doing anything, clone this git repository to both the server (for the TTN listener etc.) and your local machine (for programming the IoT device):
 
-### Server
- - The `bash ./build setup` should have already installed the necessary dependencies.
- - Create `settings.toml` in the root of the repository if it doesn't exist already.
- - Review `server/settings.default.toml` and adjust `settings.toml` to fill in the values for the required fields, not forgetting the encryption key generated when setting up the IoT device.
+```bash
+git clone https://git.starbeamrainbowlabs.com/sbrl/Msc-Summer-Project.git
+```
 
-### Client-side web interface
+_If you have somehow obtained a static copy of the code (e.g. through the University's marking system), then skip the above and use that instead._
 
+Next, `cd` to the root of the repository and then run the setup task of the build script:
+
+```bash
+./build setup
+```
+
+On Windows, this should be run in _Git Bash_ (accessible from the start menu when _Git_ is installed - don't forget to `cd` to the root of the repository).
+
+
+### Step 1: Build a device
+First, a device must be built and provisioned. See `HARDWARE.md` in this repository for detailed instructions on how to do this.
+
+Once built, copy `iot/main/settings.custom.cpp.example` to `iot/main/settings.custom.cpp` and follow the instructions fill in the fields there. To do this, you'll need to register the device using ABP (Activation By Personalisation) on _The Things Network_. [This guide](https://www.thethingsnetwork.org/docs/devices/registration.html) tells you how to do this.
+
+It is suggested that the following Bash one-liner be used to generate a new encryption key in the right formats:
+
+```bash
+head -c16 /dev/urandom | od -tx1 | awk '{ gsub(/^0+\s+/, "", $0); toml=$0; gsub(/\s+/, "", toml); print("settings.toml format: " toml); arduino=toupper($0); gsub(/\s+/, ", 0x", arduino); print("settings.custom.cpp format: 0x" arduino); exit }'
+```
+
+_(Paste it into a terminal and hit enter)_
+
+Then, review `iot/main/settings.h` to make sure it matches your setup (e.g. all the pin numbers are correct)
+
+Next, copy the folders in `iot/libraries` to your Arduino IDE libraries folder.
+
+Finally, open `iot/main/main.ino` in the Arduino IDE and program the IoT device itself.
+
+### Step 2: The Things Network Setup
+_This step should be completed on the server._
+
+Once a device has been constructed, running the _The Things Network_ listener is next. This requires giving the system the _The Things Network_ credentials.
+
+Edit the file called `settings.toml` in the root of this repository (or create it if it doesn't exist), and make sure it contains the following:
+
+```toml
+[ttn]
+app_id = "{APPLICATION_ID}"
+access_key = "{TTN_ACCESS_ID}"
+
+encryption_key = "{ENCRYPTION_KEY_FROM_STEP_1}"
+
+port = 8883
+tls = true
+
+devices = [ "{DEVICE_NAME_FROM_TTN}" ]
+```
+
+The `app_id` and `access_key` can be obtained from _The Things Network Console_:
+
+![](images/TTN-Main.png)
+
+The device name can be obtained from step #1, when you registered the device with _The Things Network_. Alternatively, if the device is already registered, it can be obtained from the device list if you click the "X registered devices" text.
+
+With `settings.toml` filled in, you can now start TTN Listener:
+
+```bash
+./build ttn-listener
+```
+
+....it will display an error message if you forgot a value.
+
+Now that the TTN listener is running, the IoT device can be carried around and data collected.
+
+### Step 3: Processing the data
+_This step should be completed on the server._
+
+Once data has been collected by the IoT device, it can then be processed by the Node.js server application.
+
+Copy the `DATA.TSV` file from the IoT device's microSD card to the root of the repository on the server. It is suggested that `scp` (Linux) or [WinSCP](https://winscp.net/) be used to do this.
+
+Next, fold `DATA.TSV` into the database like so:
+
+```bash
+./build process-data
+```
+
+Then, train the AIs on the collected data like this:
+
+```bash
+./build train-ai
+```
+
+The architecture of the neural networks trained can be customised by editing `settings.toml`. Check `server/settings.default.toml` for a guide on the different options available.
+
+### Step 4: Viewing the AI output
+With the AIs trained, the browser-based web interface can be used to display the output as a map. Any static web server will do, but the build script has one built-in (if PHP is installed) for convenience:
+
+```bash
+./build server
+```
+
+If an alternative server is to be used, it should serve the `app/` directory that can be found in the root of this repository.
 
 ## Credits
  - AI: [TensorFlow.js](https://www.tensorflow.org/js/)

build

@@ -197,9 +197,14 @@ task_server() {
 	fi
 	
 	task_begin "Starting development server";
-	php -S "[::1]:40382" -t "app" &
-	exit_code=$?;
-	[[ "${exit_code}" -eq "0" ]] && echo $! >/tmp/summer-project-dev-server.pid;
+	php -S "[::]:40382" -t "app" &
+	exit_code_a=$?;
+	pid_a=$!;
+	php -S "127.0.0.1:40382" -t "app" &
+	exit_code_b=$?;
+	pid_b=$!;
+	[[ "${exit_code_a}" -eq "0" ]] && echo "${pid_a}" >/tmp/summer-project-dev-server-v6.pid;
+	[[ "${exit_code_b}" -eq "0" ]] && echo "${pid_b}" >/tmp/summer-project-dev-server-v4.pid;
 	task_end $?; # Should be 0 unless php died for some reason
 	sleep 1;
 }
@@ -207,13 +212,14 @@ task_server() {
 task_server-stop() {
 	task_begin "Stopping development server";
 	
-	if [ ! -f "/tmp/summer-project-dev-server.pid" ]; then
+	if [ ! -f "/tmp/summer-project-dev-server-v6.pid" ]; then
 		echo -e "${HC}${FRED}Error: The development server doesn't appear to be running, so it can't be stopped. Have you tried running ./build dev-server?${RESET}";
 		return 1;
 	fi
 	
-	kill "$(cat /tmp/summer-project-dev-server.pid)";
-	rm /tmp/summer-project-dev-server.pid;
+	kill "$(cat /tmp/summer-project-dev-server-v4.pid)";
+	kill "$(cat /tmp/summer-project-dev-server-v6.pid)";
+	rm /tmp/summer-project-dev-server-v[46].pid;
 	
 	task_end $?;
 }

server/settings.default.toml

@@ -1,6 +1,9 @@
 # Default settings file.
 # 
-# DO NOT EDIT THIS FILE. Instead edit ../settings.toml (or create it if it doesn't exist yet).
+# **********************************************************
+# ***************** DO NOT EDIT THIS FILE. *****************
+# **********************************************************
+# Instead edit ../settings.toml (or create it if it doesn't exist yet).
 
 program_name = "LoRaWAN Signal Mapper"
 version = "v0.1"