171 lines
7.3 KiB
Markdown
171 lines
7.3 KiB
Markdown
# Msc-Summer-Project
|
|
|
|
> My Msc Summer Project
|
|
|
|
This repository contains my Masters-level summer project. The title is _LoRaWAN Signal Mapping_. The software contained here provides a complete system for mapping and visualising the signal coverage of [The Things Network](https://thethingsnetwork.org/).
|
|
|
|
## Structure
|
|
The system is split up into 2 primary parts:
|
|
|
|
1. An Arduino program that collects the data.
|
|
2. A Node.js program that stores and analyses the data
|
|
|
|
It's best explained with the aid of a diagram or two:
|
|
|
|
|
|
|
|
The above flowchart describes the workflow when using the system:
|
|
|
|
1. First, the IoT device is turned on and the TTN Listener is launched.
|
|
2. Then, the IoT device is taken around the target area that needs mapping. This can be done by anyone - the device does not require operation beyond turning it on and off once provisioned.
|
|
3. Once the device has been carried around the target area, the `DATA.TSV` file on the IoT device's microSD card is copied off and placed on the server.
|
|
4. The server data processor is then run to fold the data into the database.
|
|
5. The AI trainer is now run on the collected data
|
|
6. The data is displayed in a web browser, with the help of a web server
|
|
|
|
The flow of data during use can be shown using a diagram:
|
|
|
|
|
|
|
|
## System requirements
|
|
This software has a number of requirements in order to function properly:
|
|
|
|
- A Linux-based server (_Ubuntu Server LTS_ is recommended), with the following installed:
|
|
+ Node.js v10+ with npm 6+ - see [this website](https://github.com/nodesource/distributions) for instructions on how to install a recent version of Node.js (for the various application server stuff)
|
|
+ Git (for cloning the code repository)
|
|
+ `awk` (for text processing by the build script)
|
|
+ Bash v4+ (for the build script)
|
|
+ PHP (for the temporary test web server; not otherwise used any static web server will do)
|
|
- Arduino IDE (for programming the IoT device)
|
|
- Git (for cloning the code repository)
|
|
-
|
|
|
|
## Getting Started
|
|
|
|
### Step 0: Initial setup
|
|
Before doing anything, clone this git repository to both the server (for the TTN listener etc.) and your local machine (for programming the IoT device):
|
|
|
|
```bash
|
|
git clone https://git.starbeamrainbowlabs.com/sbrl/Msc-Summer-Project.git
|
|
```
|
|
|
|
_If you have somehow obtained a static copy of the code (e.g. through the University's marking system), then skip the above and use that instead._
|
|
|
|
Next, `cd` to the root of the repository and then run the setup task of the build script:
|
|
|
|
```bash
|
|
./build setup
|
|
```
|
|
|
|
On Windows, this should be run in _Git Bash_ (accessible from the start menu when _Git_ is installed - don't forget to `cd` to the root of the repository).
|
|
|
|
|
|
### Step 1: Build a device
|
|
First, a device must be built and provisioned. See `HARDWARE.md` in this repository for detailed instructions on how to do this.
|
|
|
|
Once built, copy `iot/main/settings.custom.cpp.example` to `iot/main/settings.custom.cpp` and follow the instructions fill in the fields there. To do this, you'll need to register the device using ABP (Activation By Personalisation) on _The Things Network_. [This guide](https://www.thethingsnetwork.org/docs/devices/registration.html) tells you how to do this.
|
|
|
|
It is suggested that the following Bash one-liner be used to generate a new encryption key in the right formats:
|
|
|
|
```bash
|
|
head -c16 /dev/urandom | od -tx1 | awk '{ gsub(/^0+\s+/, "", $0); toml=$0; gsub(/\s+/, "", toml); print("settings.toml format: " toml); arduino=toupper($0); gsub(/\s+/, ", 0x", arduino); print("settings.custom.cpp format: 0x" arduino); exit }'
|
|
```
|
|
|
|
_(Paste it into a terminal and hit enter)_
|
|
|
|
Then, review `iot/main/settings.h` to make sure it matches your setup (e.g. all the pin numbers are correct)
|
|
|
|
Next, copy the folders in `iot/libraries` to your Arduino IDE libraries folder.
|
|
|
|
Finally, open `iot/main/main.ino` in the Arduino IDE and program the IoT device itself.
|
|
|
|
### Step 2: The Things Network Setup
|
|
_This step should be completed on the server._
|
|
|
|
Once a device has been constructed, running the _The Things Network_ listener is next. This requires giving the system the _The Things Network_ credentials.
|
|
|
|
Edit the file called `settings.toml` in the root of this repository (or create it if it doesn't exist), and make sure it contains the following:
|
|
|
|
```toml
|
|
[ttn]
|
|
app_id = "{APPLICATION_ID}"
|
|
access_key = "{TTN_ACCESS_ID}"
|
|
|
|
encryption_key = "{ENCRYPTION_KEY_FROM_STEP_1}"
|
|
|
|
port = 8883
|
|
tls = true
|
|
|
|
devices = [ "{DEVICE_NAME_FROM_TTN}" ]
|
|
```
|
|
|
|
The `app_id` and `access_key` can be obtained from _The Things Network Console_:
|
|
|
|
|
|
|
|
The device name can be obtained from step #1, when you registered the device with _The Things Network_. Alternatively, if the device is already registered, it can be obtained from the device list if you click the "X registered devices" text.
|
|
|
|
With `settings.toml` filled in, you can now start TTN Listener:
|
|
|
|
```bash
|
|
./build ttn-listener
|
|
```
|
|
|
|
....it will display an error message if you forgot a value.
|
|
|
|
Now that the TTN listener is running, the IoT device can be carried around and data collected.
|
|
|
|
### Step 3: Processing the data
|
|
_This step should be completed on the server._
|
|
|
|
Once data has been collected by the IoT device, it can then be processed by the Node.js server application.
|
|
|
|
Copy the `DATA.TSV` file from the IoT device's microSD card to the root of the repository on the server. It is suggested that `scp` (Linux) or [WinSCP](https://winscp.net/) be used to do this.
|
|
|
|
Next, fold `DATA.TSV` into the database like so:
|
|
|
|
```bash
|
|
./build process-data
|
|
```
|
|
|
|
Then, train the AIs on the collected data like this:
|
|
|
|
```bash
|
|
./build train-ai
|
|
```
|
|
|
|
The architecture of the neural networks trained can be customised by editing `settings.toml`. Check `server/settings.default.toml` for a guide on the different options available.
|
|
|
|
### Step 4: Viewing the AI output
|
|
With the AIs trained, the browser-based web interface can be used to display the output as a map. Any static web server will do, but the build script has one built-in (if PHP is installed) for convenience:
|
|
|
|
```bash
|
|
./build server
|
|
```
|
|
|
|
If an alternative server is to be used, it should serve the `app/` directory that can be found in the root of this repository.
|
|
|
|
## Credits
|
|
- Build Engine: [Lantern Build Engine](https://gitlab.com/sbrl/lantern-build-engine) (written by me; proof available upon request)
|
|
|
|
### IoT Device
|
|
- Random number generation: [Entropy](https://github.com/taoyuan/Entropy)
|
|
- LoRa Modem Driver: [LMiC](https://github.com/matthijskooijman/arduino-lmic)
|
|
- MicroSD card access: [SdFat](https://github.com/greiman/SdFat)
|
|
- GPS Decoder: [TinyGPS](https://github.com/mikalhart/TinyGPS)
|
|
- Free memory analyser: [Arduino-MemoryFree](https://github.com/mpflaga/Arduino-MemoryFree/)
|
|
|
|
### Node.js Server
|
|
- AI: [Brain.js](https://www.npmjs.com/package/brain.js)
|
|
- Database access: [better-sqlite3](https://github.com/JoshuaWise/better-sqlite3/)
|
|
- Encryption: [aes-js](https://www.npmjs.com/package/aes-js)
|
|
- The Things Network Access: [async-mqtt](https://github.com/mqttjs/async-mqtt)
|
|
- Dependency Injection: [Awilix](https://www.npmjs.com/package/awilix)
|
|
|
|
### Web Interface
|
|
- Mapping Library: [Leaflet](https://leafletjs.com/)
|
|
- [Loading Animation](https://github.com/SamHerbert/SVG-Loaders)
|
|
- Packaging system: [Rollup](https://rollupjs.org/)
|
|
|
|
## Useful Links
|
|
- Entropy extraction from the watchdog timer vs the internal clock: https://github.com/taoyuan/Entropy
|