LoRaWAN-Signal-Mapping/msc-project-md-pdf.ikopCXS

204 lines
14 KiB
Plaintext
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
<head>
<meta charset="utf-8" />
<meta name="generator" content="pandoc" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
<title>LoRaWAN Signal Mapping - User Manual</title>
<style type="text/css">
code{white-space: pre-wrap;}
span.smallcaps{font-variant: small-caps;}
span.underline{text-decoration: underline;}
div.column{display: inline-block; vertical-align: top; width: 50%;}
</style>
<style type="text/css">
a.sourceLine { display: inline-block; line-height: 1.25; }
a.sourceLine { pointer-events: none; color: inherit; text-decoration: inherit; }
a.sourceLine:empty { height: 1.2em; }
.sourceCode { overflow: visible; }
code.sourceCode { white-space: pre; position: relative; }
div.sourceCode { margin: 1em 0; }
pre.sourceCode { margin: 0; }
@media screen {
div.sourceCode { overflow: auto; }
}
@media print {
code.sourceCode { white-space: pre-wrap; }
a.sourceLine { text-indent: -1em; padding-left: 1em; }
}
pre.numberSource a.sourceLine
{ position: relative; left: -4em; }
pre.numberSource a.sourceLine::before
{ content: attr(title);
position: relative; left: -1em; text-align: right; vertical-align: baseline;
border: none; pointer-events: all; display: inline-block;
-webkit-touch-callout: none; -webkit-user-select: none;
-khtml-user-select: none; -moz-user-select: none;
-ms-user-select: none; user-select: none;
padding: 0 4px; width: 4em;
color: #aaaaaa;
}
pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; }
div.sourceCode
{ }
@media screen {
a.sourceLine::before { text-decoration: underline; }
}
code span.al { color: #ff0000; font-weight: bold; } /* Alert */
code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
code span.at { color: #7d9029; } /* Attribute */
code span.bn { color: #40a070; } /* BaseN */
code span.bu { } /* BuiltIn */
code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
code span.ch { color: #4070a0; } /* Char */
code span.cn { color: #880000; } /* Constant */
code span.co { color: #60a0b0; font-style: italic; } /* Comment */
code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
code span.do { color: #ba2121; font-style: italic; } /* Documentation */
code span.dt { color: #902000; } /* DataType */
code span.dv { color: #40a070; } /* DecVal */
code span.er { color: #ff0000; font-weight: bold; } /* Error */
code span.ex { } /* Extension */
code span.fl { color: #40a070; } /* Float */
code span.fu { color: #06287e; } /* Function */
code span.im { } /* Import */
code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
code span.kw { color: #007020; font-weight: bold; } /* Keyword */
code span.op { color: #666666; } /* Operator */
code span.ot { color: #007020; } /* Other */
code span.pp { color: #bc7a00; } /* Preprocessor */
code span.sc { color: #4070a0; } /* SpecialChar */
code span.ss { color: #bb6688; } /* SpecialString */
code span.st { color: #4070a0; } /* String */
code span.va { color: #19177c; } /* Variable */
code span.vs { color: #4070a0; } /* VerbatimString */
code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
</style>
</head>
<body>
<header>
<h1 class="title">LoRaWAN Signal Mapping - User Manual</h1>
</header>
<h1 id="msc-summer-project">Msc-Summer-Project</h1>
<blockquote>
<p>My Msc Summer Project</p>
</blockquote>
<p>This repository contains my Masters-level summer project. The title is <em>LoRaWAN Signal Mapping</em>. The software contained here provides a complete system for mapping and visualising the signal coverage of <a href="https://thethingsnetwork.org/">The Things Network</a>.</p>
<h2 id="structure">Structure</h2>
<p>The system is split up into 2 primary parts:</p>
<ol type="1">
<li>An Arduino program that collects the data.</li>
<li>A Node.js program that stores and analyses the data</li>
</ol>
<p>Its best explained with the aid of a diagram or two:</p>
<p><img src="images/Manual%20Diagrams-Workflow.png" /></p>
<p>The above flowchart describes the workflow when using the system:</p>
<ol type="1">
<li>First, the IoT device is turned on and the TTN Listener is launched.</li>
<li>Then, the IoT device is taken around the target area that needs mapping. This can be done by anyone - the device does not require operation beyond turning it on and off once provisioned.</li>
<li>Once the device has been carried around the target area, the <code>DATA.TSV</code> file on the IoT devices microSD card is copied off and placed on the server.</li>
<li>The server data processor is then run to fold the data into the database.</li>
<li>The AI trainer is now run on the collected data</li>
<li>The data is displayed in a web browser, with the help of a web server</li>
</ol>
<p>The flow of data during use can be shown using a diagram:</p>
<p><img src="images/Manual%20Diagrams-Data%20Flow.png" /></p>
<h2 id="system-requirements">System requirements</h2>
<p>This software has a number of requirements in order to function properly:</p>
<ul>
<li>A Linux-based server (<em>Ubuntu Server LTS</em> is recommended), with the following installed:
<ul>
<li>Node.js v10+ with npm 6+ - see <a href="https://github.com/nodesource/distributions">this website</a> for instructions on how to install a recent version of Node.js (for the various application server stuff)</li>
<li>Git (for cloning the code repository)</li>
<li><code>awk</code> (for text processing by the build script)</li>
<li>Bash v4+ (for the build script)</li>
<li>PHP (for the temporary test web server; not otherwise used any static web server will do)</li>
</ul></li>
<li>Arduino IDE (for programming the IoT device)</li>
<li>Git (for cloning the code repository)</li>
<li></li>
</ul>
<h2 id="getting-started">Getting Started</h2>
<h3 id="step-0-initial-setup">Step 0: Initial setup</h3>
<p>Before doing anything, clone this git repository to both the server (for the TTN listener etc.) and your local machine (for programming the IoT device):</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode bash"><code class="sourceCode bash"><a class="sourceLine" id="cb1-1" title="1"><span class="fu">git</span> clone https://git.starbeamrainbowlabs.com/sbrl/Msc-Summer-Project.git</a></code></pre></div>
<p><em>If you have somehow obtained a static copy of the code (e.g. through the Universitys marking system), then skip the above and use that instead.</em></p>
<p>Next, <code>cd</code> to the root of the repository and then run the setup task of the build script:</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode bash"><code class="sourceCode bash"><a class="sourceLine" id="cb2-1" title="1"><span class="ex">./build</span> setup</a></code></pre></div>
<p>On Windows, this should be run in <em>Git Bash</em> (accessible from the start menu when <em>Git</em> is installed - dont forget to <code>cd</code> to the root of the repository).</p>
<h3 id="step-1-build-a-device">Step 1: Build a device</h3>
<p>First, a device must be built and provisioned. See <code>HARDWARE.md</code> in this repository for detailed instructions on how to do this.</p>
<p>Once built, copy <code>iot/main/settings.custom.cpp.example</code> to <code>iot/main/settings.custom.cpp</code> and follow the instructions fill in the fields there. To do this, youll need to register the device using ABP (Activation By Personalisation) on <em>The Things Network</em>. <a href="https://www.thethingsnetwork.org/docs/devices/registration.html">This guide</a> tells you how to do this.</p>
<p>It is suggested that the following Bash one-liner be used to generate a new encryption key in the right formats:</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode bash"><code class="sourceCode bash"><a class="sourceLine" id="cb3-1" title="1"><span class="fu">head</span> -c16 /dev/urandom <span class="kw">|</span> <span class="fu">od</span> -tx1 <span class="kw">|</span> <span class="fu">awk</span> <span class="st">&#39;{ gsub(/^0+\s+/, &quot;&quot;, $0); toml=$0; gsub(/\s+/, &quot;&quot;, toml); print(&quot;settings.toml format: &quot; toml); arduino=toupper($0); gsub(/\s+/, &quot;, 0x&quot;, arduino); print(&quot;settings.custom.cpp format: 0x&quot; arduino); exit }&#39;</span></a></code></pre></div>
<p><em>(Paste it into a terminal and hit enter)</em></p>
<p>Then, review <code>iot/main/settings.h</code> to make sure it matches your setup (e.g. all the pin numbers are correct)</p>
<p>Next, copy the folders in <code>iot/libraries</code> to your Arduino IDE libraries folder.</p>
<p>Finally, open <code>iot/main/main.ino</code> in the Arduino IDE and program the IoT device itself.</p>
<h3 id="step-2-the-things-network-setup">Step 2: The Things Network Setup</h3>
<p><em>This step should be completed on the server.</em></p>
<p>Once a device has been constructed, running the <em>The Things Network</em> listener is next. This requires giving the system the <em>The Things Network</em> credentials.</p>
<p>Edit the file called <code>settings.toml</code> in the root of this repository (or create it if it doesnt exist), and make sure it contains the following:</p>
<pre class="toml"><code>[ttn]
app_id = &quot;{APPLICATION_ID}&quot;
access_key = &quot;{TTN_ACCESS_ID}&quot;
encryption_key = &quot;{ENCRYPTION_KEY_FROM_STEP_1}&quot;
port = 8883
tls = true
devices = [ &quot;{DEVICE_NAME_FROM_TTN}&quot; ]</code></pre>
<p>The <code>app_id</code> and <code>access_key</code> can be obtained from <em>The Things Network Console</em>:</p>
<p><img src="images/TTN-Main.png" /></p>
<p>The device name can be obtained from step #1, when you registered the device with <em>The Things Network</em>. Alternatively, if the device is already registered, it can be obtained from the device list if you click the “X registered devices” text.</p>
<p>With <code>settings.toml</code> filled in, you can now start TTN Listener:</p>
<div class="sourceCode" id="cb5"><pre class="sourceCode bash"><code class="sourceCode bash"><a class="sourceLine" id="cb5-1" title="1"><span class="ex">./build</span> ttn-listener</a></code></pre></div>
<p>….it will display an error message if you forgot a value.</p>
<p>Now that the TTN listener is running, the IoT device can be carried around and data collected.</p>
<h3 id="step-3-processing-the-data">Step 3: Processing the data</h3>
<p><em>This step should be completed on the server.</em></p>
<p>Once data has been collected by the IoT device, it can then be processed by the Node.js server application.</p>
<p>Copy the <code>DATA.TSV</code> file from the IoT devices microSD card to the root of the repository on the server. It is suggested that <code>scp</code> (Linux) or <a href="https://winscp.net/">WinSCP</a> be used to do this.</p>
<p>Next, fold <code>DATA.TSV</code> into the database like so:</p>
<div class="sourceCode" id="cb6"><pre class="sourceCode bash"><code class="sourceCode bash"><a class="sourceLine" id="cb6-1" title="1"><span class="ex">./build</span> process-data</a></code></pre></div>
<p>Then, train the AIs on the collected data like this:</p>
<div class="sourceCode" id="cb7"><pre class="sourceCode bash"><code class="sourceCode bash"><a class="sourceLine" id="cb7-1" title="1"><span class="ex">./build</span> train-ai</a></code></pre></div>
<p>The architecture of the neural networks trained can be customised by editing <code>settings.toml</code>. Check <code>server/settings.default.toml</code> for a guide on the different options available.</p>
<h3 id="step-4-viewing-the-ai-output">Step 4: Viewing the AI output</h3>
<p>With the AIs trained, the browser-based web interface can be used to display the output as a map. Any static web server will do, but the build script has one built-in (if PHP is installed) for convenience:</p>
<div class="sourceCode" id="cb8"><pre class="sourceCode bash"><code class="sourceCode bash"><a class="sourceLine" id="cb8-1" title="1"><span class="ex">./build</span> server</a></code></pre></div>
<p>If an alternative server is to be used, it should serve the <code>app/</code> directory that can be found in the root of this repository.</p>
<h2 id="credits">Credits</h2>
<ul>
<li>Build Engine: <a href="https://gitlab.com/sbrl/lantern-build-engine">Lantern Build Engine</a> (written by me; proof available upon request)</li>
</ul>
<h3 id="iot-device">IoT Device</h3>
<ul>
<li>Random number generation: <a href="https://github.com/taoyuan/Entropy">Entropy</a></li>
<li>LoRa Modem Driver: <a href="https://github.com/matthijskooijman/arduino-lmic">LMiC</a></li>
<li>MicroSD card access: <a href="https://github.com/greiman/SdFat">SdFat</a></li>
<li>GPS Decoder: <a href="https://github.com/mikalhart/TinyGPS">TinyGPS</a></li>
<li>Free memory analyser: <a href="https://github.com/mpflaga/Arduino-MemoryFree/">Arduino-MemoryFree</a></li>
</ul>
<h3 id="node.js-server">Node.js Server</h3>
<ul>
<li>AI: <a href="https://www.npmjs.com/package/brain.js">Brain.js</a></li>
<li>Database access: <a href="https://github.com/JoshuaWise/better-sqlite3/">better-sqlite3</a></li>
<li>Encryption: <a href="https://www.npmjs.com/package/aes-js">aes-js</a></li>
<li>The Things Network Access: <a href="https://github.com/mqttjs/async-mqtt">async-mqtt</a></li>
<li>Dependency Injection: <a href="https://www.npmjs.com/package/awilix">Awilix</a></li>
</ul>
<h3 id="web-interface">Web Interface</h3>
<ul>
<li>Mapping Library: <a href="https://leafletjs.com/">Leaflet</a></li>
<li><a href="https://github.com/SamHerbert/SVG-Loaders">Loading Animation</a></li>
<li>Packaging system: <a href="https://rollupjs.org/">Rollup</a></li>
</ul>
<h2 id="useful-links">Useful Links</h2>
<ul>
<li>Entropy extraction from the watchdog timer vs the internal clock: https://github.com/taoyuan/Entropy</li>
</ul>
</body>
</html>