TTN MQTT & device management (#8) #13

noerw · 2017-11-06T21:08:51Z

implements the feature that was suggested in #8.
Also featuring a rewrite of error handling code

will the oSeM-frontend or -backend call the webhook to trigger the TTN registration?
- (scalable) authentication mechanism! (currently token auth from env variable)

TODO

update docker build
update jsdoc
end to end tests with thethingsnetwork (ttnctl)
update README: document device registration & uplink flow
update osem db schema: add box.integrations.ttn.dev_eui, make app_id, dev_id, port tuple unique
add device registration call to GET ttnintegration/v1.1/ttndevice/:boxId at PUT & POST mainapi/boxes/:boxId handler
add new env vars to sensebox/OSeM-compose

closes #12

Now using Error subclasses to separate different response types instead of a homebrew `err.code` check. This allows to move the selection of errortype from the calling promise chain to the throwing function, improving consistency and readability.

TODO: authentication. right now theres an IP whitelist. proper authentication would allow queries directly from the oSeM frontend, which would provide a smoother integration in the end..

ubergesundheit · 2017-11-29T08:49:53Z

lib/database.js

+        const p = box.integrations.ttn.port;
+
+        return (p === port || p === undefined);
+      })[0];


The app_id and dev_id are not enforced to be unique on the model schema. You should add the requirement to the main project

Indeed, the app_id, dev_id, port tuple is assumed to be unique..

ubergesundheit · 2017-11-29T08:54:06Z

lib/ttn.js

+const log = createLogger('ttn');
+
+const onUplinkMessage = function onUplinkMessage (dev_id, message) {
+  const { app_id, port } = message;


Duplicate code to the httpIntegrationHandler ?

This is because the response handling is different & I wanted to have different loggers for each protocol, and I didn't consider 10 duplicated lines to be worth an additional abstraction

ubergesundheit · 2017-11-29T09:14:58Z

Just pushed a docker image of this branch to sensebox/ttn-osem-integration:feature_mqtt

ubergesundheit · 2017-11-29T09:29:35Z

Running the image results in a Cannot find module babel-runtime/regenerator error


> [email protected] start /usr/src/app
> node lib/index.js | bunyan -o short

module.js:471
    throw err;
    ^

Error: Cannot find module 'babel-runtime/regenerator'
    at Function.Module._resolveFilename (module.js:469:15)
    at Function.Module._load (module.js:417:25)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/usr/src/app/node_modules/ttn/dist/index.js:8:20)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/usr/src/app/lib/ttn.js:6:36)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)

noerw · 2017-11-29T12:40:12Z

I have this dependency in node_modules on my local installation.. sigh, shouldn't yarn.lock protect us from stuff like this?

See TheThingsArchive/node-app-sdk@e1eca3f, it seems the babel people (™) have messed with semver or something.
Instead of adding this dependency here, lets use the ttn beta release for now, it only contains changes from my PR anyway.

noerw · 2017-11-29T12:41:46Z

Also, I can finally test this with actual LoRa hardware in the next days. Will add end-to-end tests then

ubergesundheit · 2017-11-29T12:43:41Z

Very good!
There is an additional error related to alpine/muslibc/glibc for the dependency grpc of the ttn package..

Error: Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /usr/src/app/node_modules/grpc/src/node/extension_binary/node-v48-linux-x64/grpc_node.node)
    at Error (native)
    at Object.Module._extensions..node (module.js:597:18)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/usr/src/app/node_modules/grpc/src/node/src/grpc_extension.js:30:15)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/usr/src/app/node_modules/grpc/src/node/src/client.js:38:12)

removes need to set the dev_id at all for mqtt boxes

noerw · 2018-02-05T07:53:00Z

@ubergesundheit Hey, this is ready, implementation wise.
For these changes to be useful, they have to be integrated with the main API.
I cannot do that atm, due to lack of time and understanding of the new package architecture.

If required I can add comprehensive documentation on the required changes, a short list is in the TODO at the top of this PR already.

ubergesundheit · 2018-02-05T08:38:54Z

Alright! Are there any changes to the angular app needed?

noerw · 2018-02-05T09:24:55Z

yeah. the user should be able to select between his own TTN device with http integration (current solution), or the auto-creation of a device with our application.
If he chooses the latter, he may provide an dev_eui, and gets the lora keys (otaa and abp?) back in the completed registration overview.

noerw · 2018-02-05T09:30:36Z

alternatively, a full integration with a Lora sketch containing the Lora keys comes to mind.
would this be feasible, or does the lora hardware require custom solutions anyway?

ubergesundheit · 2018-02-05T09:31:51Z

With the new senseBox MCU in the pipeline, a LoRa Bee addon is planned. The generation of a full working sketch should be possible then.

noerw and others added 11 commits September 16, 2017 14:33

use bunyan for logging

00b1379

closes #12

connect to ttn mqtt broker

52c3a03

handle mqtt uplink messages

11f8990

fix errorhandling (mostly)

a53a30c

fix errorhandling (for real!)

8a80f20

Now using Error subclasses to separate different response types instead of a homebrew `err.code` check. This allows to move the selection of errortype from the calling promise chain to the throwing function, improving consistency and readability.

send response before logging, lint

f98d39e

update opensensemap API

5dbe568

add hook for registration of new devices.

be3d70f

TODO: authentication. right now theres an IP whitelist. proper authentication would allow queries directly from the oSeM frontend, which would provide a smoother integration in the end..

Merge branch 'feature/mqtt' into develop

44c17b2

apply correct keys, comments

941f147

change route auth to Authorization header token

fc9c445

noerw added the work in progress label Nov 6, 2017

ubergesundheit reviewed Nov 29, 2017

View reviewed changes

noerw added 11 commits December 2, 2017 19:25

cleanup & fix for missing auth token

baeb09d

update ttn to v2.3.0-alpha.3

26351c9

fix docker build

c8cec3f

improve error handling

9762ba5

add tests for /v1.1/ttndevice/:boxId

26a7690

add mqtt end2end tests

3a4bce3

mqtt: match box via boxID

dfee586

removes need to set the dev_id at all for mqtt boxes

update jsdoc

d34cd9f

use standard HTTP codes

1605e5b

minor readme fix

f6c0911

update deps

b37c5a3

noerw added help wanted and removed work in progress labels Feb 5, 2018

This was referenced Feb 11, 2018

support new ttn-integration sensebox/openSenseMap-API#146

Merged

add env variables for new ttn-integration sensebox/OSeM-compose#1

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TTN MQTT & device management (#8) #13

TTN MQTT & device management (#8) #13

noerw commented Nov 6, 2017 •

edited

Loading

ubergesundheit Nov 29, 2017

noerw Nov 29, 2017

ubergesundheit Nov 29, 2017

noerw Nov 29, 2017

ubergesundheit Nov 29, 2017

ubergesundheit commented Nov 29, 2017 •

edited

Loading

ubergesundheit commented Nov 29, 2017

noerw commented Nov 29, 2017 •

edited

Loading

noerw commented Nov 29, 2017

ubergesundheit commented Nov 29, 2017

noerw commented Feb 5, 2018

ubergesundheit commented Feb 5, 2018

noerw commented Feb 5, 2018

noerw commented Feb 5, 2018

ubergesundheit commented Feb 5, 2018

TTN MQTT & device management (#8) #13

Are you sure you want to change the base?

TTN MQTT & device management (#8) #13

Conversation

noerw commented Nov 6, 2017 • edited Loading

ubergesundheit Nov 29, 2017

Choose a reason for hiding this comment

noerw Nov 29, 2017

Choose a reason for hiding this comment

ubergesundheit Nov 29, 2017

Choose a reason for hiding this comment

noerw Nov 29, 2017

Choose a reason for hiding this comment

ubergesundheit Nov 29, 2017

Choose a reason for hiding this comment

ubergesundheit commented Nov 29, 2017 • edited Loading

ubergesundheit commented Nov 29, 2017

noerw commented Nov 29, 2017 • edited Loading

noerw commented Nov 29, 2017

ubergesundheit commented Nov 29, 2017

noerw commented Feb 5, 2018

ubergesundheit commented Feb 5, 2018

noerw commented Feb 5, 2018

noerw commented Feb 5, 2018

ubergesundheit commented Feb 5, 2018

noerw commented Nov 6, 2017 •

edited

Loading

ubergesundheit commented Nov 29, 2017 •

edited

Loading

noerw commented Nov 29, 2017 •

edited

Loading