Storing data from ESP 32 device, through gateway into AWS IOT Core for Lorawan, and later management

[emilioher]

Hello
I use Arduino ide for sketches in an esp32.
Connect sensors to esp32 node, and send their readings to AWS IOT Core for Lorawan for storage, via a RAK Gateway with Basic Station, for a smart agriculture project.

I have connected a first device with hello World Message.
It is encoded binary base64.
Need to connect 3 sensors and see readings going through well.

question1: Can I leave payloads in binary base64, or must I convert? Objective: store in S3 or Dynamo, an create AWS grafana dashboard with data.

question 2: Do you recommend S3, Kinesis Firehouse or Dynamo DB for storing info? knowing that my intention, with time, is to use images from drone/satellite imaging to combine with sensors data, and proccess with AWS IA/machine learning algorythms services to improve farming tecnique.

Question 3: If I store sensor data in s3 or Dynamo, will AWS recognise and separate data from different devices, sensors and times or do I have to create code for that?
Thanks for assistance
Emilio Hernandez

[asvand]

Q1: you should convert fron base64 to binary, and then run a binary decoder to transform to JSON. After this is done, you can store in S3 or DynamoDB

Q3: You will have to create code for that

Q2: sorry it's too generic question to recommend specific service

[emilioher]

Hello Andrei

thanks for feedback
Q1: Can you guide on how making the decoding to Json?
Q2: Which language must I use? I understand for storing only and use grafana to see, no need for code, only to integrate with imagery. IS that right?
Q3: What I mean is: If I have 1 gateway, 13 esp32 nodes, and 8 sensors per node, totalling 104 sensors, creating data into AWS through AWS IOT Core for LOrawan, will AWS store each sensor data separately? For instance: if I set 3 soil humidity sensors to each node at different soil depths, will AWS keep the data from each sensor identified, so that I can know the depth of each reading?
Thanks

[dizcza]

Q1: you should convert fron base64 to binary, and then run a binary decoder to transform to JSON. After this is done, you can store in S3 or DynamoDB

I think you refer to the following line:

aws-iot-core-lorawan/transform_binary_payload/src-iotrule-transformation/app.py

Line 178 in 75f2b63

result = eval(conversion_function_name)(input_base64, fPort)

The eval function poses security issues as it can execute arbitrary code passed by the user. Mapping all available decoder functions and their names to a dict and therefore restricting the user actions to execute one of the predefined functions will be a safer approach.

[asvand]

Q1: this sample explains it in detail https://github.com/aws-samples/aws-iot-core-lorawan/tree/main/transform_binary_payload
Q2: please have understanding that I can't provide a recommendation for programming language choices. You can find an overview of tools for each programming langugage here: https://aws.amazon.com/tools/
Q3: technically, AWS will not keep any sensor data unless you build your solution in a way it does. When building a solution, it's up to you to decide how AWS will store the sensor data. You can find an example here: https://github.com/aws-samples/aws-iot-core-lorawan/tree/main/timestream

[emilioher]

Thanks Andrei

I am working on this info and come back as I advance or block.
Please dont close conversation, and stand by, though it will take me time.
Thanks
Emilio

[asvand]

Sure, looking forward to hear how it worked out! I also recommend you to go through https://iotwireless.workshop.aws/en/ workshop as it will cover examples you asked for

[emilioher]

Thanks a lot!!

[emilioher]

Hello Andrei
I am having trouble to understand how to send sensor data from my device via LMIC to AWS.
I am using a lyligo lora esp32, hello world is getting through all right, but I have tried several examples and do not manage to get bmp 085 temperature readings through
would you help me break my block?
I attach code used.
I include the libreries (.h)
THe bmp file
create the mydata box
the Lmicdata2 sentence
and send the job but I crash problem after problem
If you can help me mature my code to get my first sensor data through to AWS, I appreciate.

[emilioher]

Ensayo_01102021.txt
THis is my last sketch today.
Get temperature value of bmp sensor in serial port, but in MQTT test of AWS doesnt get through. Any Suggestion?

[emilioher]

[jshakar]

Hi @emilioher

I am trying to setup an esp32 with lora capability to communicate to aws through raspi running basic station. I have been unable to get the esp32 node to successfully transmit the join request and get picked up by the basic station. Could i ask you some questions on your setup?

[emilioher]

Hello Happy Ned Year. Sure, but that was long time ago and It has been years I dont use that technology. I can try BEST regards Enviado desde Outlook para Android<https://aka.ms/AAb9ysg>

…

________________________________ From: jshakar ***@***.***> Sent: Tuesday, January 7, 2025 6:40:28 AM To: aws-samples/aws-iot-core-lorawan ***@***.***> Cc: emilioher ***@***.***>; Mention ***@***.***> Subject: Re: [aws-samples/aws-iot-core-lorawan] Storing data from ESP 32 device, through gateway into AWS IOT Core for Lorawan, and later management (Discussion #24) Hi @emilioher<https://github.com/emilioher> I am trying to setup an esp32 with lora capability to communicate to aws through raspi running basic station. I have been unable to get the esp32 node to successfully transmit the join request and get picked up by the basic station. Could i ask you some questions on your setup? — Reply to this email directly, view it on GitHub<#24 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ARUG3WD54LUG2BZQ6QZDX3D2JNSEZAVCNFSM6AAAAABUW7ABY6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNZVG4ZTCMI>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[jshakar]

Thanks for getting back to me. It is much appreciated.

I have been able to setup basic station on my gateway and its shows as connected in AWS. It is not receiving any loraWAN message from the node. They are both configured to the right freq bands, and all that.

I was able to confirm the node (TTGO T-Beam esp32 based mcu) does tansmit the join request but it doesnt get pick up my the gateway. (it doesn't even show up in the station logs).

Was there any special setup you needed to do to get the node to talk with the gateway? I've added all the profiles and roles and certificates and all that jazz on the aws side.

I am using zero'd-out EUI's for testing (DevEUI, AppKey and AppEUI) since AWS lets you define them instead of generating them for you like TTN.

Did you have to do anything special for the dev, app and key euis?

Any special setup on your rak gateway to receive lorawan messages from nodes or anything?

Lastly, did your RAK concentrator has LED status lights? The "config" light on my WM1302 concentrator stays on even after configuration (see below).

Sorry for the scatter brain questions. I've been trying to debug this issue for weeks. Any help is much appreciated. Thank you for your time!