Proposal for Cloud Integration: Expanding EMS-Gateway Use

[FieteRensen]

Hello Proddy,
Hello Michael,

I've been a satisfied customer of your excellent EMS-ESP-Gateway for three years.
Just last week, I ordered another three gateways.

What do I want from you?
I would like to install your EMS-GATEWAY in many other houses every year.
What I'm missing is a cloud connection, similar to openenergymonitor.org, where data from your gateways can be uploaded.

Who am I?
I'm a software developer at a bank, specializing in data analysis, and I live in the northern part of Germany, in the town of Oldenburg.

After my initial successful DIY-Heatpump installations, I started advising homeowners on planning their heat pumps for their existing heating systems.

What's my problem?
I can calculate the heating load for each room from the construction documents, select the appropriate radiators, and suggest a suitable heat pump. However, I'm missing a lot of data from the heating system's bus system to assess the quality of the existing heating system.

Which data do I want to upload to a cloud?

• Flow temperature
• Return temperature
• Burner modulation
• Heating circuit pump modulation
• DWH temperature
• Circulation pump

What conclusions about the quality of the heating system can be drawn from this data?

• Whether the temperature difference between the flow and return temperatures is sufficiently low for a heat pump.
• Whether the cycle runtimes are sufficiently long on very cold days and warmer days.
• Whether the flow temperature is below 45 degrees on very cold days.
• Whether the domestic hot water storage tank is charged a maximum of three times a day.

Why should data be uploaded to the cloud?
Unfortunately, I cannot use your EMS-ESP-GATEWAY in other people's homes because they don't have a Home Assistant or another home automation system.
Additionally, I would like the data to be stored in a highly standardized manner so that a regional benchmark can be drawn across different houses.

Through which channels can I have the EMS-Gateway installed?
I have contacts with local craftsmen who maintain and install Buderus boilers.
I want your EMS-Gateway to be installed on every Bosch boiler system that is being converted to a heat pump.
For a prototype test, I can currently provide 6 houses in the region.
Possibly, through the craftsmen, I can provide even more houses.

Can I provide technical support?
I've been a Java programmer for 20 years; perhaps I can assist you with a cloud and database solution.
Maybe I can also find a statistician/mathematician for data analysis.

[proddy]

This needs some thought, and what first comes to mind is for you to host an MQTT broker in the cloud and have EMS-ESP publish to that? In each household you would only need to open a port and it'll be read-only data going out.

[MichaelDvP]

Search the internet for "private mqtt cloud" and you'll find a lot of managed services and tutorials for aws/azure/etc.

[bbqkees]

Yes the easiest way to get what you need is to use an external MQTT broker.

The whole idea of EMS-ESP and the EMS Gateway is to provide local access to the boiler without use of the cloud.
But with an external/public broker (which you secure to the max) you can still create cloud access.
Just make sure the router at the customer allows the connection to be open.

[FieteRensen]

Thank you for your responses.

The question is whether you want to open your product to more user groups.

How many boilers from your relatives, friends, and colleagues are compatible with your EMS-Gateway?

How many households have the technical know-how to install and operate a Home Assistant or others?

In my opinion, it's essential to run heating monitoring to ensure the transition to a heat pump doesn't end in disaster.

An alternative to a solution through your EMS-ESP-Gateway is a cloud solution by Bosch.
If Bosch even desires it, the data would no longer be available to the community.

So far, Bosch and the respective craftsmen have benefited greatly from poorly planned heat pumps.

The customer has to pay for the unnecessarily high electricity costs.

I will try to feed the Mqtt-Service from openenergymonitor.org with data from the EMS-Gateway project.
I'll probably encounter authentication problems with the Mqtt-Service of Openenergymonitor.org.

I will likely first direct the data from the EMS-Gateway to my local Mqtt-Broker.
Then convert your Json-Payload and subsequently publish it to the Mqtt-Broker of openenergymonitor.org

[FieteRensen]

I just discovered that the three Gateway S32 (Standard WiFi Edition V2.0) units I bought a few days ago don't have an S32-S3 inside.

With my current Esp-Gateway, I couldn't test a secure MQTT connection to a cloud provider.

Additionally, I found out that openenergymonitor.org doesn't host an MQTT broker. The MQTT broker is only available for local installations.

As an alternative, I looked into the Thingsboard project. Thingsboard also doesn't provide an MQTT broker for publishing IoT sensor metrics. Instead, IoT metrics have to be uploaded via a REST call.

[bbqkees]

Your older Gateway is probably an ESP8266 one with very old firmware 2.1, for which development stopped 2 years ago.
The S32 with firmware 3.6 has so much more features. Just try testing with MQTT with the S32 and not the old Gateway.

There are numerous MQTT cloud providers out there.
Once you have set one up, and send all the data from all customer Gateways to that service, you can connect your own MQTT client to it and retrieve the data in your own application. You could probably even use something simple like NodeRed to put it in the right format.

Once it is setup all Gateways will just keep publishing to the broker continuously.

[FieteRensen]

My three-year-old ESP-Gateway is not connected to the boiler.

I tested the new ESP-Gateway you delivered to me on Saturday with version 3.6 release. In the release notes for version 3.6, it states the following:
"Added support for TLS in MQTT (ESP32-S3 only) #1178"

Ticket #1178 notes that only on the ESP32-S3 is there a corresponding text field for uploading a certificate.

Overall, I'm impressed by the progress from the old to the new ESP-Gateway. You've done a great job integrating it with Home Assistant.

The public MQTT broker is just one step in the journey.
Using a secured MQTT broker costs 200 euros per month.
For the prototype phase, this is too expensive.

I want to use a platform that has a database and a dashboard. Unfortunately, I can only supply the cloud platforms I mentioned with data via a secured REST-call.

Does the ESP32-S3 have enough hardware reserves for such a REST-call as an alternative to the previous MQTT-calls?

https://emoncms.org/site/api#input

[proddy]

Does the ESP32-S3 have enough hardware reserves for such a REST-call as an alternative to the previous MQTT-calls?

Yes, but it would mean finding a new webserver that supports SSL. See #36 . You have 20+ years OO programming so maybe work on it together?

[FieteRensen]

That sounds good.

Currently, I'm a complete novice in the microcontroller world.
For the past 20 years, I've been programming with Java, PL/SQL, and PHP.
A few days ago, I started copying and modifying code for the https://github.com/Zeppelin500/MBusino board.
Over the past three weeks, I've read a lot of code for projects supporting the DS18B20 sensors.

Can you recommend the version of the ESP32-S3 that I can best use for testing the emsesp project?