SEND/RECEIVE HEX commands over mqtt

[lobocobra]

What I intend to do
Over telnet, I can with "log t" and SEND write data directly to the bus and read the answer.
=> I tried to evaluate the data with NODE-RED but failed

I need to do this as I have a floor-heating and thus HC2 is used and not the coded HC1
=> At the end I want to be able to change values of my heating in Openhab.

Solution path

1. I exchange 0x3D with 0x47/0x48, to have HC2 and add a telnet command to switch between the HC1/2
   PRO: simpler to achieve
   CONTRA: I would like to change more setting in my heating, than are currently coded
2. I try to add an MQTT functionality that enables me to SEND HEX Code to the EMS-bus and analyze the answer in NODE-RED.

I opt currently for option 2, but as I have not yet analyzed the code, I want to ask if anyone has a better idea, how I could solve my problem.
=> Does anyone already SEND HEX date over NODE-RED to the WEMOS?
=> Does anyone see a better solution, when I want to send HEX commands to EMS?

[proddy]

I’m thinking the easiest way is to extend the telnet ‘theromstat’ Command with an optional parameter like hc1 or hc2 etc, which always defaults to hc1 if not specified or the current thermostat does not support multiple heating circuits.

Then implement a new mqtt message type which uses a JSON payload containing the heating circuit, command and value.

Is this for a RC35 type thermostat?

[lobocobra]

Yes it is a RC35 type thermostat.
=> Due to that 0x3D needs to be replaced with 0x47
=> Depending what else I want to query I need also 0x48 (but those actions need anyway a coding)
You are right the easiest way to adapt it is by adding HC2 as a telnet command and link it to a MQTT.

Probably this is the first thing I run for to have a working solution.

What I intend to do
Currently I use a SONOFF POW2 to calculate how much oil is left in the tank. All works perfect and I adapted the ARENDST Firmware.
https://github.com/lobocobra/sonoff-shutter
=> But with a POW2 I can not change the temperature of the heating.
=> In conclusion it would be cool if I can send telegram by MQTT. Main Issue will probably be to catch the response.

But I will start with the solution proposed by you.

[lobocobra]

I fail already compiling the standard firmware :(

VSC claims....
Arduino Git/EMS-ESP/src/ems-esp.ino:823:11: error: StaticJsonBuffer is a class from ArduinoJson 5. Please see arduinojson.org/upgrade to learn how to upgrade your program to ArduinoJson version 6

So I concluded I need to install in arduino the arduinojson 5.11.... still same outcome. :(
=> Do I need to install an old Arduino IDE for the platformIO?

Compiling .pioenvs\d1_mini\src\ems-esp.ino.cpp.o
C:/Arduino Git/EMS-ESP/src/ems-esp.ino:823:11: error: StaticJsonBuffer is a class from ArduinoJson 5. Please see arduinojson.org/upgrade to learn how to upgrade your program to ArduinoJson version 6
ok = true;
^
Compiling .pioenvs\d1_mini\libb98\myESP\MyESP.cpp.o
C:/Arduino Git/EMS-ESP/src/ems-esp.ino: In function 'void publishValues(bool)':
C:/Arduino Git/EMS-ESP/src/ems-esp.ino:823:21: error: expected primary-expression before '<' token
ok = true;
^
C:/Arduino Git/EMS-ESP/src/ems-esp.ino:823:37: error: 'jsonBuffer' was not declared in this scope
ok = true;
^
C:/Arduino Git/EMS-ESP/src/ems-esp.ino:891:23: error: 'ArduinoJson::JsonObject' has no member named 'measureLength'
myDebug("MQTT topic: thermostat mode value %s", message);
^
C:/Arduino Git/EMS-ESP/src/ems-esp.ino:893:16: error: 'ArduinoJson::JsonObject' has no member named 'printTo'
ems_setThermostatMode(2);
^
C:/Arduino Git/EMS-ESP/src/ems-esp.ino:1005:31: error: 'ArduinoJson::JsonObject' has no member named 'measureLength'
ems_doReadCommand(scanThermostat_count, EMS_Thermostat.type_id);
^
C:/Arduino Git/EMS-ESP/src/ems-esp.ino:1007:24: error: 'ArduinoJson::JsonObject' has no member named 'printTo'
}
^
lib\myESP\MyESP.cpp:974:11: error: StaticJsonBuffer is a class from ArduinoJson 5. Please see arduinojson.org/upgrade to learn how to upgrade your program to ArduinoJson version 6
StaticJsonBuffer<SPIFFS_MAXSIZE> jsonBuffer;
^
lib\myESP\MyESP.cpp:1013:11: error: StaticJsonBuffer is a class from ArduinoJson 5. Please see arduinojson.org/upgrade to learn how to upgrade your program to ArduinoJson version 6
StaticJsonBuffer<SPIFFS_MAXSIZE> jsonBuffer;
^
*** [.pioenvs\d1_mini\src\ems-esp.ino.cpp.o] Error 1
In file included from C:\users\rrs.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266WiFi\src/WiFiClient.h:25:0,
from C:\users\rrs.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266WiFi\src/ESP8266WiFi.h:39,
from C:\users\rrs.platformio\packages\framework-arduinoespressif8266\libraries\ArduinoOTA/ArduinoOTA.h:4,
from lib\myESP\MyESP.h:13,
from lib\myESP\MyESP.cpp:11:
c:\users\rrs.platformio\packages\toolchain-xtensa\xtensa-lx106-elf\include\c++\4.8.2\bits\stl_vector.h: In member function 'std::vector<_Tp, _Alloc>::size_type std::vector<_Tp, _Alloc>::_M_check_len(std::vector<_Tp, _Alloc>::size_type, const char*)
const':
C:\users\rrs.platformio\packages\framework-arduinoespressif8266\cores\esp8266/Arduino.h:253:18: error: expected unqualified-id before '(' token
#define max(a,b) ((a)>(b)?(a):(b))
^
c:\users\rrs.platformio\packages\toolchain-xtensa\xtensa-lx106-elf\include\c++\4.8.2\bits\stl_bvector.h: In member function
'std::vector<bool, _Alloc>::size_type std::vector<bool, _Alloc>::_M_check_len(std::vector<bool, _Alloc>::size_type, const char*) const':
C:\users\rrs.platformio\packages\framework-arduinoespressif8266\cores\esp8266/Arduino.h:253:18: error: expected unqualified-id before '(' token
#define max(a,b) ((a)>(b)?(a):(b))
^
lib\myESP\MyESP.cpp: In member function 'bool MyESP::_fs_loadConfig()':
lib\myESP\MyESP.cpp:974:21: error: expected primary-expression before '<' token
StaticJsonBuffer<SPIFFS_MAXSIZE> jsonBuffer;
^
lib\myESP\MyESP.cpp:974:38: error: 'jsonBuffer' was not declared in this scope
StaticJsonBuffer<SPIFFS_MAXSIZE> jsonBuffer;
^
lib\myESP\MyESP.cpp: In member function 'bool MyESP::fs_saveConfig()':
lib\myESP\MyESP.cpp:1013:21: error: expected primary-expression before '<' token
StaticJsonBuffer<SPIFFS_MAXSIZE> jsonBuffer;
^
lib\myESP\MyESP.cpp:1013:38: error: 'jsonBuffer' was not declared in this scope
StaticJsonBuffer<SPIFFS_MAXSIZE> jsonBuffer;
^
lib\myESP\MyESP.cpp:1032:10: error: 'ArduinoJson::JsonObject' has no member named 'printTo'
json.printTo(configFile);
^
*** [.pioenvs\d1_mini\libb98\myESP\MyESP.cpp.o] Error 1
================================================ [ERROR] Took 2.54 seconds ================================================

======================================================== [SUMMARY] ========================================================
Environment nodemcuv2 [SKIP]
Environment d1_mini [ERROR]

[proddy]

Hi @lobocobra you should have created a separate issue for this as not to confuse it with the enhancement you're making to handle multiple heating circuits.

But, yes, looks Benoit @ ArduinoJson has finally unleashed the mighty V6 of his json library which is now the default standard in platformio. I'll need to make a few adjustments as the library handles documents differently and I'll probably decide to serialize for performance while I'm at it. In the meantime to keep it all working with v5 try replacing the ArduinoJson in platformio.ini lib_deps section with https://github.com/bblanchon/ArduinoJson#v5.13.5

[Gert05]

Yep, I can confirm that V6 causes problems. I’ve been struggling with vsc for a couple of days, until I finally copied ArduinoJson v5.13.5 into the library directory. This is a working alternative to the lib_deps solution mentioned above.

[proddy]

I've fixed this in 1.5.5. Just running some stress tests and will upload shortly.

[lobocobra]

Many thanks! I started to doubt, that I am even able to compile a standard firmware :-)

[lobocobra]

@proddy
I made a new console command and can already change the "Heating Circuit 1/2".
=> But I fail to save the value to SPIFF

My thought was that I can extend bool FSCallback(MYESP_FSACTION action, const JsonObject json) and I added:
json["hc2"] = EMSESP_Status.hc_number;

and I want to read with...
// lobocobra start if (!(EMSESP_Status.hc_number = json["hc2"])) { EMSESP_Status.hc_number = EMSESP_HC2; // set default } //lobocobra end

But I get a wired number from "hc2".
=> Can you point me to the section, where I can extend the reading/writing code, so that the heating circuit can be reloaded after reboot?

I saw that the code for writing to SPIFF is in MyESP.cpp, but I simply want to add 1-2 variables.
=> Thanks for a hint, where to change the code....

[proddy]

That should work. What is the variable hc_number defined as? It may be something wrong in my code but will do a check myself today.

Another thing though, I’m not sure this is the best approach. Depending on the thermostat type we should be able to detect and read out all heating circuit values and send/receive via MQTT topics. No need to preset the hc in spiffs, right?

[lobocobra]

Many thanks for the fast answer.
=> I based now on v1.5.5 and it works now perfectly. I probably made an error somewhere on my side.

Approach
My approach is currently to replace #define EMS_TYPE_RC35Set 0x3D by a variable, which should allow dynamic changes to the adress depending on the thermostat/situation.
=> I will make a working version and then I'll rewrite it anyhow (once I understand the code better).

As I can now save/read values and exteded the telnet command, I will attack the mqtt extention part
=> I need to subscribe/publish, the new value and also extend the send command to extend by mqtt raw data.

[lobocobra]

I have now a working version and can change the heating circuit 2 over mqtt (and thus read the right data).

Before I make a pull request, I will continue to test it.
=> https://github.com/lobocobra/EMS-ESP-HC2

What is still missing
I extend now the code to send raw data over mqtt

[proddy]

nice. Can you do the PR on the 'dev' branch when ready?

[lobocobra]

Hi proody

Yes of course, I will do it.

Following things are working

• switching of heating circuit console/mqtt and save the status
• executing mqtt raw message (answer is not yet implemented, its fire and forget so far)
• thermostat mode is now working
• thermostat temp is working for the night-temp

not yet solved issues
I assume it has to do with the fact, that I have a floor heating and no boiler. This seems to be a combination, which is not yet reflected in the code.... working on it....

• thermostat temp is not working for day-temp as the DAY bit is not recognized
  => checked it for 2hours, but the checked flag is in my system never changing (day/night)
• info (thermostat is randomly updated, sometimes with wrong values)
  => invested 2h but I did not get it yet, where the data is taken (should be the 60sec message)
• thermostat mqtt is not sending the gathered values
  => it seems that the system is not recognizing my thermostat, if I switch the heat-circuit, so I missed one place to change (I guess)

features to implement

• mqtt raw must replay to mqtt
• turn-on / off boiler section, if there is no boiler
• calculate liter of oil remaining in the tank of the heating (currently running on sonoff)
• full error code display and mqtt sending

So I have still alot to do, but luckly the code really good commented. That helps much.

[lobocobra]

@proddy
Ha! Just when I had all working against DEV1.6.0b3, I saw that the code evolved to DEV1.6.0b6.
=> Unfortunately there are some major changes between, the 2 versions compared to what I changed and on my small laptop screen, the 3-way merge was a mission impossible and ended +/- ok but showed wrong temps in the console. I doubt that I can do a merge like this. But I will try tomorrow.

Anyhow... for those interested you can find my working changes against 1.6.0b3 here:
https://github.com/lobocobra/EMS-ESP-HC2 (!!! only in DEV):

• MQTT send raw is working.
• Heating Circuit of RC35 can now be changed on the fly without reboot.
• MWTT messages are now working and were extended if you have a floor-heating.
• The thermostat commands work now with HC2 and the console show the right values.
• Found some minor issues and fixed them, to make all work with floor heating
  => Maybe anyone with a bigger screen can merge my 1.6.0b3 with the latest DEV.

[proddy]

I'll take a look and see about merging some of your updates in the dev branch (which I'm calling 1.7)

[proddy]

I have some questions on the design

• Sending raw telegrams via MQTT to the ems-esp I think is dangerous. There is no validation or way to handle the responses correctly, which could have unknown effects if done carefully. Although I understand what you're trying to do with the Sonoff POW I just don't see this a valid use case to add to the code for others.

• Handling multiple heating circuits on an RC35 is useful. I think here the correct approach is to query all the circuits to see which are active and then report all the temperatures just like we do with the normal thermostat. Then have an optional parameter in the thermostat temp xx command and MQTT to select which HC the user wants. Hope that makes sense.

The major change between 1.6b3 and 1.6b6 is that way the values are stored. I removed floats completely and store the raw values from the telegram, and then render to floating point when sending to MQTT. This is for speed optimization and memory usage. I can help get your dev branch working

[lobocobra]

Hi Proddy

As you saw probably from my code, I am far away from your programming skills. But its fun to learn from your code. My big programming days, were 30 years ago :-), when I thougt that command.com was programmed lousy and I re-programmed it.

MQTT
I agree with your statement. The code represents a risk for most and its easier to code the needed logic into the firmware, rather than to do it with node-red or similar externally. Now that I understand the code better, I see also no use case for other users.
=> My thoughts were to have a back-door for the future. Similar to your console "SEND xx" which I love.

Multiple heating circuits
Of course it would be the best to check, which of the 4 possible circuits are active. But personally I doubt, that someone with a heating with 4 active circuits will even bother to program something.
=> This left me with 2 realistic use cases...... they have a floor-heating (HC2) or a regular heating (HC1). Of course I can be mistaken, but my aim was to reactivate your standard commands like thermostat temp , which worked again perfectly in version 1.6b3.

The part which did the trick for me was here....

/**
 * Find the pointer to the EMS_Types array for a given type ID
 */
int _ems_findType(uint8_t type) {
    uint8_t i         = 0;
    bool    typeFound = false;
    // scan through known ID types
    while (i < _EMS_Types_max) {
        //lobocobra start we accept also if we have 47 or 48 instead of 3D or 3E
        if ((EMS_Types[i].type == type) || (EMS_Types[i].type == type -10 )) {
        //lobocobra end
            typeFound = true; // we have a match
            break;
        }
        i++;
    }

Now as this section is gone in the newest version; I would either have to start from scratch or simply hard code the HC2 address into the code, which avoids all the coding.
=> I would be happy if you could help me in making the HC-switch working again.

[proddy]

Ok, you put quite some work into this so let's break it down into chunks and program it together under the dev branch.

• I can see the sense in that only one of the two Heating Circuits will ever be used. So let the user define which one they wish to use (1 or 2) and store this as a setting just like you did. It will default to 1 if using a thermostat that doesn't support multiple heating circuits. We don't need to change the MQTT code as there will only be one set of values. The 'set thermostat' command will always respond to the chosen HC. For this feature let's create a new GitHub issue and use that for tracking.
• Turning off the boiler section if there is no boiler is also a good idea. I was thinking of the boiler_type is unset (value 0x00) then displaying the data and sending via MQTT is ignored. Does this make sense? Let's create a GitHub issue for this two.
• Send raw HEX over MQTT we agreed is not needed, so suggest we close this enhancement issue.

Am I missing anything?

[lobocobra]

Sounds great! All 3 points make sense.

I would only propose to add following mqtt commands:

        rootThermostat[THERMOSTAT_daytemp]  =       _float_to_char(s, EMS_Thermostat.daytemp);
        rootThermostat[THERMOSTAT_nighttemp]  =     _float_to_char(s, EMS_Thermostat.nighttemp);
        rootThermostat[THERMOSTAT_holidaytemp]  =   _float_to_char(s, EMS_Thermostat.holidaytemp);
        rootThermostat[THERMOSTAT_heatingtype]  =   _int_to_char(s, EMS_Thermostat.heatingtype);
        rootThermostat[THERMOSTAT_circuitcalctemp]= _int_to_char(s, EMS_Thermostat.circuitcalctemp);

The enable Openhab (or similar) to change the basic settings of the heating and to run a script at the end of the vacation, to slowely heat up the house.