[mqtt] Treat incoming empty string as NULL for most types

[ccutrer]

Empty strings are often received when deleting retained topics when a device goes offline, or as the result of a transformation that is missing a value (such as a "scene" event from zwave-js-ui, which sends JSON with a timestamp and the scene value, then immediately sends a value to the topic with only a timestamp).

For string channels, add a configuration value to allow setting a specific string for treating as NULL, since empty string can make sense for that type.

[kaikreuzer]

Thanks for this enhancement.
Just a question: Are you sure that UNDEF is the right state here and not maybe NULL?
UNDEF is meant for ambiguous situations, when there actually IS a real state out there, we just do not know how to represent it in the item. NULL means that there isn't any state at all.

[ccutrer]

No I'm not sure at all. I'm never quite clear on which of whose is which. The intended purpose here is "we know that the item does not currently have a state" not "we don't know what the state is." I thought UNDEF was what I wanted, since NULL is what items start out as when first created or on boot, before persistence has restored them (and thus we don't know what their state is, or if we even have a state).

[kaikreuzer]

Well, the question is how we want treat the incoming value. If we say that the current state of the item is probably right and we simply want to ignore the incoming empty string, then UNDEF is probably indeed the right option. If the empty string means that a topic does not exist anymore and there is no representation "in the outside world" anymore, NULL would be better.
If the first case is what we deal with here, all is fine.

[ccutrer]

Okay, I think we want use NULL then. UNDEF is used for NaN, which does match your description, and this is for either "the device is gone, so no longer has a state" or "there is no state." Either of which is a definitive representation.

[apella12]

Background: I have been running Zwave-JS-UI (zwave2mqtt) using generic MQTT things because of issues with the HA configs (i.e #15145). As noted above, HA scenes configs have momentary values, but revert to empty. I addressed (no WARN messages) with the empty Scene topic problem in the generic MQTT by using a string channel, only getting the "value" and using MAP (=OFF).

I recently set up a test environment mirroring one production instance using the HA configs. The HA Scene configs are pulled into OH as a number channel; mqtt:homeassistant_zwavejs2mqtt_5F0xf1c43721_5Fnode8:6035d9536c:zwavejs2mqtt_5F0xf1c43721_5Fnode8:zwavejs2mqtt_5F0xf1c43721_5F8_2D91_2D0_2Dscene_2D006#sensor (Number)
The channels are;
{"state_topic":"zwave1/nodeID_8/central_scene/endpoint_0/scene/007","value_template":"{{ value_json.value | default('') }}","state_class":"measurement","availability":[{"payload_available":"true","payload_not_available":"false","topic":"zwave1/nodeID_8/status","value_template":"{{'true' if value_json.value else 'false'}}"},{"topic":"zwave1/_CLIENTS/ZWAVE_GATEWAY-zwave-js-ui/status","value_template":"{{'online' if value_json.value else 'offline'}}"},{"payload_available":"true","payload_not_available":"false","topic":"zwave1/driver/status"}],"availability_mode":"all","json_attributes_topic":"zwave1/nodeID_8/central_scene/endpoint_0/scene/007","device":{"identifiers":["zwavejs2mqtt_0xf1c43721_node8"],"manufacturer":"Fibargroup","model":"Keyfob (FGKF601)","name":"nodeID_8","sw_version":"3.2"},"name":"nodeID_8_scene_state_scene_007","unique_id":"zwavejs2mqtt_0xf1c43721_8-91-0-scene-007"}

And the WARN sequence (just 1 example). I have 26 scene channels (not all used).

2024-01-28 13:48:31.350 [DEBUG] [.internal.JinjaTransformationService] - about to transform '{"time":1706458126275}' by the function '{{ value_json.value | default('') }}'
2024-01-28 13:48:31.351 [DEBUG] [.internal.JinjaTransformationService] - transformation resulted in ''
2024-01-28 13:48:31.352 [WARN ] [ab.binding.mqtt.generic.ChannelState] - Command '' from channel 'mqtt:homeassistant_zwavejs2mqtt_5F0xf1c43721_5Fnode59:6035d9536c:zwavejs2mqtt_5F0xf1c43721_5Fnode59:zwavejs2mqtt_5F0xf1c43721_5F59_2D91_2D0_2Dscene_2D001#sensor' not supported by type 'NumberValue': null

Anyway I was going to create an issue, but saw this PR and wanted to ask if this code was going to fix the Scene problem? If not, is there a way to change the HA Scene config from Number to String in OH? Despite all the complexity, all the Scenes contain "91" as that is the Zwave command class enum for central_scene.

[ccutrer]

@apella12 : yes, this will fix that error in the logs. And is one of several reasons I wrote this PR. I also use zwave-js-ui.

[kaikreuzer]

lgtm, thanks!

[stedon81]

Hi ErikDB87,
thanks very much for this achievement! Exactly I was looking and hoping for!
However, actually your change in the mqtt binding violates the way how the Homie Convention says to interprete the values. (There they explicitly say that an empty value is forbidden: "An empty string ("") is not a valid payload").
Therefore I asked the community around the Homie Convention to come up with a specification for those cases (homieiot/convention#284). I would strongly hope that they realize this as simple as your idea is.

Cheers,
Stefan D.

[ccutrer]

Heh, yes, I was very aware the Homie convention forbids empty payloads (presumably because that's how you "delete" retained topics in MQTT). But given their language, one should never encounter an empty payload, and thus doesn't specify how one should handle it if it does happen - it's simply error handling. For example, it does not say "a client should retain the prior value when an empty string is received."

I was not aware of that discussion, or even the existence of an upcoming Homie 5, so thank you for that link. I stopped looking at the spec when it had not changed for years, except for some minor changes that they didn't actually change the version number for.

I do agree with you on the alerts topic though- the spec is so vague as to the contents of its payloads that it's essentially useless.

[openhab-bot]

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/mqtt-discussion/154139/13

[openhab-bot]

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/ignoring-empty-mqtt-payload/154554/9