Ua-deltaframe Messages don't contain ExtensionFields

[hvetter-de]

Describe the bug
I have tested several messaging profiles from the opc publisher module to find how how the ExtensionFields look in the different profiles.
I now have the issue, that the ExtensionFields doesn't appear for example the "FullNetworkMessages" profile.
Only message profile where I can see the ExtensionFields is "FullSamples".

Here it is documented that several profiles should support ExtensionFields: https://github.com/Azure/Industrial-IoT/blob/release/2.9.0/docs/opc-publisher/messageformats.md#messaging-profiles-supported-by-opc-publisher
But I can only get it working for "FullSamples"

To Reproduce
I have an Iot Edge device with these modules:
$edgeAgent, $edgeHub, opcplc, opc-publisher

The publisher is configured with this container create options:

{
  "Cmd": [
    "--PkiRootPath=/mount/pki",
    "--PublishedNodesFile=/appdata/pn.json",
    "--CreatePublishFileIfNotExistKey=false",
    "--AutoAcceptUntrustedCertificates",
    "--AddAppCertToTrustedStore=true",
    "--LogLevel=Debug",
    "--strict",
    "--fm=true",
    "--mm=FullNetworkMessages",
    "--me=JsonReversible",
    "--bs=2"
  ],
  "HostConfig": {
    "Binds": [
      "/mnt/opcpublisher/pki:/mount/pki",
      "/mnt/opcpublisher/appdata:/appdata"
    ],
    "CapDrop": [
      "CHOWN",
      "SETUID"
    ]
  }
}

Endpoints configuration returned from "GetConfiguredEndpoints_V1" direct method:

{
    "status": 200,
    "payload": {
        "endpoints": [
            {
                "Version": 1,
                "LastChangeTimespan": "2023-10-20T09:16:22.0490753Z",
                "DataSetWriterId": "DataFlow1",
                "DataSetWriterGroup": "Asset1",
                "EndpointUrl": "opc.tcp://opcplc:50000",
                "DataSetExtensionFields": {
                    "FactoryId": "11",
                    "ExternalEquipmentId": "1234"
                },
                "UseSecurity": true,
                "OpcAuthenticationMode": "Anonymous"
            }
        ]
    }
}

Expected behavior
I expected that the FullNetworkMessages would also contain the ExtensionFields.
I have tested the FullSamples profile and received the ExtensionFields.
But for FullNetworkMessages it is not working.
FullSamples-JSONReversible Sample:

{
    "NodeId": "nsu=http://microsoft.com/Opc/OpcPlc/;s=LongString10kB",
    "EndpointUrl": "opc.tcp://opcplc:50000/",
    "ApplicationUri": "urn:OpcPlc:opcpl",
    "DisplayName": "nsu=http://microsoft.com/Opc/OpcPlc/;s=LongString10kB",
    "Timestamp": "2023-10-20T09:10:34.9683776Z",
    "Value": {
      "Value": {
        "Type": 12,
        "Body": "HHHHHHHHHHHHHHHHHHHHH"
      },
      "SourceTimestamp": "2023-10-20T09:10:34.2671998Z",
      "ServerTimestamp": "2023-10-20T09:10:34.2672417Z"
    },
    "SequenceNumber": 212,
    "ExtensionFields": {
      "WriterGroupId": "Asset1",
      "FactoryId": "[FactoryId, { \"Value\": \"11\", \"Raw\": \"11\", \"IsObject\": False, \"IsArray\": False, \"IsBoolean\": False, \"IsString\": True, \"IsBytes\": False, \"IsDecimal\": True, \"IsDouble\": True, \"IsFloat\": True, \"IsInt64\": True, \"IsUInt64\": True, \"IsInteger\": True, \"IsGuid\": False, \"IsDateTime\": False, \"IsTimeSpan\": True }]",
      "ExternalEquipmentId": "[ExternalEquipmentId, { \"Value\": \"1234\", \"Raw\": \"1234\", \"IsObject\": False, \"IsArray\": True, \"IsBoolean\": False, \"IsString\": True, \"IsBytes\": True, \"IsDecimal\": True, \"IsDouble\": True, \"IsFloat\": True, \"IsInt64\": True, \"IsUInt64\": True, \"IsInteger\": True, \"IsGuid\": False, \"IsDateTime\": False, \"IsTimeSpan\": True }]"
    }
  }

opc-publisher image: mcr.microsoft.com/iotedge/opc-publisher:2.9.2

[marcschier]

Can you post the message content of the FullNetworkMessage when using extension fields? The extension fields look like regular fields of the data set message object, as if they came from the OPC UA server. There is no particular extension fields "object" then, like we had in Samples encoding.

You uncovered a bug though for samples, the string is incorrect.

[hvetter-de]

Edit: I have collected DataSetMessages, FullNetworkMessages, FullSamples and PubSub messages with JSON and JSONReversible profile. All with ExtensionFields configured. I can upload all the messages if needed.

Cfg:

"Cmd": [
    "--PkiRootPath=/mount/pki",
    "--PublishedNodesFile=/appdata/pn.json",
    "--CreatePublishFileIfNotExistKey=false",
    "--AutoAcceptUntrustedCertificates",
    "--AddAppCertToTrustedStore=true",
    "--LogLevel=Debug",
    "--strict",
    "--fm=true",
    "--mm=FullNetworkMessages",
    "--me=JsonReversible",
    "--bs=2"
  ]

Msg:

{
  "MessageId": "f54c075e-a455-45c3-82c5-f49b79860a23",
  "MessageType": "ua-data",
  "PublisherId": "iothub-xxxxx_device_publisher29poc_module_opc-publisher",
  "DataSetWriterGroup": "Asset1",
  "Messages": [
    {
      "DataSetWriterId": 1,
      "SequenceNumber": 2512,
      "MetaDataVersion": {
        "MajorVersion": 1222365517,
        "MinorVersion": 997070557
      },
      "Timestamp": "2023-10-20T12:45:43.7435731Z",
      "MessageType": "ua-deltaframe",
      "DataSetWriterName": "DataFlow1",
      "Payload": {
        "nsu=http://microsoft.com/Opc/OpcPlc/;s=LongString10kB": {
          "Value": {
            "Type": 12,
            "Body": "NNNNN"
          },
          "SourceTimestamp": "2023-10-20T12:45:43.2682162Z",
          "ServerTimestamp": "2023-10-20T12:45:43.2682305Z"
        },
        "nsu=http://microsoft.com/Opc/OpcPlc/;s=FastUInt1": {
          "Value": {
            "Type": 7,
            "Body": 135234
          },
          "SourceTimestamp": "2023-10-20T12:45:43.6645184Z",
          "ServerTimestamp": "2023-10-20T12:45:43.6645279Z"
        }
      }
    },
    {
      "DataSetWriterId": 1,
      "SequenceNumber": 2513,
      "MetaDataVersion": {
        "MajorVersion": 1222365517,
        "MinorVersion": 997070557
      },
      "Timestamp": "2023-10-20T12:45:48.7526074Z",
      "MessageType": "ua-deltaframe",
      "DataSetWriterName": "DataFlow1",
      "Payload": {
        "nsu=http://microsoft.com/Opc/OpcPlc/;s=LongString10kB": {
          "Value": {
            "Type": 12,
            "Body": "NNNNN"
          },
          "SourceTimestamp": "2023-10-20T12:45:48.2679673Z",
          "ServerTimestamp": "2023-10-20T12:45:48.2679787Z"
        },
        "nsu=http://microsoft.com/Opc/OpcPlc/;s=FastUInt1": {
          "Value": {
            "Type": 7,
            "Body": 135284
          },
          "SourceTimestamp": "2023-10-20T12:45:48.6666901Z",
          "ServerTimestamp": "2023-10-20T12:45:48.6667005Z"
        }
      }
    }
  ]
}

[marcschier]

Thanks for the details. I will take a look at this so it gets addressed before the 2.9.3 release. We have tests for extension fields,, but maybe a case not covered.

[marcschier]

Update: I added another test that uses the same command line arguments, and it verifies that extension fields working but only as part of the key frame messages:

Industrial-IoT/src/Azure.IIoT.OpcUa.Publisher.Module/tests/Sdk/ReferenceServer/BasicPubSubIntegrationTests.cs

Line 338 in 5dfae19

public async Task CanSendKeyFramesWithExtensionFieldsToIoTHubTestJsonReversible()

If there are no key frame messages configured/sent then you will not see extension fields in delta frame messages. This logically makes sense as you won't expect changes in extension fields, yet works against it's usefulness for contextualization.

We can change this by adding a cli switch to pack extension fields always into all messages that are sent. In the meantime, would key frames work for you? Then you can set "DataSetKeyFrameCount": x where x is the number of delta messages until a key frame is sent.

[hvetter-de]

I am ill at the moment, that's why my answer is delayed. We need the Extensionfields in every message for more context. A new config for sending the Extensionfields in every msg will be a perfect solution. I will try it with the DataSetKeyFrameCount cfg you mentioned next week.

Thanks for your support :)

[hvetter-de]

The DataSetKeyFrameCount cfg works, thanks for you help.

[marcschier]

Updated documentation in #2103. Closing for now.

[hvetter-de]

@marcschier
I apologize, but setting DataSetKeyFrameCount=x is not working for us.
Because our processes are not working if all nodes are sent every time, even if they did not change.

We need a way to get the ExtensionFields with every DeltaMessage, which only sends the nodes which have been changed.

Maybe there is also another MessageFormat which fits our needs?
If not, is it possible to implement the config you mentioned earlier to always send the ExtensionFields?