[inputs.gnmi] Got empty metric-name for response

[rc-networks]

Relevant telegraf.conf

[[inputs.gnmi]]
  addresses = ["${HOST_ADDRESS5}"]
  username = "${USERNAME}"
  password = "${PASSWORD}"
  encoding = "json_ietf"
  redial = "10s"

  [inputs.gnmi.aliases]
    origin = "rfc7951"
    interfaces-counter = "/Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics"
  

# interface counters
  [[inputs.gnmi.subscription]]
    name = "interfaces-counter"
    origin = "rfc7951"
    path = "/Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics"
    subscription_mode = "sample"
    sample_interval = "5s"

[[outputs.file]]
  data_format = "json"
  files = ["stdout"]

Logs from Telegraf

2024-02-16T07:35:03Z D! [inputs.gnmi] Connection to gNMI device company.device.test:50052 established
2024-02-16T07:35:04Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/in-discards-64
2024-02-16T07:35:04Z W! [inputs.gnmi] Got empty metric-name for response, usually indicating
configuration issues as the response cannot be related to any subscription.
Please open an issue on https://github.com/influxdata/telegraf including your
device model and the following response data:
timestamp:1708069483623763000 update:{path:{origin:"rfc7951" elem:{name:"Cisco-IOS-XE-interfaces-oper:interfaces"} elem:{name:"interface" key:{key:"name" value:"AppGigabitEthernet1/0/1"}} elem:{name:"statistics"}} val:{json_ietf_val:"{\"discontinuity-time\":\"2023-11-15T05:50:40+00:00\",\"in-octets\":\"272\",\"in-unicast-pkts\":\"0\",\"in-broadcast-pkts\":\"0\",\"in-multicast-pkts\":\"0\",\"in-discards\":0,\"in-errors\":63,\"in-unknown-protos\":0,\"out-octets\":494244605,\"out-unicast-pkts\":\"5509764\",\"out-broadcast-pkts\":\"0\",\"out-multicast-pkts\":\"4705581\",\"out-discards\":\"0\",\"out-errors\":\"0\",\"rx-pps\":\"0\",\"rx-kbps\":\"0\",\"tx-pps\":\"0\",\"tx-kbps\":\"0\",\"num-flaps\":\"0\",\"in-crc-errors\":\"0\",\"in-discards-64\":\"0\",\"in-errors-64\":\"63\",\"in-unknown-protos-64\":\"0\",\"out-octets-64\":\"494244605\"}"}} update:{path:{origin:"rfc7951" elem:{name:"Cisco-IOS-XE-interfaces-oper:interfaces"} elem:{name:"interface" key:{key:"name" value:"FortyGigabitEthernet1/1/1"}} elem:{name:"statistics"}} val:{json_ietf_val:"{\"discontinuity-time\":\"2023-11-15T05:50:40+00:00\",\"in-octets\":\"0\",\"in-unicast-pkts\":\"0\",\"in-broadcast-pkts\":\"0\",\"in-multicast-pkts\":\"0\",\"in-discards\":0,\"in-errors\":0,\"in-unknown-protos\":0,\"out-octets\":0,\"out-unicast-pkts\":\"0\",\"out-broadcast-pkts\":\"0\",\"out-multicast-pkts\":\"0\",\"out-discards\":\"0\",\"out-errors\":\"0\",\"rx-pps\":\"0\",\"rx-kbps\":\"0\",\"tx-pps\":\"0\",\"tx-kbps\":\"0\",\"num-flaps\":\"0\",\"in-crc-errors\":\"0\",\"in-discards-64\":\"0\",\"in-errors-64\":\"0\",\"in-unknown-protos-64\":\"0\",\"out-octets-64\":\"0\"}"}}

...output omitted for brevity

System info

Telegraf 1.29.4 (git: HEAD@4441c4ed)

Docker

No response

Steps to reproduce

1. Install

curl -s https://repos.influxdata.com/influxdata-archive_compat.key > influxdata-archive_compat.key
echo '393e8779c89ac8d958f81f942f9ad7fb82a25e133faddaf92e15b16e6ac9ce4c influxdata-archive_compat.key' | sha256sum -c && cat influxdata-archive_compat.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/influxdata-archive_compat.gpg > /dev/null
echo 'deb [signed-by=/etc/apt/trusted.gpg.d/influxdata-archive_compat.gpg] https://repos.influxdata.com/debian stable main' | sudo tee /etc/apt/sources.list.d/influxdata.list
sudo apt-get update && sudo apt-get install telegraf

2. Enable gnxi telemetry service on the cisco ios-xe device
3. Add debug = true on your telegraf.conf
4. Run telegraf -config telegraf.conf
5. Check the logs

Expected behavior

{"fields":{"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/discontinuity_time":"2023-11-15T05:50:40+00:00","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_broadcast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_crc_errors":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_discards":0,"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_discards_64":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_errors":63,"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_errors_64":"63","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_multicast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_octets":"272","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_unicast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_unknown_protos":0,"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_unknown_protos_64":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/num_flaps":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_broadcast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_discards":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_errors":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_multicast_pkts":"4705581","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_octets":494244605,"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_octets_64":"494244605","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_unicast_pkts":"5509764","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/rx_kbps":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/rx_pps":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/tx_kbps":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/tx_pps":"0"},"name":"interfaces-counter","tags":{"host":"dev-server","name":"AppGigabitEthernet1/0/1","source":"company.device.test"},"timestamp":1708069483}
{"fields":{"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/discontinuity_time":"2023-11-15T05:50:40+00:00","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_broadcast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_crc_errors":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_discards":0,"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_discards_64":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_errors":0,"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_errors_64":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_multicast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_octets":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_unicast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_unknown_protos":0,"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_unknown_protos_64":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/num_flaps":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_broadcast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_discards":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_errors":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_multicast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_octets":0,"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_octets_64":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_unicast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/rx_kbps":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/rx_pps":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/tx_kbps":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/tx_pps":"0"},"name":"interfaces-counter","tags":{"host":"dev-server","name":"FortyGigabitEthernet1/1/1","source":"company.device.test"},"timestamp":1708069483}

Actual behavior

{"fields":{"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/discontinuity_time":"2023-11-15T05:50:40+00:00","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_broadcast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_crc_errors":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_discards":0,"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_discards_64":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_errors":63,"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_errors_64":"63","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_multicast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_octets":"272","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_unicast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_unknown_protos":0,"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_unknown_protos_64":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/num_flaps":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_broadcast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_discards":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_errors":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_multicast_pkts":"4705581","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_octets":494244605,"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_octets_64":"494244605","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_unicast_pkts":"5509764","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/rx_kbps":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/rx_pps":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/tx_kbps":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/tx_pps":"0"},"name":"","tags":{"host":"dev-server","name":"AppGigabitEthernet1/0/1","source":"company.device.test"},"timestamp":1708069483}
{"fields":{"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/discontinuity_time":"2023-11-15T05:50:40+00:00","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_broadcast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_crc_errors":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_discards":0,"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_discards_64":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_errors":0,"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_errors_64":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_multicast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_octets":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_unicast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_unknown_protos":0,"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_unknown_protos_64":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/num_flaps":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_broadcast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_discards":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_errors":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_multicast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_octets":0,"Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_octets_64":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_unicast_pkts":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/rx_kbps":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/rx_pps":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/tx_kbps":"0","Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/tx_pps":"0"},"name":"","tags":{"host":"dev-server","name":"FortyGigabitEthernet1/1/1","source":"company.device.test"},"timestamp":1708069483}

...output omitted for brevity

Additional info

Already tested:

  [inputs.gnmi.aliases]
    origin = "rfc7951"
    interfaces-counter = "/Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/(.*)"

and

    interfaces-counter = "/Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/(.*)"

The above does not work.

If you comment out data_format = "json" on the [[outputs.file]], you get:

...output omitted for brevity
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/in-unicast-pkts
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/in-unknown-protos-64
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/rx-kbps
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/in-crc-errors
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/in-errors
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/out-discards
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/tx-pps
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/out-errors
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/in-errors-64
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/out-octets-64
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/in-unknown-protos
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/in-broadcast-pkts
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/out-multicast-pkts
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/out-unicast-pkts
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/in-discards
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/rx-pps
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/num-flaps
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/in-discards-64
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/out-octets
2024-02-16T08:07:32Z D! [inputs.gnmi] No measurement alias for gNMI path: /Cisco-IOS-XE-interfaces-oper:interfaces/interface/statistics/in-octets
2024-02-16T08:07:37Z D! [outputs.file] Could not serialize metric: invalid name
2024-02-16T08:07:37Z D! [outputs.file] Could not serialize metric: invalid name
2024-02-16T08:07:37Z D! [outputs.file] Could not serialize metric: invalid name
2024-02-16T08:07:37Z D! [outputs.file] Could not serialize metric: invalid name
2024-02-16T08:07:37Z D! [outputs.file] Could not serialize metric: invalid name
2024-02-16T08:07:37Z D! [outputs.file] Could not serialize metric: invalid name
2024-02-16T08:07:37Z D! [outputs.file] Could not serialize metric: invalid name
2024-02-16T08:07:37Z D! [outputs.file] Could not serialize metric: invalid name
2024-02-16T08:07:37Z D! [outputs.file] Could not serialize metric: invalid name
2024-02-16T08:07:37Z D! [outputs.file] Could not serialize metric: invalid name
2024-02-16T08:07:37Z D! [outputs.file] Could not serialize metric: invalid name
2024-02-16T08:07:37Z D! [outputs.file] Could not serialize metric: invalid name
...output omitted for brevity

Device Detail:

Switch Ports Model              SW Version        SW Image              Mode   
------ ----- -----              ----------        ----------            ----   
*    1 65    C9300-48UXM        17.09.04a         CAT9K_IOSXE           INSTALL

Cleaner output here:

{'fields': {'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/discontinuity_time': '2023-11-15T05:50:39+00:00',
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_broadcast_pkts': '0',
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_crc_errors': '0',
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_discards': 0,
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_discards_64': '0',
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_errors': 63,
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_errors_64': '63',
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_multicast_pkts': '0',
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_octets': '272',
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_unicast_pkts': '0',
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_unknown_protos': 0,
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/in_unknown_protos_64': '0',
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/num_flaps': '0',
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_broadcast_pkts': '0',
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_discards': '0',
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_errors': '0',
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_multicast_pkts': '4704885',
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_octets': 494171461,
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_octets_64': '494171461',
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/out_unicast_pkts': '5508949',
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/rx_kbps': '0',
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/rx_pps': '0',
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/tx_kbps': '0',
  'Cisco_IOS_XE_interfaces_oper:interfaces/interface/statistics/tx_pps': '0'},
 'name': '',
 'tags': {'host': 'dev-server',
  'name': 'AppGigabitEthernet1/0/1',
  'source': 'company.device.test'},
 'timestamp': 1708068293}

[rc-networks]

This issue is similar to #13052 but with no known working solution. or is there?

[srebhan]

@rc-networks can you please test the binary in PR #14838 and let me know if this fixed the issue!?!

[rc-networks]

Awesome!
Tested linux_amd64.tar.gz and it's working as expected!

When will this be part of the official release?

[srebhan]

I think it can be part of the next bug-fix-release which is due tomorrow. ;-)