SNMP input unknown ifAlias_1

[Gnoale]

Actual behavior:

I try to migrate from an environment where a container run Telegraf 1.5 to an environment where a VM run Telegraf 1.10.

The Telegraf configuration is identical as the InfluxDB environment.

I use Influxdb Relay to provide HA and request buffering.

When the data is written from the new environment, only the ifAlias_1 field is populated, when the data is written from the original environment, both ifAlias TAG and ifAlias_1 field are populated.

I don't know when or how was created this ifAlias_1 and why Telegraf 1.10 doesn't write in the appropriate TAG ifAlias.

• Example with overlapping points written from the 2 agents (value are not exactly similar but that illustrate).

Request filtered on host=70b61eba915f : that's the telegraf agent 1.5.

select * from interface_snmp where agent_host='cpe1-mgmt1.mediam.fr-avs.intercloud.fr' and host='70b61eba915f' AND time >='2019-04-16T16:00:00Z' AND time <='2019-04-16T17:00:00Z';

name: interface_snmp
time                agent_host                             environment host         hostname                ifAlias                                ifAlias_1                              ifConnectorPresent ifCounterDiscontinuityTime ifHCInBroadcastPkts ifHCInMulticastPkts ifHCInOctets ifHCInUcastPkts ifHCOutBroadcastPkts ifHCOutMulticastPkts ifHCOutOctets ifHCOutUcastPkts ifHighSpeed ifInBroadcastPkts ifInMulticastPkts ifLinkUpDownTrapEnable ifName         ifOutBroadcastPkts ifOutMulticastPkts ifPromiscuousMode
----                ----------                             ----------- ----         --------                -------                                
1555430414000000000 cpe1-mgmt1.mediam.fr-avs.intercloud.fr             70b61eba915f cpe1-mgmt.mediam.fr-avs                                                                               2                  0                          0                   0                   162344298    419496          0                    0                    0             0                1000        0                 0                 1                      ge-0/0/8.32767 0                  0                  2
1555430414000000000 cpe1-mgmt1.mediam.fr-avs.intercloud.fr             70b61eba915f cpe1-mgmt.mediam.fr-avs VLAN-ADMIN:INTERCLOUD                  VLAN-ADMIN:INTERCLOUD                  2                  0                          0                   0                   8498878133   114587671       0                    0                    30257389700   139020460        1000        0                 0                 1                      ge-0/0/8.1     0                  0                  2
...

Request filtered on host=svcmon0-ld5 : that's the telegraf agent 1.10.

select * from interface_snmp where agent_host='cpe1-mgmt1.mediam.fr-avs.intercloud.fr' and host='svcmon0-ld5' AND time >='2019-04-16T16:00:00Z' AND time <='2019-04-16T17:00:00Z';

time                agent_host                             environment host        hostname                ifAlias ifAlias_1                              ifConnectorPresent ifCounterDiscontinuityTime ifHCInBroadcastPkts ifHCInMulticastPkts ifHCInOctets ifHCInUcastPkts ifHCOutBroadcastPkts ifHCOutMulticastPkts ifHCOutOctets ifHCOutUcastPkts ifHighSpeed ifInBroadcastPkts ifInMulticastPkts ifLinkUpDownTrapEnable ifName      ifOutBroadcastPkts ifOutMulticastPkts ifPromiscuousMode
----                ----------                             ----------- ----        --------                ------- ---------                              ------------------ -------------------------- ------------------- ------------------- ------------ --------------- -------------------- -------------------- ------------- ---------------- ----------- ----------------- ----------------- ---------------------- ------      ------------------ ------------------ -----------------
1555433772000000000 cpe1-mgmt1.mediam.fr-avs.intercloud.fr             svcmon0-ld5 cpe1-mgmt.mediam.fr-avs                                                1                  0                          0                   0                   0            0               0                    0                    0             0                1000        0                 0                 1                      ge-0/0/13   0                  0                  2
1555433772000000000 cpe1-mgmt1.mediam.fr-avs.intercloud.fr             svcmon0-ld5 cpe1-mgmt.mediam.fr-avs         VLAN-ADMIN:INTERCLOUD                  2                  0                          0                   0                   8503445083   114664704       0                    0                    30283124920   139144433        1000        0                 0                 1                      ge-0/0/8.1  0                  0                  2

Expected behavior:

The snmp value should be written in the ifAlias tag and not in an ifAlias_1 field.

System info:

• Telegraf host :

Linux svcmon0-ld5 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Telegraf 1.10.0 (git: HEAD fe33ee89)

• Old Telegraf host :

Docker image telegraf:1.5-alpine

Telegraf v1.5.3 (git: release-1.5 1e519698)

• InfluxDB host :

Linux pa2-server2.intercloud.fr 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:53:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

InfluxDB v1.7.4 (git: 1.7 ef77e72f435b71b1ad6da7d6a6a4c4a262439379)

Steps to reproduce:

I didn't managed to reproduce the issue on a similar environment : same Telegraf instance and Relay setup, different InfluxDB server (running the same version and databases).

I don't know from where come this field, I checked the configuration and nothings refers it.

Relevant telegraf.conf:

[[inputs.snmp]]
  agents = [
"cpe1-mgmt1.mediam.fr-crb.intercloud.fr",
"cpe1-mgmt1.mediam.fr-avs.intercloud.fr"
 ]
  version = 2
  interval = "120s"
  community = "public"
  name_suffix = "_snmp"
  name = "system"

  [[inputs.snmp.field]]
    name = "hostname"
    oid = "RFC1213-MIB::sysName.0"
    is_tag = true

  [[inputs.snmp.field]]
    name = "memory"
    oid = "JUNIPER-MIB::jnxBoxKernelMemoryUsedPercent.0"

  [[inputs.snmp.table]]
    name = "interface"
    inherit_tags = [ "hostname" ]
    oid = "IF-MIB::ifXTable"

    # Interface tag - used to identify interface in metrics database
    [[inputs.snmp.table.field]]
      name = "ifAlias"
      oid = "IF-MIB::ifAlias"
      is_tag = true

  [[inputs.snmp.table]]
    name = "errors"
    inherit_tags = [ "hostname" ]
    oid = "IF-MIB::ifTable"

    # Interface tag - used to identify interface in metrics database
    [[inputs.snmp.table.field]]
      name = "ifAlias"
      oid = "IF-MIB::ifAlias"
      is_tag = true

  [[inputs.snmp.table]]
    name = "cpu"
    inherit_tags = [ "hostname" ]
    oid = "JUNIPER-MIB::jnxOperatingTable"

    [[inputs.snmp.table.field]]
      name = "jnxOperatingDescr"
      oid = "JUNIPER-MIB::jnxOperatingDescr"
      is_tag = true

Additional info:

[danielnelson]

Can you run these commands and add the output:

snmptable -v2c -c public cpe1-mgmt1.mediam.fr-crb.intercloud.fr:161 IF-MIB::ifXTable
telegraf --input-filter snmp --test

[Gnoale]

SNMP table: IF-MIB::ifXTable

         ifName ifInMulticastPkts ifInBroadcastPkts ifOutMulticastPkts ifOutBroadcastPkts   ifHCInOctets ifHCInUcastPkts ifHCInMulticastPkts ifHCInBroadcastPkts  ifHCOutOctets ifHCOutUcastPkts ifHCOutMulticastPkts ifHCOutBroadcastPkts ifLinkUpDownTrapEnable ifHighSpeed ifPromiscuousMode ifConnectorPresent                            ifAlias ifCounterDiscontinuityTime
           fxp0                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled           0             false               true                                                  0:0:00:00.00
           fxp2                 0                 0                  0                  0              0               0                   0                   0      363906250          7278125                    0                    0               disabled           0             false               true                                                  0:0:00:00.00
            lsi                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0               disabled           0             false              false                                                  0:0:00:00.00
            lo0                 0                 0                  0                  0     3175195214        68446652                   0                   0     3175195214         68446652                    0                    0                enabled           0             false              false                                                  0:0:00:00.00
            tap                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled           0             false              false                                                  0:0:00:00.00
            gre                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled           0             false              false                                                  0:0:00:00.00
           ipip                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled           0             false              false                                                  0:0:00:00.00
           pime                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0               disabled           0             false              false                                                  0:0:00:00.00
           pimd                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0               disabled           0             false              false                                                  0:0:00:00.00
           mtun                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled           0             false              false                                                  0:0:00:00.00
         fxp2.0                 0                 0                  0                  0              0               0                   0                   0      262012536          7278126                    0                    0                enabled           0             false              false                                                  0:0:00:00.00
      lo0.16384                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled           0             false              false                                                  0:0:00:00.00
      lo0.16385                 0                 0                  0                  0     3175188506        68446619                   0                   0     3175188506         68446619                    0                    0                enabled           0             false              false                                                  0:0:00:00.00
      lo0.32768                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0               disabled           0             false              false                                                  0:0:00:00.00
            pp0                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled           0             false              false                                                  0:0:00:00.00
            irb                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled           0             false              false                                                  0:0:00:00.00
           vtep                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled           0             false              false                                                  0:0:00:00.00
            st0                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0               disabled           0             false              false                                                  0:0:00:00.00
           ppd0                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled         800             false              false                                                  0:0:00:00.00
           ppe0                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled         800             false              false                                                  0:0:00:00.00
           vlan                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0               disabled       10000             false              false                                                  0:0:00:00.00
       ge-0/0/0                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled        1000             false               true                                                  0:0:00:00.00
       ge-0/0/1                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled        1000             false               true                                                  0:0:00:00.00
           jsrv                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0               disabled           0             false              false                                                  0:0:00:00.00
       ge-0/0/2                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled        1000             false               true                                                  0:0:00:00.00
         jsrv.1                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0               disabled        1000             false              false                                                  0:0:00:00.00
       ge-0/0/3                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled        1000             false               true                                                  0:0:00:00.00
       ge-0/0/4                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled        1000             false               true                                                  0:0:00:00.00
       ge-0/0/5                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled        1000             false               true                                                  0:0:00:00.00
       ge-0/0/6                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled        1000             false               true                                                  0:0:00:00.00
       ge-0/0/7                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled        1000             false               true                                                  0:0:00:00.00
     ge-0/0/8.2                 0                 0                  0                  0  1035690773819      1251354914                   0                   0   863935672522       1235039284                    0                    0                enabled        1000             false              false           IC:MEDIAM-006:VR:AWS-DEV               0:0:00:00.00
       ge-0/0/8                 0            111255                  0              49754 11813183488529     11445902851                   0              111256  1806194696438       7533181478                    0                49754                enabled        1000             false               true               IC::MEDIAM-006:LIAZO               0:0:00:00.00
       ge-0/0/9            146851                 3                  0              12123  1780646353147      7430502297              146851                   3 11808654363037      11369055865                    0                12123                enabled        1000             false               true                         CST:MEDIAM               0:0:00:00.00
     ge-0/0/8.3                 0                 0                  0                  0     2585916841         3155197                   0                   0      606881451          2208585                    0                    0                enabled        1000             false              false       IC:MEDIAM-006:VR:AWS-RECETTE               0:0:00:00.00
      ge-0/0/10                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled        1000             false               true                      CST:MEDIAM-PA               0:0:00:00.00
      ge-0/0/11                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled        1000             false               true                                                  0:0:00:00.00
     ge-0/0/8.4                 0                 0                  0                  0   155457129082       179462101                   0                   0   102560303878        171755268                    0                    0                enabled        1000             false              false          IC:MEDIAM-006:VR:AWS-DDSM               0:0:00:00.00
      ge-0/0/12                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled        1000             false               true                                                  0:0:00:00.00
      ge-0/0/13                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled        1000             false               true                                                  0:0:00:00.00
      ge-0/0/14                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled        1000             false               true                                                  0:0:00:00.00
      ge-0/0/15                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled        1000             false               true                                                  0:0:00:00.00
       sp-0/0/0                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled         800             false              false                                                  0:0:00:00.00
     ge-0/0/8.5                 0                 0                  0                  0    69371305548       138020849                   0                   0   198789327723        200515701                    0                    0                enabled        1000             false              false       IC:MEDIAM-006:VR:AWS-PREPROD               0:0:00:00.00
       gr-0/0/0                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0               disabled         800             false              false                                                  0:0:00:00.00
       ip-0/0/0                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0               disabled         800             false              false                                                  0:0:00:00.00
      lsq-0/0/0                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled         622             false              false                                                  0:0:00:00.00
       mt-0/0/0                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled         800             false              false                                                  0:0:00:00.00
       lt-0/0/0                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled         800             false              false                                                  0:0:00:00.00
     ge-0/0/8.6                 0                 0                  0                  0 10142853260334      9433579546                   0                   0   589741053013       5615908643                    0                    0                enabled        1000             false              false          IC:MEDIAM-006:VR:AWS-PROD               0:0:00:00.00
     ge-0/0/8.7                 0                 0                  0                  0   401041053604       349720989                   0                   0    19698929322        193069803                    0                    0                enabled        1000             false              false          IC:MEDIAM-006:VR:AWS-MGMT               0:0:00:00.00
     ge-0/0/9.0                 0                 0                  0                  0  1780303405963      7424239622                   0                   0 11807333271782      11358580712                    0                    0                enabled        1000             false              false                  CST:MEDIAM-AWS-UK               0:0:00:00.00
     ge-0/0/8.8                 0                 0                  0                  0       31741689          517766                   0                   0       41859150           524266                    0                    0                enabled        1000             false              false          IC:MEDIAM-006:VR:AWS-INNO               0:0:00:00.00
    ge-0/0/10.0                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled        1000             false              false                                                  0:0:00:00.00
    ge-0/0/8.12                 0                 0                  0                  0       40003074          534169                   0                   0       43343870           541517                    0                    0                enabled        1000             false              false     IC:MEDIAM-PA-006:VR:AWS-PA-DEV               0:0:00:00.00
    ge-0/0/8.13                 0                 0                  0                  0       31451231          517750                   0                   0       42434874           530152                    0                    0                enabled        1000             false              false IC:MEDIAM-PA-006:VR:AWS-PA-RECETTE               0:0:00:00.00
    ge-0/0/8.14                 0                 0                  0                  0       31734560          517719                   0                   0       42474522           533422                    0                    0                enabled        1000             false              false    IC:MEDIAM-PA-006:VR:AWS-PA-DDSM               0:0:00:00.00
    ge-0/0/8.15                 0                 0                  0                  0      833496856         7969401                   0                   0      168299115          2979838                    0                    0                enabled        1000             false              false IC:MEDIAM-PA-006:VR:AWS-PA-PREPROD               0:0:00:00.00
    ge-0/0/8.16                 0                 0                  0                  0       37180384          530109                   0                   0       42563514           533509                    0                    0                enabled        1000             false              false    IC:MEDIAM-PA-006:VR:AWS-PA-PROD               0:0:00:00.00
     sp-0/0/0.0                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled         800             false              false                                                  0:0:00:00.00
 sp-0/0/0.16383                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled         800             false              false                                                  0:0:00:00.00
     ge-0/0/8.1                 0                 0                  0                  0     4827933309        75198649                   0                   0    21014451813        101853183                    0                    0                enabled        1000             false              false              VLAN-ADMIN:INTERCLOUD               0:0:00:00.00
 ge-0/0/8.32767                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled        1000             false              false                                                  0:0:00:00.00
    ge-0/0/8.17                 0                 0                  0                  0       31741423          517732                   0                   0       41478595           521364                    0                    0                enabled        1000             false              false    IC:MEDIAM-PA-006:VR:AWS-PA-MGMT               0:0:00:00.00
    ge-0/0/8.18                 0                 0                  0                  0       31741843          517732                   0                   0       44064600           543009                    0                    0                enabled        1000             false              false    IC:MEDIAM-PA-006:VR:AWS-PA-INNO               0:0:00:00.00
     ge-0/0/9.1                 0                 0                  0                  0      342947120         5535333                   0                   0     1081010655         10487506                    0                    0                enabled        1000             false              false                  CST:MEDIAM-AWS-FR               0:0:00:00.00
 ge-0/0/9.32767                 0                 0                  0                  0              0               0                   0                   0              0                0                    0                    0                enabled        1000             false              false                                                  0:0:00:00.00
    ge-0/0/8.19                 0                 0                  0                  0      287027310         3899724                   0                   0     5025636172          6174079                    0                    0                enabled        1000             false              false           IC:MEDIAM-006:VR:AWS-PRM               0:0:00:00.00

[Gnoale]

2019-04-18T08:02:29Z I! Starting Telegraf 1.10.0
2019-04-18T08:02:29Z I! Using config file: /etc/telegraf/telegraf.conf

[danielnelson]

I think I see whats going on here, In the past using is_tag would result in the item being sent as both a field and as a tag, but it was changed in #4203 to only send the tag. The sending of a tag and file with the same name is what causes InfluxDB to show the two columns, one corresponds to the tag and the other the field. Since they have the same name the query adds _1 to the end to differentiate them.

This is the same issue described in #4326 (comment). I briefly describe a workaround there: you will need to use the ::tag suffix to select specifically the tag column.

select ifAlias::tag,ifHighSpeed from interface_snmp

If you are using * you could also do this and only look at the ifAlias column ignoring any _1, _2 columns:

select ifAlias::tag,* from interface_snmp

Once your current shard expires any queries against the new data will work with the original query again, but you will probably want to keep the workaround in place until all old data has been removed by a retention policy.

Closing the issue, but let me know if you have any follow up questions.

[Gnoale]

Hello thanks for looking into it, I understand your workaround but the thing is my new Telegraf agent doesn't write any data in the ifAlias TAG, I cannot request on a field.

[danielnelson]

The ifAlias tag should still be created, but the ifAlias field is now gone. Here is a influx session that shows how this behaves.

Setup in a new database so as not to mess up real data.

> create database xyzzy
> use xyzzy

Insert a fake point in the way Telegraf 1.5 does, both field and tag are added with same name:

> insert interface,ifAlias=eth0 ifAlias="eth0",value=42

Insert a fake point in the way Telegraf 1.10 does, only the tag:

> insert interface,ifAlias=eth0 value=42

Here we can see the field and tag have the same name:

> show field keys from interface
name: interface
fieldKey fieldType
-------- ---------
ifAlias  string
value    float
> show tag keys from interface
name: interface
tagKey
------
ifAlias

Now if we query all, they appear as ifAlias and ifAlias_1, even though they were actually both stored as ifAlias:

> select * from interface
name: interface
time                ifAlias ifAlias_1 value
----                ------- --------- -----
1555722688287104345 eth0    eth0      42
1555722695827905721         eth0      42

It may be more clear with this query using aliases, you can see it is the tag that is missing:

> select ifAlias::tag as t,ifAlias::field as f,value from interface
name: interface
time                t    f    value
----                -    -    -----
1555722688287104345 eth0 eth0 42
1555722695827905721 eth0      42

It might be easiest if we work through an actual query, if you could show me one that is giving you trouble I think I will be able to help you get it working.

[Gnoale]

Thanks for the precision !
I didn't know it was possible to specify the type, in Grafana I changed my request and it works perfectly "ifAlias"::tag =~