Failed to retrieve conntrack stats on an armv5 based Synology NAS

[bruzzz]

Host operating system: output of uname -a

Linux DS 2.6.32.12 #25556 Thu Jul 1 14:25:48 CST 2021 armv5tel GNU/Linux synology_88f6282_411

node_exporter version: output of node_exporter --version

node_exporter, version 1.5.0 (branch: HEAD, revision: prometheus/node_exporter@1b48970)
build user: root@6e7732a7b81b
build date: 20221129-19:00:13
go version: go1.19.3
platform: linux/arm

As originally reported in this ticket, the conntrack collector on the node_exporter seems to have an issue because of some expected fields beeing missing.

The error meassage with node_exporter v 1.2.2 looks as following (node_exporter log output):

ts=2023-03-13T19:04:46.789Z caller=collector.go:169 level=error msg="collector failed" name=conntrack duration_seconds=0.000592692 err="failed to retrieve conntrack stats: failed to read conntrack stats from \"/proc/net/stat/nf_conntrack\": invalid conntrackstat entry, missing fields"

With version 1.5.0 it seems to be even worse:

ts=2023-03-13T21:26:20.985Z caller=collector.go:169 level=error msg="collector failed" name=conntrack duration_seconds=0.000755877 err="failed to retrieve conntrack stats: failed to read conntrack stats from \"/proc/net/stat/nf_conntrack\": invalid conntrackstat entry, missing fields"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x66de9c]

goroutine 59 [running]:
github.com/prometheus/node_exporter/collector.parseNetlinkStats({0x1209ec0, 0x3, 0x3}, 0x105dda8, {0x8be790, 0x105dd70})
        /home/dietpi/node_exporter/node_exporter-1.5.0/collector/netdev_linux.go:69 +0x1dc
github.com/prometheus/node_exporter/collector.netlinkStats(0x105dda8, {0x8be790, 0x105dd70})
        /home/dietpi/node_exporter/node_exporter-1.5.0/collector/netdev_linux.go:52 +0x114
github.com/prometheus/node_exporter/collector.getNetDevStats(0x105dda8, {0x8be790, 0x105dd70})
        /home/dietpi/node_exporter/node_exporter-1.5.0/collector/netdev_linux.go:35 +0x40
github.com/prometheus/node_exporter/collector.(*netDevCollector).Update(0x105dda0, 0x11b6a80)
        /home/dietpi/node_exporter/node_exporter-1.5.0/collector/netdev_common.go:113 +0x34
github.com/prometheus/node_exporter/collector.execute({0x7be599, 0x6}, {0x8bebf0, 0x105dda0}, 0x11b6a80, {0x8be7f4, 0x11144a0})
        /home/dietpi/node_exporter/node_exporter-1.5.0/collector/collector.go:161 +0x40
github.com/prometheus/node_exporter/collector.NodeCollector.Collect.func1({0x7be599, 0x6}, {0x8bebf0, 0x105dda0})
        /home/dietpi/node_exporter/node_exporter-1.5.0/collector/collector.go:152 +0x54
created by github.com/prometheus/node_exporter/collector.NodeCollector.Collect
        /home/dietpi/node_exporter/node_exporter-1.5.0/collector/collector.go:151 +0x7c

Content of /proc/net/stat/nf_conntrack:

cat /proc/net/stat/nf_conntrack

entries  searched found new invalid ignore delete delete_list insert insert_failed drop early_drop icmp_error  expect_new expect_create expect_delete
0000002b  0003159f 02e6786a 00142562 0001bf93 00e1a051 00142537 000b8fe0 000b900b 00000000 00000000 00000000 0001b46a  00000000 00000000 00000000

[dswarbrick]

You're actually seeing two separate errors in v1.5.0.

The original "invalid conntrackstat entry, missing fields" error is present in both v1.2.2 and v1.5.0, since procfs expects /proc/net/stat/nf_conntrack to contain 17 fields, which apparently it doesn't on 2.6 kernels.

The panic that you see in v1.5.0 is due to node_exporter v1.4.0 and later using kernel netlink to get netdev stats (prometheus/node_exporter#2074). You could try reverting to the previous method of parsing /proc/net/dev by specifying --no-collector.netdev.netlink.

[dswarbrick]

Your /proc/net/stat/nf_conntrack appears to be missing one field at the end (search_restart), based on the test fixtures used to test net_conntrackstat.go:

entries  searched found new invalid ignore delete delete_list insert insert_failed drop early_drop icmp_error  expect_new expect_create expect_delete search_restart
00000021  00000000 00000000 00000000 00000003 0000588a 00000000 00000000 00000000 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

According to iproute manpages:

search_restart Number of conntrack table lookups which had to be restarted due to hashtable resizes.

This field first appeared in kernel 2.6.35.

[bruzzz]

The panic that you see in v1.5.0 is due to node_exporter v1.4.0 and later using kernel netlink to get netdev stats (prometheus/node_exporter#2074). You could try reverting to the previous method of parsing /proc/net/dev by specifying --no-collector.netdev.netlink.

Now I got that. Thanks for explaining! I just tried it and yes, I can confirm, that using the flags --no-collector.netdev.netlink and --no-collector.conntrack the self build v1.5.0 works fine as well (self build, because of the go issue).