nProbe rpi crashing with zmq settings

[tnichols1998]

This looks to be similar to (or a regression of) #174.

I've configured ntopng (pro license) and nprobe on a raspberry pi 4 to receive netflow traffic from the DD-WRT based router (using sflow). The DD-WRT host points sflow traffic to the nprobe on the rpi on port 2205, and nprobe should collect the flows and forward via zmq to ntopng on the same rpi.

However, when I configure the zmq settings for nprobe, it crashes on startup with a SEGV and no error message. I haven't purchased a license for nprobe yet as I want to prove function first.

nprobe is stable for me without the zmq options set.

Here are my configs, and the nprobe output to daemon.log:

Thanks for the help

nprobe.conf ----------------
-i=none
-n=none
-3=2055
-b=1
--zmq="tcp://127.0.0.1:5556"
--zmq-probe-mode
-T="@NTOPNG@"

ntopng.conf ----------------
-G=/var/run/ntopng.pid
-i=tcp://127.0.0.1:5556c
-m=192.168.1.0/24

daemon.log [nprobe] ----------------
Jul 3 14:59:51 ntop systemd[1]: nprobe.service: Service RestartSec=5s expired, scheduling restart.
Jul 3 14:59:51 ntop systemd[1]: nprobe.service: Scheduled restart job, restart counter is at 73.
Jul 3 14:59:51 ntop systemd[1]: Stopped nprobe extensible NetFlow v5/v9/IPFIX probe/collector for IPv4/v6.
Jul 3 14:59:51 ntop systemd[1]: Starting nprobe extensible NetFlow v5/v9/IPFIX probe/collector for IPv4/v6...
Jul 3 14:59:51 ntop systemd[1]: Started nprobe extensible NetFlow v5/v9/IPFIX probe/collector for IPv4/v6.
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:5054] Reading configuration file /run/nprobe.conf
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [plugin.c:177] No plugins found in ./plugins
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [plugin.c:185] Loading 23 plugins [.so] from /usr/local/lib/nprobe/plugins
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:4620] ERROR: Invalid license (/etc/nprobe.license) [Missing license file]
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:4627] ERROR: *****************************************************
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:4628] ERROR: ** **
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:4629] ERROR: ** Switching to DEMO MODE (missing valid license) **
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:4630] ERROR: ** **
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:4632] ERROR: ** Purchase your license at **
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:4633] ERROR: ** https://shop.ntop.org/ **
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:4634] ERROR: ** **
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:4636] ERROR: *****************************************************
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:6677] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ?
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:6680] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ?
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:6771] Flow cache is disabled in flow collection mode
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:6774] Welcome to nProbe v.9.1.200629 ($Revision: 6903 $) for armv7l-unknown-linux-gnueabihf
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:6785] Running on Raspbian GNU/Linux 10 (buster)
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:6796] [LICENSE] nProbe SystemId: 4491C28A5E6BA0A5
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:6867] Sample rate [packet: 1][flow collection/export: 1/1]
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:9734] ERROR: ***************************************************************
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:9735] ERROR: * NOTE: This is a DEMO version limited to 25000 flows export. *
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:9736] ERROR: ***************************************************************
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:9743] Welcome to nProbe v.9.1.200629 for armv7l-unknown-linux-gnueabihf
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:8557] WARNING: Adding %EXPORTER_IPV4_ADDRESS to the template as nProbe is working as collector
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:8675] Using NetFlow Packet Payload Len: 1472
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:8595] @NTOPNG@ expanded to " %IN_SRC_MAC %OUT_DST_MAC %INPUT_SNMP %OUTPUT_SNMP %SRC_VLAN %IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV6_SRC_ADDR %IPV6_DST_ADDR %IP_PROTOCOL_VERSION %PROTOCOL %L7_PROTO %IN_BYTES %IN_PKTS %OUT_BYTES %OUT_PKTS %FIRST_SWITCHED %LAST_SWITCHED %CLIENT_TCP_FLAGS %SERVER_TCP_FLAGS %EXPORTER_IPV4_ADDRESS"
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:8710] Flow export type: bidirectional flows
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [plugin.c:1171] 0 plugin(s) enabled
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:9177] Each flow is 104 bytes long
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:9178] The # flows per packet has been set to 13
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:9181] IP TOS is ignored
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:10026] Flows ASs will not be computed (no GeoDB files loaded)
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:10131] Not capturing packet from interface (collector mode)
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [util.c:5117] Initializing ZMQ as client
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [util.c:5190] Exporting flows towards ZMQ endpoint tcp://127.0.0.1:5556
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [util.c:4114] Enlarged socket buffer [echo 8388608 > /proc/sys/net/core/rmem_max]
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [util.c:4155] nProbe changed user to 'nprobe'
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [collect.c:192] Flow collector listening on port 2055 (IPv4/v6)
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [export.c:540] Using TLV as serialization format
Jul 3 14:59:51 ntop nprobe[24756]: 03/Jul/2020 14:59:51 [nprobe.c:10394] nProbe started successfully
Jul 3 14:59:52 ntop nprobe[24756]: 03/Jul/2020 14:59:52 [nprobe.c:3612] ---------------------------------
Jul 3 14:59:52 ntop nprobe[24756]: 03/Jul/2020 14:59:52 [nprobe.c:3631] L7 Proto Diff Total
Jul 3 14:59:52 ntop nprobe[24756]: 03/Jul/2020 14:59:52 [nprobe.c:3645] #011Unknown/0 12.14 KB 12.14 KB
Jul 3 14:59:52 ntop nprobe[24756]: 03/Jul/2020 14:59:52 [nprobe.c:3664] Flows exports (including drops) [1 flows][avg: 1.0 flows/sec][latest 1 sec avg: 1.0 flows/sec]
Jul 3 14:59:52 ntop nprobe[24756]: 03/Jul/2020 14:59:52 [nprobe.c:3672] Flow drops [export queue full: 0]
Jul 3 14:59:52 ntop nprobe[24756]: 03/Jul/2020 14:59:52 [nprobe.c:3675] Packet drops [too many flow buckets: 0]
Jul 3 14:59:52 ntop nprobe[24756]: 03/Jul/2020 14:59:52 [nprobe.c:3678] Flow Buckets [active: 1][allocated: 1][toBeExported: 0]
Jul 3 14:59:52 ntop nprobe[24756]: 03/Jul/2020 14:59:52 [nprobe.c:3682] Export Queue [current: 0][max: 512000][fill level: 0.0%]
Jul 3 14:59:52 ntop nprobe[24756]: 03/Jul/2020 14:59:52 [nprobe.c:3712] ZMQ Export [1 exporters][1 flows][total avg: 9.97 Kb/sec][236.0 bytes/flow][latest 1 sec avg: 9.97 Kb/sec]
Jul 3 14:59:52 ntop nprobe[24756]: 03/Jul/2020 14:59:52 [nprobe.c:3774] Collector Threads: [1 pkts@0]
Jul 3 14:59:52 ntop nprobe[24756]: 03/Jul/2020 14:59:52 [nprobe.c:3428] Processed packets: 0 (max bucket search: 0)
Jul 3 14:59:52 ntop nprobe[24756]: 03/Jul/2020 14:59:52 [nprobe.c:3411] Fragment queue length: 0
Jul 3 14:59:52 ntop nprobe[24756]: 03/Jul/2020 14:59:52 [nprobe.c:3439] UDP collection stats: [collected pkts: 1][UDP socket drops: 0]
Jul 3 14:59:52 ntop nprobe[24756]: 03/Jul/2020 14:59:52 [nprobe.c:3446] Flow collection stats: [processed: 2][dropped (holes in collected flow sequence): 0]
Jul 3 14:59:52 ntop nprobe[24756]: 03/Jul/2020 14:59:52 [nprobe.c:3452] Flow export stats: [0 bytes/0 pkts][0 flows/0 pkts sent]
Jul 3 14:59:52 ntop nprobe[24756]: 03/Jul/2020 14:59:52 [nprobe.c:3458] Flow export drop stats: [0 bytes/0 pkts][0 flows]
Jul 3 14:59:52 ntop nprobe[24756]: 03/Jul/2020 14:59:52 [nprobe.c:3463] Total flow stats: [0 bytes/0 pkts][0 flows/0 pkts sent]
Jul 3 14:59:54 ntop systemd[1]: nprobe.service: Main process exited, code=killed, status=11/SEGV
Jul 3 14:59:54 ntop systemd[1]: nprobe.service: Failed with result 'signal'.

daemon.log [ntopng] ------------------------------
Jul 3 14:44:04 ntop systemd[1]: Starting ntopng high-speed web-based traffic monitoring and analysis tool...
Jul 3 14:44:04 ntop systemd[1]: Started ntopng high-speed web-based traffic monitoring and analysis tool.
Jul 3 14:44:04 ntop ntopng[21947]: 03/Jul/2020 14:44:04 [Ntop.cpp:2254] Setting local networks to 192.168.1.0/24
Jul 3 14:44:04 ntop ntopng[21947]: 03/Jul/2020 14:44:04 [Redis.cpp:157] Successfully connected to redis 127.0.0.1:6379@0
Jul 3 14:44:04 ntop ntopng[21947]: 03/Jul/2020 14:44:04 [Redis.cpp:157] Successfully connected to redis 127.0.0.1:6379@0
Jul 3 14:44:04 ntop ntopng[21947]: 03/Jul/2020 14:44:04 [NtopPro.cpp:299] [LICENSE] Reading license from /etc/ntopng.license
Jul 3 14:44:04 ntop ntopng[21947]: 03/Jul/2020 14:44:04 [NtopPro.cpp:429] [LICENSE] /etc/ntopng.license: found valid Professional Embedded license
Jul 3 14:44:05 ntop ntopng[21947]: 03/Jul/2020 14:44:05 [Ntop.cpp:2359] Registered interface tcp://127.0.0.1:5556c [id: 8]
Jul 3 14:44:05 ntop ntopng[21947]: 03/Jul/2020 14:44:05 [main.cpp:316] PID stored in file /var/run/ntopng.pid
Jul 3 14:44:05 ntop ntopng[21947]: 03/Jul/2020 14:44:05 [Geolocation.cpp:150] Running without geolocation support.
Jul 3 14:44:05 ntop ntopng[21947]: 03/Jul/2020 14:44:05 [Geolocation.cpp:151] To enable geolocation follow the instructions at
Jul 3 14:44:05 ntop ntopng[21947]: 03/Jul/2020 14:44:05 [Geolocation.cpp:152] https://github.com/ntop/ntopng/blob/dev/doc/README.geolocation.md
Jul 3 14:44:06 ntop ntopng[21947]: 03/Jul/2020 14:44:06 [HTTPserver.cpp:1498] Web server dirs [/usr/share/ntopng/httpdocs][/usr/share/ntopng/scripts]
Jul 3 14:44:06 ntop ntopng[21947]: 03/Jul/2020 14:44:06 [HTTPserver.cpp:1501] HTTP server listening on 3000
Jul 3 14:44:06 ntop ntopng[21947]: 03/Jul/2020 14:44:06 [Utils.cpp:761] User changed to ntopng
Jul 3 14:44:06 ntop ntopng[21947]: 03/Jul/2020 14:44:06 [main.cpp:386] Working directory: /var/lib/ntopng
Jul 3 14:44:06 ntop ntopng[21947]: 03/Jul/2020 14:44:06 [main.cpp:388] Scripts/HTML pages directory: /usr/share/ntopng
Jul 3 14:44:06 ntop ntopng[21947]: 03/Jul/2020 14:44:06 [Ntop.cpp:455] Welcome to ntopng armv7l v.4.1.200629 - (C) 1998-20 ntop.org
Jul 3 14:44:06 ntop ntopng[21947]: 03/Jul/2020 14:44:06 [Ntop.cpp:465] Built on Raspbian GNU/Linux 10 (buster)
Jul 3 14:44:06 ntop ntopng[21947]: 03/Jul/2020 14:44:06 [NtopPro.cpp:699] [LICENSE] System Id:#[removed]
Jul 3 14:44:06 ntop ntopng[21947]: 03/Jul/2020 14:44:06 [NtopPro.cpp:700] [LICENSE] Edition:#011Professional Embedded
Jul 3 14:44:06 ntop ntopng[21947]: 03/Jul/2020 14:44:06 [NtopPro.cpp:701] [LICENSE] License Type:#011Permanent License
[license removed for email]
Jul 3 14:44:06 ntop ntopng[21947]: 03/Jul/2020 14:44:06 [NtopPro.cpp:725] [LICENSE] Maintenance:#011Until Thu Jul 1 12:34:46 2021 [362 days left]
Jul 3 14:44:06 ntop ntopng[21947]: 03/Jul/2020 14:44:06 [PeriodicActivities.cpp:105] Started periodic activities loop...
Jul 3 14:44:12 ntop ntopng[21947]: 03/Jul/2020 14:44:12 [PeriodicActivities.cpp:165] Each periodic activity script will use 2 threads
Jul 3 14:44:12 ntop ntopng[21947]: 03/Jul/2020 14:44:12 [NetworkInterface.cpp:2358] Started packet polling on interface tcp://127.0.0.1:5556c [id: 8]...
Jul 3 14:44:12 ntop ntopng[21947]: 03/Jul/2020 14:44:12 [ZMQCollectorInterface.cpp:255] Collecting flows on tcp://127.0.0.1:5556c

pi@ntop:~ $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 57G 6.6G 48G 13% /
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 2.0G 57M 1.9G 3% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/mmcblk0p6 253M 52M 202M 21% /boot
tmpfs 391M 0 391M 0% /run/user/1000

pi@ntop:~ $ free -mh
total used free shared buff/cache available
Mem: 3.8Gi 301Mi 2.8Gi 76Mi 780Mi 3.3Gi
Swap: 99Mi 0B 99Mi

pi@ntop:~ $ uname -a
Linux ntop 4.19.118-v7l+ #1311 SMP Mon Apr 27 14:26:42 BST 2020 armv7l GNU/Linux

pi@ntop:~ $ cat /etc/release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

[tnichols1998]

I've created a core dump and made it available here: https://www.dropbox.com/s/2pvwvuaw7epsnuy/core.gz?dl=0

[tnichols1998]

A bit more detail on troubleshooting. If I run nprobe in a docker container on my mac (x86) and forward the packets via zmq to ntopng on the raspberry pi, the problem moves to ntopng. The nprobe binary on the Mac runs fine and shows packets/flows processed. But ntopng segfaults when those flows appear on the in the zmq queue.

[tnichols1998]

gdb backtrace output:

`pi@ntop:~ $ gdb /usr/local/bin/nprobe -c core
GNU gdb (Raspbian 8.2.1-2) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/nprobe...(no debugging symbols found)...done.
[New LWP 2949]
[New LWP 2947]
[New LWP 2948]
[New LWP 2951]
[New LWP 2946]
[New LWP 2952]
[New LWP 2950]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `/usr/local/bin/nprobe --debug -i none -n none -3 2055 -b 1 --zmq tcp://127.0.0.'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0xb6def994 in ndpi_guess_protocol_id () from /usr/local/lib/libnprobe-9.1.200629.so
[Current thread is 1 (Thread 0xb2cfedf0 (LWP 2949))]
(gdb) bt
#0 0xb6def994 in ndpi_guess_protocol_id () from /usr/local/lib/libnprobe-9.1.200629.so
#1 0xb6dfa2cc in ndpi_guess_undetected_protocol () from /usr/local/lib/libnprobe-9.1.200629.so
#2 0xb6d78bf8 in setPayload () from /usr/local/lib/libnprobe-9.1.200629.so
#3 0xb6d828f8 in processFlowPacket () from /usr/local/lib/libnprobe-9.1.200629.so
#4 0xb6d6c8a4 in ?? () from /usr/local/lib/libnprobe-9.1.200629.so
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) info threads
Id Target Id Frame

• 1 Thread 0xb2cfedf0 (LWP 2949) 0xb6def994 in ndpi_guess_protocol_id () from /usr/local/lib/libnprobe-9.1.200629.so
  2 Thread 0xb4e50df0 (LWP 2947) 0xb68789d0 in epoll_wait (epfd=, events=0xb4e4f718, maxevents=256,
  timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  3 Thread 0xb3e4fdf0 (LWP 2948) 0xb68789d0 in epoll_wait (epfd=, events=0xb3e4e718, maxevents=256,
  timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  4 Thread 0xb0cfcdf0 (LWP 2951) __GI___nanosleep (remaining=0xb0cfc69c, requested_time=0xb0cfc69c)
  at ../sysdeps/unix/sysv/linux/nanosleep.c:28
  5 Thread 0xb6fdf240 (LWP 2946) __GI___nanosleep (remaining=0xbed7a17c, requested_time=0xbed7a17c)
  at ../sysdeps/unix/sysv/linux/nanosleep.c:28
  6 Thread 0xafcfbdf0 (LWP 2952) __GI___nanosleep (remaining=0xafcfb184, requested_time=0xafcfb184)
  at ../sysdeps/unix/sysv/linux/nanosleep.c:28
  7 Thread 0xb1cfddf0 (LWP 2950) futex_wait_cancelable (private=0, expected=0, futex_word=0xb59dd9f8)
  at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  (gdb)
  `

[simonemainardi]

An issue potentially affecting this has been resolved. @lucaderi can you please trigger a rebuild for the raspbian buster package?

[tnichols1998]

Updated package seems to have resolved the issue. I will watch throughout the day and send an update.

Thanks!

[simonemainardi]

thanks for reporting.