Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ntopng segfaults ntopng-3.9.190823-7061.x86_64 #2781

Closed
bmansvk opened this issue Aug 23, 2019 · 13 comments
Closed

ntopng segfaults ntopng-3.9.190823-7061.x86_64 #2781

bmansvk opened this issue Aug 23, 2019 · 13 comments
Assignees
@emanuele-f
Labels
Bug In Progress

Comments

@bmansvk
Copy link

bmansvk commented Aug 23, 2019

Latest update of ntopng starts to segfault when using GUI (opening monitored interface in the GUI).

ntopng[14713]: segfault at 7f67dc950000 ip 00007f68607622ca sp 00007f6821ffa068 error 6 in libc-2.17.so[7f68606c7000+1c2000]

No problem with version ntopng-3.9.190819-7032.x86_64 and below.

OS Centos 7 x86_64
@emanuele-f
Copy link
Contributor

Can you get a stack trace as described in https://github.com/ntop/ntopng/blob/dev/doc/README.crash ?

@bmansvk
Copy link
Author

bmansvk commented Aug 23, 2019 •
edited
Loading

gdb --args ntopng /etc/ntopng/ntopng.conf

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /usr/bin/ntopng...done.
(gdb) run
Starting program: /usr/local/bin/ntopng /etc/ntopng/ntopng.conf
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
23/Aug/2019 10:04:41 [Ntop.cpp:1906] Setting local networks to 192.168.0.0/16,10.0.0.0/8
23/Aug/2019 10:04:41 [Redis.cpp:152] Successfully connected to redis 127.0.0.1:6379@0
23/Aug/2019 10:04:41 [Redis.cpp:152] Successfully connected to redis 127.0.0.1:6379@0
23/Aug/2019 10:04:41 [NtopPro.cpp:309] [LICENSE] Reading license from /etc/ntopng.license
23/Aug/2019 10:04:41 [NtopPro.cpp:386] [LICENSE] /etc/ntopng.license: found valid Professional license
Detaching after fork from child process 642.
23/Aug/2019 10:04:42 [PF_RINGInterface.cpp:53] Reading packets from PF_RING v.7.5.0 interface eth1...
23/Aug/2019 10:04:42 [Ntop.cpp:2004] Registered interface eth1 [id: 0]
23/Aug/2019 10:04:42 [main.cpp:321] PID stored in file /var/run/ntopng.pid
23/Aug/2019 10:04:42 [Utils.cpp:739] User changed to ntopng
[New Thread 0x7fffe67ba700 (LWP 644)]
[New Thread 0x7fffe5fb9700 (LWP 645)]
[New Thread 0x7fffe57b8700 (LWP 646)]
[New Thread 0x7fffe4fb7700 (LWP 647)]
[New Thread 0x7fffd7fff700 (LWP 648)]
[New Thread 0x7fffd77fe700 (LWP 649)]
23/Aug/2019 10:04:42 [HTTPserver.cpp:1316] Web server dirs [/usr/share/ntopng/httpdocs][/usr/share/ntopng/scripts]
23/Aug/2019 10:04:42 [HTTPserver.cpp:1319] HTTP server listening on 3000
23/Aug/2019 10:04:42 [main.cpp:396] Working directory: /var/lib/ntopng
23/Aug/2019 10:04:42 [main.cpp:398] Scripts/HTML pages directory: /usr/share/ntopng
23/Aug/2019 10:04:42 [Ntop.cpp:370] Welcome to ntopng x86_64 v.3.9.190823 - (C) 1998-19 ntop.org
23/Aug/2019 10:04:42 [NtopPro.cpp:657] [LICENSE] System Id: 12056C62B205A206
23/Aug/2019 10:04:42 [NtopPro.cpp:658] [LICENSE] Edition: Professional
23/Aug/2019 10:04:42 [NtopPro.cpp:659] [LICENSE] License Type: Permanent License
23/Aug/2019 10:04:42 [NtopPro.cpp:683] [LICENSE] Maintenance: Until Tue Jul 28 11:15:36 2020 [340 days left]
23/Aug/2019 10:04:42 [Ntop.cpp:729] Adding fe80::5054:ff:fefb:5a1c/128 as IPv6 interface address for eth1
23/Aug/2019 10:04:42 [Ntop.cpp:738] Adding fe80::5054:ff:fefb:5a1c/64 as IPv6 local network for eth1
[New Thread 0x7fffd6ffd700 (LWP 650)]
[New Thread 0x7fffd67fc700 (LWP 651)]
23/Aug/2019 10:04:42 [PeriodicActivities.cpp:72] Started periodic activities loop...
Detaching after fork from child process 652.
Detaching after fork from child process 654.
Detaching after fork from child process 660.
Detaching after fork from child process 662.
23/Aug/2019 10:04:43 [PeriodicActivities.cpp:114] Each periodic activity script will use 2 threads
[New Thread 0x7fffd51a7700 (LWP 668)]
[New Thread 0x7fffd49a6700 (LWP 669)]
[New Thread 0x7fffbffff700 (LWP 670)]
[New Thread 0x7fffbf7fe700 (LWP 671)]
[New Thread 0x7fffbeffd700 (LWP 672)]
[New Thread 0x7fffbe7fc700 (LWP 673)]
[New Thread 0x7fffbdffb700 (LWP 674)]
[New Thread 0x7fffbd7fa700 (LWP 675)]
[New Thread 0x7fffbcff9700 (LWP 676)]
[New Thread 0x7fffb7fff700 (LWP 677)]
[New Thread 0x7fffb77fe700 (LWP 678)]
[New Thread 0x7fffb6ffd700 (LWP 679)]
[New Thread 0x7fffb67fc700 (LWP 680)]
[New Thread 0x7fffb5ffb700 (LWP 681)]
[New Thread 0x7fffb57fa700 (LWP 682)]
[New Thread 0x7fffb4ff9700 (LWP 683)]
23/Aug/2019 10:04:43 [NetworkInterface.cpp:2778] Started packet polling on interface eth1 [id: 0]...
23/Aug/2019 10:04:44 [NetworkInterface.cpp:2114] Invalid packet received [len: 2762][max len: 1518].
23/Aug/2019 10:04:44 [NetworkInterface.cpp:2116] WARNING: If you have TSO/GRO enabled, please disable it
23/Aug/2019 10:04:44 [NetworkInterface.cpp:2118] WARNING: Use sudo ethtool -K eth1 gro off gso off tso off

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffb4ff9700 (LWP 683)]
0x00007ffff43742bc in __strncpy_sse2_unaligned () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 cairo-1.15.12-3.el7.x86_64 cyrus-sasl-lib-2.1.26-23.el7.x86_64 elfutils-libelf-0.172-2.el7.x86_64 elfutils-libs-0.172-2.el7.x86_64 expat-2.1.0-10.el7_3.x86_64 fontconfig-2.13.0-4.3.el7.x86_64 freetype-2.8-12.el7_6.1.x86_64 fribidi-1.0.2-1.el7.x86_64 glib2-2.56.1-4.el7_6.x86_64 glibc-2.17-260.el7_6.6.x86_64 gmp-6.0.0-15.el7.x86_64 gnutls-3.3.29-9.el7_6.x86_64 graphite2-1.3.10-1.el7_3.x86_64 harfbuzz-1.7.5-2.el7.x86_64 hiredis-0.12.1-1.el7.x86_64 json-c-0.11-4.el7_0.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-37.el7_6.x86_64 libX11-1.6.5-2.el7.x86_64 libXau-1.0.8-2.1.el7.x86_64 libXext-1.3.3-3.el7.x86_64 libXrender-0.9.10-1.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libcap-2.22-9.el7.x86_64 libcom_err-1.42.9-13.el7.x86_64 libcurl-7.29.0-51.el7_6.3.x86_64 libffi-3.0.13-18.el7.x86_64 libgcc-4.8.5-36.el7_6.2.x86_64 libglvnd-1.0.1-0.8.git5baa1e5.el7.x86_64 libglvnd-egl-1.0.1-0.8.git5baa1e5.el7.x86_64 libglvnd-glx-1.0.1-0.8.git5baa1e5.el7.x86_64 libidn-1.28-4.el7.x86_64 libmaxminddb-1.2.0-1.el7.x86_64 libpng-1.5.13-7.el7_2.x86_64 libselinux-2.5-14.1.el7.x86_64 libsodium-1.0.18-1.el7.x86_64 libssh2-1.4.3-12.el7_6.3.x86_64 libstdc++-4.8.5-36.el7_6.2.x86_64 libtasn1-4.10-1.el7.x86_64 libthai-0.1.14-9.el7.x86_64 libuuid-2.23.2-59.el7_6.1.x86_64 libxcb-1.13-1.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 libzstd-1.4.2-1.el7.x86_64 mariadb-libs-5.5.60-1.el7_5.x86_64 nettle-2.7.1-8.el7.x86_64 nspr-4.19.0-1.el7_5.x86_64 nss-3.36.0-7.1.el7_6.x86_64 nss-softokn-freebl-3.36.0-5.el7_5.x86_64 nss-util-3.36.0-1.1.el7_6.x86_64 openldap-2.4.44-21.el7_6.x86_64 openpgm-5.2.122-2.el7.x86_64 openssl-libs-1.0.2k-16.el7_6.1.x86_64 p11-kit-0.23.5-3.el7.x86_64 pango-1.42.4-2.el7_6.x86_64 pcre-8.32-17.el7.x86_64 pixman-0.34.0-1.el7.x86_64 radcli-1.2.11-1.el7.x86_64 rrdtool-1.4.8-9.el7.x86_64 sqlite-3.7.17-8.el7.x86_64 systemd-libs-219-62.el7_6.9.x86_64 xz-libs-5.2.2-1.el7.x86_64 zeromq-4.1.4-6.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) bt
#0 0x00007ffff43742bc in __strncpy_sse2_unaligned () from /lib64/libc.so.6
#1 0x000000000054a787 in concat_hash_string (
buf=buf@entry=0x7fffa0c31e70 "[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman"..., client_hash=client_hash@entry=0 '\000', packet=0x7fffa647ab30, packet=0x7fffa647ab30)
at protocols/ssh.c:138
#2 0x000000000054ab02 in ndpi_search_ssh_tcp (ndpi_struct=0xea3b40, flow=0x7fffa647a810) at protocols/ssh.c:261
#3 0x000000000053c28a in check_ndpi_tcp_flow_func (ndpi_struct=0xea3b40, flow=0x7fffa647a810,
ndpi_selection_packet=0x7fffb4ff829c) at ndpi_main.c:3872
#4 0x000000000053c708 in ndpi_detection_process_packet (ndpi_struct=0xea3b40, flow=0x7fffa647a810,
packet=, packetlen=, current_tick_l=, src=0x7fffa647b0b0,
dst=0x7fffa647b1d0) at ndpi_main.c:4632
#5 0x000000000042ad0c in NetworkInterface::processPacket (this=0xcebfd0, bridge_iface_idx=1, ingressPacket=true,
when=0x7fffb4ff8870, packet_time=1566547488902, eth=0x7fffeaf2d524, vlan_id=0, iph=0x7fffeaf2d532, ip6=0x0,
ip_offset=14, len_on_wire=1514, h=0x7fffb4ff8870, packet=0x7fffeaf2d524 "RT", ndpiProtocol=0x7fffb4ff8956,
srcHost=0x7fffb4ff8948, dstHost=0x7fffb4ff8940, hostFlow=0x7fffb4ff8938) at src/NetworkInterface.cpp:1781
#6 0x000000000042cbd7 in NetworkInterface::dissectPacket (this=0xcebfd0, bridge_iface_idx=1, ingressPacket=true,
sender_mac=0x0, h=0x7fffb4ff8870, packet=0x7fffeaf2d524 "RT", ndpiProtocol=0x7fffb4ff8956,
srcHost=0x7fffb4ff8948, dstHost=0x7fffb4ff8940, flow=0x7fffb4ff8938) at src/NetworkInterface.cpp:2422
#7 0x000000000050ab24 in PF_RINGInterface::singlePacketPollLoop (this=0xcebfd0) at src/PF_RINGInterface.cpp:161
#8 0x000000000050af00 in packetPollLoop (ptr=0xcebfd0) at src/PF_RINGInterface.cpp:239
#9 0x00007ffff6943dd5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007ffff43d702d in clone () from /lib64/libc.so.6

@emanuele-f emanuele-f self-assigned this Aug 23, 2019
@emanuele-f
Copy link
Contributor

Thank you. Can you provide a pcap file with ssh traffic to reproduce the crash? You should be able to get it with sudo tcpdump -i your_nic -s0 -w ssh.pcap tcp port 22

@bmansvk
Copy link
Author

bmansvk commented Aug 23, 2019

ssh.pcap.zip

@emanuele-f
Copy link
Contributor

The very first packets of the connections are needed to reproduce the crash. Please follow the following steps:

  • start the tcpdump command above
  • start ntopng and wait for the crash
  • stop the tcpdump command and get the pcap

If you prefer, you can send me the pcap at [email protected]

@emanuele-f
Copy link
Contributor

I was unable to reproduce but some boundary checks where missing, ntop/nDPI@1231b81 may have solved the issue. Please wait one hour until the new ntopng package is available and report if the crash is fixed

@bmansvk
Copy link
Author

bmansvk commented Aug 23, 2019

Yes, I did it as you described:

  1. started pcap dump
  2. started ntopng in debug mode
  3. stopped tcpdump when ntopng crashed
    I done it over ssh connection (so there is no SSH handshake).

I repeated tcpdump using this scenario:

  1. started pcap dump on local tty terminal
  2. started ntopng on local tty terminal
  3. connected to host using ssh and waited for the crash
  4. stopped tcpdump
    This time there is full SSH/TCP connection logged
    ssh2.pcap.zip

@bmansvk
Copy link
Author

bmansvk commented Aug 23, 2019

I was unable to reproduce but some boundary checks where missing, ntop/nDPI@1231b81 may have solved the issue. Please wait one hour until the new ntopng package is available and report if the crash is fixed

I will try and let you know.

@emanuele-f
Copy link
Contributor

Could not reproduce the crash with the pcap above and pre-fix code. Anyway, let's wait until you test the new package.

@bmansvk
Copy link
Author

bmansvk commented Aug 23, 2019

Crashed again. See log: scrash.txt

Installed versions:
Aug 23 12:31:41 Updated: ntopng-data-3.9.190823-7062.noarch
Aug 23 12:31:43 Updated: ntopng-3.9.190823-7062.x86_64

Maybe is SSH but on different port than 22 (so the packet which caused crash, isn't using TCP/22). The network which is monitored using ntopng is mess.

@emanuele-f
Copy link
Contributor

There is an issue with packages, I'll write here when the new package with the fix above is available.

@emanuele-f
Copy link
Contributor

The new package is now available

@bmansvk
Copy link
Author

bmansvk commented Aug 25, 2019

It seems that is fixed now. Running for two days without segfault.Thanks.

@bmansvk bmansvk closed this as completed Aug 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@emanuele-f emanuele-f

Labels
Bug In Progress
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bmansvk @emanuele-f