ton of dropped and flows issues

[Niko78]

hi,
since few builds I have ton of dropped flows... I was suspecting alerts. In the same time I had increasing of flows without any reason (I increase limit agan and again each day, but in ntop it increased also) ; in restarting ntop process flows were back ok less than 25k instead of more than 2M

so I decided to remove alerts from preferences to check, so I put it to Off.
I wanted to "flush alerts data" but got an error (see screenshot error)... in refreshing main page, no header/footer on ntop (see screenshot error2) ; I had to restart processes to be back ok

On one of my server, error to see "flows" (see screenshot error3) and from logs I have that (see screenshot error4)

any idea ?
this morning no changes in disabling alerts, still ore than 2M flows :-( while it's not correct value
thanks

nicolas

fyi :
Version | 3.9.200128 (8774) - Enterprise Edition
Ubuntu 18.04.3 LTS
Debian buster/sid [x86_64][Ubuntu 18.04.3 LTS] - 64 bit
ntopng --max-num-flows "2000000" --max-num-hosts "500000" --dns-mode "3" --dump-flows "[hidden]" --local-networks "161.48.0.0/19,100.64.0.0/10,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16" --http-port "3000" --data-dir "/storage/tmp" --interface "tcp://172.31.1.245:5556" --dont-change-user --online-license-check

[cardigliano]

Please provide a screenshot of the active flows chart for 1 day, which is available in the historical chart in the Interface.
http://127.0.0.1:3000/lua/if_stats.lua?page=historical&version=4&ts_schema=iface%3Aflows&zoom=1d

[cardigliano]

Please note that "flush alerts data" has been fixed (a new package will be available in ~1h)

[Niko78]

added information requested
fyi this information is false cause if I restart ntop process, flows are back OK

[cardigliano]

@Niko78 it seems inactive flows are not purged from memory, any chance we can ssh to this system to debug it?

[Niko78]

nope but I could share my screen

[Niko78]

Please note that "flush alerts data" has been fixed (a new package will be available in ~1h)

I just upgraded this morning, i can confirm now no error... but after that header/bottom disapeared like I reported before (I need to restart ntop again)

nothing in flow tab also as reported got error : SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data
in syslog in the same time
Jan 30 08:52:22 localhost ntopng: [LuaEngine.cpp:12514] WARNING: Script failure [/usr/share/ntopng/scripts/lua/get_flows_data.lua][/usr/share/ntopng/scripts/lua/get_flows_data.lua:247: attempt to index a nil value (field 'status_normal')]
Jan 30 08:52:22 localhost ntopng[1936]: 30/Jan/2020 08:52:22 [LuaEngine.cpp:12514] WARNING: Script failure [/usr/share/ntopng/scripts/lua/get_flows_data.lua][/usr/share/ntopng/scripts/lua/get_flows_data.lua:247: attempt to index a nil value (field 'status_normal')]

in syslog I have ton of lines like that :
Jan 30 08:43:42 localhost ntopng: [LuaEngine.cpp:12009] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/system/housekeeping.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:43:42 localhost ntopng[31259]: 30/Jan/2020 08:43:42 [LuaEngine.cpp:12009] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/system/housekeeping.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:43:43 localhost ntopng: [LuaEngine.cpp:12514] WARNING: Script failure [/usr/share/ntopng/scripts/lua/rest/get/interface/data.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:43:43 localhost ntopng[31259]: 30/Jan/2020 08:43:43 [LuaEngine.cpp:12514] WARNING: Script failure [/usr/share/ntopng/scripts/lua/rest/get/interface/data.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:43:45 localhost ntopng: [LuaEngine.cpp:12009] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/system/housekeeping.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:43:45 localhost ntopng[31259]: 30/Jan/2020 08:43:45 [LuaEngine.cpp:12009] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/system/housekeeping.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:43:48 localhost ntopng: [LuaEngine.cpp:12009] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/system/housekeeping.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:43:48 localhost ntopng[31259]: 30/Jan/2020 08:43:48 [LuaEngine.cpp:12009] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/system/housekeeping.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:43:51 localhost ntopng: [LuaEngine.cpp:12009] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/system/housekeeping.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:43:51 localhost ntopng[31259]: 30/Jan/2020 08:43:51 [LuaEngine.cpp:12009] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/system/housekeeping.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:43:53 localhost ntopng: [LuaEngine.cpp:12514] WARNING: Script failure [/usr/share/ntopng/scripts/lua/rest/get/interface/data.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:43:53 localhost ntopng[31259]: 30/Jan/2020 08:43:53 [LuaEngine.cpp:12514] WARNING: Script failure [/usr/share/ntopng/scripts/lua/rest/get/interface/data.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:43:54 localhost ntopng: [LuaEngine.cpp:12009] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/system/housekeeping.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:43:54 localhost ntopng[31259]: 30/Jan/2020 08:43:54 [LuaEngine.cpp:12009] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/system/housekeeping.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:43:57 localhost ntopng: [LuaEngine.cpp:12009] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/system/housekeeping.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:43:57 localhost ntopng[31259]: 30/Jan/2020 08:43:57 [LuaEngine.cpp:12009] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/system/housekeeping.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:44:00 localhost ntopng: [LuaEngine.cpp:12009] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/system/housekeeping.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:44:00 localhost ntopng[31259]: 30/Jan/2020 08:44:00 [LuaEngine.cpp:12009] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/system/housekeeping.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:44:00 localhost ntopng: [LuaEngine.cpp:12009] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/interface/minute.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:44:00 localhost ntopng[31259]: 30/Jan/2020 08:44:00 [LuaEngine.cpp:12009] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/interface/minute.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:44:00 localhost ntopng: [AlertCheckLuaEngine.cpp:72] WARNING: Script failure[/usr/share/ntopng/scripts/callbacks/system/system.lua] [/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:44:00 localhost ntopng[31259]: 30/Jan/2020 08:44:00 [AlertCheckLuaEngine.cpp:72] WARNING: Script failure[/usr/share/ntopng/scripts/callbacks/system/system.lua] [/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:45:00 localhost ntopng: [LuaEngine.cpp:10959] WARNING: Script failure /usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)
Jan 30 08:45:00 localhost ntopng[31259]: 30/Jan/2020 08:45:00 [LuaEngine.cpp:10959] WARNING: Script failure /usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)
Jan 30 08:45:00 localhost ntopng: [5min.lua:10] [flow_consts.lua:59] ERROR: Error loading status definition from /storage/tmp/plugins/status_definitions/status_normal.lua
Jan 30 08:45:00 localhost ntopng[31259]: 30/Jan/2020 08:45:00 [5min.lua:10] [flow_consts.lua:59] ERROR: Error loading status definition from /storage/tmp/plugins/status_definitions/status_normal.lua
Jan 30 08:45:00 localhost ntopng: [LuaEngine.cpp:10959] WARNING: Script failure /usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)
Jan 30 08:45:00 localhost ntopng[31259]: 30/Jan/2020 08:45:00 [LuaEngine.cpp:10959] WARNING: Script failure /usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)
Jan 30 08:45:00 localhost ntopng: [5min.lua:10] [flow_consts.lua:59] ERROR: Error loading status definition from /storage/tmp/plugins/status_definitions/status_not_purged.lua
Jan 30 08:45:00 localhost ntopng[31259]: 30/Jan/2020 08:45:00 [5min.lua:10] [flow_consts.lua:59] ERROR: Error loading status definition from /storage/tmp/plugins/status_definitions/status_not_purged.lua
Jan 30 08:45:00 localhost ntopng: [LuaEngine.cpp:12009] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/system/5min.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]
Jan 30 08:45:00 localhost ntopng[31259]: 30/Jan/2020 08:45:00 [LuaEngine.cpp:12009] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/system/5min.lua][/usr/share/ntopng/scripts/lua/modules/alert_consts.lua:225: bad argument #3 to 'format' (number expected, got nil)]

[Niko78]

fyi I have that on 2 different server
server with drop flow, I have that in syslog

Jan 30 08:59:01 localhost cento[1140]: 30/Jan/2020 08:59:01 [NetworkInterface.cpp:1325] [lo] [1'439 pps/0.00 Gbps][3/25/0/512'000 act/exp/drop/max flows][0 RX pkt drops][3 active flows]
Jan 30 08:59:01 localhost cento[1140]: 30/Jan/2020 08:59:01 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:02 localhost cento[1140]: 30/Jan/2020 08:59:02 [NetworkInterface.cpp:1325] [lo] [483 pps/0.00 Gbps][5/25/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:02 localhost cento[1140]: 30/Jan/2020 08:59:02 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:03 localhost cento[1140]: 30/Jan/2020 08:59:03 [NetworkInterface.cpp:1325] [lo] [313 pps/0.00 Gbps][5/25/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:03 localhost cento[1140]: 30/Jan/2020 08:59:03 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:04 localhost cento[1140]: 30/Jan/2020 08:59:04 [NetworkInterface.cpp:1325] [lo] [465 pps/0.00 Gbps][5/25/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:04 localhost cento[1140]: 30/Jan/2020 08:59:04 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:05 localhost cento[1140]: 30/Jan/2020 08:59:05 [NetworkInterface.cpp:1325] [lo] [831 pps/0.00 Gbps][5/25/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:05 localhost cento[1140]: 30/Jan/2020 08:59:05 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:06 localhost cento[1140]: 30/Jan/2020 08:59:06 [NetworkInterface.cpp:1325] [lo] [15 pps/0.00 Gbps][5/25/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:06 localhost cento[1140]: 30/Jan/2020 08:59:06 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:07 localhost cento[1140]: 30/Jan/2020 08:59:07 [NetworkInterface.cpp:1325] [lo] [581 pps/0.00 Gbps][5/25/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:07 localhost cento[1140]: 30/Jan/2020 08:59:07 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:08 localhost cento[1140]: 30/Jan/2020 08:59:08 [NetworkInterface.cpp:1325] [lo] [861 pps/0.00 Gbps][5/25/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:08 localhost cento[1140]: 30/Jan/2020 08:59:08 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:09 localhost cento[1140]: 30/Jan/2020 08:59:09 [NetworkInterface.cpp:1325] [lo] [225 pps/0.00 Gbps][5/25/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:09 localhost cento[1140]: 30/Jan/2020 08:59:09 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:10 localhost cento[1140]: 30/Jan/2020 08:59:10 [NetworkInterface.cpp:1325] [lo] [481 pps/0.00 Gbps][5/25/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:10 localhost cento[1140]: 30/Jan/2020 08:59:10 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:11 localhost cento[1140]: 30/Jan/2020 08:59:11 [NetworkInterface.cpp:1325] [lo] [485 pps/0.00 Gbps][5/25/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:11 localhost cento[1140]: 30/Jan/2020 08:59:11 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:12 localhost cento[1140]: 30/Jan/2020 08:59:12 [NetworkInterface.cpp:1325] [lo] [433 pps/0.00 Gbps][5/25/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:12 localhost cento[1140]: 30/Jan/2020 08:59:12 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:13 localhost cento[1140]: 30/Jan/2020 08:59:13 [NetworkInterface.cpp:1325] [lo] [305 pps/0.00 Gbps][5/25/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:13 localhost cento[1140]: 30/Jan/2020 08:59:13 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:14 localhost cento[1140]: 30/Jan/2020 08:59:14 [NetworkInterface.cpp:1325] [lo] [243 pps/0.00 Gbps][5/25/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:14 localhost cento[1140]: 30/Jan/2020 08:59:14 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:15 localhost cento[1140]: 30/Jan/2020 08:59:15 [NetworkInterface.cpp:1325] [lo] [303 pps/0.00 Gbps][5/25/0/512'000 act/exp/drop/max flows][0 RX pkt drops][6 active flows]
Jan 30 08:59:15 localhost cento[1140]: 30/Jan/2020 08:59:15 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:16 localhost cento[1140]: 30/Jan/2020 08:59:16 [NetworkInterface.cpp:1325] [lo] [457 pps/0.00 Gbps][5/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:16 localhost cento[1140]: 30/Jan/2020 08:59:16 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:17 localhost cento[1140]: 30/Jan/2020 08:59:17 [NetworkInterface.cpp:1325] [lo] [189 pps/0.00 Gbps][5/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:17 localhost cento[1140]: 30/Jan/2020 08:59:17 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:18 localhost cento[1140]: 30/Jan/2020 08:59:18 [NetworkInterface.cpp:1325] [lo] [103 pps/0.00 Gbps][5/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:18 localhost cento[1140]: 30/Jan/2020 08:59:18 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:19 localhost cento[1140]: 30/Jan/2020 08:59:19 [NetworkInterface.cpp:1325] [lo] [397 pps/0.00 Gbps][5/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:19 localhost cento[1140]: 30/Jan/2020 08:59:19 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:20 localhost cento[1140]: 30/Jan/2020 08:59:20 [NetworkInterface.cpp:1325] [lo] [489 pps/0.00 Gbps][5/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:20 localhost cento[1140]: 30/Jan/2020 08:59:20 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:21 localhost cento[1140]: 30/Jan/2020 08:59:21 [NetworkInterface.cpp:1325] [lo] [523 pps/0.00 Gbps][5/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:21 localhost cento[1140]: 30/Jan/2020 08:59:21 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:22 localhost cento[1140]: 30/Jan/2020 08:59:22 [NetworkInterface.cpp:1325] [lo] [435 pps/0.00 Gbps][5/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:22 localhost cento[1140]: 30/Jan/2020 08:59:22 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:23 localhost cento[1140]: 30/Jan/2020 08:59:23 [NetworkInterface.cpp:1325] [lo] [213 pps/0.00 Gbps][5/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:23 localhost cento[1140]: 30/Jan/2020 08:59:23 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:24 localhost cento[1140]: 30/Jan/2020 08:59:24 [NetworkInterface.cpp:1325] [lo] [391 pps/0.00 Gbps][5/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:24 localhost cento[1140]: 30/Jan/2020 08:59:24 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:25 localhost cento[1140]: 30/Jan/2020 08:59:25 [NetworkInterface.cpp:1325] [lo] [895 pps/0.00 Gbps][5/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][5 active flows]
Jan 30 08:59:25 localhost cento[1140]: 30/Jan/2020 08:59:25 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:26 localhost cento[1140]: 30/Jan/2020 08:59:26 [NetworkInterface.cpp:1325] [lo] [19 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:26 localhost cento[1140]: 30/Jan/2020 08:59:26 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:27 localhost cento[1140]: 30/Jan/2020 08:59:27 [NetworkInterface.cpp:1325] [lo] [699 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:27 localhost cento[1140]: 30/Jan/2020 08:59:27 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:28 localhost cento[1140]: 30/Jan/2020 08:59:28 [NetworkInterface.cpp:1325] [lo] [357 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:28 localhost cento[1140]: 30/Jan/2020 08:59:28 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:29 localhost cento[1140]: 30/Jan/2020 08:59:29 [NetworkInterface.cpp:1325] [lo] [273 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:29 localhost cento[1140]: 30/Jan/2020 08:59:29 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:30 localhost cento[1140]: 30/Jan/2020 08:59:30 [NetworkInterface.cpp:1325] [lo] [529 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:30 localhost cento[1140]: 30/Jan/2020 08:59:30 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:31 localhost cento[1140]: 30/Jan/2020 08:59:31 [NetworkInterface.cpp:1325] [lo] [333 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:31 localhost cento[1140]: 30/Jan/2020 08:59:31 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:32 localhost cento[1140]: 30/Jan/2020 08:59:32 [NetworkInterface.cpp:1325] [lo] [515 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:32 localhost cento[1140]: 30/Jan/2020 08:59:32 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:33 localhost cento[1140]: 30/Jan/2020 08:59:33 [NetworkInterface.cpp:1325] [lo] [183 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:33 localhost cento[1140]: 30/Jan/2020 08:59:33 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:34 localhost cento[1140]: 30/Jan/2020 08:59:34 [NetworkInterface.cpp:1325] [lo] [477 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:34 localhost cento[1140]: 30/Jan/2020 08:59:34 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:35 localhost cento[1140]: 30/Jan/2020 08:59:35 [NetworkInterface.cpp:1325] [lo] [519 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:35 localhost cento[1140]: 30/Jan/2020 08:59:35 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:36 localhost cento[1140]: 30/Jan/2020 08:59:36 [NetworkInterface.cpp:1325] [lo] [519 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:36 localhost cento[1140]: 30/Jan/2020 08:59:36 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:37 localhost cento[1140]: 30/Jan/2020 08:59:37 [NetworkInterface.cpp:1325] [lo] [577 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:37 localhost cento[1140]: 30/Jan/2020 08:59:37 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:38 localhost cento[1140]: 30/Jan/2020 08:59:38 [NetworkInterface.cpp:1325] [lo] [551 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:38 localhost cento[1140]: 30/Jan/2020 08:59:38 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:39 localhost cento[1140]: 30/Jan/2020 08:59:39 [NetworkInterface.cpp:1325] [lo] [433 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:39 localhost cento[1140]: 30/Jan/2020 08:59:39 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:40 localhost cento[1140]: 30/Jan/2020 08:59:40 [NetworkInterface.cpp:1325] [lo] [277 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:40 localhost cento[1140]: 30/Jan/2020 08:59:40 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:41 localhost cento[1140]: 30/Jan/2020 08:59:41 [NetworkInterface.cpp:1325] [lo] [419 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:41 localhost cento[1140]: 30/Jan/2020 08:59:41 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:42 localhost cento[1140]: 30/Jan/2020 08:59:42 [NetworkInterface.cpp:1325] [lo] [347 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:42 localhost cento[1140]: 30/Jan/2020 08:59:42 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:43 localhost cento[1140]: 30/Jan/2020 08:59:43 [NetworkInterface.cpp:1325] [lo] [701 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:43 localhost cento[1140]: 30/Jan/2020 08:59:43 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:44 localhost cento[1140]: 30/Jan/2020 08:59:44 [NetworkInterface.cpp:1325] [lo] [187 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:44 localhost cento[1140]: 30/Jan/2020 08:59:44 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:45 localhost cento[1140]: 30/Jan/2020 08:59:45 [NetworkInterface.cpp:1325] [lo] [771 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:45 localhost cento[1140]: 30/Jan/2020 08:59:45 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:46 localhost cento[1140]: 30/Jan/2020 08:59:46 [NetworkInterface.cpp:1325] [lo] [249 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:46 localhost cento[1140]: 30/Jan/2020 08:59:46 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:47 localhost cento[1140]: 30/Jan/2020 08:59:47 [NetworkInterface.cpp:1325] [lo] [159 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:47 localhost cento[1140]: 30/Jan/2020 08:59:47 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:48 localhost cento[1140]: 30/Jan/2020 08:59:48 [NetworkInterface.cpp:1325] [lo] [691 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:48 localhost cento[1140]: 30/Jan/2020 08:59:48 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:49 localhost cento[1140]: 30/Jan/2020 08:59:49 [NetworkInterface.cpp:1325] [lo] [703 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:49 localhost cento[1140]: 30/Jan/2020 08:59:49 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:50 localhost cento[1140]: 30/Jan/2020 08:59:50 [NetworkInterface.cpp:1325] [lo] [415 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:50 localhost cento[1140]: 30/Jan/2020 08:59:50 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:51 localhost cento[1140]: 30/Jan/2020 08:59:51 [NetworkInterface.cpp:1325] [lo] [863 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:51 localhost cento[1140]: 30/Jan/2020 08:59:51 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:52 localhost cento[1140]: 30/Jan/2020 08:59:52 [NetworkInterface.cpp:1325] [lo] [425 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:52 localhost cento[1140]: 30/Jan/2020 08:59:52 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:53 localhost cento[1140]: 30/Jan/2020 08:59:53 [NetworkInterface.cpp:1325] [lo] [219 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:53 localhost cento[1140]: 30/Jan/2020 08:59:53 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:54 localhost cento[1140]: 30/Jan/2020 08:59:54 [NetworkInterface.cpp:1325] [lo] [323 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:54 localhost cento[1140]: 30/Jan/2020 08:59:54 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:55 localhost cento[1140]: 30/Jan/2020 08:59:55 [NetworkInterface.cpp:1325] [lo] [369 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:55 localhost cento[1140]: 30/Jan/2020 08:59:55 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:56 localhost cento[1140]: 30/Jan/2020 08:59:56 [NetworkInterface.cpp:1325] [lo] [273 pps/0.00 Gbps][7/26/0/512'000 act/exp/drop/max flows][0 RX pkt drops][7 active flows]
Jan 30 08:59:56 localhost cento[1140]: 30/Jan/2020 08:59:56 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:57 localhost cento[1140]: 30/Jan/2020 08:59:57 [NetworkInterface.cpp:1295] WARNING: Demo mode expired
Jan 30 08:59:57 localhost cento[1140]: 30/Jan/2020 08:59:57 [PF_RINGInterface.cpp:421] Terminated packet polling for lo
Jan 30 08:59:57 localhost cento[1140]: 30/Jan/2020 08:59:57 [PF_RINGInterface.cpp:427] [61.00 K Pkts/10.10 MB]
Jan 30 08:59:57 localhost cento[1140]: 30/Jan/2020 08:59:57 [FlowExporter.cpp:47] Terminated flow exporter for lo
Jan 30 08:59:57 localhost cento[1140]: 30/Jan/2020 08:59:57 [FlowExporter.cpp:107] [lo] Flow Export Queue Len [0 tot queued][0/0 IPv4/v6][Drops queue too long 0/0 IPv4/v6][0/0 tot exported/dropped flows]
Jan 30 08:59:57 localhost kernel: [146565.924968] device lo left promiscuous mode
Jan 30 09:00:00 localhost ntopng: [LuaEngine.cpp:10959] WARNING: Script failure cannot open /storage/tmp/plugins/plugins_metadata.lua: No such file or directory
Jan 30 09:00:00 localhost ntopng[1831]: 30/Jan/2020 09:00:00 [LuaEngine.cpp:10959] WARNING: Script failure cannot open /storage/tmp/plugins/plugins_metadata.lua: No such file or directory
Jan 30 09:00:00 localhost ntopng[1831]: 30/Jan/2020 09:00:00 [AlertCheckLuaEngine.cpp:80] WARNING: Script failure[/usr/share/ntopng/scripts/callbacks/system/system.lua] [/usr/share/ntopng/scripts/lua/modules/plugins_utils.lua:690: attempt to index a nil value (upvalue 'METADATA')]
Jan 30 09:00:00 localhost ntopng: [AlertCheckLuaEngine.cpp:80] WARNING: Script failure[/usr/share/ntopng/scripts/callbacks/system/system.lua] [/usr/share/ntopng/scripts/lua/modules/plugins_utils.lua:690: attempt to index a nil value (upvalue 'METADATA')]

[cardigliano]

@Niko78 we pushed another fix for the issue you reported, please update again (a new package is available now)

[Niko78]

@cardigliano I can confirm now it seems to be much better, I can see flows and no error in syslog
now let's see if flows increase or if they are released from memory like you said
thanks for fix

[Niko78]

@cardigliano this morning, flows increased again and seem to be frozen to 2600001
it doesn't change since more than 1 hour

[cardigliano]

Flow purging is not working, we definitely need to debug this. Please write to cardigliano at ntop.org and we will schedule a remote session. Thank you.

[simonemainardi]

Seems issue is due to MySQL. Turning MySQL export off causes the system to properly expire and idle flows.

[simonemainardi]

should be fixed with 37975c1 and companion pro commit

[Niko78]

it seems to be ok now
still drops (zmq/flows)... but it's an other problem
thanks

[simonemainardi]

thanks for reporting