Memory leak ntopng 5.7.230804 since June/July 2023

[notyourcommy]

Environment:

• OS name: Centos Stream
• OS version: 8
• Architecture: amd64
• ntopng version/revision: 5.7.230804 [Enterprise/Professional build]

What happened:
Since ~June/July 2023 we are observing memory leak (it eats all the ram (32GB) in a day and then OOM kills it)

Ntop/Nprobe config
cat /etc/ntopng/ntopng.conf
-G=/var/run/ntopng.pid
-F="clickhouse;...;ntopng;ntop;midiland"
-i "tcp://127.0.0.1:5556"
-n 3
-x 700000
-X 700000
--ignore-vlans
-v 2
-m

cat /etc/nprobe/nprobe-sflow.conf
-i none
--collector-port 6343
--ntopng "zmq://127.0.0.1:5556"
-–flow-deduplication 15
-n none
-G=/var/run/nprobe-sflow.pid
-T="%IN_SRC_MAC %OUT_DST_MAC %INPUT_SNMP %OUTPUT_SNMP %SRC_VLAN %IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV6_SRC_ADDR %IPV6_DST_ADDR %IP_PROTOCOL_VERSION %PROTOCOL %L7_PROTO %IN_BYTES %IN_PKTS %OUT_BYTES %OUT_PKTS %FIRST_SWITCHED %LAST_SWITCHED %CLIENT_TCP_FLAGS %SERVER_TCP_FLAGS %L7_PROTO_RISK %L7_RISK_SCORE %EXPORTER_IPV4_ADDRESS %DIRECTION%POST_NAT_SRC_IPV4_ADDR %POST_NAT_DST_IPV4_ADDR %POST_NAPT_SRC_TRANSPORT_PORT %POST_NAPT_DST_TRANSPORT_PORT %BGP_PREV_ADJACENT_ASN %BGP_NEXT_ADJACENT_ASN %IPV4_BGP_NEXT_HOP %SRC_AS %DST_AS"

[notyourcommy]

[288859.648241] crond invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[288859.649488] CPU: 0 PID: 809931 Comm: crond Kdump: loaded Tainted: G OE --------- - - 4.18.0-500.el8.x86_64 #1
[288859.650663] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org 04/01/2014
[288859.651878] Call Trace:
[288859.653141] dump_stack+0x41/0x60
[288859.653583] dump_header+0x4a/0x1df
[288859.653998] oom_kill_process.cold.33+0xb/0x10
[288859.654497] out_of_memory+0x1bd/0x4e0
[288859.654909] __alloc_pages_slowpath+0xbf0/0xcd0
[288859.655435] __alloc_pages_nodemask+0x2e2/0x330
[288859.655947] pagecache_get_page+0xce/0x310
[288859.656395] filemap_fault+0x6c8/0xa30
[288859.656820] ? __mod_memcg_lruvec_state+0x4a/0xc0
[288859.657340] ? __mod_lruvec_page_state+0x5e/0x80
[288859.657853] ? page_add_file_rmap+0x99/0x150
[288859.658317] ? alloc_set_pte+0xb8/0x3e0
[288859.658749] ? xas_load+0x8/0x80
[288859.659113] ? _cond_resched+0x15/0x30
[288859.659537] __xfs_filemap_fault+0x6d/0x200 [xfs]
[288859.660319] __do_fault+0x38/0xc0
[288859.660715] handle_pte_fault+0x55d/0x880
[288859.661167] __handle_mm_fault+0x365/0x6c0
[288859.661627] handle_mm_fault+0xca/0x2a0
[288859.662046] __do_page_fault+0x1f0/0x460
[288859.662489] ? page_fault+0x8/0x30
[288859.662877] do_page_fault+0x37/0x130
[288859.663279] ? page_fault+0x8/0x30
[288859.663667] page_fault+0x1e/0x30
[288859.664054] RIP: 0033:0x7f00ed778bda
[288859.664455] Code: 1d 23 80 21 00 48 89 44 24 28 41 83 ec 01 0f 88 7c 00 00 00 44 89 e3 44 8b 74 24 24 48 8b 4c 24 18 d1 fb 48 63 c3 49 0f af c6 <8b> 74 01 04 41 39 f5 76 5d 45 31 d2 4c 89 5c 24 30 48 8b 6c 24 08
[288859.666420] RSP: 002b:00007ffe0c8fd0c0 EFLAGS: 00010202
[288859.666991] RAX: 00000000000013b0 RBX: 00000000000000d2 RCX: 00007f00ed98b030
[288859.667742] RDX: 0000000000000000 RSI: 00007ffe0c8fd160 RDI: 000000000000000e
[288859.668491] RBP: 00007f00ed787ad7 R08: 00007f00ed990690 R09: 00007f00ddc60deb
[288859.669256] R10: 000055bc1d150008 R11: 0000000000000002 R12: 00000000000001a5
[288859.670017] R13: 0000000000004e83 R14: 0000000000000018 R15: 0000000000000000
[288859.670787] Mem-Info:
[288859.671050] active_anon:7103 inactive_anon:7928585 isolated_anon:0
active_file:0 inactive_file:893 isolated_file:31
unevictable:77114 dirty:0 writeback:0
slab_reclaimable:10791 slab_unreclaimable:19767
mapped:81473 shmem:6265 pagetables:32172 bounce:0
free:52419 free_pcp:1849 free_cma:0
[288859.674508] Node 0 active_anon:28412kB inactive_anon:31714340kB active_file:0kB inactive_file:2968kB unevictable:308456kB isolated(anon):0kB isolated(file):124kB mapped:325892kB dirty:0kB writeback:0kB shmem:25060kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 2836480kB writeback_tmp:0kB kernel_stack:9488kB pagetables:128688kB all_unreclaimable? no
[288859.677805] Node 0 DMA free:11264kB min:28kB low:40kB high:52kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[288859.680225] lowmem_reserve[]: 0 2663 31792 31792 31792
[288859.680799] Node 0 DMA32 free:121836kB min:5656kB low:8380kB high:11104kB active_anon:5612kB inactive_anon:2651224kB active_file:0kB inactive_file:880kB unevictable:0kB writepending:0kB present:3129192kB managed:2801512kB mlocked:0kB bounce:0kB free_pcp:6384kB local_pcp:792kB free_cma:0kB
[288859.683483] lowmem_reserve[]: 0 0 29129 29129 29129
[288859.684030] Node 0 Normal free:77584kB min:94660kB low:124488kB high:154316kB active_anon:22800kB inactive_anon:29063224kB active_file:128kB inactive_file:3540kB unevictable:308456kB writepending:0kB present:30408704kB managed:29836492kB mlocked:308456kB bounce:0kB free_pcp:1212kB local_pcp:144kB free_cma:0kB
[288859.686894] lowmem_reserve[]: 0 0 0 0 0
[288859.687323] Node 0 DMA: 04kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 11024kB (U) 12048kB (M) 24096kB (M) = 11264kB
[288859.688558] Node 0 DMA32: 3934kB (UME) 3788kB (UE) 52716kB (UME) 16832kB (UME) 8264kB (UME) 73128kB (UME) 104256kB (UME) 61512kB (UME) 301024kB (UME) 02048kB 04096kB = 121572kB
[288859.690342] Node 0 Normal: 48734kB (UME) 24208kB (UMEH) 176216kB (UME) 31232kB (M) 064kB 1128kB (M) 1256kB (M) 1512kB (M) 11024kB (M) 02048kB 0*4096kB = 78948kB
[288859.691953] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[288859.692893] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[288859.693810] 84330 total pagecache pages
[288859.694238] 0 pages in swap cache
[288859.694606] Swap cache stats: add 0, delete 0, find 0/0
[288859.695177] Free swap = 0kB
[288859.695500] Total swap = 0kB
[288859.695851] 8388472 pages RAM
[288859.696194] 0 pages HighMem/MovableOnly
[288859.696624] 225131 pages reserved
[288859.697008] 0 pages hwpoisoned
[288859.697362] Tasks state (memory values in pages):
[288859.697892] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[288859.698831] [ 632] 0 632 26550 4881 270336 0 0 systemd-journal
[288859.699813] [ 658] 0 658 24788 744 217088 0 -1000 systemd-udevd
[288859.700758] [ 687] 32 687 16812 182 180224 0 0 rpcbind
[288859.701671] [ 696] 0 696 32709 176 135168 0 -1000 auditd
[288859.702549] [ 705] 0 705 12163 91 147456 0 0 sedispatch
[288859.703485] [ 743] 2 743 95740 861 233472 0 0 rngd
[288859.704362] [ 744] 81 744 19015 236 155648 0 -900 dbus-daemon
[288859.705312] [ 752] 0 752 147632 1130 380928 0 0 NetworkManager
[288859.706300] [ 753] 0 753 20820 540 184320 0 0 systemd-logind
[288859.707275] [ 755] 0 755 31244 164 147456 0 0 irqbalance
[288859.708205] [ 764] 992 764 35019 204 176128 0 0 chronyd
[288859.709116] [ 773] 0 773 19162 233 192512 0 -1000 sshd
[288859.709986] [ 777] 0 777 174676 4693 462848 0 0 tuned
[288859.710851] [ 778] 990 778 442278 318837 2674688 0 0 redis-server
[288859.711803] [ 801] 0 801 77899 165 184320 0 0 gssproxy
[288859.712716] [ 837] 0 837 58914 223 98304 0 0 crond
[288859.713611] [ 838] 0 838 54903 29 65536 0 0 agetty
[288859.714529] [ 839] 0 839 6533 49 94208 0 0 atd
[288859.715403] [ 887] 0 887 58525 2512 221184 0 0 rsyslogd
[288859.716329] [ 890] 985 890 175106 20857 692224 0 0 clckhouse-watch
[288859.717322] [ 892] 0 892 66016 2386 151552 0 0 munin-node
[288859.718258] [ 918] 998 918 506732 1796 389120 0 0 polkitd
[288859.719164] [ 1092] 985 1092 10764020 152078 61779968 0 0 clickhouse-serv
[288859.720143] [ 397308] 988 397308 183896 60630 983040 0 0 nprobe@6343
[288859.721100] [ 397330] 989 397330 8043456 7386159 60317696 0 0 ntopng-main
[288859.722042] [ 809925] 989 809925 56163 57 86016 0 0 sh
[288859.722895] [ 809927] 989 809927 222800 33285 1089536 0 0 clickhouse-clie
[288859.723873] [ 809931] 0 809931 65182 235 122880 0 0 crond
[288859.724770] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/ntopng.service,task=ntopng-main,pid=397330,uid=989
[288859.726489] Out of memory: Killed process 397330 (ntopng-main) total-vm:32173824kB, anon-rss:29544636kB, file-rss:0kB, shmem-rss:0kB, UID:989 pgtables:58904kB oom_score_adj:0
[288862.872661] oom_reaper: reaped process 397330 (ntopng-main), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

[lucaderi]

@notyourcommy I think the bug is fixed. Overnight new packages will be built: please test them and report.

[notyourcommy]

5.7.230808 ? Still leaking unfortunately.
But I see new update (5.7.230808-21268). I'll try also this one.

[lucaderi]

How do you know it's leaking: this is because memory increases or OOM? Can you please share pictures of the memory used and number of flows/hosts?

[notyourcommy]

Ntop uses all memory then OOM kills it.
[root@ntop centos]# ps axu | grep ntop
ntopng 1494664 166 40.9 16180932 13386064 ? Ssl aug08 1169:01 /usr/bin/ntopng /run/ntopng.conf
root 1611083 0.0 0.0 221940 1092 pts/0 S+ 10:20 0:00 grep --color=auto ntop
[root@ntop centos]# pmap -x 1494664 | grep total
total kB 16180936 13390772 13290864
[root@ntop centos]#
Thats after 10 hours.

20k local hosts, 200k remote hosts, 400k flows

[lucaderi]

The number of hosts and flows is steady? I will run a new testing session, hoping to be able to reproduce the issue otherwise I need your help. Stay tuned

[notyourcommy]

Number of hosts/flows depends on the time of day. I wrote you the "prime time" numbers. (~AVG MAX)

[lucaderi]

I have pushed a fix that is included in the build that will be ready at 15:45 CET. Please update at your convenience and report

[notyourcommy]

Upgrading to 5.7.230809-21278
Will report back in ~12-20hours

[lucaderi]

@notyourcommy Any news?

[notyourcommy]

@notyourcommy Any news?

Unfortunately still leaking - 30GB in ~20 hours.
I can give access to the server if it will help.

[Pk aug 11 16:54:53 2023] ntopng-S-active invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[Pk aug 11 16:54:53 2023] CPU: 5 PID: 1741596 Comm: ntopng-S-active Kdump: loaded Tainted: G OE --------- - - 4.18.0-500.el8.x86_64 #1
[Pk aug 11 16:54:53 2023] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org 04/01/2014
[Pk aug 11 16:54:53 2023] Call Trace:
[Pk aug 11 16:54:53 2023] dump_stack+0x41/0x60
[Pk aug 11 16:54:53 2023] dump_header+0x4a/0x1df
[Pk aug 11 16:54:53 2023] oom_kill_process.cold.33+0xb/0x10
[Pk aug 11 16:54:53 2023] out_of_memory+0x1bd/0x4e0
[Pk aug 11 16:54:53 2023] __alloc_pages_slowpath+0xbf0/0xcd0
[Pk aug 11 16:54:53 2023] __alloc_pages_nodemask+0x2e2/0x330
[Pk aug 11 16:54:53 2023] pagecache_get_page+0xce/0x310
[Pk aug 11 16:54:53 2023] filemap_fault+0x6c8/0xa30
[Pk aug 11 16:54:53 2023] ? _raw_spin_lock+0x1e/0x30
[Pk aug 11 16:54:53 2023] ? alloc_set_pte+0x1f1/0x3e0
[Pk aug 11 16:54:53 2023] ? _cond_resched+0x15/0x30
[Pk aug 11 16:54:53 2023] __xfs_filemap_fault+0x6d/0x200 [xfs]
[Pk aug 11 16:54:53 2023] __do_fault+0x38/0xc0
[Pk aug 11 16:54:53 2023] handle_pte_fault+0x55d/0x880
[Pk aug 11 16:54:53 2023] __handle_mm_fault+0x365/0x6c0
[Pk aug 11 16:54:53 2023] handle_mm_fault+0xca/0x2a0
[Pk aug 11 16:54:53 2023] __do_page_fault+0x1f0/0x460
[Pk aug 11 16:54:53 2023] do_page_fault+0x37/0x130
[Pk aug 11 16:54:53 2023] ? page_fault+0x8/0x30
[Pk aug 11 16:54:53 2023] page_fault+0x1e/0x30
[Pk aug 11 16:54:53 2023] RIP: 0033:0x60f020
[Pk aug 11 16:54:53 2023] Code: Unable to access opcode bytes at RIP 0x60eff6.
[Pk aug 11 16:54:53 2023] RSP: 002b:00007f46c77fb4d8 EFLAGS: 00010246
[Pk aug 11 16:54:53 2023] RAX: 0000000000000000 RBX: 00007f467c130238 RCX: 00007f467c0123d0
[Pk aug 11 16:54:53 2023] RDX: 00000000000003d0 RSI: 00007f467c0123b0 RDI: 00007f467c130238
[Pk aug 11 16:54:53 2023] RBP: 00007f467c0123b0 R08: 00007f467c0123d0 R09: 0000000000000000
[Pk aug 11 16:54:53 2023] R10: 00007f46c77fafb5 R11: 00007f467c012360 R12: 00007f467c2696b0
[Pk aug 11 16:54:53 2023] R13: 0000000000000004 R14: 000000000060f020 R15: 00007f467c0feea0
[Pk aug 11 16:54:53 2023] Mem-Info:
[Pk aug 11 16:54:53 2023] active_anon:798 inactive_anon:8032753 isolated_anon:0
active_file:318 inactive_file:1216 isolated_file:31
unevictable:0 dirty:0 writeback:0
slab_reclaimable:8546 slab_unreclaimable:20217
mapped:7293 shmem:10365 pagetables:18192 bounce:0
free:50214 free_pcp:801 free_cma:0
[Pk aug 11 16:54:53 2023] Node 0 active_anon:3192kB inactive_anon:32131012kB active_file:940kB inactive_file:4000kB unevictable:0kB isolated(anon):0kB isolated(file):124kB mapped:28880kB dirty:0kB writeback:0kB shmem:41460kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 16756736kB writeback_tmp:0kB kernel_stack:4448kB pagetables:72768kB all_unreclaimable? no
[Pk aug 11 16:54:53 2023] Node 0 DMA free:11264kB min:28kB low:40kB high:52kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Pk aug 11 16:54:53 2023] lowmem_reserve[]: 0 2663 31792 31792 31792
[Pk aug 11 16:54:53 2023] Node 0 DMA32 free:121996kB min:5656kB low:8380kB high:11104kB active_anon:0kB inactive_anon:2665872kB active_file:52kB inactive_file:1332kB unevictable:0kB writepending:0kB present:3129192kB managed:2801512kB mlocked:0kB bounce:0kB free_pcp:2696kB local_pcp:304kB free_cma:0kB
[Pk aug 11 16:54:53 2023] lowmem_reserve[]: 0 0 29129 29129 29129
[Pk aug 11 16:54:53 2023] Node 0 Normal free:68100kB min:65988kB low:95816kB high:125644kB active_anon:3192kB inactive_anon:29465140kB active_file:604kB inactive_file:1460kB unevictable:0kB writepending:0kB present:30408704kB managed:29836492kB mlocked:0kB bounce:0kB free_pcp:1220kB local_pcp:152kB free_cma:0kB
[Pk aug 11 16:54:53 2023] lowmem_reserve[]: 0 0 0 0 0
[Pk aug 11 16:54:53 2023] Node 0 DMA: 04kB 08kB 016kB 032kB 064kB 0128kB 0256kB 0512kB 11024kB (U) 12048kB (M) 24096kB (M) = 11264kB
[Pk aug 11 16:54:53 2023] Node 0 DMA32: 2474kB (UE) 3048kB (UE) 41116kB (UE) 37932kB (UME) 31264kB (UME) 174128kB (UME) 95256kB (UME) 36512kB (UME) 141024kB (UME) 02048kB 04096kB = 121452kB
[Pk aug 11 16:54:53 2023] Node 0 Normal: 12154kB (UMEH) 11278kB (UMEH) 91316kB (UME) 24332kB (ME) 13164kB (ME) 86128kB (ME) 44256kB (ME) 3512kB (M) 01024kB 02048kB 0*4096kB = 68452kB
[Pk aug 11 16:54:53 2023] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Pk aug 11 16:54:53 2023] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Pk aug 11 16:54:53 2023] 11470 total pagecache pages
[Pk aug 11 16:54:53 2023] 0 pages in swap cache
[Pk aug 11 16:54:53 2023] Swap cache stats: add 0, delete 0, find 0/0
[Pk aug 11 16:54:53 2023] Free swap = 0kB
[Pk aug 11 16:54:53 2023] Total swap = 0kB
[Pk aug 11 16:54:53 2023] 8388472 pages RAM
[Pk aug 11 16:54:53 2023] 0 pages HighMem/MovableOnly
[Pk aug 11 16:54:53 2023] 225131 pages reserved
[Pk aug 11 16:54:53 2023] 0 pages hwpoisoned
[Pk aug 11 16:54:53 2023] Tasks state (memory values in pages):
[Pk aug 11 16:54:53 2023] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[Pk aug 11 16:54:53 2023] [ 632] 0 632 33808 10635 327680 0 0 systemd-journal
[Pk aug 11 16:54:53 2023] [ 658] 0 658 24788 744 217088 0 -1000 systemd-udevd
[Pk aug 11 16:54:53 2023] [ 687] 32 687 16812 182 180224 0 0 rpcbind
[Pk aug 11 16:54:53 2023] [ 696] 0 696 32709 176 135168 0 -1000 auditd
[Pk aug 11 16:54:53 2023] [ 705] 0 705 12163 94 147456 0 0 sedispatch
[Pk aug 11 16:54:53 2023] [ 743] 2 743 95740 866 233472 0 0 rngd
[Pk aug 11 16:54:53 2023] [ 744] 81 744 19015 241 155648 0 -900 dbus-daemon
[Pk aug 11 16:54:53 2023] [ 752] 0 752 147632 1157 380928 0 0 NetworkManager
[Pk aug 11 16:54:53 2023] [ 753] 0 753 20820 541 184320 0 0 systemd-logind
[Pk aug 11 16:54:53 2023] [ 755] 0 755 31244 164 147456 0 0 irqbalance
[Pk aug 11 16:54:53 2023] [ 764] 992 764 35019 204 176128 0 0 chronyd
[Pk aug 11 16:54:53 2023] [ 773] 0 773 19162 233 192512 0 -1000 sshd
[Pk aug 11 16:54:53 2023] [ 777] 0 777 174740 4748 462848 0 0 tuned
[Pk aug 11 16:54:53 2023] [ 778] 990 778 444838 330915 2793472 0 0 redis-server
[Pk aug 11 16:54:53 2023] [ 801] 0 801 77899 165 184320 0 0 gssproxy
[Pk aug 11 16:54:53 2023] [ 837] 0 837 58914 225 98304 0 0 crond
[Pk aug 11 16:54:53 2023] [ 838] 0 838 54903 29 65536 0 0 agetty
[Pk aug 11 16:54:53 2023] [ 839] 0 839 6533 49 94208 0 0 atd
[Pk aug 11 16:54:53 2023] [ 887] 0 887 64657 5453 266240 0 0 rsyslogd
[Pk aug 11 16:54:53 2023] [ 892] 0 892 66016 2386 151552 0 0 munin-node
[Pk aug 11 16:54:53 2023] [ 918] 998 918 506732 1797 389120 0 0 polkitd
[Pk aug 11 16:54:53 2023] [1740985] 989 1740985 8360454 7573210 62767104 0 0 ntopng-main
[Pk aug 11 16:54:53 2023] [1741046] 988 1741046 183896 66538 1040384 0 0 nprobe@6343
[Pk aug 11 16:54:53 2023] [2159843] 0 2159843 37029 327 319488 0 0 sshd
[Pk aug 11 16:54:53 2023] [2159848] 1000 2159848 22368 331 204800 0 0 systemd
[Pk aug 11 16:54:53 2023] [2159851] 1000 2159851 78025 1338 339968 0 0 (sd-pam)
[Pk aug 11 16:54:53 2023] [2159859] 1000 2159859 37029 327 315392 0 0 sshd
[Pk aug 11 16:54:53 2023] [2159860] 1000 2159860 56229 146 94208 0 0 bash
[Pk aug 11 16:54:53 2023] [2159892] 0 2159892 86051 263 307200 0 0 sudo
[Pk aug 11 16:54:53 2023] [2159894] 0 2159894 84612 238 286720 0 0 su
[Pk aug 11 16:54:53 2023] [2159895] 0 2159895 56229 173 86016 0 0 bash
[Pk aug 11 16:54:53 2023] [2163493] 0 2163493 188417 29706 974848 0 0 dnf
[Pk aug 11 16:54:53 2023] [2163615] 989 2163615 56163 58 69632 0 0 sh
[Pk aug 11 16:54:53 2023] [2163616] 989 2163616 54306 17 61440 0 0 cat
[Pk aug 11 16:54:53 2023] [2163617] 989 2163617 180301 4731 794624 0 0 clickhouse-clie
[Pk aug 11 16:54:53 2023] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/ntopng.service,task=ntopng-main,pid=1740985,uid=989
[Pk aug 11 16:54:53 2023] Out of memory: Killed process 1740985 (ntopng-main) total-vm:33441816kB, anon-rss:30292840kB, file-rss:0kB, shmem-rss:0kB, UID:989 pgtables:61296kB oom_score_adj:0
[Pk aug 11 16:54:56 2023] oom_reaper: reaped process 1740985 (ntopng-main), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

[lucaderi]

After debugging the root cause was InfluxDB that was using most of the available memory