[Linux] Segfault in memory_maps() when monitored process uses hundreds of GBs of memory

[wdh]

Summary

• OS: Linux
• Architecture: 64bit
• Psutil version: 5.9.0+
• Python version: 3.8+
• Type: core

Description

If the monitored process uses hundreds of GBs of memory (for example, a large impalad process), a segfault can occur in the memory_maps() function.

Given a large enough process, the following script is sufficient to reproduce with psutil 5.9.0+ on Python 3.8, 3.10 and 3.12:

import psutil
import sys
pid = int(sys.argv[1])
process = psutil.Process(pid)
mm = psutil.Process.memory_maps(process)

Inspection of the resulting core dump with gdb shows that the backtrace contains > 65,500 calls to map_next() (this output from Python 3.10.11):

#0 0x0007ffb3625b6a0 in map_next (lz=0x7ffb2d7777c0) at Python/bltinmodule.c:1341
...
#65503 0x00007ffb3625b6a6 in map_next (lz=0x7ffb2c2e4be0) at Python/bltinmodule.c:1341
#65504 0x00007ffb36188bed in list_extend (iterable=<optimized out>, self=0x7ffb2c2d7500) at Objects/listobject.c:960
#65505 _PyList_Extend (self=0x7ffb2c2d7500, iterable=<optimized out>) at Objects/listobject.c:1000
...

[giampaolo]

Mmm weird. Process.memory_maps() implementation is pure python, it doesn't use any C extension. Judging from the gdb core dump the error originates from cPython itself (list.extend()). You may have found a cPython bug.

[wdh]

Yes - understood and agreed on both counts. The attached PR serves simply as a mitigation to allow psutil to work for the above use case, given that the underlying issue in cPython does not sound as though it will be going away any time soon, if ever (see e.g. python/cpython#103503)

[giampaolo]

Interesting. In #2497 you say that the PR "mitigates" this issue. What does it mean exactly? That it happens less often or...?

[wdh]

Sorry if that was ambiguous - I mean "works around the underlying cPython issue, so the segfault is avoided 100% of the time, allowing psutil to be used to monitor these large processes".

[giampaolo]

Fixed in #2497.