[3.12] gh-116510: Fix a crash due to shared immortal interned strings.

[nascheme]

Fix a crash caused by immortal interned strings being shared between sub-interpreters that use basic single-phase init. In that case, the string can be used by an interpreter that outlives the interpeter that created and
interned it. For interpreters that share obmalloc state, also share the interned dict with the main interpreter.

• Issue: CPython 3.12 embedded in WeeChat causes segfault on subsequent calls to Py_EndInterpreter #116510

[nascheme]

This is a small change code-wise but I'm not sure about the full implications of it. It does fix a bug but might create other bugs or problems, I'm not confident.

One downside to this change is that if many sub-interpreters are created and they create a lot of immortal interned strings, those strings will remain alive in the main interpreter state. So, it could cause a lot more memory to be used. My gut feeling is that's okay since we shouldn't be immortalizing strings unless we are fairly sure they won't cause memory bloat, even in a single interpreter case. E.g. it happens to variable names appearing in Python source code.

[ericsnowcurrently]

Mostly LGTM

I think there are just a couple spots where we need to clear the PyInterpreterState field for the shared interned dict case.

[bedevere-app]

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

[nascheme]

I have made the requested changes; please review again

[bedevere-app]

Thanks for making the requested changes!

@ericsnowcurrently: please review the changes made to this pull request.

[ericsnowcurrently]

LGTM

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Arch Linux TraceRefs 3.12 has failed when building commit 5dd07eb.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1197/builds/893) and take a look at the build logs.
4. Check if the failure is related to this commit (5dd07eb) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1197/builds/893

Failed tests:

• test_embed
• test_syslog
• test_threading
• test_import
• test_multibytecodec
• test_atexit
• test_importlib
• test_int
• test_capi

Failed subtests:

• test_subinterps_different_ids - test.test_embed.EmbeddingTests.test_subinterps_different_ids
• test_single_init_extension_compat - test.test_import.SubinterpImportTests.test_single_init_extension_compat
• test_python_compat - test.test_import.SubinterpImportTests.test_python_compat
• test_subinterps_distinct_state - test.test_embed.EmbeddingTests.test_subinterps_distinct_state
• test_subinterps_main - test.test_embed.EmbeddingTests.test_subinterps_main
• test_multi_init_extension_compat - test.test_import.SubinterpImportTests.test_multi_init_extension_compat

Summary of the results of the build (if available):

==

Click to see traceback logs

TracebackTests.test_broken_parent_from) ... ok


TracebackTests.test_broken_submodule) ... ok


TracebackTests.test_import_bug) ... ok


TracebackTests.test_syntax_error) ... ok


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 174, in test_subinterps_main
    for run in self.run_repeated_init_and_subinterpreters():
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7f22e05ffe80\nobject refcount : 0\nobject type     : 0x55b39aff42c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7f22e05ffe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x000055b39b168908)\n\nCurrent thread 0x00007f22e0989740 (most recent call first):\n  <no Python frame>\n'


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 180, in test_subinterps_different_ids
    for run in self.run_repeated_init_and_subinterpreters():
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7f48cf32fe80\nobject refcount : 0\nobject type     : 0x557a9680d2c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7f48cf32fe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x0000557a96981908)\n\nCurrent thread 0x00007f48cf6b9740 (most recent call first):\n  <no Python frame>\n'


TracebackTests.test_exec_failure_nested) ... ok


TracebackTests.test_nonexistent_module_nested) ... ok


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1897, in test_multi_init_extension_compat
    self.check_compatible_fresh(module, strict=True)
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1817, in check_compatible_fresh
    _, out, err = script_helper.assert_python_ok('-c', textwrap.dedent(f'''
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 166, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 151, in _assert_python
    res.fail(cmd_line)
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 76, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is -11
command line: ['/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/python', '-X', 'faulthandler', '-I', '-c', '\nimport _testcapi, sys\nassert (\n    \'_testmultiphase\' in sys.builtin_module_names or\n    \'_testmultiphase\' not in sys.modules\n), repr(\'_testmultiphase\')\nret = _testcapi.run_in_subinterp_with_config(\n    "\\nimport os, sys\\n\\ntry:\\n    import _testmultiphase\\nexcept ImportError as exc:\\n    text = \'ImportError: \' + str(exc)\\nelse:\\n    text = \'okay\'\\nos.write(sys.stdout.fileno(), text.encode(\'utf-8\'))\\n",\n    **{\'allow_fork\': False, \'allow_exec\': False, \'allow_threads\': True, \'allow_daemon_threads\': False, \'use_main_obmalloc\': True, \'gil\': 1, \'check_multi_interp_extensions\': True},\n)\nassert ret == 0, ret\n']


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1886, in test_single_init_extension_compat
    self.check_incompatible_fresh(module)
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1841, in check_incompatible_fresh
    _, out, err = script_helper.assert_python_ok('-c', textwrap.dedent(f'''
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 166, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 151, in _assert_python
    res.fail(cmd_line)
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 76, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is -11
command line: ['/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/python', '-X', 'faulthandler', '-I', '-c', '\nimport _testcapi, sys\nassert \'_testsinglephase\' not in sys.modules, \'_testsinglephase\'\nret = _testcapi.run_in_subinterp_with_config(\n    "\\nimport os, sys\\n\\ntry:\\n    import _testsinglephase\\nexcept ImportError as exc:\\n    text = \'ImportError: \' + str(exc)\\nelse:\\n    text = \'okay\'\\nos.write(sys.stdout.fileno(), text.encode(\'utf-8\'))\\n",\n    **{\'allow_fork\': False, \'allow_exec\': False, \'allow_threads\': True, \'allow_daemon_threads\': False, \'use_main_obmalloc\': True, \'gil\': 1, \'check_multi_interp_extensions\': True},\n)\nassert ret == 0, ret\n']


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 188, in test_subinterps_distinct_state
    for run in self.run_repeated_init_and_subinterpreters():
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7fef8cfbfe80\nobject refcount : 0\nobject type     : 0x55da79f132c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7fef8cfbfe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x000055da7a087908)\n\nCurrent thread 0x00007fef8d34b740 (most recent call first):\n  <no Python frame>\n'


TracebackTests.test_broken_from) ... ok


TracebackTests.test_nonexistent_module) ... ok


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 174, in test_subinterps_main
    for run in self.run_repeated_init_and_subinterpreters():
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7fc0394bfe80\nobject refcount : 0\nobject type     : 0x55c76866b2c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7fc0394bfe80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x000055c7687df908)\n\nCurrent thread 0x00007fc03984b740 (most recent call first):\n  <no Python frame>\n'


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1945, in test_python_compat
    self.check_compatible_fresh(module, strict=True)
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_import/__init__.py", line 1817, in check_compatible_fresh
    _, out, err = script_helper.assert_python_ok('-c', textwrap.dedent(f'''
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 166, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 151, in _assert_python
    res.fail(cmd_line)
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/support/script_helper.py", line 76, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is -11
command line: ['/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/python', '-X', 'faulthandler', '-I', '-c', '\nimport _testcapi, sys\nassert (\n    \'threading\' in sys.builtin_module_names or\n    \'threading\' not in sys.modules\n), repr(\'threading\')\nret = _testcapi.run_in_subinterp_with_config(\n    "\\nimport os, sys\\n\\ntry:\\n    import threading\\nexcept ImportError as exc:\\n    text = \'ImportError: \' + str(exc)\\nelse:\\n    text = \'okay\'\\nos.write(sys.stdout.fileno(), text.encode(\'utf-8\'))\\n",\n    **{\'allow_fork\': False, \'allow_exec\': False, \'allow_threads\': True, \'allow_daemon_threads\': False, \'use_main_obmalloc\': True, \'gil\': 1, \'check_multi_interp_extensions\': True},\n)\nassert ret == 0, ret\n']


TracebackTests.test_unencodable_filename) ... ok


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 188, in test_subinterps_distinct_state
    for run in self.run_repeated_init_and_subinterpreters():
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7f14a6de80a0\nobject refcount : 0\nobject type     : 0x5567decca2c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7f14a6de80a0>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x00005567dee3e908)\n\nCurrent thread 0x00007f14a7168740 (most recent call first):\n  <no Python frame>\n'


Traceback (most recent call last):
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 180, in test_subinterps_different_ids
    for run in self.run_repeated_init_and_subinterpreters():
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 119, in run_repeated_init_and_subinterpreters
    out, err = self.run_embedded_interpreter("test_repeated_init_and_subinterpreters")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/3.12.pablogsal-arch-x86_64/build/Lib/test/test_embed.py", line 113, in run_embedded_interpreter
    self.assertEqual(p.returncode, returncode,
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Objects/object.c:2231: _Py_ForgetReference: Assertion failed: invalid object chain\nEnable tracemalloc to get the memory block allocation traceback\n\nobject address  : 0x7f06ef537e80\nobject refcount : 0\nobject type     : 0x5610d3a722c0\nobject type name: str\nobject repr     : <refcnt 0 at 0x7f06ef537e80>\n\nFatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed\nPython runtime state: finalizing (tstate=0x00005610d3be6908)\n\nCurrent thread 0x00007f06ef8be740 (most recent call first):\n  <no Python frame>\n'


TracebackTests.test_broken_parent) ... ok


TracebackTests.test_exec_failure) ... ok

[bedevere-app]

GH-124814 is a backport of this pull request to the 3.12 branch.