gh-112069: Add _PySet_NextEntryRef to be thread-safe.

[corona10]

• Issue: Make set thread-safe in --disable-gil builds #112069

• Add _PySet_NextEntryRef to be thread-safe.
• Use FrozenSet if possible to use _PySet_NextEntry without locking.

[corona10]

Tests / Thread sanitizer (free-threading) / Thread sanitizer (pull_request)

OKay, it's too slow.

[colesbury]

I'm not sure what the right approach here is, but as written _PySet_NextEntry still returns a borrowed reference, so the cases that were not thread-safe are probably still not thread-safe.

I don't think it makes sense to try to make individual calls to _PySet_NextEntry thread-safe. The whole loop of while(_PySet_NextEntry(...)) is generally what needs to be thread-safe. For example, the comment in set_next says:

cpython/Objects/setobject.c

Lines 477 to 478 in fccedbd

	* CAUTION: In general, it isn't safe to use set_next in a loop that
	* mutates the table.

I think we should add locking in the use sites of _PySet_NextEntry as necessary. Most look like they're already okay to me: either they already do locking (listobject.c and dictobject.c) or operate on private or immutable data (pylifecycle.c, codeobject.c).

The two cases that concern me and may need locking are:

• _pickle.c
• marshal.c

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @corona10 for commit 5d57a56 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

[corona10]

buildbot/AMD64 RHEL7 Refleaks PR: Unrelated

In file included from ./Include/internal/mimalloc/mimalloc/types.h:24:0,
                 from ./Include/internal/pycore_mimalloc.h:40,
                 from ./Include/internal/pycore_interp.h:30,
                 from ./Include/internal/pycore_runtime.h:17,
                 from ./Include/internal/pycore_pystate.h:12,
                 from Parser/myreadline.c:14:
./Include/internal/mimalloc/mimalloc/atomic.h:39:23: fatal error: stdatomic.h: No such file or directory
 #include <stdatomic.h>
                       ^
compilation terminated.
In file included from ./Include/internal/mimalloc/mimalloc/types.h:24:0,
                 from ./Include/internal/pycore_mimalloc.h:40,
                 from ./Include/internal/pycore_interp.h:30,
                 from ./Include/internal/pycore_runtime.h:17,
                 from ./Include/internal/pycore_long.h:13,
                 from Objects/boolobject.c:4:
./Include/internal/mimalloc/mimalloc/atomic.h:39:23: fatal error: stdatomic.h: No such file or directory
 #include <stdatomic.h>
                       ^

[colesbury]

• We should not change the reference count behavior of _PySet_NextEntry as it's exposed publicly and used outside of CPython (like in Cython). If needed, add a new function _PySet_NextEntryRef.
• I don't think _PyFrozenSet_NextEntry is needed. _PySet_NextEntry (or _PySet_NextEntryRef) looks sufficient.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @corona10 for commit 99fb7e6 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @corona10 for commit 53f949a 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @corona10 for commit 99fb7e6 🤖

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

[corona10]

test_compile leaked [2, 2, 2] references, sum=6
test_compile leaked [4, 4, 4] memory blocks, sum=12

Need to check.

[corona10]

test_compile: #118023
test_logging: #102412 (comment)

So leaks are not related to this PR.