Make set thread-safe in --disable-gil builds

[colesbury]

Feature or enhancement

The set object is not currently thread-safe in --disable-gil builds. We should make it thread-safe by using the "critical section" API. to acquire the per-object locks around operations. There should be no effective change in the default build (i.e., with GIL) because critical sections are no-ops in the default build.

Notes:

1. Unlike dict and list, I don't think it's worth the complexity to try to "optimistically avoid locking" around any set operation (except set_len). We could consider doing this in the future if there is a performance justification, but not for 3.13.
2. set_len can avoid locking and instead use relaxed atomics for reading the "used" field. Note that writes to "used" should then also use relaxed atomics.
3. Some operations require locking two containers (like set_merge). Some of these will need refactorings so that the critical sections macros can be added in the correct places.

For context, here is the change from the nogil-3.12 fork: colesbury/nogil-3.12@4ca2924f0d. Note that the critical section API is slightly changed in 3.13 from nogil-3.12; In 3.13 Py_BEGIN_CRITICAL_SECTION takes a PyObject instead of a PyMutex.

TODO:

• Improve set_init (see gh-112069: Make sets thread-safe with the GIL disabled #113800 (comment)). We also want to avoid locking in set_init if possible
• _PySet_NextEntry
• setiter_iternext

Linked PRs

• gh-112069: Make sets thread-safe with the GIL disabled #113800
• gh-112069: Adapt set/frozenset methods to Argument Clinic #115112
• Revert "gh-112069: Make sets thread-safe with the GIL disabled (#113800)" #116525
• gh-112069: Make setiter_iternext to be thread-safe #117935
• gh-112069: Add _PySet_NextEntryRef to be thread-safe. #117990
• gh-112069: Make PySet_GET_SIZE to be atomic safe. #118053
• gh-112069: Do not require lock if the set has never been exposed. #118069

[tomasr8]

Hi again! This one looks more challenging than the hashlib one but I'd like to try anyway :)

[colesbury]

Thanks @tomasr8!

[DinoV]

hi @tomasr8, just curious if you're still planning on working on this?

[tomasr8]

Hi! Yes, I took a break for Christmas but I have it almost working, just need to fix some tests. I could open a draft PR this weekend

[corona10]

@colesbury I am going to work on _PySet_NextEntry this week, considering the caller-side usage.

[corona10]

@colesbury Now we can close the issue right? Or more thing is left?

[colesbury]

Yeah, I think so