Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-118074: Executors in the COLD_EXITS array are not GC'able #118117

Merged
merged 4 commits into from
Apr 22, 2024

Conversation

gvanrossum
Copy link
Member

@gvanrossum gvanrossum commented Apr 20, 2024

Implement a tp_is_gc slot that tests for this.

Implement a `tp_is_gc` slot that tests for this.
@gvanrossum
Copy link
Member Author

@adoxalim Can you verify that this fix works for you? (Thanks again for the bug report.)

static int
executor_is_gc(PyObject *o)
{
if ((PyObject *)&COLD_EXITS[0] <= o && o < (PyObject *)&COLD_EXITS[UOP_MAX_TRACE_LENGTH]) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The address sanitizer warning here seems plausibly correct:

array subscript 800 is above array bounds of ‘_PyExecutorObject[200]’ [-Warray-bounds]

COLD_EXITS has a length of UOP_MAX_TRACE_LENGTH/4. If there's some other reason this is correct, maybe add a comment / a way to suppress the warning?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, you're right. That line was written by Copilot and it looked so plausible but was so wrong. :-(

@gvanrossum
Copy link
Member Author

The emulated Linux (ARM64) test failures are known, caused by the the QEMU emulation. I'm merging now.

@gvanrossum gvanrossum merged commit 1b85b34 into python:main Apr 22, 2024
50 of 54 checks passed
@gvanrossum gvanrossum deleted the fix-cold-exit-crash branch April 22, 2024 23:20
static int
executor_is_gc(PyObject *o)
{
if ((PyObject *)&COLD_EXITS[0] <= o && o < (PyObject *)&COLD_EXITS[COLD_EXIT_COUNT]) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is undefined behavior in C. Although it will mostly work in practice.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I take it this should work?

Suggested change
if ((PyObject *)&COLD_EXITS[0] <= o && o < (PyObject *)&COLD_EXITS[COLD_EXIT_COUNT]) {
if (_Py_IsImmortal(o)) {

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a similar thing to the top of executor_traverse()? (Although, really, cold exits don't have exits, so their exit_count should always be zero, right? So why bother.

Anyway, see gh-118182 for a fix.

@markshannon
Copy link
Member

Sorry for not reviewing this earlier.

I'd rather not use tp_is_gc, as it adds quite a bit of overhead to an already slow process.

As a short term fix, we could change tp_traverse to return immediately if the executor is immortal.
Longer term, we should fix this in the GC. The GC shouldn't be visiting immortal objects.

gvanrossum added a commit that referenced this pull request Apr 23, 2024
Better version of gh-118117.
Just check for immortality instead of an address range check.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants