Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-108362: Incremental GC implementation #116206

Merged
merged 16 commits into from
Mar 20, 2024

Conversation

markshannon
Copy link
Member

@markshannon markshannon commented Mar 1, 2024

Implements incremental cyclic GC.
Instead of traversing one generation on each collection, we traverse the young generation and the oldest part of the old generation. By traversing the old generation a chunk at a time, we keep pause times down a lot.

See faster-cpython/ideas#613 for the idea and algorithm.


📚 Documentation preview 📚: https://cpython-previews--116206.org.readthedocs.build/

@bedevere-app bedevere-app bot mentioned this pull request Mar 1, 2024
@markshannon
Copy link
Member Author

markshannon commented Mar 4, 2024

The build failures appear to be caused by #114751, so I think this is ready for review.

I believe the reason that the earlier version of this PR needed reverting was that it relied on the young gen being empty after a young collection. However if a finalizer creates new objects, then that assumption is false.

This version merges the young gen and increment before collection, so we don't care if the young gen isn't empty afterwards.

@markshannon markshannon added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 5, 2024
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 912ebc1 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 5, 2024
@markshannon
Copy link
Member Author

!buildbot bigmem

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 912ebc1 🤖

The command will test the builders whose names match following regular expression: bigmem

The builders matched are:

  • AMD64 Windows11 Bigmem PR

@markshannon markshannon added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 7, 2024
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 3c1ef42 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 7, 2024
@markshannon
Copy link
Member Author

!buildbot bigmem

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @markshannon for commit 3c1ef42 🤖

The command will test the builders whose names match following regular expression: bigmem

The builders matched are:

  • AMD64 Windows11 Bigmem PR

@markshannon markshannon marked this pull request as ready for review March 8, 2024 10:07
@markshannon markshannon requested review from DinoV and removed request for kumaraditya303 March 11, 2024 09:28
@markshannon
Copy link
Member Author

@DinoV
It looks like the progress calculation got reverted to the prior, incorrect, version when I merged in the move of the GC code from gcmodule.c to gc.c.

I've fixed that and added a test based on your example.
Should be good to go now.

@markshannon
Copy link
Member Author

!buildbot bigmem

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @markshannon for commit f4f04d6 🤖

The command will test the builders whose names match following regular expression: bigmem

The builders matched are:

  • AMD64 Windows11 Bigmem PR

Copy link
Contributor

@DinoV DinoV left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM now!

@markshannon
Copy link
Member Author

Note on the test failures:

  • JIT / aarch64-unknown-linux-gnu/gcc (Debug) (pull_request) passes ~450 out of 460 tests then is timed out after an hour. I'm considering that a pass.
  • Tests / CIFuzz (memory) (pull_request) is just broken and fails on everything at the moment.

@ericsnowcurrently
Copy link
Member

This may have introduced a performance regression. See gh-117108.

gvanrossum added a commit to gvanrossum/cpython that referenced this pull request Mar 23, 2024
Second half of the revert.

This reverts commit 1530932.
adorilson pushed a commit to adorilson/cpython that referenced this pull request Mar 25, 2024
diegorusso pushed a commit to diegorusso/cpython that referenced this pull request Apr 17, 2024
abhinav-upadhyay added a commit to abhinav-upadhyay/cpython that referenced this pull request May 16, 2024
In pythonGH-116206, the comment about moving reachable objects to next generation got moved from its original place to a place where there is no code below it. This PR puts it back where the actual movement of rechable objects happen to the next generation.
@markshannon markshannon deleted the incremental-gc-3 branch August 6, 2024 10:18
Yhg1s added a commit to Yhg1s/cpython that referenced this pull request Sep 29, 2024
Yhg1s added a commit to Yhg1s/cpython that referenced this pull request Sep 29, 2024
Yhg1s added a commit to Yhg1s/cpython that referenced this pull request Sep 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants