Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pageserver: add separate semaphore for L0 compaction #10780

Merged
merged 1 commit into from
Feb 12, 2025

Conversation

erikgrinaker
Copy link
Contributor

@erikgrinaker erikgrinaker commented Feb 12, 2025

Problem

L0 compaction frequently gets starved out by other background tasks and image/GC compaction. L0 compaction must be responsive to keep read amplification under control.

Touches #10694.
Resolves #10689.

Summary of changes

Use a separate semaphore for the L0-only compaction pass.

  • Add a CONCURRENT_L0_COMPACTION_TASKS semaphore and BackgroundLoopKind::L0Compaction.
  • Add a setting compaction_l0_semaphore (default off via compaction_l0_first).
  • Use the L0 semaphore when doing an OnlyL0Compaction pass.
  • Use the background semaphore when doing a regular compaction pass (which includes an initial L0 pass).
  • While waiting for the background semaphore, yield for L0 compaction if triggered.
  • Add CompactFlags::NoYield to disable L0 yielding, and set it for the HTTP API route.
  • Remove the old use_compaction_semaphore setting and compaction-scoped semaphore.
  • Remove the warning when waiting for a semaphore; it's noisy and we have metrics.

Copy link

github-actions bot commented Feb 12, 2025

7458 tests run: 7078 passed, 0 failed, 380 skipped (full report)


Flaky tests (1)

Postgres 16

Code coverage* (full report)

  • functions: 33.2% (8581 of 25870 functions)
  • lines: 49.1% (72317 of 147387 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
b01c54b at 2025-02-12T14:17:31.827Z :recycle:

@erikgrinaker erikgrinaker force-pushed the erik/l0-compaction-semaphore branch 2 times, most recently from 77e5582 to 240f6ff Compare February 12, 2025 13:09
@erikgrinaker
Copy link
Contributor Author

I ran this through CI enabled by default, which passed. Switched it back to disabled by default for now.

@erikgrinaker erikgrinaker force-pushed the erik/l0-compaction-semaphore branch from 240f6ff to b01c54b Compare February 12, 2025 13:15
@erikgrinaker erikgrinaker added this pull request to the merge queue Feb 12, 2025
Merged via the queue into main with commit f62047a Feb 12, 2025
86 checks passed
@erikgrinaker erikgrinaker deleted the erik/l0-compaction-semaphore branch February 12, 2025 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

pageserver: image creation scheduling should consider global status
2 participants