Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pageserver: generate image layers at L0/in-mem boundary (or L0/L1 boundary) #11063

Open
skyzh opened this issue Mar 3, 2025 · 0 comments
Open
Assignees
Labels
t/feature Issue type: feature, for new features or requests

Comments

@skyzh
Copy link
Member

skyzh commented Mar 3, 2025

previous attempt failed #10548 because there are cases we didn't consider, which caused errors in staging:

  • What if the LSN we selected doesn't contain any data (below all layers generated), we will get an error immediately saying DB_DIR key not exist.
  • What if we select an LSN to generate image layers that is below an existing image layer, the we might get a read error in the middle if GC decides to remove layers below the existing image layer.

Currently, we generate image layers at last_record_lsn, which causes some problems,

  • Future layers. So we would put an image layer that is above the remote/disk_consistent_lsn into the layer map, which adds a lot of complexity in the code path.
  • Overlap between in-memory and image layers needs extra work to support, and pageserver: handle in-memory layer overlaps with persistent layers #11000 fixed the case.
  • Make image layer generation more costly, if we have a lot of L0 layers between last_record_lsn and L0/L1 boundary. Every read will go through a lot of layers, versus we could have done a L0 compaction and then do image generation.

I'd like to continue to move image layer generation to a lower LSN and I think it's beneficial to do so.

@skyzh skyzh self-assigned this Mar 3, 2025
@skyzh skyzh added the t/feature Issue type: feature, for new features or requests label Mar 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
t/feature Issue type: feature, for new features or requests
Projects
None yet
Development

No branches or pull requests

1 participant