admit_long: prototype a granter for long-lived work #141501
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The granter is initially for Pebble compactions and snapshot ingests and is aware of node CPU and per-store disk bandwidth consumption. It monitors the CPU and write bandwidth of this work (read bandwidth is not explicitly modeled, but the observation of aggregate read bandwidth can correct for this oversight). The granter is integrated with MultiQueue (for snapshot ingest) such that the semaphore is generalized. It is also integrated with Pebble. cockroachdb#136615 (comment) has some experimental results. The granter can also run in a non-resource driven mode, in which case it would simply apply fixed limits on the number of compactions (or snapshot) per store and per node. There is currently no rate limiting of long-lived work, once admitted. Such rate limiting may be necessary for production deployments. Informs cockroachdb#136615 Epic: none Release note: None
![blathers[crl]](https://avatars.githubusercontent.com/in/59700?s=40&u=373d9101a12edc3847d2fb6ae78e3adcfb4ca41e&v=4){kind=avatar}
|
Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
|
This change is |
|
This is a prototype so teams other than admission-control and storage should ignore it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The granter is initially for Pebble compactions and snapshot ingests and is aware of node CPU and per-store disk bandwidth consumption. It monitors the CPU and write bandwidth of this work (read bandwidth is not explicitly modeled, but the observation of aggregate read bandwidth can correct for this oversight).
The granter is integrated with MultiQueue (for snapshot ingest) such that the semaphore is generalized. It is also integrated with Pebble.
#136615 (comment) has some experimental results.
The granter can also run in a non-resource driven mode, in which case it would simply apply fixed limits on the number of compactions (or snapshot) per store and per node.
There is currently no rate limiting of long-lived work, once admitted. Such rate limiting may be necessary for production deployments.
Informs #136615
Epic: none
Release note: None