Add recording rules to calculate Cortex scaling #278
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
owen-d
approved these changes
Mar 18, 2021
| quantile_over_time(0.99, | ||
| sum by (cluster, namespace) ( | ||
| cluster_namespace_job:cortex_distributor_received_samples:rate5m | ||
| )[24h:] |
There was a problem hiding this comment.
Just curious, why did you chose 24h as the period here? I would have chosen a smaller one. I guess the longer the interval the safer we are 🤔
There was a problem hiding this comment.
Yeah I wanted to make them all consistent (the CPU and memory are also 24hr).
This should also help prevent them from flapping, I guess at the cost of them not responding to change quite as quickly.
tomwilkie
force-pushed
the
scaling-rules
branch
4 times, most recently
from
564a959 to
5db9c40
Compare
- Update dashboard so it only shows under provisioned services and why - Add sizing rules based on limits. - Add some docs to the dashboard. Signed-off-by: Tom Wilkie <[email protected]>
tomwilkie
force-pushed
the
scaling-rules
branch
from
5db9c40 to
fbf3f98
Compare
1 task
simonswine
pushed a commit
to grafana/mimir
that referenced
this pull request
Oct 18, 2021
Add recording rules to calculate Cortex scaling
simonswine
pushed a commit
to grafana/mimir
that referenced
this pull request
Dec 20, 2021
Add recording rules to calculate Cortex scaling
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Tom Wilkie [email protected]
What this PR does:
Extracts the queries from the scaling dashboard so we can do:
And eventually alert on this.