Skip to content

Commit

Permalink
Merge pull request grafana/cortex-jsonnet#330 from grafana/series-lim…
Browse files Browse the repository at this point in the history
…it-playbook

Add a simple playbook for ingester series limit alert.
  • Loading branch information
pracucci authored Jun 17, 2021
2 parents 77718f5 + 1c214d3 commit c59c9b6
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion jsonnet/mimir-mixin/docs/playbooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,9 @@ If nothing obvious from the above, check for increased load:

### CortexIngesterReachingSeriesLimit

_TODO: this playbook has not been written yet._
First check the writes resources dashboard and scaling dashboard. The usual target is 1.5M active series per ingester and a max of 2.5M.

Scaling up the ingesters will help, but it won't resolve the alert immediately, as series are active until the next TSDB Head compaction (every 2h or so). You may also want to temporarily increase the per ingester series limit (this is a runtime reloadable config option) until that next Head compaction occurs.

### CortexIngesterReachingTenantsLimit

Expand Down

0 comments on commit c59c9b6

Please sign in to comment.