Make CortexIngesterReachingSeriesLimit warning less sensitive

As it turns out, during normal shuffle-sharding operation, the 70% mark is often exceeded, but not by much. Therefore, this change sets the new warning mark at 75%. It also increases the `for` duration to 15m as the expected reaction time for warning alerts is usually in the order of hours, so we can as well wait a bit longer to see if the problem is transient. Signed-off-by: beorn7 <[email protected]>
grafana · Jul 26, 2021 · c6eab23 · c6eab23
1 parent 0d48edf
commit c6eab23
Showing 1 changed file with 2 additions and 2 deletions.
diff --git a/cortex-mixin/alerts/alerts.libsonnet b/cortex-mixin/alerts/alerts.libsonnet
@@ -255,9 +255,9 @@
                 (cortex_ingester_memory_series / ignoring(limit) cortex_ingester_instance_limits{limit="max_series"})
                 and ignoring (limit)
                 (cortex_ingester_instance_limits{limit="max_series"} > 0)
-            ) > 0.7
+            ) > 0.75
           |||,
-          'for': '5m',
+          'for': '15m',
           labels: {
             severity: 'warning',
           },