From c6eab23ba32c52db668c0d3e6ca389718516a5ce Mon Sep 17 00:00:00 2001 From: beorn7 Date: Mon, 26 Jul 2021 19:08:00 +0200 Subject: [PATCH] Make CortexIngesterReachingSeriesLimit warning less sensitive As it turns out, during normal shuffle-sharding operation, the 70% mark is often exceeded, but not by much. Therefore, this change sets the new warning mark at 75%. It also increases the `for` duration to 15m as the expected reaction time for warning alerts is usually in the order of hours, so we can as well wait a bit longer to see if the problem is transient. Signed-off-by: beorn7 --- cortex-mixin/alerts/alerts.libsonnet | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cortex-mixin/alerts/alerts.libsonnet b/cortex-mixin/alerts/alerts.libsonnet index 9eefe7f8..4c905733 100644 --- a/cortex-mixin/alerts/alerts.libsonnet +++ b/cortex-mixin/alerts/alerts.libsonnet @@ -255,9 +255,9 @@ (cortex_ingester_memory_series / ignoring(limit) cortex_ingester_instance_limits{limit="max_series"}) and ignoring (limit) (cortex_ingester_instance_limits{limit="max_series"} > 0) - ) > 0.7 + ) > 0.75 |||, - 'for': '5m', + 'for': '15m', labels: { severity: 'warning', },