From 11f54955bb9accb568eb47199eb89288802cbb14 Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Mon, 21 Jun 2021 15:10:24 +0200 Subject: [PATCH] Addressed review comments Signed-off-by: Marco Pracucci --- CHANGELOG.md | 4 ++-- cortex-mixin/docs/playbooks.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 723b0287..fefd22e3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,8 +12,8 @@ * [CHANGE] Dashboards: added overridable `job_labels` and `cluster_labels` to the configuration object as label lists to uniquely identify jobs and clusters in the metric names and group-by lists in dashboards. #319 * [CHANGE] Dashboards: `alert_aggregation_labels` has been removed from the configuration and overriding this value has been deprecated. Instead the labels are now defined by the `cluster_labels` list, and should be overridden accordingly through that list. #319 * [CHANGE] Ingester/Ruler: set `-server.grpc-max-send-msg-size-bytes` and `-server.grpc-max-send-msg-size-bytes` to sensible default values (10MB). #326 -* [CHANGE] Renamed `CortexCompactorHasNotUploadedBlocksSinceStart` into `CortexCompactorHasNotUploadedBlocks`. #334 -* [CHANGE] Renamed `CortexCompactorRunFailed` into `CortexCompactorHasNotSuccessfullyRunCompaction`. #334 +* [CHANGE] Renamed `CortexCompactorHasNotUploadedBlocksSinceStart` to `CortexCompactorHasNotUploadedBlocks`. #334 +* [CHANGE] Renamed `CortexCompactorRunFailed` to `CortexCompactorHasNotSuccessfullyRunCompaction`. #334 * [ENHANCEMENT] cortex-mixin: Make `cluster_namespace_deployment:kube_pod_container_resource_requests_{cpu_cores,memory_bytes}:sum` backwards compatible with `kube-state-metrics` v2.0.0. #317 * [BUGFIX] Fixed `CortexIngesterHasNotShippedBlocks` alert false positive in case an ingester instance had ingested samples in the past, then no traffic was received for a long period and then it started receiving samples again. #308 * [BUGFIX] Alertmanager: fixed `--alertmanager.cluster.peers` CLI flag passed to alertmanager when HA is enabled. #329 diff --git a/cortex-mixin/docs/playbooks.md b/cortex-mixin/docs/playbooks.md index daf5eaed..1ad67bfe 100644 --- a/cortex-mixin/docs/playbooks.md +++ b/cortex-mixin/docs/playbooks.md @@ -272,14 +272,14 @@ Same as [`CortexCompactorHasNotSuccessfullyCleanedUpBlocks`](#CortexCompactorHas This alert fires when a Cortex compactor is not uploading any compacted blocks to the storage since a long time. How to **investigate**: -- If the alert `CortexCompactorHasNotSuccessfullyRunCompaction` have fired as well, then investigate that issue first +- If the alert `CortexCompactorHasNotSuccessfullyRunCompaction` has fired as well, then investigate that issue first - If the alert `CortexIngesterHasNotShippedBlocks` or `CortexIngesterHasNotShippedBlocksSinceStart` have fired as well, then investigate that issue first - Ensure ingesters are successfully shipping blocks to the storage - Look for any error in the compactor logs ### CortexCompactorHasNotSuccessfullyRunCompaction -This alert fires if the compactor is not able to successfully run a full compaction. +This alert fires if the compactor is not able to successfully compact all discovered compactable blocks. When this alert fires, the compactor may still have successfully compacted some blocks but, for some reason, other blocks compaction is consistently failing. A common case is when the compactor is trying to compact a corrupted block for a single tenant: in this case the compaction of blocks for other tenants is still working, but compaction for the affected tenant is blocked by the corrupted block.