Relaxed the hash ring heartbeat period and timeout for distributor, i…

…ngester, store-gateway and compactor. These are values which help reduce the pressure on a KV store or reduce the CPU spent by memberlist in passing messages. The tradeoff is that abrupt shutdowns/crashes of components will take longer to detect by peers. We've been running with these values at Grafana Labs for some time and haven't seen problems. Signed-off-by: Dimitar Dimitrov <[email protected]> Signed-off-by: Marco Pracucci <[email protected]>
grafana · Jan 17, 2024 · a677f2f · a677f2f
1 parent ac3cb58
commit a677f2f
Show file tree

Hide file tree

Showing 79 changed files with 766 additions and 5 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -63,6 +63,14 @@
 * [CHANGE] rollout-operator: remove default CPU limit. #7066
 * [CHANGE] Store-gateway: Increase `JAEGER_REPORTER_MAX_QUEUE_SIZE` from the default (100) to 1000, to avoid dropping tracing spans. #7068
 * [CHANGE] Query-frontend, ingester, ruler, backend and write instances: Increase `JAEGER_REPORTER_MAX_QUEUE_SIZE` from the default (100), to avoid dropping tracing spans. #7086
+* [CHANGE] Ring: relaxed the hash ring heartbeat period and timeout for distributor, ingester, store-gateway and compactor: #6860
+  * `-distributor.ring.heartbeat-period` set to `1m`
+  * `-distributor.ring.heartbeat-timeout` set to `4m`
+  * `-ingester.ring.heartbeat-period` set to `2m`
+  * `-store-gateway.sharding-ring.heartbeat-period` set to `1m`
+  * `-store-gateway.sharding-ring.heartbeat-timeout` set to `4m`
+  * `-compactor.ring.heartbeat-period` set to `1m`
+  * `-compactor.ring.heartbeat-timeout` set to `4m`
 * [FEATURE] Added support for the following root-level settings to configure the list of matchers to apply to node affinity: #6782 #6829
   * `alertmanager_node_affinity_matchers`
   * `compactor_node_affinity_matchers`

diff --git a/operations/compare-helm-with-jsonnet/components/config/write-path-config.yaml b/operations/compare-helm-with-jsonnet/components/config/write-path-config.yaml
@@ -7,7 +7,6 @@ config:
     ring:
       instance_availability_zone:
       num_tokens:
-      heartbeat_timeout:
       unregister_on_shutdown:
   distributor:
     ha_tracker:

diff --git a/operations/helm/charts/mimir-distributed/CHANGELOG.md b/operations/helm/charts/mimir-distributed/CHANGELOG.md
@@ -29,6 +29,15 @@ Entries should include a reference to the Pull Request that introduced the chang
 ## main / unreleased
 
 * [CHANGE] Rollout-operator: remove default CPU limit. #7125
+* [CHANGE] Ring: relaxed the hash ring heartbeat period and timeout for distributor, ingester, store-gateway and compactor: #6860
+  * `-distributor.ring.heartbeat-period` set to `1m`
+  * `-distributor.ring.heartbeat-timeout` set to `4m`
+  * `-ingester.ring.heartbeat-period` set to `2m`
+  * `-ingester.ring.heartbeat-timeout` set to `10m`
+  * `-store-gateway.sharding-ring.heartbeat-period` set to `1m`
+  * `-store-gateway.sharding-ring.heartbeat-timeout` set to `4m`
+  * `-compactor.ring.heartbeat-period` set to `1m`
+  * `-compactor.ring.heartbeat-timeout` set to `4m`
 * [ENHANCEMENT] Add `jaegerReporterMaxQueueSize` Helm value for all components where configuring `JAEGER_REPORTER_MAX_QUEUE_SIZE` makes sense, and override the Jaeger client's default value of 100 for components expected to generate many trace spans. #7068 #7086
 * [ENHANCEMENT] Rollout-operator: upgraded to v0.10.1. #7125
 * [ENHANCEMENT] Query-frontend: configured `-shutdown-delay`, `-server.grpc.keepalive.max-connection-age` and termination grace period to reduce the likelihood of queries hitting terminated query-frontends. #7129

diff --git a/operations/helm/charts/mimir-distributed/values.yaml b/operations/helm/charts/mimir-distributed/values.yaml
@@ -231,6 +231,13 @@ mimir:
       data_dir: "/data"
       sharding_ring:
         wait_stability_min_duration: 1m
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
+
+    distributor:
+      sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
 
     frontend:
       parallelize_shardable_queries: true
@@ -292,6 +299,8 @@ mimir:
         num_tokens: 512
         tokens_file_path: /data/tokens
         unregister_on_shutdown: false
+        heartbeat_period: 2m
+        heartbeat_timeout: 10m
         {{- if .Values.ingester.zoneAwareReplication.enabled }}
         zone_awareness_enabled: true
         {{- end }}
@@ -376,6 +385,8 @@ mimir:
         {{- if .Values.store_gateway.zoneAwareReplication.enabled }}
         kvstore:
           prefix: multi-zone/
+          heartbeat_period: 1m
+          heartbeat_timeout: 4m
         {{- end }}
         tokens_file_path: /data/tokens
         unregister_on_shutdown: false

diff --git a/...elm/tests/enterprise-https-values-generated/mimir-distributed/templates/mimir-config.yaml b/...elm/tests/enterprise-https-values-generated/mimir-distributed/templates/mimir-config.yaml
@@ -91,8 +91,14 @@ data:
       max_closing_blocks_concurrency: 2
       max_opening_blocks_concurrency: 4
       sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
         wait_stability_min_duration: 1m
       symbols_flushers_concurrency: 4
+    distributor:
+      sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
     frontend:
       cache_results: true
       grpc_client_config:
@@ -192,6 +198,8 @@ data:
     ingester:
       ring:
         final_sleep: 0s
+        heartbeat_period: 2m
+        heartbeat_timeout: 10m
         kvstore:
           store: memberlist
         num_tokens: 512
@@ -292,6 +300,8 @@ data:
     store_gateway:
       sharding_ring:
         kvstore:
+          heartbeat_period: 1m
+          heartbeat_timeout: 4m
           prefix: multi-zone/
         tokens_file_path: /data/tokens
         unregister_on_shutdown: false

diff --git a/...m/tests/gateway-enterprise-values-generated/mimir-distributed/templates/mimir-config.yaml b/...m/tests/gateway-enterprise-values-generated/mimir-distributed/templates/mimir-config.yaml
@@ -67,8 +67,14 @@ data:
       max_closing_blocks_concurrency: 2
       max_opening_blocks_concurrency: 4
       sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
         wait_stability_min_duration: 1m
       symbols_flushers_concurrency: 4
+    distributor:
+      sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
     frontend:
       parallelize_shardable_queries: true
       scheduler_address: gateway-enterprise-values-mimir-query-scheduler-headless.citestns.svc:9095
@@ -99,6 +105,8 @@ data:
     ingester:
       ring:
         final_sleep: 0s
+        heartbeat_period: 2m
+        heartbeat_timeout: 10m
         num_tokens: 512
         tokens_file_path: /data/tokens
         unregister_on_shutdown: false
@@ -147,6 +155,8 @@ data:
     store_gateway:
       sharding_ring:
         kvstore:
+          heartbeat_period: 1m
+          heartbeat_timeout: 4m
           prefix: multi-zone/
         tokens_file_path: /data/tokens
         unregister_on_shutdown: false

diff --git a/...s/helm/tests/gateway-nginx-values-generated/mimir-distributed/templates/mimir-config.yaml b/...s/helm/tests/gateway-nginx-values-generated/mimir-distributed/templates/mimir-config.yaml
@@ -49,8 +49,14 @@ data:
       max_closing_blocks_concurrency: 2
       max_opening_blocks_concurrency: 4
       sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
         wait_stability_min_duration: 1m
       symbols_flushers_concurrency: 4
+    distributor:
+      sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
     frontend:
       parallelize_shardable_queries: true
       scheduler_address: gateway-nginx-values-mimir-query-scheduler-headless.citestns.svc:9095
@@ -61,6 +67,8 @@ data:
     ingester:
       ring:
         final_sleep: 0s
+        heartbeat_period: 2m
+        heartbeat_timeout: 10m
         num_tokens: 512
         tokens_file_path: /data/tokens
         unregister_on_shutdown: false
@@ -103,6 +111,8 @@ data:
     store_gateway:
       sharding_ring:
         kvstore:
+          heartbeat_period: 1m
+          heartbeat_timeout: 4m
           prefix: multi-zone/
         tokens_file_path: /data/tokens
         unregister_on_shutdown: false

diff --git a/...elm/tests/graphite-enabled-values-generated/mimir-distributed/templates/mimir-config.yaml b/...elm/tests/graphite-enabled-values-generated/mimir-distributed/templates/mimir-config.yaml
@@ -67,8 +67,14 @@ data:
       max_closing_blocks_concurrency: 2
       max_opening_blocks_concurrency: 4
       sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
         wait_stability_min_duration: 1m
       symbols_flushers_concurrency: 4
+    distributor:
+      sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
     frontend:
       parallelize_shardable_queries: true
       scheduler_address: graphite-enabled-values-mimir-query-scheduler-headless.citestns.svc:9095
@@ -123,6 +129,8 @@ data:
     ingester:
       ring:
         final_sleep: 0s
+        heartbeat_period: 2m
+        heartbeat_timeout: 10m
         num_tokens: 512
         tokens_file_path: /data/tokens
         unregister_on_shutdown: false

diff --git a/operations/helm/tests/large-values-generated/mimir-distributed/templates/mimir-config.yaml b/operations/helm/tests/large-values-generated/mimir-distributed/templates/mimir-config.yaml
@@ -54,8 +54,14 @@ data:
       max_closing_blocks_concurrency: 2
       max_opening_blocks_concurrency: 4
       sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
         wait_stability_min_duration: 1m
       symbols_flushers_concurrency: 4
+    distributor:
+      sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
     frontend:
       cache_results: true
       parallelize_shardable_queries: true
@@ -74,6 +80,8 @@ data:
     ingester:
       ring:
         final_sleep: 0s
+        heartbeat_period: 2m
+        heartbeat_timeout: 10m
         num_tokens: 512
         tokens_file_path: /data/tokens
         unregister_on_shutdown: false
@@ -114,6 +122,8 @@ data:
     store_gateway:
       sharding_ring:
         kvstore:
+          heartbeat_period: 1m
+          heartbeat_timeout: 4m
           prefix: multi-zone/
         tokens_file_path: /data/tokens
         unregister_on_shutdown: false

diff --git a/.../helm/tests/metamonitoring-values-generated/mimir-distributed/templates/mimir-config.yaml b/.../helm/tests/metamonitoring-values-generated/mimir-distributed/templates/mimir-config.yaml
@@ -49,8 +49,14 @@ data:
       max_closing_blocks_concurrency: 2
       max_opening_blocks_concurrency: 4
       sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
         wait_stability_min_duration: 1m
       symbols_flushers_concurrency: 4
+    distributor:
+      sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
     frontend:
       parallelize_shardable_queries: true
       scheduler_address: metamonitoring-values-mimir-query-scheduler-headless.citestns.svc:9095
@@ -61,6 +67,8 @@ data:
     ingester:
       ring:
         final_sleep: 0s
+        heartbeat_period: 2m
+        heartbeat_timeout: 10m
         num_tokens: 512
         tokens_file_path: /data/tokens
         unregister_on_shutdown: false

diff --git a/...tions/helm/tests/openshift-values-generated/mimir-distributed/templates/mimir-config.yaml b/...tions/helm/tests/openshift-values-generated/mimir-distributed/templates/mimir-config.yaml
@@ -45,8 +45,14 @@ data:
       max_closing_blocks_concurrency: 2
       max_opening_blocks_concurrency: 4
       sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
         wait_stability_min_duration: 1m
       symbols_flushers_concurrency: 4
+    distributor:
+      sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
     frontend:
       parallelize_shardable_queries: true
       scheduler_address: openshift-values-mimir-query-scheduler-headless.citestns.svc:9095
@@ -77,6 +83,8 @@ data:
     ingester:
       ring:
         final_sleep: 0s
+        heartbeat_period: 2m
+        heartbeat_timeout: 10m
         num_tokens: 512
         tokens_file_path: /data/tokens
         unregister_on_shutdown: false
@@ -117,6 +125,8 @@ data:
     store_gateway:
       sharding_ring:
         kvstore:
+          heartbeat_period: 1m
+          heartbeat_timeout: 4m
           prefix: multi-zone/
         tokens_file_path: /data/tokens
         unregister_on_shutdown: false

diff --git a/.../helm/tests/scheduler-name-values-generated/mimir-distributed/templates/mimir-config.yaml b/.../helm/tests/scheduler-name-values-generated/mimir-distributed/templates/mimir-config.yaml
@@ -34,8 +34,14 @@ data:
       max_closing_blocks_concurrency: 2
       max_opening_blocks_concurrency: 4
       sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
         wait_stability_min_duration: 1m
       symbols_flushers_concurrency: 4
+    distributor:
+      sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
     frontend:
       parallelize_shardable_queries: true
       scheduler_address: scheduler-name-values-mimir-query-scheduler-headless.citestns.svc:9095
@@ -46,6 +52,8 @@ data:
     ingester:
       ring:
         final_sleep: 0s
+        heartbeat_period: 2m
+        heartbeat_timeout: 10m
         num_tokens: 512
         tokens_file_path: /data/tokens
         unregister_on_shutdown: false
@@ -80,6 +88,8 @@ data:
     store_gateway:
       sharding_ring:
         kvstore:
+          heartbeat_period: 1m
+          heartbeat_timeout: 4m
           prefix: multi-zone/
         tokens_file_path: /data/tokens
         unregister_on_shutdown: false

diff --git a/operations/helm/tests/small-values-generated/mimir-distributed/templates/mimir-config.yaml b/operations/helm/tests/small-values-generated/mimir-distributed/templates/mimir-config.yaml
@@ -54,8 +54,14 @@ data:
       max_closing_blocks_concurrency: 2
       max_opening_blocks_concurrency: 4
       sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
         wait_stability_min_duration: 1m
       symbols_flushers_concurrency: 4
+    distributor:
+      sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
     frontend:
       cache_results: true
       parallelize_shardable_queries: true
@@ -74,6 +80,8 @@ data:
     ingester:
       ring:
         final_sleep: 0s
+        heartbeat_period: 2m
+        heartbeat_timeout: 10m
         num_tokens: 512
         tokens_file_path: /data/tokens
         unregister_on_shutdown: false
@@ -114,6 +122,8 @@ data:
     store_gateway:
       sharding_ring:
         kvstore:
+          heartbeat_period: 1m
+          heartbeat_timeout: 4m
           prefix: multi-zone/
         tokens_file_path: /data/tokens
         unregister_on_shutdown: false

diff --git a/.../test-enterprise-configmap-values-generated/mimir-distributed/templates/mimir-config.yaml b/.../test-enterprise-configmap-values-generated/mimir-distributed/templates/mimir-config.yaml
@@ -67,8 +67,14 @@ data:
       max_closing_blocks_concurrency: 2
       max_opening_blocks_concurrency: 4
       sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
         wait_stability_min_duration: 1m
       symbols_flushers_concurrency: 4
+    distributor:
+      sharding_ring:
+        heartbeat_period: 1m
+        heartbeat_timeout: 4m
     frontend:
       parallelize_shardable_queries: true
       scheduler_address: test-enterprise-configmap-values-mimir-query-scheduler-headless.citestns.svc:9095
@@ -99,6 +105,8 @@ data:
     ingester:
       ring:
         final_sleep: 0s
+        heartbeat_period: 2m
+        heartbeat_timeout: 10m
         num_tokens: 512
         tokens_file_path: /data/tokens
         unregister_on_shutdown: false