Open circuit breakers on timeouts and per-instance limit errors only #7310
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Yuri Nikolic <[email protected]>
jhalterman
approved these changes
Feb 6, 2024
Signed-off-by: Yuri Nikolic <[email protected]>
I have updated the changelog and merged. Thank you @jhalterman |
beatkind
pushed a commit
to beatkind/mimir
that referenced
this pull request
Feb 13, 2024
…rafana#7310) * Open circuit breakers on timeouts and per-instance limit errors only Signed-off-by: Yuri Nikolic <[email protected]> * Update CHANGELOG Signed-off-by: Yuri Nikolic <[email protected]> --------- Signed-off-by: Yuri Nikolic <[email protected]>
dimitarvdimitrov
added a commit
that referenced
this pull request
Feb 13, 2024
* feat(helm): Adding KEDA autoscaling support * fix: update changelog * feat(helm): Porting the changes from #6971 into helm chart * feat: Adding better changelog & values documentation outlining the experimental state of the feature, using the same autoscaling rules & behaviours as jsonnet, only supporting newest version of KEDA * fix: Remove duplicate query field, add base url in CHANGELOG.md * helm: align grpc server connection lifetime settings with jsonnet (#7269) * helm: align grpc server connection lifetime settings with jsonnet Co-authored-by: Marco Pracucci <[email protected]> Signed-off-by: Vladimir Varankin <[email protected]> * helm: rebuild tests Signed-off-by: Vladimir Varankin <[email protected]> --------- Signed-off-by: Vladimir Varankin <[email protected]> Co-authored-by: Marco Pracucci <[email protected]> * querymiddleware: Fix race condition in shardActiveSeriesMiddleware (#7290) * querymiddleware: race condition in shardActiveSeriesMiddleware s2.Writer is not goroutine-safe to reuse between concurrent requests. Signed-off-by: Vladimir Varankin <[email protected]> * querymiddleware: remove flaky ResponseBodyStreamed test from shardActiveSeriesMiddleware --------- Signed-off-by: Vladimir Varankin <[email protected]> * version: add UserAgent() (#7264) Update pkg/util/version/info.go Update pkg/mimirtool/client/client.go Signed-off-by: Vladimir Varankin <[email protected]> Co-authored-by: Andy Asp <[email protected]> * helm: remove -server.grpc.keepalive.max-connection-idle from common config (#7298) Signed-off-by: Vladimir Varankin <[email protected]> * Compactor: export estimated number of compaction jobs based on bucket-index (#7299) * Compute number of compaction jobs from bucket index and export it via cortex_bucket_index_compaction_jobs metric. Signed-off-by: Peter Štibraný <[email protected]> * Add PR number. Signed-off-by: Peter Štibraný <[email protected]> * Remove unused parameter name. Signed-off-by: Peter Štibraný <[email protected]> * Make linter happy. Signed-off-by: Peter Štibraný <[email protected]> * Address review feedback. Signed-off-by: Peter Štibraný <[email protected]> * Fix tests. Signed-off-by: Peter Štibraný <[email protected]> --------- Signed-off-by: Peter Štibraný <[email protected]> * Add KubePersistentVolumeFillingUp runbook (#7297) * Add KubePersistentVolumeFillingUp runbook Co-authored-by: Arve Knudsen <[email protected]> Signed-off-by: Marco Pracucci <[email protected]> * Added CHANGELOG entry Signed-off-by: Marco Pracucci <[email protected]> * Fixed linter Signed-off-by: Marco Pracucci <[email protected]> --------- Signed-off-by: Marco Pracucci <[email protected]> Co-authored-by: Arve Knudsen <[email protected]> * Internal: remove unnecessary parameter to NoCompactionMarkFilter (#7301) * Remove unnecessary parameter to NewNoCompactionMarkFilter. Signed-off-by: Peter Štibraný <[email protected]> * Remove unnecessary parameter to NewNoCompactionMarkFilter. Signed-off-by: Peter Štibraný <[email protected]> --------- Signed-off-by: Peter Štibraný <[email protected]> * Name query metrics for easier discovery (#7302) Changes the way query metrics are named to make them easier to search for. Signed-off-by: Nick Pillitteri <[email protected]> * fix(deps): update module github.com/aws/aws-sdk-go to v1.50.11 (#7288) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * fix(deps): update module github.com/klauspost/compress to v1.17.6 (#7291) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * chore(deps): update anchore/sbom-action action to v0.15.8 (#7286) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * chore(deps): update grafana/agent docker tag to v0.39.2 (#7287) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * chore(deps): update grafana/grafana docker tag to v10.3.1 (#7292) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * fix(deps): update module github.com/failsafe-go/failsafe-go to v0.4.4 (#7289) Signed-off-by: Arve Knudsen <[email protected]> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Chore: removed unused parameter from GenerateBlockFromSpec() (#7303) * Chore: removed unused parameter from GenerateBlockFromSpec() Signed-off-by: Marco Pracucci <[email protected]> * Removed unused variables Signed-off-by: Marco Pracucci <[email protected]> --------- Signed-off-by: Marco Pracucci <[email protected]> * Update mimir-prometheus (#7293) * Update mimir-prometheus Signed-off-by: Marco Pracucci <[email protected]> * Do not shard histogram_avg() Signed-off-by: Marco Pracucci <[email protected]> * Re-vendored Signed-off-by: Marco Pracucci <[email protected]> * Fix TestBucketStore_Series_ShouldQueryBlockWithOutOfOrderChunks Signed-off-by: Marco Pracucci <[email protected]> * Fix TestGroupCompactE2E Signed-off-by: Marco Pracucci <[email protected]> * Fixed TestMultitenantCompactor_OutOfOrderCompaction Signed-off-by: Marco Pracucci <[email protected]> * Reworked TestBucketStore_Series_ShouldQueryBlockWithOutOfOrderChunks Signed-off-by: Marco Pracucci <[email protected]> * Simplify TestBucketStore_Series_ShouldQueryBlockWithOutOfOrderChunks Signed-off-by: Marco Pracucci <[email protected]> * Use filepath.Join() instead of hardcoding path separator Signed-off-by: Marco Pracucci <[email protected]> --------- Signed-off-by: Marco Pracucci <[email protected]> * Release mimir-distributed Helm chart 5.3.0-weekly.276 (#7294) * Update mimir-distributed chart to 5.2.0-weekly.276 * Update version to 5.3.0 Signed-off-by: Dimitar Dimitrov <[email protected]> --------- Signed-off-by: Dimitar Dimitrov <[email protected]> Co-authored-by: grafanabot <[email protected]> Co-authored-by: Dimitar Dimitrov <[email protected]> * Open circuit breakers on timeouts and per-instance limit errors only (#7310) * Open circuit breakers on timeouts and per-instance limit errors only Signed-off-by: Yuri Nikolic <[email protected]> * Update CHANGELOG Signed-off-by: Yuri Nikolic <[email protected]> --------- Signed-off-by: Yuri Nikolic <[email protected]> * Get rid of iterators.chunkIterator and iterators.chunkMergeIterator (#7313) * Get rid of iterators.chunkIterator and iterators.chunkMergeIterator Signed-off-by: Yuri Nikolic <[email protected]> * Fixing review findings Signed-off-by: Yuri Nikolic <[email protected]> --------- Signed-off-by: Yuri Nikolic <[email protected]> * Compactor: Language fixes (#7315) Signed-off-by: Arve Knudsen <[email protected]> * Do not register compat metrics in mimirtool (#7314) This commit does the same as we did for amtool in prometheus/alertmanager#3713. There is no need to register these metrics, so we use compat.NewMetrics(nil) instead of compat.RegisteredMetrics. * Compactor: Un-export symbols that don't need to be exported (#7317) Signed-off-by: Arve Knudsen <[email protected]> * Circuit breakers: add client.ErrCircuitBreakerOpen type (#7324) Signed-off-by: Yuri Nikolic <[email protected]> * Add mimirpb.CIRCUIT_BREAKER_OPEN error cause (#7330) * Add mimirpb.CIRCUIT_BREAKER_OPEN error cause Signed-off-by: Yuri Nikolic <[email protected]> * Fixing review findings Signed-off-by: Yuri Nikolic <[email protected]> --------- Signed-off-by: Yuri Nikolic <[email protected]> * store-gateway: remove cortex_bucket_store_blocks_loaded_by_duration (#7309) * store-gateway: remove cortex_bucket_store_blocks_loaded_by_duration The PR which added this metric 6074, the motivation was to help detect compactors which are falling behind. There is another metric `cortex_bucket_store_series_blocks_queried` added by jhalterman in 7112 which serves better the purpose of detecting uncompacted blocks. The reason is that it has an explicit label for uncompacted blocks, and it also is more sensitive to recent blocks (assuming those are queried much more often than old uncompacted blocks). Signed-off-by: Dimitar Dimitrov <[email protected]> * Add CHANGELOG.md entry Signed-off-by: Dimitar Dimitrov <[email protected]> * Update CHANGELOG.md Co-authored-by: Nick Pillitteri <[email protected]> * Update code comment Signed-off-by: Dimitar Dimitrov <[email protected]> --------- Signed-off-by: Dimitar Dimitrov <[email protected]> Co-authored-by: Nick Pillitteri <[email protected]> * ruler: don't retry on non-retriable error (#7216) * ruler: don't retry on non-retriable error Fix 6609 Signed-off-by: Vladimir Varankin <[email protected]> * Update CHANGELOG.md Co-authored-by: Dimitar Dimitrov <[email protected]> --------- Signed-off-by: Vladimir Varankin <[email protected]> Co-authored-by: Dimitar Dimitrov <[email protected]> * Update Alertmanager to f69a508 (#7332) * Update Alertmanager to f69a508 This commit updates Alertmanager to f69a508. The compat package metrics are removed, and the tests are updated to check that logs are emitted instead. * Fix lint * Fix lint again * Use strings.Trim instead of subslice * Remove integration test TestAlertmanagerMatchersMetrics * Fix lint * Prevent configurations with WebhookURLFile This commit updates the validation rules to prevent Alertmanager configurations for Discord and Microsoft Teams from using WebhookURLFile. * Add additional checks and tests for HTTPConfig * Fix missing nil check * Check for embedded HTTPConfig is not needed * Helm: add ruler specific service account (#7132) * Helm: add ruler specific service account Signed-off-by: QuantumEnigmaa <[email protected]> * add suggestions from review Signed-off-by: QuantumEnigmaa <[email protected]> * disable ruler sa by default * add rolebinding to ruler sa Signed-off-by: QuantumEnigmaa <[email protected]> * remove trailing space Signed-off-by: QuantumEnigmaa <[email protected]> * update handling of ruler sa name Signed-off-by: QuantumEnigmaa <[email protected]> * add doc comment for ruler sa name Signed-off-by: QuantumEnigmaa <[email protected]> --------- Signed-off-by: QuantumEnigmaa <[email protected]> * frontend/transport: log non-2xx replies from downstream as non-successful (#7296) Signed-off-by: Vladimir Varankin <[email protected]> * querymiddleware: Pool snappy writer in shard activity series (#7308) * querymiddleware: pool snappy writer in shard activity series Signed-off-by: Vladimir Varankin <[email protected]> * test concurrent requests in shard active series --------- Signed-off-by: Vladimir Varankin <[email protected]> * Helm: make PSP configurable (#7190) * Helm: make PSP configurable Signed-off-by: QuantumEnigmaa <[email protected]> * fix changelog Signed-off-by: QuantumEnigmaa <[email protected]> * fix psp rendering Signed-off-by: QuantumEnigmaa <[email protected]> * Update operations/helm/charts/mimir-distributed/CHANGELOG.md --------- Signed-off-by: QuantumEnigmaa <[email protected]> Co-authored-by: Dimitar Dimitrov <[email protected]> * Helm - Templatable host for gateway ingress/route (#7218) * feat: added host templating for gateway ingress * feat: added host templating for gateway route * fix: missing ctx ref * Revert "fix: missing ctx ref" This reverts commit cf956fb. * fix: missing ctx ref * ci: ran make build-helm-tests * updated CHANGELOG.md * fix: re-ordered changelog * feat: added templating to global ingress * fix: added templating to host rules * docs: values.yaml examples * test: rebuild tests according to examples --------- Co-authored-by: Dimitar Dimitrov <[email protected]> * [Docs] Update migrate-from-single-zone-with-helm.md (#7327) * Update migrate-from-single-zone-with-helm.md Fix typo beaching->breaching. * Update changelog * Always sort labels in distributors (#7326) * Always sort labels in distributors Previously, disabling metric relabelling would also disable label sorting. Fixes #7320 * Update comment about sorting * Update CHANGELOG * Do not check for ingester ring state before creating TSDB, or compacting / shipping blocks (#7322) * Do not check for ingester ring state before creating TSDB, or compacting / shipping blocks Signed-off-by: Marco Pracucci <[email protected]> * Removed unused param Signed-off-by: Marco Pracucci <[email protected]> --------- Signed-off-by: Marco Pracucci <[email protected]> * Compactor: String format compaction plan as comma separated blocks (#7321) * Compactor: Format directories as comma separated string Signed-off-by: Arve Knudsen <[email protected]> * Compactor: string format plan as comma separated blocks Signed-off-by: Arve Knudsen <[email protected]> * Harmonize log message field names Signed-off-by: Arve Knudsen <[email protected]> --------- Signed-off-by: Arve Knudsen <[email protected]> * Add a lifetime manager for Vault authentication tokens (#7337) * Initial renewal implementation * Add metrics and integration test to test token renewal * remove unneeeded comment * Update CHANGELOG.md * Use promauto * address comments * Use global const for Vault image * Add failure metric * Update metrics * Update CHANGELOG and mark as experimental * include metric names in about-versioning * fix(deps): update github.com/grafana/dskit digest to f245b48 (#7283) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Packaging: remove reload from systemd file as mimir does not take into account SIGHUP (#7345) Signed-off-by: Wilfried Roset <[email protected]> * Docs: No longer mark OTLP endpoint as experimental (#7348) Signed-off-by: Arve Knudsen <[email protected]> * Update golang.org/x/exp digest to 2c58cdc (#7352) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update module github.com/aws/aws-sdk-go to v1.50.15 (#7353) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update module github.com/minio/minio-go/v7 to v7.0.67 (#7354) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update dependency puppeteer to v21.11.0 (#7355) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update helm/kind-action action to v1.9.0 (#7357) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Update module cloud.google.com/go/storage to v1.37.0 (#7358) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Jsonnet / Helm: improve distributors graceful shutdown (#7361) * Jsonnet / Helm: improve distributors graceful shutdown Signed-off-by: Marco Pracucci <[email protected]> * Fix Helm distributor termination grace period Signed-off-by: Marco Pracucci <[email protected]> --------- Signed-off-by: Marco Pracucci <[email protected]> * Release mimir-distributed Helm chart 5.3.0-weekly.277 (#7362) Co-authored-by: grafanabot <[email protected]> * Distributor: Make `-distributor.enable-otlp-metadata-storage` flag default to true, and deprecate (#7366) * Distributor: Change -distributor.enable-otlp-metadata-storage flag's default to true and deprecate --------- Signed-off-by: Arve Knudsen <[email protected]> * Mark -ingester.limit-inflight-requests-using-grpc-method-limiter and -distributor.limit-inflight-requests-using-grpc-method-limiter as stable and enable it by default (#7360) * Mark -ingester.limit-inflight-requests-using-grpc-method-limiter and -distributor.limit-inflight-requests-using-grpc-method-limiter as stable and enable it by default Signed-off-by: Marco Pracucci <[email protected]> * Improved tests Signed-off-by: Marco Pracucci <[email protected]> * Improved tests Signed-off-by: Marco Pracucci <[email protected]> * Fixed unit tests Signed-off-by: Marco Pracucci <[email protected]> * Mark config as deprecated Signed-off-by: Marco Pracucci <[email protected]> --------- Signed-off-by: Marco Pracucci <[email protected]> * Do not consider out-of-order blocks when filtering compactable jobs (#7342) * Do not consider out-of-order blocks when filtering compactable jobs Co-authored-by: Marco Pracucci <[email protected]> * mimir: Inject span profiler into tracer (#7363) * mimir: inject span profiler into tracer Signed-off-by: Vladimir Varankin <[email protected]> * Update changelog Signed-off-by: Vladimir Varankin <[email protected]> --------- Signed-off-by: Vladimir Varankin <[email protected]> * Add experimental partitions ring lifecycler support (#7349) * Add experimental partitions ring lifecycler support Signed-off-by: Marco Pracucci <[email protected]> Co-authored-by: Dimitar Dimitrov <[email protected]> Co-authored-by: Peter Štibraný <[email protected]> * Use http.Error() Signed-off-by: Marco Pracucci <[email protected]> * Skip deprecated linter check for code that will be soon replaced Signed-off-by: Marco Pracucci <[email protected]> * Updated dskit Signed-off-by: Marco Pracucci <[email protected]> * Add partition id to logs Signed-off-by: Marco Pracucci <[email protected]> * Added comment Signed-off-by: Marco Pracucci <[email protected]> * Added TestIngester_ShouldNotCreatePartitionIfThereIsShutdownMarker Signed-off-by: Marco Pracucci <[email protected]> --------- Signed-off-by: Marco Pracucci <[email protected]> Co-authored-by: Dimitar Dimitrov <[email protected]> Co-authored-by: Peter Štibraný <[email protected]> * feat(helm): Adding KEDA autoscaling support * chore: rebase branch with main * chore: make build-helm-tests --------- Signed-off-by: Vladimir Varankin <[email protected]> Signed-off-by: Peter Štibraný <[email protected]> Signed-off-by: Marco Pracucci <[email protected]> Signed-off-by: Nick Pillitteri <[email protected]> Signed-off-by: Arve Knudsen <[email protected]> Signed-off-by: Dimitar Dimitrov <[email protected]> Signed-off-by: Yuri Nikolic <[email protected]> Signed-off-by: QuantumEnigmaa <[email protected]> Signed-off-by: Wilfried Roset <[email protected]> Co-authored-by: Vladimir Varankin <[email protected]> Co-authored-by: Marco Pracucci <[email protected]> Co-authored-by: Andy Asp <[email protected]> Co-authored-by: Peter Štibraný <[email protected]> Co-authored-by: Arve Knudsen <[email protected]> Co-authored-by: Nick Pillitteri <[email protected]> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Grot (@grafanabot) <[email protected]> Co-authored-by: grafanabot <[email protected]> Co-authored-by: Dimitar Dimitrov <[email protected]> Co-authored-by: Đurica Yuri Nikolić <[email protected]> Co-authored-by: George Robinson <[email protected]> Co-authored-by: Zirko <[email protected]> Co-authored-by: Itay Kalfon <[email protected]> Co-authored-by: Éamon Ryan <[email protected]> Co-authored-by: Patrick Oyarzun <[email protected]> Co-authored-by: Fayzal Ghantiwala <[email protected]> Co-authored-by: Wilfried ROSET <[email protected]> Co-authored-by: Jonathan Halterman <[email protected]>
7 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does
This PR does the following things:
-ingester.client.circuit-breaker.cooldown-periodfrom1mto10sChecklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX].about-versioning.mdupdated with experimental features.