Spread TSDB head compaction over the configured interval #4364
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pstibrany
reviewed
Mar 3, 2023
pkg/ingester/ingester.go
Outdated
|
|
||
| for ctx.Err() == nil { | ||
| select { | ||
| case <-ticker.C: | ||
| case <-time.After(nextPeriodicCompaction): |
There was a problem hiding this comment.
If force compaction is triggered, it will restart wait time, so subsequent compactions will be shifted (contrary to what help string says).
We could use Ticker and simply reset its duration after first compaction.
There was a problem hiding this comment.
WDYT about 6cb30be ?
|
Description makes it sound like we're spreading compactions for all TSDBs. That's not the case. All TSDBs will still be compacted at the same time, just this time is now slightly less aligned to 2h interval. |
Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
pracucci
force-pushed
the
add-initial-jitter-to-compaction-interval
branch
from
1efb6d7 to
6cb30be
Compare
I see that both PR description and CLI flag already mention that this helps across multiple ingesters only. I missed that before. Sorry for the noise. |
Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
krajorama
added a commit
that referenced
this pull request
Mar 6, 2023
* Helm: nginx HPA and tests kubeversion fixes (#4299) * Helm: fix Kubernetes override for nginx HPA The template did not take into account the override "kubeVersionOverride". Fix by using the mimir template implemented for this reason. Signed-off-by: György Krajcsovits <[email protected]> * Helm: fix missing Kubernetes version overrides in tests The golden record tests need a fixed version because helm uses the version of the default context and can produce different results between contributor's machine and the CI environment. Add logic to test build to inject the minimal version if not found in the values file. Mainly because we cannot have a version override in the small and large values files. Signed-off-by: György Krajcsovits <[email protected]> Co-authored-by: Jon Kartago Lamida <[email protected]> * Ruler: load more tenants in parallel during startup (#4258) * Ruler: load more tenants in parallel during startup * add more tests * fix lint * Apply suggestions from code review Co-authored-by: Marco Pracucci <[email protected]> * Ingester: fix OOO blocks labelling (#4297) * Ingester: fix OOO blocks labelling This fixes a bug where the OutOfOrderExternalLabel was being added to all blocks instead of the ones coming from OOO data, when the feature flag was enabled. * Changelog * PR number to changelog * Update previous changelog entry instead * Ruler: load more tenants in parallel during startup * fix context * improve unittest --------- Co-authored-by: Marco Pracucci <[email protected]> Co-authored-by: Nicolás Pazos <[email protected]> * Change language to match the math. (#4356) * Upgrade mimir-prometheus to get a fast regexp path optimization (#4357) * Upgrade mimir-prometheus to get a fast regexp path optimization Signed-off-by: Marco Pracucci <[email protected]> * Added CHANGELOG entry Signed-off-by: Marco Pracucci <[email protected]> --------- Signed-off-by: Marco Pracucci <[email protected]> * Fix typo in the docs URL for migrating from Cortex (#4358) * Remove forced paragraph break. (#4359) * Bump actions/setup-go to v3 to resolve Node.js 12 deprecation warning. (#4361) * Improve flaky `TestIngesterWithShippingDisabledDeletesBlocksOnlyAfterRetentionExpires` (#4362) * Use more specific assertion to include more information in test failures. See #4198. * Reduce flakiness of test by extending retention period. This gives the rest of the test more time to retrieve `oldBlocks` before any of the blocks is removed. * Add asynchronous validation scaffolding for block upload (#3411) * Add asynchronous validation scaffolding for block upload * addressed lint errors * Update pkg/compactor/block_upload.go Co-authored-by: Arve Knudsen <[email protected]> * Update pkg/compactor/block_upload.go Co-authored-by: Arve Knudsen <[email protected]> * Update pkg/compactor/block_upload.go Co-authored-by: Arve Knudsen <[email protected]> * Update pkg/compactor/block_upload.go Co-authored-by: Arve Knudsen <[email protected]> * Update pkg/compactor/block_upload.go Co-authored-by: Arve Knudsen <[email protected]> * Update pkg/compactor/block_upload.go Co-authored-by: Arve Knudsen <[email protected]> * enable block upload for dev testing * fixed validation errors, added debug log messages * fixed cancelled context issue * changed name of flag to disable complete block upload * addressed reviewer feedback * addressed reviewer feedback * Address some review comments, WIP * Small spacing cleanup * Transition to in-memory bucket for block finish test * Async validation test coordination, adding configuration flags * Small comment and flag fix * Swap config strategy, test still needs separation * Docs + lint * Review comments, begin separating tests * Finish validateAndComplete test * Update docs * Remove docker compose arguments * Regenerate rather than modify * Review, add test for periodicValidationUpdater * Make validateAndComplete test clearer, add upload meta check * Set missing cancelContext * Add sleep as suggested * Configure data directory * fixed compactor data dir in e2e test * Add changelog entry * Split into two entries * Missing entry number * Update CHANGELOG.md --------- Co-authored-by: Arve Knudsen <[email protected]> Co-authored-by: Andy Asp <[email protected]> Co-authored-by: Andy Asp <[email protected]> * Jsonnet: honor the minimum shard size configured (#4363) Signed-off-by: Marco Pracucci <[email protected]> * [CHANGE] Ruler: set default `evaluation-delay-duration` to 1m (#4250) * change the default evaluation delay of ruler to 1m * revert the changes in ruler test * change integration test to set ruler default value * fix integration tests * Update integration/configs.go Co-authored-by: Marco Pracucci <[email protected]> * Update CHANGELOG.md Co-authored-by: Peter Štibraný <[email protected]> --------- Co-authored-by: Marco Pracucci <[email protected]> Co-authored-by: Peter Štibraný <[email protected]> * [Chore] Update jsonnet manifest create query frontend discovery only when it is necessary (#4353) * [Chore] update jsonnet manifest, avoid setting querier.frontend-address or create query-frontend-discovery when deployement mode is microserivces or query-scheduler is enabled * linter and changelog * Helm: fix parity with jsonnet on query frontend headless service Do not generate query-frontend-headless service if query scheduler is enabled Signed-off-by: György Krajcsovits <[email protected]> * Apply suggestions from code review Co-authored-by: Marco Pracucci <[email protected]> * correct changelog * regenerate helm golden files * Update CHANGELOG.md --------- Signed-off-by: György Krajcsovits <[email protected]> Co-authored-by: György Krajcsovits <[email protected]> Co-authored-by: Marco Pracucci <[email protected]> * Remove block validation mimirtool changelog entry (#4369) * Spread TSDB head compaction over the configured interval (#4364) * Spread TSDB head compaction over the configured interval Signed-off-by: Marco Pracucci <[email protected]> * Fixed unit test Signed-off-by: Marco Pracucci <[email protected]> * Apply suggestion from code review Signed-off-by: Marco Pracucci <[email protected]> * Fix typo in CHANGELOG entry Signed-off-by: Marco Pracucci <[email protected]> * Fix typo in CHANGELOG entry Signed-off-by: Marco Pracucci <[email protected]> --------- Signed-off-by: Marco Pracucci <[email protected]> * Fix port number values. (#4368) * Ruler: change deployment max surge and max unavailable to reduce ownership spillover (#4381) * Ruler: change deployment max surge and max unavailable to reduce ownership spillover Signed-off-by: Marco Pracucci <[email protected]> * Apply suggestions from code review Co-authored-by: Dimitar Dimitrov <[email protected]> --------- Signed-off-by: Marco Pracucci <[email protected]> Co-authored-by: Dimitar Dimitrov <[email protected]> * Move "Note:" about cross-zone costs to "Costs" (#4370) This note was in an unrelated section. Signed-off-by: Oleg Zaytsev <[email protected]> * Change default -blocks-storage.tsdb.retention-period from 24h to 13h (#4382) Signed-off-by: Marco Pracucci <[email protected]> * Support histograms in pkg/storage and update other breakages (#4354) * Support histograms in pkg/storage and update other breakages Signed-off-by: Ganesh Vernekar <[email protected]> --------- Signed-off-by: György Krajcsovits <[email protected]> Signed-off-by: Marco Pracucci <[email protected]> Signed-off-by: Oleg Zaytsev <[email protected]> Signed-off-by: Ganesh Vernekar <[email protected]> Co-authored-by: Jon Kartago Lamida <[email protected]> Co-authored-by: ying-jeanne <[email protected]> Co-authored-by: Marco Pracucci <[email protected]> Co-authored-by: Nicolás Pazos <[email protected]> Co-authored-by: Ursula Kallio <[email protected]> Co-authored-by: l3ioo <[email protected]> Co-authored-by: Charles Korn <[email protected]> Co-authored-by: Vernon Miller <[email protected]> Co-authored-by: Arve Knudsen <[email protected]> Co-authored-by: Andy Asp <[email protected]> Co-authored-by: Andy Asp <[email protected]> Co-authored-by: Peter Štibraný <[email protected]> Co-authored-by: Dimitar Dimitrov <[email protected]> Co-authored-by: Oleg Zaytsev <[email protected]> Co-authored-by: Ganesh Vernekar <[email protected]>
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does
I would like to experiment spreading the TSDB head compaction over the configured interval. To make it possible, in this PR I propose two changes:
-blocks-storage.tsdb.head-compaction-intervalto a value higher than 5m (in this PR I propose a max of 15m)When running with the default config (1m interval) there should be no practical difference.
Which issue(s) this PR fixes or relates to
N/A
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]