bucket verify: fix to parse all blocks

[aymericDD]

Only the first block is parsed. This commit fixes this issue to parse all blocks present in the bucket.
Fix issue #4878 .

Signed-off-by: Aymeric [email protected]

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

• Split the function VerifyRepair in two dedicated functions verify and repair.
• Replace return statement by continue to go to the next iteration. This allow to parse all blocks from a bucket.
• Added a log in debug level to just say when a block has no issue :

level=debug ts=2021-11-19T11:50:37.47465883Z caller=index_issue.go:56 verifiers=index_known_issues verifier=index_known_issues msg="no issue" id=01FMVXMZABF699W6HC8HA2B0E3

Verification

Tested locally.

$ thanos tools bucket verify --objstore.config-file=/etc/default/thanos-storage-a01-dc1.yml -i index_known_issues  --log.level=debug                                                    
level=debug ts=2021-11-19T11:47:29.831109825Z caller=main.go:65 msg="maxprocs: Leaving GOMAXPROCS=[40]: CPU quota undefined"
level=info ts=2021-11-19T11:47:29.83165329Z caller=factory.go:49 msg="loading bucket configuration"
level=info ts=2021-11-19T11:47:29.83290754Z caller=verify.go:137 verifiers=index_known_issues msg="Starting verify task"
level=info ts=2021-11-19T11:47:29.83295293Z caller=index_issue.go:32 verifiers=index_known_issues verifier=index_known_issues msg="started verifying issue" with-repair=false
level=debug ts=2021-11-19T11:47:29.833026263Z caller=fetcher.go:323 component=block.BaseFetcher msg="fetching meta data" concurrency=32
level=info ts=2021-11-19T11:47:30.203705366Z caller=fetcher.go:481 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=370.714829ms duration_ms=370 cached=126 returned=126 partial=0
level=debug ts=2021-11-19T11:47:41.106087905Z caller=index_issue.go:140 verifiers=index_known_issues verifier=index_known_issues stats="{TotalSeries:1987113 OutOfOrderSeries:0 OutOfOrderChunks:0 DuplicatedChunks:0 OutsideChunks:0 CompleteOutsideChunks:0 Issue347OutsideChunks:0 OutOfOrderLabels:0 SeriesMinLifeDuration:0s SeriesAvgLifeDuration:1h49m34.315s SeriesMaxLifeDuration:1h59m40.005s SeriesMinLifeDurationWithoutSingleSampleSeries:19.99s SeriesAvgLifeDurationWithoutSingleSampleSeries:1h50m56.676s SeriesMaxLifeDurationWithoutSingleSampleSeries:1h59m40.005s SeriesMinChunks:1 SeriesAvgChunks:1 SeriesMaxChunks:3 TotalChunks:3732159 ChunkMinDuration:0s ChunkAvgDuration:58m20.361s ChunkMaxDuration:1h59m40.005s ChunkMinSize:988 ChunkAvgSize:997 ChunkMaxSize:1009 SingleSampleSeries:24586 SingleSampleChunks:25008 LabelNamesCount:465 MetricLabelValuesCount:2525}" id=01FMRFSCZZ7192QRXWQCF78057
level=debug ts=2021-11-19T11:47:41.106168649Z caller=index_issue.go:56 verifiers=index_known_issues verifier=index_known_issues msg="no issue" id=01FMRFSCZZ7192QRXWQCF78057
level=debug ts=2021-11-19T11:47:51.188383145Z caller=index_issue.go:140 verifiers=index_known_issues verifier=index_known_issues stats="{TotalSeries:1998916 OutOfOrderSeries:0 OutOfOrderChunks:0 DuplicatedChunks:0 OutsideChunks:0 CompleteOutsideChunks:0 Issue347OutsideChunks:0 OutOfOrderLabels:0 SeriesMinLifeDuration:0s SeriesAvgLifeDuration:1h49m9.355s SeriesMaxLifeDuration:1h59m40.034s SeriesMinLifeDurationWithoutSingleSampleSeries:19.973s SeriesAvgLifeDurationWithoutSingleSampleSeries:1h50m34.317s SeriesMaxLifeDurationWithoutSingleSampleSeries:1h59m40.034s SeriesMinChunks:1 SeriesAvgChunks:1 SeriesMaxChunks:3 TotalChunks:3753340 ChunkMinDuration:0s ChunkAvgDuration:58m7.989s ChunkMaxDuration:1h59m40.034s ChunkMinSize:941 ChunkAvgSize:982 ChunkMaxSize:1010 SingleSampleSeries:25599 SingleSampleChunks:26049 LabelNamesCount:474 MetricLabelValuesCount:2563}" id=01FMSRZRK02ECGN1MDKVSQR7RP
level=debug ts=2021-11-19T11:47:51.188463996Z caller=index_issue.go:56 verifiers=index_known_issues verifier=index_known_issues msg="no issue" id=01FMSRZRK02ECGN1MDKVSQR7RP
level=debug ts=2021-11-19T11:50:15.84308204Z caller=index_issue.go:140 verifiers=index_known_issues verifier=index_known_issues stats="{TotalSeries:24464180 OutOfOrderSeries:0 OutOfOrderChunks:0 DuplicatedChunks:0 OutsideChunks:0 CompleteOutsideChunks:0 Issue347OutsideChunks:0 OutOfOrderLabels:0 SeriesMinLifeDuration:0s SeriesAvgLifeDuration:26h30m29.054s SeriesMaxLifeDuration:335h59m40s SeriesMinLifeDurationWithoutSingleSampleSeries:1.512s SeriesAvgLifeDurationWithoutSingleSampleSeries:103h7m20.052s SeriesMaxLifeDurationWithoutSingleSampleSeries:335h59m40s SeriesMinChunks:1 SeriesAvgChunks:1 SeriesMaxChunks:4 TotalChunks:28364272 ChunkMinDuration:0s ChunkAvgDuration:22h51m47.543s ChunkMaxDuration:335h59m40s ChunkMinSize:172 ChunkAvgSize:6197 ChunkMaxSize:3758101587 SingleSampleSeries:18175544 SingleSampleChunks:18175544 LabelNamesCount:475 MetricLabelValuesCount:2466}" id=01FEKXAXEN9H4Z2858JRA6V99Y
level=debug ts=2021-11-19T11:50:15.84314986Z caller=index_issue.go:56 verifiers=index_known_issues verifier=index_known_issues msg="no issue" id=01FEKXAXEN9H4Z2858JRA6V99Y
level=debug ts=2021-11-19T11:50:26.360474951Z caller=index_issue.go:140 verifiers=index_known_issues verifier=index_known_issues stats="{TotalSeries:2000070 OutOfOrderSeries:0 OutOfOrderChunks:0 DuplicatedChunks:0 OutsideChunks:0 CompleteOutsideChunks:0 Issue347OutsideChunks:0 OutOfOrderLabels:0 SeriesMinLifeDuration:0s SeriesAvgLifeDuration:1h49m5.436s SeriesMaxLifeDuration:1h59m40s SeriesMinLifeDurationWithoutSingleSampleSeries:19.97s SeriesAvgLifeDurationWithoutSingleSampleSeries:1h50m28.86s SeriesMaxLifeDurationWithoutSingleSampleSeries:1h59m40s SeriesMinChunks:1 SeriesAvgChunks:1 SeriesMaxChunks:2 TotalChunks:3755969 ChunkMinDuration:0s ChunkAvgDuration:58m5.473s ChunkMaxDuration:1h59m40s ChunkMinSize:9223372036854775807 ChunkAvgSize:0 ChunkMaxSize:-9223372036854775808 SingleSampleSeries:25171 SingleSampleChunks:25631 LabelNamesCount:474 MetricLabelValuesCount:2563}" id=01FMSRZQ2357K745RXT4E7KJBC
level=debug ts=2021-11-19T11:50:26.360557596Z caller=index_issue.go:56 verifiers=index_known_issues verifier=index_known_issues msg="no issue" id=01FMSRZQ2357K745RXT4E7KJBC
level=debug ts=2021-11-19T11:50:37.47459854Z caller=index_issue.go:140 verifiers=index_known_issues verifier=index_known_issues stats="{TotalSeries:2026059 OutOfOrderSeries:0 OutOfOrderChunks:0 DuplicatedChunks:0 OutsideChunks:0 CompleteOutsideChunks:0 Issue347OutsideChunks:0 OutOfOrderLabels:0 SeriesMinLifeDuration:0s SeriesAvgLifeDuration:1h47m55.585s SeriesMaxLifeDuration:1h59m40s SeriesMinLifeDurationWithoutSingleSampleSeries:19.965s SeriesAvgLifeDurationWithoutSingleSampleSeries:1h49m20.31s SeriesMaxLifeDurationWithoutSingleSampleSeries:1h59m40s SeriesMinChunks:1 SeriesAvgChunks:1 SeriesMaxChunks:2 TotalChunks:3777088 ChunkMinDuration:0s ChunkAvgDuration:57m53.553s ChunkMaxDuration:1h59m40s ChunkMinSize:9223372036854775807 ChunkAvgSize:0 ChunkMaxSize:-9223372036854775808 SingleSampleSeries:26166 SingleSampleChunks:26676 LabelNamesCount:475 MetricLabelValuesCount:2565}" id=01FMVXMZABF699W6HC8HA2B0E3
level=debug ts=2021-11-19T11:50:37.47465883Z caller=index_issue.go:56 verifiers=index_known_issues verifier=index_known_issues msg="no issue" id=01FMVXMZABF699W6HC8HA2B0E3
... etc...

[aymericDD]

Anyone for a review? 😇

[aymericDD]

CHANGELOG conflict resolved. Still anyone for a review? 🥺

[bwplotka]

Amazing, thanks!

Generally quite good, just some readability nits!

[aymericDD]

All the readability nits are fixed @bwplotka 👍

[bwplotka]

Sweet, thanks!