Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

userTSDB: Avoid race condition when setting db #9708

Merged
merged 3 commits into from
Oct 28, 2024
Merged

Conversation

aknuds1
Copy link
Contributor

@aknuds1 aknuds1 commented Oct 22, 2024

What this PR does

Fix race condition in pkg/ingester.userTSDB when setting its db field.

WARNING: DATA RACE
Read at 0x00c0153be140 by goroutine 76199:
  github.com/grafana/mimir/pkg/ingester.(*userTSDB).blocksToDelete()
      /Users/arve/Projects/grafana/backend-enterprise/vendor/github.com/grafana/mimir/pkg/ingester/user_tsdb.go:364 +0x58
  github.com/grafana/mimir/pkg/ingester.(*userTSDB).blocksToDelete-fm()
      <autogenerated>:1 +0x51
  github.com/prometheus/prometheus/tsdb.(*DB).reloadBlocks()
      /Users/arve/Projects/grafana/backend-enterprise/vendor/github.com/prometheus/prometheus/tsdb/db.go:1654 +0x49e
  github.com/prometheus/prometheus/tsdb.(*DB).run()
      /Users/arve/Projects/grafana/backend-enterprise/vendor/github.com/prometheus/prometheus/tsdb/db.go:1149 +0x771
  github.com/prometheus/prometheus/tsdb.open.gowrap1()
      /Users/arve/Projects/grafana/backend-enterprise/vendor/github.com/prometheus/prometheus/tsdb/db.go:1093 +0x4f

Previous write at 0x00c0153be140 by goroutine 5042:
  github.com/grafana/mimir/pkg/ingester.(*Ingester).createTSDB()
      /Users/arve/Projects/grafana/backend-enterprise/vendor/github.com/grafana/mimir/pkg/ingester/ingester.go:2717 +0x2015
  github.com/grafana/mimir/pkg/ingester.(*Ingester).openExistingTSDB.func1()
      /Users/arve/Projects/grafana/backend-enterprise/vendor/github.com/grafana/mimir/pkg/ingester/ingester.go:2818 +0x116
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /Users/arve/Projects/grafana/backend-enterprise/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x91

Which issue(s) this PR fixes or relates to

Checklist

  • Tests updated.
  • Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • about-versioning.md updated with experimental features.

@aknuds1 aknuds1 added bug Something isn't working component/ingester labels Oct 22, 2024
@aknuds1 aknuds1 force-pushed the arve/user-tsdb-race branch from 88eea09 to e4e0267 Compare October 22, 2024 10:57
@aknuds1 aknuds1 marked this pull request as ready for review October 22, 2024 10:57
@aknuds1 aknuds1 requested a review from a team as a code owner October 22, 2024 10:57
pkg/ingester/user_tsdb.go Outdated Show resolved Hide resolved
@aknuds1 aknuds1 force-pushed the arve/user-tsdb-race branch from f2ff566 to 3862596 Compare October 28, 2024 09:35
@aknuds1 aknuds1 merged commit d14d52c into main Oct 28, 2024
29 checks passed
@aknuds1 aknuds1 deleted the arve/user-tsdb-race branch October 28, 2024 11:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working component/ingester
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants