Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/backupccl: data race in TestBackupRestoreMultiNodeRemote #96274

Closed
renatolabs opened this issue Jan 31, 2023 · 2 comments
Closed

ccl/backupccl: data race in TestBackupRestoreMultiNodeRemote #96274

renatolabs opened this issue Jan 31, 2023 · 2 comments
Assignees
@rhu713
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-disaster-recovery

Comments

@renatolabs
Copy link
Contributor

renatolabs commented Jan 31, 2023 •
edited by cockroach-jira-scripts
Loading

TestBackupRestoreMultiNodeRemote fails pretty consistently when race is enabled:

./dev test ./pkg/ccl/backupccl/ -f TestBackupRestoreMultiNodeRemote --race

Output:

==================
WARNING: DATA RACE
Read at 0x00c0019e2378 by goroutine 73980:
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*KeyRewriter).RewriteKey()
      github.com/cockroachdb/cockroach/pkg/ccl/backupccl/key_rewriter.go:249 +0x1ae
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.rewriteBackupSpanKey()
      github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:112 +0xf4
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.dbSplitAndScatterer.split()
      github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:89 +0x179
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*dbSplitAndScatterer).split()
      <autogenerated>:1 +0x104
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runGenerativeSplitAndScatter.func3()
      github.com/cockroachdb/cockroach/pkg/ccl/backupccl/generative_split_and_scatter_processor.go:302 +0x23a
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x4c
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x86

Previous write at 0x00c0019e2378 by goroutine 73981:
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*KeyRewriter).RewriteKey()
      github.com/cockroachdb/cockroach/pkg/ccl/backupccl/key_rewriter.go:250 +0x1d5
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.rewriteBackupSpanKey()
      github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:112 +0xf4
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.dbSplitAndScatterer.scatter()
      github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:117 +0xee
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*dbSplitAndScatterer).scatter()
      <autogenerated>:1 +0x10b
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runGenerativeSplitAndScatter.func3()
      github.com/cockroachdb/cockroach/pkg/ccl/backupccl/generative_split_and_scatter_processor.go:306 +0x2ce
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x4c
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x86

Goroutine 73980 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0x12e
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:167 +0x109
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runGenerativeSplitAndScatter()
      github.com/cockroachdb/cockroach/pkg/ccl/backupccl/generative_split_and_scatter_processor.go:294 +0x52c
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*generativeSplitAndScatterProcessor).Start.func2()
      github.com/cockroachdb/cockroach/pkg/ccl/backupccl/generative_split_and_scatter_processor.go:127 +0x108
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x1f6

Goroutine 73981 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0x12e
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:167 +0x109
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runGenerativeSplitAndScatter()
      github.com/cockroachdb/cockroach/pkg/ccl/backupccl/generative_split_and_scatter_processor.go:294 +0x52c
  github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*generativeSplitAndScatterProcessor).Start.func2()
      github.com/cockroachdb/cockroach/pkg/ccl/backupccl/generative_split_and_scatter_processor.go:127 +0x108
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x1f6
==================

Occurrence: https://teamcity.cockroachdb.com/viewLog.html?buildId=8526557&buildTypeId=Cockroach_BazelExtendedCi

Jira issue: CRDB-24039
@renatolabs renatolabs added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-disaster-recovery labels Jan 31, 2023
@blathers-crl
Copy link

blathers-crl bot commented Jan 31, 2023

cc @cockroachdb/disaster-recovery

@renatolabs renatolabs mentioned this issue Jan 31, 2023
Merged
@dt
Copy link
Member

dt commented Feb 21, 2023

#96313

@dt dt closed this as completed Feb 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rhu713 rhu713

Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-disaster-recovery
Projects
No open projects
Disaster Recovery Backlog
Archived in project
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dt @rhu713 @renatolabs