Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

data race found in the kv client on the relase-7.1 #10718

Closed
3AceShowHand opened this issue Mar 6, 2024 · 1 comment · Fixed by #10719
Closed

data race found in the kv client on the relase-7.1 #10718

3AceShowHand opened this issue Mar 6, 2024 · 1 comment · Fixed by #10719
Assignees
@asddongmen
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. severity/major type/bug The issue is confirmed as a bug.

Comments

@3AceShowHand
Copy link
Contributor 

==================
WARNING: DATA RACE
Read at 0x00c004305a50 by goroutine 1766:
  runtime.chansend()
      /usr/local/go/src/runtime/chan.go:160 +0x0
  github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).scheduleRegionRequest.func1()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:544 +0x7c9
  github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).scheduleRegionRequest()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:594 +0x6d6
  github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).divideAndSendEventFeedToRegions()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:894 +0xfaf
  github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).eventFeed.func5.1()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:488 +0xda
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x82

Previous write at 0x00c004305a50 by goroutine 1750:
  runtime.closechan()
      /usr/local/go/src/runtime/chan.go:357 +0x0
  github.com/pingcap/tiflow/pkg/chann.(*Chann[...]).unboundedTerminate()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/pkg/chann/chann.go:223 +0xe4
  github.com/pingcap/tiflow/pkg/chann.(*Chann[...]).unboundedProcessing()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/pkg/chann/chann.go:190 +0x54c
  github.com/pingcap/tiflow/pkg/chann.New[...].func1()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/pkg/chann/chann.go:149 +0x47

Goroutine 1766 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12e
  github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).eventFeed.func5()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:487 +0x124
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x82

Goroutine 1750 (finished) created at:
  github.com/pingcap/tiflow/pkg/chann.New[...]()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/pkg/chann/chann.go:149 +0x4ca
  github.com/pingcap/tiflow/pkg/chann.NewAutoDrainChann[...]()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/pkg/chann/drainable_chann.go:27 +0xae
  github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).eventFeed()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:449 +0x124
  github.com/pingcap/tiflow/cdc/kv.(*CDCClient).EventFeed()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:311 +0x16b
  github.com/pingcap/tiflow/cdc/puller.(*pullerImpl).Run.func1()
      /home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/puller/puller.go:142 +0x176
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x82
==================
panic: send on closed channel

goroutine 1249 [running]:
github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).scheduleRegionRequest.func1({0x0, 0x63847af382c0003, 0xc004b48258, 0x0, {0x0, 0x0, 0x0}})
	/home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:543 +0x80a
github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).scheduleRegionRequest(0xc00439b7a0, {0xaa834a8, 0xc002787ae0}, {{0x91, 0x7, 0x37}, {0x0, {0xc002d0fd40, 0x12, 0x18}, ...}, ...})
	/home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:594 +0x6d7
github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).divideAndSendEventFeedToRegions(0xc00439b7a0, {0xaa834a8, 0xc002787ae0}, {0x5e, {0xc002d0fd40, 0x12, 0x18}, {0xc002d0fd58, 0x12, 0x18}})
	/home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:894 +0xfb0
github.com/pingcap/tiflow/cdc/kv.(*eventFeedSession).eventFeed.func5.1()
	/home/jenkins/agent/workspace/pingcap/tiflow/release-7.1/pull_cdc_integration_kafka_test/tiflow/cdc/kv/client.go:488 +0xdb
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x83
created by golang.org/x/sync/errgroup.(*Group).Go
	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0x12f
@3AceShowHand 3AceShowHand added area/ticdc Issues or PRs related to TiCDC. affect-ticdc-config-docs Pull requests that affect TiCDC configuration docs. and removed affect-ticdc-config-docs Pull requests that affect TiCDC configuration docs. labels Mar 6, 2024
@asddongmen asddongmen mentioned this issue Mar 6, 2024
Merged
@asddongmen asddongmen added affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. labels Mar 6, 2024
@asddongmen asddongmen self-assigned this Mar 6, 2024
@asddongmen asddongmen added affects-6.5 This bug affects the 6.5.x(LTS) versions. severity/moderate labels Mar 6, 2024
@3AceShowHand 3AceShowHand added the type/bug The issue is confirmed as a bug. label Mar 6, 2024
@asddongmen
Copy link
Contributor

asddongmen commented Mar 6, 2024 •
edited
Loading

It seems that a new group was started here at

tiflow/cdc/kv/client.go

Line 489 in 06fcdc5

g, ctx := errgroup.WithContext(ctx)
, causing the goroutine created on line 505

tiflow/cdc/kv/client.go

Line 505 in 9b30691

g.Go(func() error {
to not be waited for by the wait here at

tiflow/cdc/kv/client.go

Line 538 in 9b30691

return g.Wait()

So a data race issue could happen.

ti-chi-bot bot pushed a commit that referenced this issue Mar 6, 2024
@asddongmen
kv (ticdc): fix kv client data race panic (#10719)
e9128ab 
close #10718
This was referenced Mar 6, 2024
Merged
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Mar 6, 2024
@ti-chi-bot
kv (ticdc): fix kv client data race panic (#10719) (#10720)
da0656d 
close #10718
ti-chi-bot bot pushed a commit that referenced this issue Mar 6, 2024
@ti-chi-bot
kv (ticdc): fix kv client data race panic (#10719) (#10722)
327dfbb 
close #10718
ti-chi-bot bot pushed a commit that referenced this issue Mar 6, 2024
@ti-chi-bot
kv (ticdc): fix kv client data race panic (#10719) (#10721)
6e3e1e6 
close #10718
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@asddongmen asddongmen

Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. severity/major type/bug The issue is confirmed as a bug.
Projects
Question and Bug Reports
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kv (ticdc): fix kv client data race panic asddongmen/tiflow
3 participants
@jebter @3AceShowHand @asddongmen