Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TiCDC changefeed stucks after cdc owner to Kafka network partition recovered #9518

Closed
fubinzh opened this issue Aug 8, 2023 · 3 comments · Fixed by #9530
Closed

TiCDC changefeed stucks after cdc owner to Kafka network partition recovered #9518

fubinzh opened this issue Aug 8, 2023 · 3 comments · Fixed by #9530
Labels
affects-7.1 This bug affects the 7.1.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. found/automation Bugs found by automation cases severity/major type/bug The issue is confirmed as a bug.

Comments

@fubinzh
Copy link

fubinzh commented Aug 8, 2023

What did you do?

  1. create kafka changefeed
  2. run workload, and at the same time inject cdc owner to Kafka network partition 40 seconds for every 5 minutes, stop the chaos injection after 1 hours.
  3. Check CDC lag after chaos injection is finished.

What did you expect to see?

CDC lag should be normal

What did you see instead?

Chaos injection recoveryed at 03:28:53, checkpoint stucks at 03:30
CDC changefeed stucks
image

Versions of the cluster

TiCDC version (execute cdc version): master

["Welcome to Change Data Capture (CDC)"] [release-version=v7.4.0-alpha] [git-hash=e61669614f12a3e1f6bfc6af38ec7c297e7ede93] [git-branch=heads/refs/tags/v7.4.0-alpha] [utc-build-time="2023-08-07 11:36:10"] [go-version="go version go1.20.7 linux/amd64"] [failpoint-build=false]
@fubinzh fubinzh added area/ticdc Issues or PRs related to TiCDC. type/bug The issue is confirmed as a bug. labels Aug 8, 2023
@fubinzh
Copy link
Author

fubinzh commented Aug 8, 2023

/found automation

@ti-chi-bot ti-chi-bot bot added the found/automation Bugs found by automation cases label Aug 8, 2023
@fubinzh
Copy link
Author

fubinzh commented Aug 8, 2023

cdc log attached
ticdc_0-1833598.log
ticdc-1_1833598.log

@fubinzh
Copy link
Author

fubinzh commented Aug 8, 2023 •
edited
Loading

/severity major

@hicqu hicqu mentioned this issue Aug 10, 2023
Merged
@hicqu hicqu added affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. and removed may-affects-5.2 may-affects-5.3 may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 labels Aug 10, 2023
ti-chi-bot bot pushed a commit that referenced this issue Aug 14, 2023
@hicqu
sink(cdc): fix internal retry algothrim (#9530)
bd210f8 
close #9518
This was referenced Aug 14, 2023
Closed
Merged
@hicqu hicqu removed the affects-6.5 This bug affects the 6.5.x(LTS) versions. label Aug 14, 2023
@ti-chi-bot ti-chi-bot mentioned this issue Aug 15, 2023
Merged
hicqu added a commit to ti-chi-bot/tiflow that referenced this issue Aug 15, 2023
@hicqu
sink(cdc): fix internal retry algothrim (pingcap#9530)
3c22f63 
close pingcap#9518

Signed-off-by: qupeng <[email protected]>
ti-chi-bot bot pushed a commit that referenced this issue Aug 15, 2023
@ti-chi-bot
sink(cdc): fix internal retry algothrim (#9530) (#9567)
a9f2071 
close #9518
ti-chi-bot bot pushed a commit that referenced this issue Aug 16, 2023
@ti-chi-bot
sink(cdc): fix internal retry algothrim (#9530) (#9572)
425b10a 
ref #9272, close #9518
3AceShowHand pushed a commit to 3AceShowHand/tiflow that referenced this issue Aug 29, 2023
@hicqu @3AceShowHand
sink(cdc): fix internal retry algothrim (pingcap#9530)
5a56540 
close pingcap#9518
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-7.1 This bug affects the 7.1.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. found/automation Bugs found by automation cases severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

sink(cdc): fix internal retry algothrim hicqu/ticdc
2 participants
@fubinzh @hicqu