Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

after inject pdleader network partition last for 10m,ticdc changefeed lag is more and more #8479

Closed
Lily2025 opened this issue Mar 9, 2023 · 9 comments

Comments

@Lily2025
Copy link

Lily2025 commented Mar 9, 2023

What did you do?

1、create changefeed
2、run workload tpcc with 1000 warehouse and 32 thread
3、inject pdleader network partition last for 10m
[2023/03/08 15:24:23.579 +08:00] [INFO] [chaos.go:150] ["Run chaos"] [name=network_partition] [selectors="[endless-ha-test-tps-1655438-1-603/tc-pd-1]"] [SelectorsRetainPolicy(selectors)="[endless-ha-test-tps-1655438-1-603/tc-pd-1]"]

What did you expect to see?

ticdc changefeed lag less than 30s

What did you see instead?

after inject pdleader network partition last for 10m,ticdc changefeed lag is more and more

clinic: https://clinic.pingcap.com.cn/portal/#/orgs/31/clusters/7208067513104263723?from=1678259639&to=1678261493
logs: http://minio.pingcap.net:38888/buckets/test-infra-testground/browse/YXJjaGl2ZS9lbmRsZXNzLWhhLXRlc3QtdHBzLTE2NTI3NTQtMS0yMjc=
6fa13ad5-4571-4441-b9f6-aa61a28c1b85

Versions of the cluster

./cdc version
Release Version: v6.7.0-alpha
Git Commit Hash: f43dfb2
Git Branch: heads/refs/tags/v6.7.0-alpha
UTC Build Time: 2023-03-07 11:34:11
Go Version: go version go1.20.1 linux/amd64
Failpoint Build: false

current status of DM cluster (execute query-status <task-name> in dmctl)

(paste current status of DM cluster here)
@Lily2025 Lily2025 added area/dm Issues or PRs related to DM. type/bug The issue is confirmed as a bug. labels Mar 9, 2023
@Lily2025
Copy link
Author

Lily2025 commented Mar 9, 2023

/remove-area dm
/area ticdc

@ti-chi-bot ti-chi-bot added area/ticdc Issues or PRs related to TiCDC. and removed area/dm Issues or PRs related to DM. labels Mar 9, 2023
@Lily2025 Lily2025 changed the title after inject pdleader network partition last for 10m,pitr lag is more and more after inject pdleader network partition last for 10m,changefeed lag is more and more Mar 9, 2023
@Lily2025 Lily2025 changed the title after inject pdleader network partition last for 10m,changefeed lag is more and more after inject pdleader network partition last for 10m,ticdc changefeed lag is more and more Mar 9, 2023
@fubinzh
Copy link

fubinzh commented Mar 13, 2023

/severity critical

@fubinzh
Copy link

fubinzh commented Mar 13, 2023

/remove-severity critical

@fubinzh
Copy link

fubinzh commented Mar 13, 2023

/severity major

@nongfushanquan
Copy link
Contributor

/assign @overvenus

@VelocityLight VelocityLight added the affects-7.1 This bug affects the 7.1.x(LTS) versions. label Apr 20, 2023
@nongfushanquan
Copy link
Contributor

duplicated with #8808 and fixed by #9118

@nongfushanquan
Copy link
Contributor

/close

@ti-chi-bot ti-chi-bot bot closed this as completed Jun 9, 2023
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jun 9, 2023

@nongfushanquan: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@nongfushanquan
Copy link
Contributor

/remove-label affects-7.1

@ti-chi-bot ti-chi-bot bot removed the affects-7.1 This bug affects the 7.1.x(LTS) versions. label Oct 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants