TiCDC admin job blocked, and etcd worker may be stuck

[3AceShowHand]

What did you do?

tiup cluster deploy lingjin-kafka-cluster-test v5.5.0-nightly-20220104 kafka-cluster-testing.yaml -p -u root

tiup ctl:v5.5.0-nightly-20220104 cdc changefeed create --pd=http://172.16.7.51:2379 --sink-uri="kafka://172.16.6.111:9092/normal-incremental-test?protocol=open-protocol"

tiup ctl:v5.5.0-nightly-20220104 cdc changefeed pause --pd=http://172.16.7.51:2379 --changefeed-id="bbadf644-73eb-4d07-a8f3-1f245f82b9a9"

What did you expect to see?

changefeed's state become stopped

What did you see instead?

it is still normal.

blocked for a few minutes, and then become stopped.

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Git Hash:

fc70dbde8

[3AceShowHand]

cdc.log

[3AceShowHand]

it looks that the admin job was handled, but it was stucked when try to update the etcd.

[asddongmen]

[2022/01/04 15:18:56.439 +08:00] [WARN] [client.go:226] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=3m55.999651547s]
[2022/01/04 15:18:56.992 +08:00] [WARN] [etcd_worker.go:242] ["EtcdWorker reactor tick took too long"] [duration=4m5.165662023s]

It seems that the owner is stucked.
The EtcdWorker reactor here is owner.

[3AceShowHand]

kafka min.insync.replicas is set to 2, and replication-factor is 1

[3AceShowHand]

server.log

[3AceShowHand]

cdc-126.log

[Rustin170506]

Duplicate of #4241