CDC consumes too much cpu, and lag goes up from 2s to 4s every few hours

[fubinzh]

What did you do?

1. TiDB cluster with 2 CDC nodes (24c32g)
   cdc config:

 config: |
      log-file = "/var/lib/ticdc/log/ticdc.log"

      [debug]
        [debug.kv-client]
          frontier-concurrent = 32

TiKV cdc.min-ts-interval = "50ms"
2. CDC simiple protocol changefeed create to sync 4k tables (throughput <20k, rowwidth<10k)

What did you expect to see?

CDC lag should be stable

What did you see instead?

CDC lag goes up from 2s to 4s every few hours

Versions of the cluster

Upstream TiKV version (execute tikv-server --version):

[root@upstream-tikv-0 /]# /tikv-server -V
TiKV
Release Version:   8.1.0
Edition:           Community
Git Commit Hash:   6ebf7ef3a1627a59062ddfbe32b33006df881950

TiCDC version (execute cdc version):

[root@upstream-ticdc-0 /]# /cdc version
Release Version: v8.1.0-lock-test
Git Commit Hash: 3afce276e3a79788be4e294bd4f322d844cffea7
Git Branch: heads/refs/tags/v8.1.0-lock-test
UTC Build Time: 2024-05-10 03:24:05
Go Version: go version go1.21.6 linux/amd64
Failpoint Build: false

[fubinzh]

/severity major

[hicqu]

lag increasings always keep about 20 minutes, and intervals are always about 6h. I think there are some period changes.

[hicqu]

I find some doubts:

Maybe we can adjust some parameters to fix it.