table scheduler is changefeed level, which could lead to imbalance in some scenarios

[amyangfei]

Is your feature request related to a problem?

1. Setup a TiDB cluster with 3 TiCDC nodes. Create 10 tables in upstream
2. Create 10 changefeeds, each changefeed replicates one table.

test with cdc master@pingcap/ticdc@39cfb3e

Describe the feature you'd like

The tables replication stream can be dispatched to 3 TiCDC nodes evenly.
But we will observe the tables are not even among 3 TiCDC nodes. This is caused by the table scheduler is in changefeed level, ref: https://github.com/pingcap/ticdc/blob/39cfb3eda4fee3a340a8b8dfd618195967455359/cdc/owner/changefeed.go#L76-L79
and the scheduler algorithm is based on min workload with a random map accessing. https://github.com/pingcap/ticdc/blob/39cfb3eda4fee3a340a8b8dfd618195967455359/cdc/owner/scheduler.go#L206

workloads is a map with some undeterminable accessing, but not enough.

workload ratio is 7:2:1 among three TiCDC nodes.

/tidb/cdc/task/workload/95f9f6e2-7edd-41cf-ad12-c5e91e20c624/test-cf-1
{"57":{"workload":1}}
/tidb/cdc/task/workload/ce10785f-0c81-4882-b0ee-d4b702178549/test-cf-2
{"57":{"workload":1}}
/tidb/cdc/task/workload/ce10785f-0c81-4882-b0ee-d4b702178549/test-cf-3
{"57":{"workload":1}}
/tidb/cdc/task/workload/dbc97296-ecb8-457c-9d62-9f89220635ad/test-cf-10
{"57":{"workload":1}}
/tidb/cdc/task/workload/dbc97296-ecb8-457c-9d62-9f89220635ad/test-cf-4
{"57":{"workload":1}}
/tidb/cdc/task/workload/dbc97296-ecb8-457c-9d62-9f89220635ad/test-cf-5
{"57":{"workload":1}}
/tidb/cdc/task/workload/dbc97296-ecb8-457c-9d62-9f89220635ad/test-cf-6
{"57":{"workload":1}}
/tidb/cdc/task/workload/dbc97296-ecb8-457c-9d62-9f89220635ad/test-cf-7
{"57":{"workload":1}}
/tidb/cdc/task/workload/dbc97296-ecb8-457c-9d62-9f89220635ad/test-cf-8
{"57":{"workload":1}}
/tidb/cdc/task/workload/dbc97296-ecb8-457c-9d62-9f89220635ad/test-cf-9
{"57":{"workload":1}}

Describe alternatives you've considered

In the short term, we can add more random factors to changefeed scheduler, such as
In the long term, we can add better scheduler algorithm.

Teachability, Documentation, Adoption, Migration Strategy

No response

[amyangfei]

The randomness optimization is added since v5.4.0