scheduler(cdc): add randomness to the scheduler(V2)

[liuzix]

What problem does this PR solve?

Issue Number: close #4203 close #4204 ref #3654

• Since the scheduler is per changefeed, when the changefeeds are many and tables in each changefeed are few, workload imbalance among nodes is likely to happen (table scheduler is changefeed level, which could lead to imbalance in some scenarios #3654).
• If the tables are created in an order that correlates to the workload, the workload can be imbalanced (Scheduler v2 may produce imbalance workload in some scenarios  #4203).
• When the total number of tables is large, finding a target to dispatch a table can be slow and consumes a lot of CPU (Owner is blocked by scheduler v2 when there are many tables need to be scheduled #4204).

What is changed and how it works?

• Introduced randomness to scheduler/balancer to 1) add some randomness in choosing dispatch target, 2) add some randomness in choosing victims when rebalancing.
• Avoided copying table records when trying to find victims when rebalancing.

Check List

Tests

• Unit test
• Integration test

Performance

• A benchmark is added to verify the fix for Owner is blocked by scheduler v2 when there are many tables need to be scheduled #4204
• Without the changes

goos: linux
goarch: amd64
pkg: github.com/pingcap/tiflow/cdc/scheduler
cpu: Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz
BenchmarkAddTable
BenchmarkAddTable-16    	   10000	    519251 ns/op
PASS

• With the changes

goos: linux
goarch: amd64
pkg: github.com/pingcap/tiflow/cdc/scheduler
cpu: Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz
BenchmarkAddTable
BenchmarkAddTable-16    	   76887	     14000 ns/op
PASS

Side effects

• Increased code complexity

Related changes

• Need to cherry-pick to the release branch

Release note

None

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• amyangfei
• overvenus

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[codecov-commenter]

Codecov Report

Merging #4202 (f1d560d) into master (08da001) will increase coverage by 0.0146%.
The diff coverage is 53.0550%.

Flag	Coverage Δ
cdc	58.5891% <57.2872%> (-0.0557%)	⬇️
dm	52.2819% <42.2746%> (+0.0273%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master      #4202        +/-   ##
================================================
+ Coverage   55.1722%   55.1868%   +0.0146%     
================================================
  Files           485        489         +4     
  Lines         59829      60605       +776     
================================================
+ Hits          33009      33446       +437     
- Misses        23484      23792       +308     
- Partials       3336       3367        +31     

[overvenus]

Could you add a benchmark about addTable?

[liuzix]

/run-leak-tests

[overvenus]

=== RUN   TestBalancerFindTargetTied
    balancer_test.go:234: 
        	Error Trace:	balancer_test.go:234
        	Error:      	Should be false
        	Test:       	TestBalancerFindTargetTied
--- FAIL: TestBalancerFindTargetTied (0.00s)

test fails

[liuzix]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 5f40346

[overvenus]

/run-verify
/run-leak-test

[ti-chi-bot]

In response to a cherrypick label: new pull request created: #4236.