lightning: generate and send region job more smoothly

[lance6716]

What problem does this PR solve?

Issue Number: close #42456

Problem Summary:

What is changed and how it works?

1. in the old PR we try to generate all region jobs at once, then process jobs, and generate jobs for unfinished range again, then process again... In order to reduce the lag caused by generating, we now start processing jobs once a key range has generated job, and start generating job once the job need to be generated again, like the region is changed. Also, these work are done concurrently.
2. unfinishedRange is removed because we don't have a barrier to stop all processing before use unfinishedRange to generate job. Now the after the job is initially generated, the key range of job will be totally finished by worker, or shrank by partitially ingestion. The job may also be splitted into multiple job whose total key range equals to the original job.
3. In order to let the workers schedule the backoffing job for later processing, a regionJobRetryer is introduced. Now the workflow becomes

	/*
	   [prepareAndSendJob]-----jobToWorkerCh--->[workers]
	                        ^                       |
	                        |                jobFromWorkerCh
	                        |                       |
	                        |                       v
	               [regionJobRetryer]<--[dispatchJobGoroutine]-->done
	*/

Reviews please help check if there's deadlock and if there's a better way to implement it.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• D3Hunter
• gozssky

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[lance6716]

@gozssky Can you give some suggestions on the pattern? Currently regionJob has different stages, multiple workers hold the resouces (pd client, ...) that can advance the stage, and producer/consumer are communicated by channel

[lance6716]

I can think of two design patterns

one is using goroutines to acquire "worker resource" and then start job, when the job need to sleep some period to retry, release the worker resource. like

for _, j := range jobs {
  w := workerPool.Acquire()
  go func() {j.run(w); w.Release()}
}

and

// when meet error
j.w.Release()
sleep()
j.w := workerPool.Acquire()

In this method, I want to rely on golang's goroutine scheduling to keep N system thread where N is worker count. But I'm afraid we will spawn too many sleeping goroutines and affect the runtime.

In the refactoring PR I want to do manually scheduling, where try to keep the N worker goroutines always busy and put the job back when the job needs retry. To solve #42456, I should put every time consuming work in worker to make use of concurrency, and remove the barrier in the single-thread producer goroutine. Do you have any suggestions? @gozssky @D3Hunter

[lance6716]

/run-integration-br-test

[lance6716]

/run-integration-br-test

[lance6716]

/retest

[lance6716]

/run-integration-br-test

[lance6716]

/run-integration-br-test

[lance6716]

/hold

need to fix #42780 (comment)

[lance6716]

/unhold

[lance6716]

/run-integration-br-test

[lance6716]

/run-integration-br-test

[lance6716]

/lgtm cancel

fix a problem that sending to putBackCh will block retryer.push 45b208d

because retryer.push must access the inner queue, sending to putBackCh must not access the inner queue to avoid blocking the former. So we should use a temporary toPutBack to reduce the locking scope. @D3Hunter @gozssky

@D3Hunter @gozssky PTAL

[lance6716]

/run-integration-br-test

[lance6716]

lightning integration test passed https://ci.pingcap.net/blue/organizations/jenkins/br_ghpr_unit_and_integration_test/detail/br_ghpr_unit_and_integration_test/15798/pipeline

[lance6716]

/retest

[lance6716]

/run-integration-br-test

[lance6716]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 3f90366

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-7.0: #43035.