scheduler: allow balance-region-scheduler create multiple operators #4008

bufferflies · 2021-08-19T06:51:07Z

Signed-off-by: bufferflies [email protected]

What problem does this PR solve?

fix #3744, scheduler can generate some operators in scheduler time

What is changed and how it works?

Check List

Tests

Unit test
Manual test (add detailed scripts or steps below)

Code changes

Side effects

Related changes

Release note

Please add a release note.

If you don't think this PR needs a release note then fill it with None.

Signed-off-by: bufferflies <[email protected]>

ti-chi-bot · 2021-08-19T06:51:08Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

disksing

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: bufferflies <[email protected]>

… feature/schedulerOps

Signed-off-by: bufferflies <[email protected]>

codecov · 2021-08-20T11:15:56Z

Codecov Report

Merging #4008 (91b82a4) into master (5e0c212) will decrease coverage by 0.02%.
The diff coverage is 92.00%.

@@            Coverage Diff             @@
##           master    #4008      +/-   ##
==========================================
- Coverage   74.70%   74.67%   -0.03%     
==========================================
  Files         260      260              
  Lines       26625    26638      +13     
==========================================
+ Hits        19889    19893       +4     
- Misses       4950     4956       +6     
- Partials     1786     1789       +3

Flag	Coverage Δ
unittests	`74.67% <92.00%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/schedulers/balance_region.go	`88.63% <85.71%> (-0.54%)`	⬇️
server/schedule/operator_controller.go	`85.59% <100.00%> (+1.24%)`	⬆️
server/schedulers/shuffle_hot_region.go	`55.55% <0.00%> (-10.11%)`	⬇️
server/id/id.go	`76.19% <0.00%> (-4.77%)`	⬇️
server/region_syncer/client.go	`77.46% <0.00%> (-4.23%)`	⬇️
pkg/etcdutil/etcdutil.go	`84.70% <0.00%> (-3.53%)`	⬇️
pkg/dashboard/adapter/manager.go	`79.78% <0.00%> (-3.20%)`	⬇️
server/server.go	`71.40% <0.00%> (-0.45%)`	⬇️
server/cluster/cluster.go	`82.33% <0.00%> (-0.36%)`	⬇️
client/client.go	`72.50% <0.00%> (-0.14%)`	⬇️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5e0c212...91b82a4. Read the comment docs.

server/cluster/coordinator_test.go

server/schedule/operator_controller.go

Signed-off-by: bufferflies <[email protected]>

server/schedulers/balance_test.go

server/schedulers/hot_region_config.go

Signed-off-by: bufferflies <[email protected]>

server/schedulers/balance_test.go

Signed-off-by: bufferflies <[email protected]>

server/schedulers/balance_region.go

Signed-off-by: bufferflies <[email protected]>

… feature/schedulerOps

Signed-off-by: bufferflies <[email protected]>

rleungx · 2021-11-29T04:37:51Z

server/cluster/coordinator_test.go

@@ -987,6 +987,7 @@ func (s *testOperatorControllerSuite) TestStoreOverloaded(c *C) {
 	opt.SetAllStoresLimit(storelimit.RemovePeer, 600)
 	time.Sleep(time.Second)
 	for i := 0; i < 10; i++ {
+


remove this line

rleungx · 2021-11-29T04:38:08Z

server/cluster/coordinator_test.go

@@ -1024,10 +1025,10 @@ func (s *testOperatorControllerSuite) TestStoreOverloadedWithReplace(c *C) {
 	c.Assert(oc.AddOperator(op2), IsTrue)
 	op3 := newTestOperator(1, tc.GetRegion(2).GetRegionEpoch(), operator.OpRegion, operator.AddPeer{ToStore: 1, PeerID: 3})
 	c.Assert(oc.AddOperator(op3), IsFalse)
-	c.Assert(lb.Schedule(tc), IsNil)
+	c.Assert(len(lb.Schedule(tc)), Equals, 0)


rleungx · 2021-11-29T05:14:47Z

server/schedulers/balance_region.go

@@ -51,6 +58,7 @@ func init() {
 		if err := decoder(conf); err != nil {
 			return nil, err
 		}
+		log.Info("conf", zap.Any("config", conf))


prefer to make it more clear, like balance region config?

ti-chi-bot · 2022-01-15T03:12:32Z

@bufferflies: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

CabinfeverB · 2022-02-16T10:11:15Z

server/schedulers/balance_region.go

@@ -43,6 +43,13 @@ func init() {
 			}
 			conf.Ranges = ranges
 			conf.Name = BalanceRegionName
+			conf.Batch = 1
+			if len(args) >= 3 {


args could be greater than 3 because there could be multi ranges

AndreMouche · 2022-03-10T07:14:34Z

server/schedule/operator_controller.go

+			oc.PromoteWaitingOperator()
+		} else {
+			for i := 0; i < added; i++ {
+				oc.PromoteWaitingOperator()


should we check ops[id] is merge?

AndreMouche · 2022-03-10T07:20:48Z

server/schedule/operator_controller.go

@@ -313,7 +312,17 @@ func (oc *OperatorController) AddWaitingOperator(ops ...*operator.Operator) int

 	oc.Unlock()
 	operatorWaitCounter.WithLabelValues(ops[0].Desc(), "promote-add").Inc()
-	oc.PromoteWaitingOperator()
+
+	if added > 0 {


return as soon as possible so the maintainer needn't keep too many conditions in mind. For example, here we could return when we find added=0 so we needn't keep in mind the condition add=0 anymore.

AndreMouche

Rest LGTM

…rs (#4652) ref #4008, ref #4610 speed up balance leader by batch Signed-off-by: Cabinfever_B <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

nolouch · 2022-07-18T05:53:16Z

close it? I think the current bottleneck is the store limit. you can reopen it if needed.

scheduler can generate many ops by once

c456412

Signed-off-by: bufferflies <[email protected]>

ti-chi-bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Aug 19, 2021

ti-chi-bot requested review from disksing and nolouch August 19, 2021 06:51

bufferflies added 7 commits August 19, 2021 19:57

pass ut

dc7f5aa

Signed-off-by: bufferflies <[email protected]>

pass ut

f10f7ca

Signed-off-by: bufferflies <[email protected]>

promote dependence added

e05127a

Signed-off-by: bufferflies <[email protected]>

promote dependence added

d5c5484

Signed-off-by: bufferflies <[email protected]>

remove unlock

ed52884

Signed-off-by: bufferflies <[email protected]>

pass ut

a9bf43a

Signed-off-by: bufferflies <[email protected]>

Merge branch 'master' into feature/schedulerOps

8cfbb8c

bufferflies changed the title ~~scheduler can generate many ops by once~~ [scheduler]:scheduler can generate many ops by once Aug 20, 2021

bufferflies added 3 commits August 20, 2021 14:15

Merge branch 'feature/schedulerOps' of github.com:bufferflies/pd into…

e94e0ff

… feature/schedulerOps

Merge branch 'master' into feature/schedulerOps

bb137a6

pass ut

5aef395

Signed-off-by: bufferflies <[email protected]>

bufferflies changed the title ~~[scheduler]:scheduler can generate many ops by once~~ scheduler:scheduler can generate many ops by once Aug 20, 2021

bufferflies added 2 commits August 20, 2021 18:11

pass ut

b6aea3f

Signed-off-by: bufferflies <[email protected]>

pass ut

6ec852e

Signed-off-by: bufferflies <[email protected]>

disksing reviewed Aug 23, 2021

View reviewed changes

server/cluster/coordinator_test.go Outdated Show resolved Hide resolved

server/cluster/coordinator_test.go Outdated Show resolved Hide resolved

server/schedule/operator_controller.go Outdated Show resolved Hide resolved

disksing changed the title ~~scheduler:scheduler can generate many ops by once~~ scheduler: allow balance-region-scheduler create multiple operators Aug 23, 2021

assert wrong use

4d1fe4b

Signed-off-by: bufferflies <[email protected]>

disksing reviewed Aug 24, 2021

View reviewed changes

server/schedulers/balance_test.go Outdated Show resolved Hide resolved

server/schedulers/hot_region_config.go Outdated Show resolved Hide resolved

pass ut

0067b49

Signed-off-by: bufferflies <[email protected]>

disksing reviewed Aug 24, 2021

View reviewed changes

server/schedulers/balance_test.go Outdated Show resolved Hide resolved

server/schedulers/balance_test.go Outdated Show resolved Hide resolved

server/schedulers/balance_test.go Outdated Show resolved Hide resolved

bufferflies added 2 commits August 24, 2021 16:56

pass ut

ca34525

Signed-off-by: bufferflies <[email protected]>

Merge branch 'master' into feature/schedulerOps

4dcac50

disksing approved these changes Aug 25, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Aug 25, 2021

rleungx reviewed Aug 25, 2021

View reviewed changes

server/schedulers/balance_region.go Show resolved Hide resolved

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 29, 2021

bufferflies added 2 commits September 2, 2021 16:00

add batch influence and add batch test

294da5b

Signed-off-by: bufferflies <[email protected]>

Merge branch 'feature/schedulerOps' of github.com:bufferflies/pd into…

23a6878

… feature/schedulerOps

bufferflies requested a review from rleungx September 22, 2021 02:56

resolve conflict

0923ba8

Signed-off-by: bufferflies <[email protected]>

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 11, 2021

add annotation

91b82a4

Signed-off-by: bufferflies <[email protected]>

rleungx reviewed Nov 29, 2021

View reviewed changes

nolouch mentioned this pull request Dec 14, 2021

Improve the speed of the balance-leader-scheduler #4454

Closed

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 15, 2022

nolouch mentioned this pull request Jan 25, 2022

Speed up balance leader #4610

Closed

CabinfeverB reviewed Feb 16, 2022

View reviewed changes

CabinfeverB mentioned this pull request Feb 18, 2022

scheduler: allow balance-leader-scheduler generate multiple operators #4652

Merged

AndreMouche reviewed Mar 10, 2022

View reviewed changes

nolouch closed this Jul 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

scheduler: allow balance-region-scheduler create multiple operators #4008

scheduler: allow balance-region-scheduler create multiple operators #4008

bufferflies commented Aug 19, 2021

ti-chi-bot commented Aug 19, 2021 •

edited

Loading

codecov bot commented Aug 20, 2021 •

edited

Loading

rleungx Nov 29, 2021

rleungx Nov 29, 2021

rleungx Nov 29, 2021

ti-chi-bot commented Jan 15, 2022

CabinfeverB Feb 16, 2022

AndreMouche Mar 10, 2022

AndreMouche Mar 10, 2022

AndreMouche left a comment

nolouch commented Jul 18, 2022 •

edited

Loading

scheduler: allow balance-region-scheduler create multiple operators #4008

scheduler: allow balance-region-scheduler create multiple operators #4008

Conversation

bufferflies commented Aug 19, 2021

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Aug 19, 2021 • edited Loading

codecov bot commented Aug 20, 2021 • edited Loading

Codecov Report

rleungx Nov 29, 2021

Choose a reason for hiding this comment

rleungx Nov 29, 2021

Choose a reason for hiding this comment

rleungx Nov 29, 2021

Choose a reason for hiding this comment

ti-chi-bot commented Jan 15, 2022

CabinfeverB Feb 16, 2022

Choose a reason for hiding this comment

AndreMouche Mar 10, 2022

Choose a reason for hiding this comment

AndreMouche Mar 10, 2022

Choose a reason for hiding this comment

AndreMouche left a comment

Choose a reason for hiding this comment

nolouch commented Jul 18, 2022 • edited Loading

ti-chi-bot commented Aug 19, 2021 •

edited

Loading

codecov bot commented Aug 20, 2021 •

edited

Loading

nolouch commented Jul 18, 2022 •

edited

Loading