checker: add disconnected check when fix orphan peers #7240

lhy1024 · 2023-10-20T03:53:53Z

What problem does this PR solve?

Issue Number: Close #7249

What is changed and how does it work?

Check List

Tests

Unit test

Release note

None.

ti-chi-bot · 2023-10-20T03:53:56Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

Connor1996
nolouch

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: lhy1024 <[email protected]>

lhy1024 · 2023-10-20T09:26:27Z

pkg/schedule/checker/rule_checker.go

@@ -494,6 +504,9 @@ loopFits:
 					return operator.CreatePromoteLearnerOperatorAndRemovePeer("replace-down-peer-with-orphan-peer", c.cluster, region, orphanPeer, pinDownPeer)
 				case orphanPeerRole == metapb.PeerRole_Voter && destRole == metapb.PeerRole_Learner:
 					return operator.CreateDemoteLearnerOperatorAndRemovePeer("replace-down-peer-with-orphan-peer", c.cluster, region, orphanPeer, pinDownPeer)
+				case orphanPeerRole == metapb.PeerRole_Voter && destRole == metapb.PeerRole_Voter &&
+					c.cluster.GetStore(pinDownPeer.GetStoreId()).IsDisconnected():
+					return operator.CreateRemovePeerOperator("remove-orphan-peer", c.cluster, 0, region, pinDownPeer.GetStoreId())
 				default:
 					// destRole should not same with orphanPeerRole. if role is same, it fit with orphanPeer should be better than now.


I am not sure whether we may remove this comment. cc @nolouch

Connor1996 · 2023-10-23T04:00:51Z

pkg/schedule/checker/rule_checker.go

+			// Isdisconnected is more strictly than IsUnhealthy.
+			if c.cluster.GetStore(p.GetStoreId()).IsDisconnected() {
+				hasUnhealthyFit = true
+				pinDownPeer = p


if it's disconnected, we don't need to hurry to remove it.

how about skipping the disconnected store when fixing orphan peers, only removing them when the store is down or the store is normal?

forget about it, replacing the disconnect store's peer with orphan peer is reasonable.

may meet issues like #4045. we should make sure the replacement peer is healthy (orphan peer).

these peers are in disconnect or down store, which they will change to down peer.

@lhy1024 I mean make sure the remaining peer is healthy.

I added check about disconnect store

Connor1996

LGTM

Connor1996

Please update PR description

lhy1024 · 2023-10-25T04:46:39Z

PTAL @nolouch

pkg/schedule/checker/rule_checker.go

Signed-off-by: lhy1024 <[email protected]>

nolouch · 2023-10-26T08:25:04Z

/merge

ti-chi-bot · 2023-10-26T08:25:06Z

@nolouch: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-10-26T08:25:08Z

This pull request has been accepted and is ready to merge.

Commit hash: 618407f

close tikv#7249 Signed-off-by: lhy1024 <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: lhy1024 <[email protected]>

close #7249 Signed-off-by: lhy1024 <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

close tikv#7249 Signed-off-by: lhy1024 <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

close tikv#7249 Signed-off-by: lhy1024 <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: lhy1024 <[email protected]>

ti-chi-bot · 2024-01-23T17:52:01Z

/cherry-pick release-6.5

close tikv#7249 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2024-01-23T17:52:38Z

In response to a cherrypick label: new pull request created to branch release-6.5: #7751.

ti-chi-bot · 2024-01-23T17:53:14Z

@ti-chi-bot: new pull request could not be created: failed to create pull request against tikv/pd#release-6.5 from head ti-chi-bot:cherry-pick-7240-to-release-6.5: status code 422 not one of [201], body: {"message":"Validation Failed","errors":[{"resource":"PullRequest","code":"custom","message":"A pull request already exists for ti-chi-bot:cherry-pick-7240-to-release-6.5."}],"documentation_url":"https://docs.github.com/rest/pulls/pulls#create-a-pull-request"}

In response to this:

/cherry-pick release-6.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

close tikv#7249 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2024-01-30T04:36:33Z

/cherry-pick release-7.5

ti-chi-bot · 2024-01-30T04:37:08Z

In response to a cherrypick label: new pull request created to branch release-7.5: #7776.

close tikv#7249 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2024-01-30T04:37:44Z

@ti-chi-bot: new pull request could not be created: failed to create pull request against tikv/pd#release-7.5 from head ti-chi-bot:cherry-pick-7240-to-release-7.5: status code 422 not one of [201], body: {"message":"Validation Failed","errors":[{"resource":"PullRequest","code":"custom","message":"A pull request already exists for ti-chi-bot:cherry-pick-7240-to-release-7.5."}],"documentation_url":"https://docs.github.com/rest/pulls/pulls#create-a-pull-request"}

In response to this:

/cherry-pick release-7.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

close tikv#7249 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2024-02-02T19:51:09Z

In response to a cherrypick label: new pull request created to branch release-7.1: #7797.

close tikv#7249 Signed-off-by: ti-chi-bot <[email protected]>

lhy1024 requested a review from Connor1996 October 20, 2023 03:53

ti-chi-bot bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/needs-triage-completed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 20, 2023

lhy1024 force-pushed the check-disconnect branch from 83ad962 to 7791814 Compare October 20, 2023 07:09

ti-chi-bot bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 20, 2023

checker: add disconnected check when fix orphan peers

508824c

Signed-off-by: lhy1024 <[email protected]>

lhy1024 force-pushed the check-disconnect branch from 1316a22 to 508824c Compare October 20, 2023 09:23

lhy1024 commented Oct 20, 2023

View reviewed changes

lhy1024 requested a review from nolouch October 20, 2023 09:26

Connor1996 reviewed Oct 23, 2023

View reviewed changes

Connor1996 approved these changes Oct 23, 2023

View reviewed changes

ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label Oct 23, 2023

Connor1996 reviewed Oct 23, 2023

View reviewed changes

ti-chi-bot bot removed do-not-merge/needs-linked-issue do-not-merge/needs-triage-completed labels Oct 25, 2023

nolouch reviewed Oct 26, 2023

View reviewed changes

pkg/schedule/checker/rule_checker.go Outdated Show resolved Hide resolved

address comments

618407f

Signed-off-by: lhy1024 <[email protected]>

lhy1024 force-pushed the check-disconnect branch from 382c2f1 to 618407f Compare October 26, 2023 08:03

lhy1024 requested a review from nolouch October 26, 2023 08:08

nolouch approved these changes Oct 26, 2023

View reviewed changes

ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Oct 26, 2023

ti-chi-bot bot merged commit f9b476d into tikv:master Oct 26, 2023

lhy1024 deleted the check-disconnect branch October 26, 2023 09:04

lhy1024 mentioned this pull request Oct 30, 2023

checker: add disconnected check when fix orphan peers (#7240) #7286

Merged

lhy1024 mentioned this pull request Nov 8, 2023

checker: reduces the probability of deleting normal peers when the store becomes unavailable (#7249) #7330

Merged

lhy1024 mentioned this pull request Nov 8, 2023

checker: reduces the probability of deleting normal peers when the store becomes unavailable (#7249) #7334

Merged

lhy1024 mentioned this pull request Nov 8, 2023

checker: reduces the probability of deleting normal peers when the store becomes unavailable (#7249) #7337

Merged

ti-chi-bot added the needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. label Jan 23, 2024

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Jan 23, 2024

This is an automated cherry-pick of tikv#7240

3d9d326

close tikv#7249 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Jan 23, 2024

checker: add disconnected check when fix orphan peers (#7240) #7751

Closed

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Jan 23, 2024

This is an automated cherry-pick of tikv#7240

6ef8570

close tikv#7249 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot added the needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. label Jan 30, 2024

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Jan 30, 2024

This is an automated cherry-pick of tikv#7240

4f9d241

close tikv#7249 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Jan 30, 2024

checker: add disconnected check when fix orphan peers (#7240) #7776

Closed

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Jan 30, 2024

This is an automated cherry-pick of tikv#7240

5799824

close tikv#7249 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot added the needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. label Feb 2, 2024

ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this pull request Feb 2, 2024

This is an automated cherry-pick of tikv#7240

0d29602

close tikv#7249 Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Feb 2, 2024

checker: add disconnected check when fix orphan peers (#7240) #7797

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

checker: add disconnected check when fix orphan peers #7240

checker: add disconnected check when fix orphan peers #7240

lhy1024 commented Oct 20, 2023 •

edited

Loading

ti-chi-bot bot commented Oct 20, 2023 •

edited

Loading

lhy1024 Oct 20, 2023

Connor1996 Oct 23, 2023

lhy1024 Oct 23, 2023

Connor1996 Oct 23, 2023

nolouch Oct 26, 2023

lhy1024 Oct 26, 2023

nolouch Oct 26, 2023 •

edited

Loading

lhy1024 Oct 26, 2023

Connor1996 left a comment

Connor1996 left a comment

lhy1024 commented Oct 25, 2023

nolouch commented Oct 26, 2023

ti-chi-bot bot commented Oct 26, 2023

ti-chi-bot bot commented Oct 26, 2023

ti-chi-bot commented Jan 23, 2024

ti-chi-bot commented Jan 23, 2024

ti-chi-bot commented Jan 23, 2024

ti-chi-bot commented Jan 30, 2024

ti-chi-bot commented Jan 30, 2024

ti-chi-bot commented Jan 30, 2024

ti-chi-bot commented Feb 2, 2024

checker: add disconnected check when fix orphan peers #7240

checker: add disconnected check when fix orphan peers #7240

Conversation

lhy1024 commented Oct 20, 2023 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Oct 20, 2023 • edited Loading

lhy1024 Oct 20, 2023

Choose a reason for hiding this comment

Connor1996 Oct 23, 2023

Choose a reason for hiding this comment

lhy1024 Oct 23, 2023

Choose a reason for hiding this comment

Connor1996 Oct 23, 2023

Choose a reason for hiding this comment

nolouch Oct 26, 2023

Choose a reason for hiding this comment

lhy1024 Oct 26, 2023

Choose a reason for hiding this comment

nolouch Oct 26, 2023 • edited Loading

Choose a reason for hiding this comment

lhy1024 Oct 26, 2023

Choose a reason for hiding this comment

Connor1996 left a comment

Choose a reason for hiding this comment

Connor1996 left a comment

Choose a reason for hiding this comment

lhy1024 commented Oct 25, 2023

nolouch commented Oct 26, 2023

ti-chi-bot bot commented Oct 26, 2023

ti-chi-bot bot commented Oct 26, 2023

ti-chi-bot commented Jan 23, 2024

ti-chi-bot commented Jan 23, 2024

ti-chi-bot commented Jan 23, 2024

ti-chi-bot commented Jan 30, 2024

ti-chi-bot commented Jan 30, 2024

ti-chi-bot commented Jan 30, 2024

ti-chi-bot commented Feb 2, 2024

lhy1024 commented Oct 20, 2023 •

edited

Loading

ti-chi-bot bot commented Oct 20, 2023 •

edited

Loading

nolouch Oct 26, 2023 •

edited

Loading