fix panic when restart after removeMember #13479
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ahrtr
reviewed
Nov 19, 2021
|
PTAL @ptabor |
@serathius @ptabor This issue can't be reproduced on 3.5.1, so no need to get it included in the plans for v3.5.2. Please see my comments issuecomment-973757868 and issuecomment-975024314. But I agree that it is indeed an issue in the main branch (3.6). |
ahrtr
reviewed
Jan 26, 2022
| @@ -339,24 +339,29 @@ func (t *Transport) RemoveAllPeers() { | |||
|
|
|||
| // the caller of this function must have the peers mutex. | |||
| func (t *Transport) removePeer(id types.ID) { | |||
| if peer, ok := t.peers[id]; ok { | |||
| peer, ok := t.peers[id] | |||
There was a problem hiding this comment.
You'd better add a comment something like below:
etcd may remove a member again on startup due to WAL files replaying.
|
Please also squash the two commits and rebase this PR. |
// etcd may remove a member again on startup due to WAL files replaying. // (see: etcd-io#13479)
|
Three submits are inserted with a lot of Push, Squash failed, and re-raised a PR to integrate commit. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I used the ETCD3.5 version to stably reproduce this problem, and the process is divided into the following steps:
use this code quickly do an auto snapshot: Do snapshot now #13474
#13466
The logic of the overall error is basically clear. The key is to take a snapshot before deleting the node.
Then the biggest problem is in the above code. If the index of the snapshot is less than the ConsistentIndex, start with the old backend. The old backend has the latest member information (including the deleted node information), but the WAL playback is based on the snapshotIndex. Play back at the starting position, so the removeMember raft changeConfig will be played back.