raft: group gets stuck when leader is removed

[tbg]

See the "document a problem" commit in #11037.

When a leader is removed via a conf change, it will retain its leadership indefinitely (continues to heartbeat followers). However, it does not accept incoming proposals.

[xiang90]

The expected behavior is what? The removed leader should stop heartbeating and keep silence?

[tbg]

At the very least, the leader should immediately step down when it applies the configuration change. But I think even more so, we probably want to send an MsgTimeoutNow to one follower that has helped commit the config change (i.e. the one with the largest Match), which in the very common case would minimize the disruption.

[xiang90]

@tbg

send an MsgTimeoutNow to one follower that has helped commit the config change (i.e. the one with the largest Match), which in the very common case would minimize the disruption.

For etcd, we did this in the application layer I believe.

[jingyih]

Yes, this is handled in application layer. When leader is removed, during its stop process, it will transfer leadership to the longest connected voting member in cluster.

Ref:

etcd/etcdserver/server.go

Lines 1502 to 1503 in 9b29151

	func (s *EtcdServer) Stop() {
	if err := s.TransferLeadership(); err != nil {

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.