Randomise leader election

[CAJan93]

Is your feature request related to a problem? Please describe.

I noticed that we are currently always waiting for the same amount of time during leader election (see here)

let mut tick_interval = tokio::time::interval(Duration::from_secs(lease_time / 2));
loop {
    tick_interval.tick().await;

Maybe it makes sense to randomise this a bit and give different instances a slightly different tick_interval. This way we would space out requests a bit.

CC @yezizp2012

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

Raft also uses randomised timeouts during leader election to avoid collisions

[ArshErgon]

@CAJan93 whats the minimum value lease_time can have? and can you tell me how to test that particular file?

[yezizp2012]

Well, I think it's not that necessary to do this, because: 1. eventually we will replace it with APIs of etcd lib or (some big refactoring); 2. even in current implementation, etcd will guarantee only one node will register successfully, it will ensure sequential writing of each requests.

[CAJan93]

I agree, it is not needed, but it would still reduce the likelihood of simultaneous election-runs that may then be repeated. Therefore I introduced this randomisation in #6466
(it really is just very few lines of code)