etcdserver: ensure hardstate is persisten before applying committed entries

[ahrtr]

// If the hardstate isn't empty, we must sync the hardstate to WAL file before
// applying the committed entries. Otherwise, etcd may have already responded
// to the client, but the hardstate isn't persisted to the WAL file. When etcd
// starts again, it loads the stale hardstate, and accordingly a linearizable
// request may get the stale readIndex (actually committed Index), eventually
// it won't wait for the latest applied data before responding to the client.

The solution is similar to #14400. Need to evaluate the impact on performance.

[serathius]

Hmm, it's an interesting that etcd didn't require flushing move of commit index.

Etcd starts from sending a publish request that requires a quorum, it doesn't even wait for the request to be processed, just for a proposal. So, to avoid going back in time, etcd just requires getting a quorum. This is enough for a the member to get the commit index from leader and maybe apply the entries from before the crash.

I see two problems:

• --wait-cluster-ready-timeout totally circumvents this mechanism allowing etcd to start serving data before it's ready.
• There is no mechanism that prevents serving before commit index was applied, waiting for quorum might not be enough.

[serathius]

Also "ensure hardstate is persisted before applying committed entries" might be stricter then required. We just need to "ensure hardstate is persisted before responsing to user".

[ahrtr]

Won't continue to work on this.

#16666 (comment)