Distribute information about who's leader for a given partition

[tillrohrmann]

For the ingress to communicate with the leading partition processor of a given partition, we need to propagate leadership changes between the Restate nodes. There are different ways to implement this functionality:

Gossiping leadership changes

The leadership information could be transient information that gets propagated/aggregated via gossiping leadership changes to other Restate nodes. Every node could gossip about the set of known leaders it is running and those the node has heard about from other nodes. The information could be (PartitionId, EpochNumber, GenerationalNodeId). Using the EpochNumber nodes can discern what's the latest information.

Until we implement a gossip based cluster state protocol, we could piggy-back the current ClusterState on the heartbeat requests that are being sent by the cluster controller.

Communicate leaders via the PartitionTable

Alternatively, we could propagate the leadership information as part of the PartitionTable. The cluster controller could update this metadata whenever it learns about newly running leaders for a given partition. The downside is that updating this information will entail a write to the MetadataStore.

[tillrohrmann]

Fixed via #2166. The current solution is to retrieve the leader information from the SchedulingPlan stored in the metadata store.