Come up with a better scheme for dealing with consul outages

[tomwilkie]

We should be able to operate if consul goes away. This probably means heartbeating the ingesters ourselves, or something like that.

[bboreham]

I was thinking about this - in a Kubernetes install we could just store the ring as an object in the api-server, and forget Consul.

[bboreham]

I imagine the benefit of Consul (or equivalent) is that all callers have a consistent view of the world. If callers have an inconsistent view then queries can go to ingesters that don't have all the samples.

So what does "heartbeating the ingesters ourselves" mean? "Use a last-known-consistent copy of the ring, and check heartbeats from ingesters are still consistent with that" ?

[tomwilkie]

Consul is the SPOF for a cortex cluster, and most of the time we don't actively need its coordination - the ring only changes when we add and remove nodes.

Except we use consul for heartbeats, to stop sending reads/writes to dead ingesters. So this issues is about replacing that heartbeat mechanism with an alternative, p2p one. Then cortex should be able to survive long consul outages without problem.

[bboreham]

OK, so "Use a last-known-consistent copy of the ring, make heartbeat calls directly to ingesters in that ring"?

[kfox1111]

+1 to seeing if a kubernetes crd would be sufficient. Much lower administration cost when already running in k8s.

[rverma-nikiai]

+1 for kubernetes crd, as an alternative we can use dynamodb as well.

[gouthamve]

One solution: We have healthcheck / heartbeats in the distributors to the ingesters. if consul is down, use the healthcheck info to coast until it is back up.

[friedrichg]

With memberlist and dynamodb we can close this as done. No longer needed to run single consul.