ha-mode-exactly should ensure we have exactly N mirrors when down nodes come back up

[gsogol]

Per request of @tsaleh:

With an ha policy, I can specify to keep 2 copies. However, as nodes go down or new one come up, the policy does not know whether or not you have 1 now and you need to automatically create a copy on a new node. However, if a 2nd node comes back up, you now have 3 copies so you need to rebalance back to 2. It seems @simonmacmullen filed #26463 to help with this request.

[michaelklishin]

26463 is fixed in 3.5.0. The title says

Make ha-mode=exactly start up new mirrors when old ones go down

So what is left to be done? Ensuring we don't have more than N mirrors at any given time?

[gsogol]

if it's done, then yeah, If I specified 2, it should be kept at 2.

[gsogol]

Thanks

[Ayanda-D]

hi @michaelklishin. we've been running some tests to reproduce this issue. We're able to reproduce it when mirror nodes come back up simultaneously / in a near simultaneous manner. Frequency of occurance is very low and nondeterministic. We want to find out if we should still spend time fixing this in light of these test outcomes?

[michaelklishin]

@Ayanda-D yes, the probability of this is quite low. Given that we will be moving to Raft after 3.6.0, this is indeed a fair question. I'll discuss this with the team.

[Ayanda-D]

Okay. Thanks.

[videlalvaro]

I think agreeing on the number of copies is per-se a group-membership/consensus problem best to be solved with something like Raft

[gsogol]

IMHO, the definition of done should be whether or not it always works. We are experiencing this even with 3.5. Your testing from most production setups may be different. Please test with after your rework to Raft but it should work 100% of the time, not 99%.

[michaelklishin]

@gsogol we completely agree that it should work correctly 100% of the time. Our question really is, should we try to solve this with an ad-hoc protocol (something we're moving away from) or after we introduce Raft into the core.

[gsogol]

I agree that it wouldn't make sense to do this now if you're planning big changes in 3.6. Waiting makes sense. Just wanted to clarify that low occurrences are still had.

On Aug 10, 2015, at 8:22 AM, Michael Klishin [email protected] wrote:

@gsogol we completely agree that it should work correctly 100% of the time. Our question really is, should we try to solve this with an ad-hoc protocol (something we're moving away from) or after we introduce Raft into the core.

—
Reply to this email directly or view it on GitHub.

[michaelklishin]

@gsogol thanks for the feedback! We'll postpone this until after we move this part to Raft. This may be 3.6.0 or 3.7.0.

[mkuratczyk]

This is a very old issue and yet there are still known issues when it comes to classic queue mirroring and maintaining exact number of replicas, eg. #2737. Closing the issue, as quorum queues should be used instead.