Previously, it was possible for a satisfiable voter constraint to never be satisfied when: 1. There were a correct number of `VOTER` and `NON_VOTER` replicas. 2. All existing replicas were necessary to satisfy a replica constraint, or voter constraint. The allocator relies on the `RebalanceVoter` path to resolve voter constraint violations when there are a correct number of each replica type. Candidates which are `necessary` to satisfy a constraint are ranked higher as rebalance targets than those which are not. Under most circumstances this leads to constraint conformance. However, when every existing replica is necessary to satisfy a replica constraint, and a voter constraint is unsatisfied -- `RebalanceVoter` would not consider swapping a `VOTER` and `NON_VOTER` to satisfy the constraint. For example, consider a setup where there are two stores, one in locality `a` and the other `b`, where some range has the following config and initial placement: ``` replicas = a(non-voter) b(voter) constraints = a:1 b:1 voter_constraints = a:1 ``` In this example, the only satisfiable placement is `a(voter)` `b(non-voter)`, which would require promoting `a(non-voter) -> a(voter)`, and demoting `b(voter)->b(non-voter)`. However, both are necessary to satisfy `constraints` leading to no rebalance occurring. Add an additional field to the allocator candidate struct, which is used to sort rebalance candidates. The new field, `voterNecessary` is sorted strictly after `necessary`, but before `diversityScore`. The `voterNecessary` field can be true only when rebalancing voters, and when the rebalance candidate is necessary to satisfy a voter constraint, the rebalance candidate already has a non-voter, and the existing voter is not necessary to satisfy *any* voter constraint. Note these rebalances are turned into swaps (promotion and demotion) in `plan.ReplicationChangesForRebalance`, so incur no snapshots. Fixes: cockroachdb#98020 Fixes: cockroachdb#106559 Fixes: cockroachdb#108127 Release note (bug fix): Voter constraints which were never satisfied due to all existing replicas being considered necessary to satisfy a replica constraint, will now be satisfied by promoting existing non-voters.

Commits

Commits on Oct 3, 2023

kvserver: add detail to should queue non voter test

kvserver: skip should queue non voter test under stressrace

Commits on Oct 6, 2023

asim: add voter constraint violation reproduction

allocator: prioritize non-voter promotion to satisfy voter-constraints