release-22.1: add checkpointing for raftAppliedIndexTermMigration

[erikgrinaker]

The raftAppliedIndexTermMigration upgrade migration could be
unreliable. It iterates over all ranges and runs a Migrate request
which must be applied on all replicas. However, if any ranges merge or
replicas are unavailable, the migration fails and starts over from the
beginning. In large clusters with many ranges, this meant that it might
never complete.

This patch makes the upgrade more robust, by retrying each Migrate
request 5 times, and checkpointing the progress after every fifth batch
(1000 ranges), allowing resumption on failure. At some point this should
be made part of the migration infrastructure.

Resolves #84073.

Release note (bug fix): the 22.1 upgrade migration "21.2-56: populate
RangeAppliedState.RaftAppliedIndexTerm for all ranges" is now more
resilient to failures, by checkpointing its progress in case it fails
and must be restarted. This migration must be applied across all
ranges and replicas in the system, and can fail with 'operation "wait
for Migrate application" timed out' if any replicas are temporarily
unavailable, which is increasingly likely to happen in large clusters
with many ranges -- previously, this would restart the migration from
the start, and might never make it all the way through.

Release justification: makes upgrades more robust.

[blathers-crl]

Thanks for opening a backport.

Please check the backport criteria before merging:

• Patches should only be created for serious issues or test-only changes.
• Patches should not break backwards-compatibility.
• Patches should change as little code as possible.
• Patches should not change on-disk formats or node communication protocols.
• Patches should not add new functionality.
• Patches must not add, edit, or otherwise modify cluster versions; or add version gates.

If some of the basic criteria cannot be satisfied, ensure that the exceptional criteria are satisfied within.

• There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
• The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
• New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters.
• The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.

Add a brief release justification to the body of your PR to justify this backport.

Some other things to consider:

• What did we do to ensure that a user that doesn’t know & care about this backport, has no idea that it happened?
• Will this work in a cluster of mixed patch versions? Did we test that?
• If a user upgrades a patch version, uses this feature, and then downgrades, what happens?

[cockroach-teamcity]

This change is 

[irfansharif]

Is this worth adding a test for? (I assume we've manually verified the change.)

[erikgrinaker]

Is this worth adding a test for? (I assume we've manually verified the change.)

I hear ya, but it's a pretty annoying test to write, since it's going to need a bunch of testing knobs and stuff -- it'd be easier to test if we generalized this, but I don't really have time to get sidetracked before stability.

And yes, I've manually verified that this works both on migration errors, coordinator failures, and pause/resume.

[irfansharif]

A few more words in the release note for the observed symptoms before this patch would be helpful me thinks.

[erikgrinaker]

A few more words in the release note for the observed symptoms before this patch would be helpful me thinks.

How's this?

[irfansharif]

It's great, thanks!