Stuck retrying Migration to 21.2-56: "populate RangeAppliedState.RaftAppliedIndexTerm for all ranges"

[wzrdtales]

Describe the problem

After upgrading to 22.1, several migrations were running through fine. Since 10 hours it is however stuck with 21.2-56 and doesn't finish on it. (Retrying the 9th time now)

To Reproduce

Upgrade to 22.1

On kubernetes.

Jira issue: CRDB-16140

[blathers-crl]

Hello, I am Blathers. I am here to help you get the issue triaged.

Hoot - a bug! Though bugs are the bane of my existence, rest assured the wretched thing will get the best of care here.

I was unable to automatically find someone to ping.

If we have not gotten back to your issue within a few business days, you can try the following:

• Join our community slack channel and ask on #cockroachdb.
• Try find someone from here if you know they worked closely on the area and CC them.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[wzrdtales]

10th retry now...

[wzrdtales]

I went through the nodes and their logs and found this now on one node:

so this seems to be a display bug, but also the progress is very slow, not sure if intended. With that speed it will probably take around 3-4 days to finish.

[irfansharif]

With that speed it will probably take around 3-4 days to finish.

@wzrdtales How many total ranges do you have in this cluster? Judging from that screenshot we're churning through ~3800 ranges per minute. With the estimate of 3-4 days, are you saying this cluster has 16,416,000+ ranges?

I'm more interested in the retry loop itself rather than how long each attempt takes. Do you happen to have logs around where these retries occur? If it's because of timeouts in applying the Migrate request, I wonder if we should bump kv.migration.migrate_application_timeout to give it ample time to proceed. Say to something like 5m?

but also the progress is very slow, not sure if intended

It is partially intended, though the retry behavior not so much. The slow pace is to pace the internal migrations such that they're non-disruptive to foreground traffic. We want to avoid a thundering herd of work on upgrades.

[wzrdtales]

I was calculating the whole runtime, it ended up needing around 24 hours in total I think. It resolved on its own, but in a weird status which left us worried (as you noticed :))

With the estimate of 3-4 days, are you saying this cluster has 16,416,000+ ranges?

Not yet that big, "only" 300k+ ranges.

[irfansharif]

I'm glad it resolved, I'll close this issue. I'll add a suggestion to #72931 to let operators control the pacing here more directly, if they want to speed up the migrate parallelism for clusters with large range counts while keeping an eye out on foreground impact. I'll also note that some of these migrations are intended to be long-running and operating over the timescales you're observing: #48843.