upgrade: retry errors when dialing instances

[healthy-pod]

Release note: None
Epic: none
Closes #108860

[blathers-crl]

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[cockroach-teamcity]

This change is 

[ajstorm]

Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @healthy-pod and @stevendanna)

---

pkg/upgrade/upgradecluster/tenant_cluster.go line 238 at r1 (raw file):

				MaxBackoff:     10 * time.Millisecond,
			}
			if err := retry.WithMaxAttempts(ctx, retryOpts, retryOpts.MaxRetries+1, func() error {

Nit: you can save the +1 here by making MaxRetries above 3.

Can you also add a comment here which describes that we're adding this retry to make our tests succeed and that, strictly speaking, it's not a change that is likely to benefit customers as much as it benefits us?

[healthy-pod]

Nit: you can save the +1 here by making MaxRetries above 3.

The value above is the number of retries but the value passed to the function call is the number of attempts hence why it's maxRetries+1.

[healthy-pod]

bors r=ajstorm

[craig]

Timed out.

[stevendanna]

bors retry

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)