Add a retry when getting ts from PD for validating read ts

[MyonKeminta]

This PR adds a retry to ValidateReadTS to avoid a false-positive when the checked ts comes from a different source other than the current pdOracle. This can happen in some non-TiDB caller (e.g. BR, lightning).

The ValidateReadTS function may use getCurrentTSForValidation to get a ts, which internally uses a singleflight. So it's possible that a later call to ValidateReadTS reuses an older on-the-way single flight and returns an older ts. If the ts to be checked comes from outside the current pdOralce (so that it's not cached by the low resolution ts), it may cause false positive.

The solution is to add a retry: when ValidateReadTS check failed with a ts returned by getCurrentTSForValidation, try it one more time. So when the second call to getCurrentTSForValidation must uses a later-started singleflight, inwhich the GetTS RPC to PD is strictly later than the call. If the given ts is valid, it can be guaranteed by PD's TSO's monotonicity that the second ts returned by getCurrentTSForValidation must be greater than the given ts.

Ref: pingcap/tidb#59758 , this PR can avoid this problem in master branch where the validation to read ts takes effect to all read.

[cfzjywxk]

LGTM for working around the BR/Lightning compatible issue

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cfzjywxk, ekexium

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [cfzjywxk,ekexium]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2025-02-28 09:09:13.486134099 +0000 UTC m=+666.615053843: ☑️ agreed by cfzjywxk.
• 2025-03-03 11:24:38.045586975 +0000 UTC m=+267991.174506716: ☑️ agreed by ekexium.