tso: unexpected timestamp fallback in TestMixedTSODeployment

[binshi-bing]

Flaky Test

Which jobs are failing

panic: global timestamp fallback, newly acquired ts (1682453235378, 1) is less or equal to last one (1682453236989, 2526)

goroutine 610965 [running]:
github.com/tikv/pd/client.(*tsoClient).compareAndSwapTS(0xc0025805a0, {0x41a6177, 0x6}, 0x187ba0706b2, 0x1, 0x0, 0x1)
/home/runner/work/pd/pd/client/tso_dispatcher.go:746 +0x46f
github.com/tikv/pd/client.(*tsoClient).processRequests(0xc0025805a0, {0x50aaae0, 0xc006dc1380}, {0x41a6177, 0x6}, 0xc0134ff0e0, {0x0, 0x0, 0x0})
/home/runner/work/pd/pd/client/tso_dispatcher.go:719 +0x4e5
github.com/tikv/pd/client.(*tsoClient).handleDispatcher(0xc0025805a0, {0x50c52f0?, 0xc0134ff090}, {0x41a6177, 0x6}, 0xc0134ff0e0)
/home/runner/work/pd/pd/client/tso_dispatcher.go:444 +0x1b25
created by github.com/tikv/pd/client.(*tsoClient).createTSODispatcher
/home/runner/work/pd/pd/client/tso_dispatcher.go:289 +0x4f7
FAIL github.com/tikv/pd/tests/integrations/tso 183.581s

CI link

https://github.com/tikv/pd/actions/runs/4801648211/jobs/8544142348?pr=6321

Reason for failure (if possible)

unclear

Anything else

[rleungx]

I think this issue has already been resolved? @binshi-bing

[binshi-bing]

We didn't really have this issue before. The one we had was related to mismatched test configuration which resulted into same election name used across election members, which was fixed by we using Advertise_Url to generated unique election name and QA team using different Advertise_Url for different tso pods. This test case failed due to something else.

[binshi-bing]

I think this issue has already been resolved? @binshi-bing

If you meant timestamp fallback issue, see my comment above.
If you meant TestMixedTSODeployment timeout issue caused by http server close stuck issue, this issue is totally different, and my fix should be nothing to do with this issue as my fix for timeout issue is tso.Server.Close() and this CI failure job's log shows that the error happened before calling tso.Server.Close().

It seems that this is a real timestamp fallback issue when upgrading the PD cluster with mixed PD and API mode.

[rleungx]

Fixed by #6740