[lightning] no retries when pd timeout

[fubinzh]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

It is observed that there is no retry when pd timeout

2. What did you expect to see? (Required)

There should be retry mechanism when pd connection timeout during lightning import

3. What did you see instead (Required)

Lightning import fails immediately when PD server timeout

2022/01/05 03:12:47.110 +00:00] [INFO] [restore.go:444] ["the whole procedure start"]
[2022/01/05 03:12:47.111 +00:00] [INFO] [restore.go:748] ["restore all schema start"]
[2022/01/05 03:14:30.165 +00:00] [INFO] [restore.go:767] ["restore all schema completed"] [takeTime=1m43.053950663s] []
[2022/01/05 03:14:42.712 +00:00] [ERROR] [restore.go:462] ["run failed"] [step=2] [error="Error 9001: PD server timeout"]
[2022/01/05 03:14:42.712 +00:00] [ERROR] [restore.go:473] ["the whole procedure failed"] [takeTime=1m55.602514675s] [error="Error 9001: PD server timeout"]
[2022/01/05 03:14:42.713 +00:00] [WARN] [local.go:501] ["remove local db file failed"] [error="unlinkat /tmp/sorted-kv-dir: device or resource busy"]

4. What is your TiDB version? (Required)

/ # /tidb-lightning -V
Release Version: v5.4.0
Git Commit Hash: 974b578
Git Branch: heads/refs/tags/v5.4.0
Go Version: go1.16.4
UTC Build Time: 2022-01-03 10:01:05
Race Enabled: false

[sleepymole]

Similar to #28223.