batchClient may timeout due to use recycled Idle connArray

[lysu]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

not ease to reproduce, but it should be happen when:

• recycleIdleConnArray running after connection long time without any request, and will log like this

[2020/12/04 09:47:08.776 +08:00] [INFO] [client_batch.go:655] ["recycle idle connection"] [target=10.142.201.27:20172]
[2020/12/04 09:47:08.778 +08:00] [INFO] [client_batch.go:655] ["recycle idle connection"] [target=10.142.201.27:20171]
[2020/12/04 09:47:14.994 +08:00] [INFO] [region_cache.go:833] ["invalidate region cache due to cannot find peer when updating leader"] [regionID=344196] [currIdx=2] [leaderStoreID=7]

• another question arrival at the same time as recycleIdleConnArray running

2. What did you expect to see? (Required)

need request success or fail(and retry succcess) quickly

3. What did you see instead (Required)

it will block a long time and timeout like

[2020/12/04 09:48:08.775 +08:00] [WARN] [expensivequery.go:168] [expensive_query] [cost_time=60.035280495s] [conn_id=1805330] [user=chain_fact] [database=supply_chain_factory] [txn_start_ts=0] [mem_max="2261000 Bytes (2.1562576293945312 MB)"] [sql="i

[2020/12/04 09:48:09.152 +08:00] [WARN] [client_batch.go:617] ["send request is cancelled"] [to=10.142.200.29:20171] [cause="context canceled"]

4. What is your TiDB version? (Required)

master

[ti-srebot]

Please edit this comment or add a new comment to complete the following information

Not a bug

1. Remove the 'type/bug' label
2. Add notes to indicate why it is not a bug

Duplicate bug

1. Add the 'type/duplicate' label
2. Add the link to the original bug

Bug

Note: Make Sure that 'component', and 'severity' labels are added
Example for how to fill out the template: #20100

1. Root Cause Analysis (RCA) (optional)

2. Symptom (optional)

3. All Trigger Conditions (optional)

4. Workaround (optional)

5. Affected versions

[v3.0.0:v3.0.20], [v4.0.0:v4.0.9], [v5.0.0-rc]

6. Fixed versions

v4.0.10