store/tikv: fix a concurrency bug that may cause the batchClient timeout

[tiancaiamao]

What problem does this PR solve?

closes #22334

Problem Summary:

The recycleIdleConnArray() logic has a bug: when one goroutine getConnArray() and the other goroutine recycle the idle connection, the prior goroutine may get a stale batchConn which is closed already.

sendBatchRequest() using that stale batchConn would block until timeout.

++++++++++++++++++++++++++++++++++++++++++++++++++++++
g1: connArray := getConnArray(addr, enableBatch)
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
                                                                                                            g2: c.Lock()    
                                                                                                            g2: conn := c.conns[addr]
                                                                                                            g2: Unlock()
                                                                                                            g2: conn.Close()
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
g1: sendBatchRequest(connArray)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

What is changed and how it works?

What's Changed:

RLock()
conn := getConArray()
RUnlock()
sendBatchRequest(conn)

This is not enough to protect the conn from been recycle and close.
Now the whole sending process is protected by the read lock, and modify conn map should obtain the write lock.

How it Works:

As long as the sending operation hold the read lock, the recycle connection operation need to wait to obtain the write lock.

Related changes

• Need to cherry-pick to the release branch

Maybe we can cherry-pick it to 5.0, it's rare to see this bug in the production environment.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Release note

• fix a concurrency bug that may cause the batch client timeout

[sre-bot]

No release note, Please follow https://github.com/pingcap/community/blob/master/contributors/release-note-checker.md

[lysu]

/bench

[lysu]

LGTM

[hicqu]

LGTM

[tiancaiamao]

/merge

[ti-srebot]

/run-all-tests

[ti-srebot]

cherry pick to release-3.0 in PR #22335

[ti-srebot]

cherry pick to release-4.0 in PR #22336

[ti-srebot]

cherry pick to release-5.0-rc in PR #22337

[zhangjinpeng87]

Does this will impact the send speed?