store/tikv: fix a concurrency bug that may cause the batchClient timeout (#22239)

[ti-srebot]

cherry-pick #22239 to release-3.0
You can switch your code base to this Pull Request by using git-extras:

# In tidb repo:
git pr 22335

After apply modifications, you can push your change to this PR via:

git push [email protected]:ti-srebot/tidb.git pr/22335:ti-srebot:release-3.0-ae7e43249a35

---

What problem does this PR solve?

closes #22334

Problem Summary:

The recycleIdleConnArray() logic has a bug: when one goroutine getConnArray() and the other goroutine recycle the idle connection, the prior goroutine may get a stale batchConn which is closed already.

sendBatchRequest() using that stale batchConn would block until timeout.

++++++++++++++++++++++++++++++++++++++++++++++++++++++
g1: connArray := getConnArray(addr, enableBatch)
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
                                                                                                            g2: c.Lock()    
                                                                                                            g2: conn := c.conns[addr]
                                                                                                            g2: Unlock()
                                                                                                            g2: conn.Close()
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
g1: sendBatchRequest(connArray)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

What is changed and how it works?

What's Changed:

RLock()
conn := getConArray()
RUnlock()
sendBatchRequest(conn)

This is not enough to protect the conn from been recycle and close.
Now the whole sending process is protected by the read lock, and modify conn map should obtain the write lock.

How it Works:

As long as the sending operation hold the read lock, the recycle connection operation need to wait to obtain the write lock.

Related changes

• Need to cherry-pick to the release branch

Maybe we can cherry-pick it to 5.0, it's rare to see this bug in the production environment.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Release note

• fix a concurrency bug that may cause the batch client timeout

[ti-srebot]

/run-all-tests

[ti-srebot]

@tiancaiamao you're already a collaborator in bot's repo.

[lysu]

LGTM

[tiancaiamao]

PTAL @hicqu