Merge #6422: perf: add in quorumBaseBlockIndexCache to reduce cs_main contention

3d67771 refactor: add in quorumBaseBlockIndexCache to reduce cs_main contention (pasta)

Pull request description:

  ## Issue being fixed or feature implemented
  subset of #6418; only includes the new quorumBaseBlockIndexCache, doesn't include the caching of the chain-tip, as that introduced regressions I'm still debugging.

  ## What was done?
  introduce a LRU cache for quorumHash -> const CBlockIndex*; this should significantly reduce cs_main contention during high transaction load.

  ## How Has This Been Tested?
  Ran tests locally; let's see CI happy, and I also intend to run this on a testnet MN first and see the level of contention reduction

  ## Breaking Changes
  None

  ## Checklist:
    _Go over all the following points, and put an `x` in all the boxes that apply._
  - [ ] I have performed a self-review of my own code
  - [ ] I have commented my code, particularly in hard-to-understand areas
  - [ ] I have added or updated relevant unit/integration/functional/e2e tests
  - [ ] I have made corresponding changes to the documentation
  - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

ACKs for top commit:
  UdjinM6:
    utACK 3d67771
  knst:
    utACK 3d67771

Tree-SHA512: dbb4bdafed095397ca0e12dbd8bba25c108d199538387c71b1ff4285af821f9d9ad0ad4426407a015528270f3c163fa66ce91755efb1c8a7a90fd7cb70a918bc

Author: PastaPastaPasta

src/llmq/quorums.cpp

@@ -633,7 +633,20 @@ std::vector<CQuorumCPtr> CQuorumManager::ScanQuorums(Consensus::LLMQType llmqTyp
 
 CQuorumCPtr CQuorumManager::GetQuorum(Consensus::LLMQType llmqType, const uint256& quorumHash) const
 {
-    const CBlockIndex* pQuorumBaseBlockIndex = WITH_LOCK(cs_main, return m_chainstate.m_blockman.LookupBlockIndex(quorumHash));
+    const CBlockIndex* pQuorumBaseBlockIndex = [&]() {
+        // Lock contention may still be high here; consider using a shared lock
+        // We cannot hold cs_quorumBaseBlockIndexCache the whole time as that creates lock-order inversion with cs_main;
+        // We cannot aquire cs_main if we have cs_quorumBaseBlockIndexCache held
+        const CBlockIndex* pindex;
+        if (!WITH_LOCK(cs_quorumBaseBlockIndexCache, return quorumBaseBlockIndexCache.get(quorumHash, pindex))) {
+            pindex = WITH_LOCK(cs_main, return m_chainstate.m_blockman.LookupBlockIndex(quorumHash));
+            if (pindex) {
+                LOCK(cs_quorumBaseBlockIndexCache);
+                quorumBaseBlockIndexCache.insert(quorumHash, pindex);
+            }
+        }
+        return pindex;
+    }();
     if (!pQuorumBaseBlockIndex) {
         LogPrint(BCLog::LLMQ, "CQuorumManager::%s -- block %s not found\n", __func__, quorumHash.ToString());
         return nullptr;

src/llmq/quorums.h

@@ -252,6 +252,10 @@ class CQuorumManager
     mutable Mutex cs_cleanup;
     mutable std::map<Consensus::LLMQType, unordered_lru_cache<uint256, uint256, StaticSaltedHasher>> cleanupQuorumsCache GUARDED_BY(cs_cleanup);
 
+    mutable Mutex cs_quorumBaseBlockIndexCache;
+    // On mainnet, we have around 62 quorums active at any point; let's cache a little more than double that to be safe.
+    mutable unordered_lru_cache<uint256 /*quorum_hash*/, const CBlockIndex* /*pindex*/, StaticSaltedHasher, 128 /*max_size*/> quorumBaseBlockIndexCache;
+
     mutable ctpl::thread_pool workerPool;
     mutable CThreadInterrupt quorumThreadInterrupt;