Investigate cycilic increase in Committee fetch time on butteflynet

[masih]

Committee fetch time cycles almost exactly every 12hrs increasing from mere milliseconds to ~30ms and back down again. Observed on Butterflynet. Understand why.

The pattern matches heap object allocation:

Link to Grafana dashboard.

[masih]

allocs from preminer-1:

[masih]

Looks like this is related to the default powerTableFrequency the frequency by which power table of an instance is stored, currently set to every 1440 instances (in steady state this is twice a day). The frequency matches the length of cyclic pattern.

This means to get the power table of an instance as part of GetCommittee calls we are traversing half a day worth of finality certs to cumulatively calculate the target power table.

We may need to increase this frequency considering the sensitivity to instance misalignment if GetCommittee is slow and/or arbitrarily slow across a large network. This specific scenario was observed in previous round of passive testing as "Slow GetCommittee" calls, and was partially mitigated by introducing caching layers at consensus input; see #784.

The disadvantage of lower frequency is higher storage footprint of certstore.

[masih]

A way to significantly speed up certstore.GetPowerTable (in exchange for a small memory footprint) is to cache the latest known powertable in certStore. This would make calls to GetPowerTable for an instance made by consensus inputs in steady state very quick. This call makes up the vast majority of certstore workload.

@Stebalien Do you recall why we didn't do this to begin with? i.e.: Cache the latest powertable, and fallback to getting range from instance mod frequency to instance if latest is not cached.

[masih]

Also unclear why consensus inputs is not using the powertable cache for getting committee. Never mind; that's just the cache of power table CIDs.

[Stebalien]

Cache the latest powertable, and fallback to getting range from instance mod frequency to instance if latest is not cached.

Because I didn't want to complicate the initial implementation and/or get it wrong. But you're right, we absolutely should cache recent values.

I personally wouldn't store more intermediate values unless we really need to.

[masih]

Captured fixing this in a separate issue as investigation is complete.