kvflowcontrol: surface per-replication stream metrics

[irfansharif]

Is your feature request related to a problem? Please describe.

There's a lack of metrics for every <source-node, dest-store> pair in kvflowcontrol, which can make it hard to diagnose exactly how writes are being being shaped. Log statements printed every 30s could miss token exhaustion.

Some notes from internal discussions:

• If we have a 10-node cluster, we’re adding 10*10 metrics at most, since most cluster setups are small.
• Look into adding more into the logs themselves. Maybe capture all streams that were blocked at any point in the last 30s, throughput achieved in that 30s delta, etc. So we have something historical.

Jira issue: CRDB-31711

[sumeerbhola]

We shouldn't forget tenant-id, since that's also part of the key.

[aadityasondhi]

Summarizing some offline discussions.

These metrics are too high to always be on for all clusters. https://cockroachlabs.slack.com/archives/C01CNRP6TSN/p1695138592720659.

We essentially have three options:

1. Export these separately on a different Prometheus endpoint. This can be problematic for customers since they will need to set up Prometheus to scrape multiple endpoints. Not included in tsdump/debug zip.
2. Doing an opt-in cluster setting, which defaults to off. It would trigger registering/unregistering of these metrics.
3. System table with a TTL and storing snapshots of these metrics.

After discussing with AC folks, I will try implementing Option 2. It is opt-in, so not always on. It also allows us to easily get this data through our regular metrics workflow (i.e. it will be in TSDB). Turning this cluster setting off would stop us from recording any further data for these metrics into TSDB.

Option 1 is tedious from a customer standpoint and will not let us access these metrics for self-hosted. The system table approach seems weird for this kind of data.

[aadityasondhi]

De prioritized for now. We have logs that surface the information. Unclear if metrics are required.