admission, storage: prototype AC limiter for concurrent compactions

[itsbilal]

Prototype a slot-based admission control granter/requester well as an interface and integration point with Pebble to be able to limit concurrent compactions (and in the future, being able to pace compactions).

This granter will take into account CPU and IO resources used by a compaction, and Pebble will call into the interface to update the granter with the amount of IO taken up by a compaction. A linear model can be used for calculating IO tokens to deduct from compaction bytes read/written. Flushes as well as the first compaction in a store could be chosen to always run without permission.

Informs cockroachdb/pebble#1329 as well as #74697.

Jira issue: CRDB-45164

[sumeerbhola]

Some results from a prototype, running in a setting where disk bandwidth was a bottleneck (400MiB/s provisioned bandwidth and 8vCPU), with workload ./workload run kv --init --histograms=perf/stats.json --concurrency=512 --splits=1000 --duration=120m0s --read-percent=0 --min-block-bytes=8192 --max-block-bytes=8192 on a single node cluster.

The cluster setting allowed one compaction to run without permission, and the granter can run more if resources permit. And the read+write bandwidth utilization goal was initially set to 0.6, then lowered to 0.4 and then increased to 0.8. Since this was local SSD, there is a hack to limit compaction bandwidth to 40MiB/s (otherwise a single compaction + flush + WAL writes can consume 240MiB/s, which doesn't allow us to demonstrate any control by this granter).

The granter is forecasting and estimating every 100ms. The first graph shows that we are ~4.32 compactions until 14:20:30 (aforementioned setting 0.6), then reduce (setting 0.4) and then increase (setting 0.8) at 14:26. The estimated and observed CPU due to compactions is low and consistent with each other.

The following graph shows that the forecast admit_long.disk_write.forecast and actual disk.write.bytes are in agreement, and close to the configured (0.6 * 400 = 240MiB). And the observed writes from compactions are in agreement with the estimated writes.

[sumeerbhola]

Prototype is done