run Layer::get_value_reconstruct_data in spawn_blocking

[problame]

This PR concludes the "async Layer::get_value_reconstruct_data" project.

The problem we're solving is that, before this patch, we'd execute Layer::get_value_reconstruct_data on the tokio executor threads.
This function is IO- and/or CPU-intensive.
The IO is using VirtualFile / std::fs; hence it's blocking.
This results in unfairness towards other tokio tasks, especially under (disk) load.

Some context can be found at #4154
where I suspect (but can't prove) load spikes of logical size calculation to
cause heavy eviction skew.

Sadly we don't have tokio runtime/scheduler metrics to quantify the unfairness.
But generally, we know blocking the executor threads on std::fs IO is bad.
So, let's have this change and watch out for severe perf regressions in staging & during rollout.

Changes

• rename Layer::get_value_reconstruct_data to Layer::get_value_reconstruct_data_blocking
• add a new blanket impl'd Layer::get_value_reconstruct_data async_trait method that runs get_value_reconstruct_data_blocking inside spawn_blocking.
• The spawn_blocking requires 'static lifetime of the captured variables; hence I had to change the data flow to move the ValueReconstructState into and back out of get_value_reconstruct_data instead of passing a reference. It's a small struct, so I don't expect a big performance penalty.

Performance

Fundamentally, the code changes cause the following performance-relevant changes:

• Latency & allocations: each get_value_reconstruct_data call now makes a short-lived allocation because async_trait is just sugar for boxed futures under the hood
• Latency: spawn_blocking adds some latency because it needs to move the work to a thread pool
  • using spawn_blocking plus the existing synchronous code inside is probably more efficient better than switching all the synchronous code to tokio::fs because each tokio::fs call does spawn_blocking under the hood.
• Throughput: the spawn_blocking thread pool is much larger than the async executor thread pool. Hence, as long as the disks can keep up, which they should according to AWS specs, we will be able to deliver higher get_value_reconstruct_data throughput.
• Disk IOPS utilization: we will see higher disk utilization if we get more throughput. Not a problem because the disks in prod are currently under-utilized, according to node_exporter metrics & the AWS specs.
• CPU utilization: at higher throughput, CPU utilization will be higher.

Slightly higher latency under regular load is acceptable given the throughput gains and expected better fairness during disk load peaks, such as logical size calculation peaks uncovered in #4154.

Full Stack Of Preliminary PRs

This PR builds on top of the following preliminary PRs

1. Clean-ups

• tenant_map_insert: don't expose the vacant entry to the closure #4316
• refactor responsibility for tenant/timeline activation #4317
• refactor: eliminate global storage_broker client state #4318
• refactor: make timeline activation infallible #4319
• refactor TenantState transitions #4321
• Note: these were mostly to find an alternative to allow async code inside Tenant::state.send_modify #4291, which I thought we'd need in my original plan where we would need to convert Tenant::timelines into an async locking primitive (make Tenant::timelines a tokio::sync::RwLock #4333). In reviews, we walked away from that, but these cleanups were still quite useful.

2. Don't hold timelines lock over load layer map #4364
3. make Delta{Value,Key}Iter Send #4472
4. refactor check_checkpoint_distance to prepare for async Timeline::layers #4476
5. make TimelineWriter Send by using tokio::sync Mutex internally #4477
6. init_empty_layer_map: use try_write #4485
7. turn Timeline::layers into tokio::sync::RwLock #4441

[skyzh]

I would prefer merge and observe the performance difference in prod/staging at some time

[github-actions]

1016 tests run: 974 passed, 0 failed, 42 skipped (full report)

---

Flaky tests (1)

Postgres 14

• test_crafted_wal_end[last_wal_record_crossing_segment]: ✅ debug

_{The comment gets automatically updated with the latest test results
9afb589 at 2023-06-23T15:00:04.219Z :recycle:}

[LizardWizzard]

Looks good to me. Minor nits. Note there is run_benchmarks label so you can run benchmarks before merging. I havent used it but some people did.

Two comments point to possible optimizations, feel free to close them (we can get back to them later if needed).