fix(db): Fix root cause of RocksDB misbehavior

[slowli]

What ❔

Fixes (hopefully) the root cause of recent RocksDB misbehavior. After recent refactoring, it improperly cancels background compactions / flushes each time a RocksDB instance is dropped, which leads to compactions / flushes being constantly interrupted. (After refactoring, RocksDB instances are Cloneeable and act essentially as Arc wrappers; a new instance is created and dropped, in particular, when processing each chunk of L1 batches.) A proper solution, implemented by this PR, is to cancel background work when dropping an internal, non-Arc'd DB instance.

Why ❔

That's a bug that makes RocksDB behave very unstably, as witnessed by recent issues with it.

Checklist

• PR title corresponds to the body of PR (we generate changelog entries from PRs).
• Code has been formatted via zk fmt and zk lint.

[slowli]

In hindsight, the issue was obvious; recent RocksDB logs are littered with "Shutdown: canceling all background work" logs, which I couldn't attribute previously.

[codecov]

Codecov Report

Attention: 8 lines in your changes are missing coverage. Please review.

Comparison is base (f048485) 35.63% compared to head (72f2ae7) 35.58%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #301      +/-   ##
==========================================
- Coverage   35.63%   35.58%   -0.06%     
==========================================
  Files         520      520              
  Lines       28350    28359       +9     
==========================================
- Hits        10103    10092      -11     
- Misses      18247    18267      +20     

Files	Coverage Δ
core/lib/storage/src/metrics.rs	25.00% <0.00%> (-1.67%)	⬇️
core/lib/storage/src/db.rs	50.46% <45.45%> (-1.45%)	⬇️

... and 24 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[popzxc]

I hoped that the PR would be titled RocksDB Wars: Episode IV -- The Stalled Writes Strike Back

Nice and certainly makes sense. If this works out, would it make sense to roll back the new code added in the previous 3 PRs? Less code is generally better than more code.

[slowli]

@popzxc We have write stalls reported for the state keeper cache, so I think we should leave this logic as-is for now. New RocksDB metrics also make sense, IMO; for one, they'll allow to estimate whether this fix works.