refactor(storage): use ReadOptions and WriteOptions in StateStore. #3216
Conversation
Codecov Report
@@ Coverage Diff @@
## main #3216 +/- ##
========================================
Coverage 73.74% 73.74%
========================================
Files 768 768
Lines 105736 106454 +718
========================================
+ Hits 77972 78507 +535
- Misses 27764 27947 +183
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more |
|
Why there's a |
It is cell table id and will be used to decide the compaction group during state store's read/write. |
So is this PR a continuation of #1812? It still seems to expose too many implementation details. 😢 |
|
Seems that we have a lot of merge conflict right now. |
StateTable has table_id and StateStore needs table_id for reads/writes. The issue here is how to pass StateTable's table_id to StateStore. How about putting table_id in |
But state store's read interfaces still require table_id? |
zwang28
force-pushed
the
wangzheng/refactor_state_store
branch
from
b7f3808 to
cc40b26
Compare
|
I resolved the conflict and left the test unfixed for now. |
You are right. We either pass |
|
@skyzh Would you please take a look at this refactor? |
| let val_size = match store | ||
| .get( | ||
| &key, | ||
| ReadOptions { |
There was a problem hiding this comment.
how about use the same ReadOptions with Hummock? (for prefetch)
There was a problem hiding this comment.
You mean reusing the existed hummock ReadOptions for state store interfaces?
I think it's a concept in state store layer(although other state store impl doesn't make use of it)?
There was a problem hiding this comment.
No. In the future we may need executor to tell us which it is a large scan request so that storage can decide whether we need to prefetch more data from object storage.
src/stream/src/executor/hash_agg.rs
Outdated
| // --- Flush states to the state store --- | ||
| // Some state will have the correct output only after their internal states have been | ||
| // fully flushed. | ||
| let (write_batch, dirty_cnt) = { | ||
| let mut write_batch = store.start_write_batch(); | ||
| let mut write_batch = store.start_write_batch(WriteOptions { epoch, table_id }); |
There was a problem hiding this comment.
I'm afraid this is not correct. HashAgg uses a lot of tables, we should produce multiple write batches in hash agg.
There was a problem hiding this comment.
Right both hash_agg.rs and global_simple_agg.rs have been refactored after this PR. Will fix.
| @@ -202,6 +203,7 @@ impl<S: StateStore> SimpleAggExecutor<S> { | |||
| ) -> StreamExecutorResult<Option<StreamChunk>> { | |||
| // The state store of each keyspace is the same so just need the first. | |||
| let store = keyspace[0].state_store(); | |||
| let table_id = keyspace[0].table_id(); | |||
# Conflicts: # src/ctl/src/cmd_impl/hummock/list_kv.rs # src/storage/src/hummock/state_store.rs # src/storage/src/store.rs # src/stream/src/executor/global_simple_agg.rs # src/stream/src/executor/hash_agg.rs
There was a problem hiding this comment.
LGTM. cc @BugenZhao are you working on string agg now? Seems that this PR might affect how string_agg stores and reads things. As it is not covered by our e2e test, you might want to pay attention to this PR.
# Conflicts: # src/storage/src/table/cell_based_table.rs # src/storage/src/table/test_relational_table.rs
What's changed and what's your intention?
Use ReadOptions and WriteOptions to wrap parameters for StateStore.
store.rs, but lots of call sites are affected.Checklist
./risedev check(or alias,./risedev c)Refer to a related PR or issue link (optional)
#2065