Tracking: Hummock Block Format

[Li0k]

This issue tracks the progress of Hummock Block Format support in RisingWave kernel.

We plan to introduce two new features:

1. Flexible length encoding
2. Customized compression for TableId and Epoch

Any ideas and suggestions are welcome!

Related RFC: risingwavelabs/rfcs#56

Progress

Basic

• Statistical information for key_length and epoch_count in block / sst feat(storage): monitor avg_key_size and avg_epoch_count #8297
• TableId compression on block level #8293
• Flexible length KeyPrefix encoding #8378 feat(storage): flexible KeyPrefix encoding for Block #8379
• Epoch compression on the bottom level (remove Epoch from encoding)
• Move block meta offset to the end of block meta

Expand

- [ ] Epoch dictionary encoding

• Introduce dictionary encoding for the whole block when using zstd #8584

[Li0k]

epoch dictionary encoding

TLDR:

1. Introduces complexity and some performance degradation, with insignificant space savings
2. Version compatibility issues

#8605 (comment)

[hzxa21]

Future work are tracked in separate issues.