Extend lfs_fs_gc to compact metadata, compact_thresh #913
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Renamed lfs.free -> lfs.lookahead - Renamed lfs.free.off -> lfs.lookahead.start - Renamed lfs.free.i -> lfs.lookahead.next - Renamed lfs.free.ack -> lfs.lookahead.ckpoint - Renamed lfs_alloc_ack -> lfs_alloc_ckpoint These have been named a bit confusingly, and I think the new names make their relevant purposes a bit clearer. At the very it's clear lfs.lookahead is related to the lookahead buffer. (and doesn't imply a closed free-bitmap).
Some of this is just better documentation, some of this is reworking the logic to be more intention driven... if that makes sense...
This drops the lookahead buffer from operating on 32-bit words to operating on 8-bit bytes, and removes any alignment requirement. This may have some minor performance impact, but it is unlikely to be significant when you consider IO overhead. The original motivation for 32-bit alignment was an attempt at future-proofing in case we wanted some more complex on-disk data structure. This never happened, and even if it did, it could have been added via additional config options. This has been a significant pain point for users, since providing word-aligned byte-sized buffers in C can be a bit annoying.
geky
added
enhancement
performance
needs minor version
This was referenced Dec 21, 2023
Tests passed ✓, Code: 16964 B (+0.9%), Stack: 1432 B (-1.1%), Structs: 804 B (+0.5%)
|
Tests passed ✓, Code: 16964 B (+0.9%), Stack: 1432 B (-1.1%), Structs: 804 B (+0.5%)
|
The only reason we needed this alignment was for the lookahead buffer. Now that the lookahead buffer is relaxed to operate on bytes, we can relax our malloc alignment requirement all the way down to the byte level, since we mainly use lfs_malloc to allocate byte-level buffers. This does introduce a risk that we might need word-level mallocs in the future. If that happens we will need to decide if changing the malloc alignment is a breaking change, or gate alignment requirements behind user provided defines. Found by HiFiPhile.
geky
added
next minor
and removed
needs minor version
This extends lfs_fs_gc to now handle three things: 1. Calls mkconsistent if not already consistent 2. Compacts metadata > compact_thresh 3. Populates the block allocator Which should be all of the janitorial work that can be done without additional on-disk data structures. Normally, metadata compaction occurs when an mdir is full, and results in mdirs that are at most block_size/2. Now, if you call lfs_fs_gc, littlefs will eagerly compact any mdirs that exceed the compact_thresh configuration option. Because the resulting mdirs are at most block_size/2, it only makes sense for compact_thresh to be >= block_size/2 and <= block_size. Additionally, there are some special values: - compact_thresh=0 => defaults to ~88% block_size, may change - compact_thresh=-1 => disables metadata compaction during lfs_fs_gc Note that compact_thresh only affects lfs_fs_gc. Normal compactions still only occur when full.
Tests passed ✓, Code: 16972 B, Stack: 1432 B, Structs: 804 B
|
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This extends
lfs_fs_gcto now handle three things:compact_threshWhich should be all of the janitorial work that can be done without additional on-disk data structures.
Normally, metadata compaction occurs when an mdir is full, and results in mdirs that are at most
block_size/2.Now, if you call lfs_fs_gc, littlefs will eagerly compact any mdirs that exceed the
compact_threshconfiguration option. Because the resulting mdirs are at mostblock_size/2, it only makes sense forcompact_threshto be >=block_size/2and <=block_size.Additionally, there are some special values:
compact_thresh=0=> defaults to ~88%block_size, may change in the futurecompact_thresh=-1=> disables metadata compaction duringlfs_fs_gcNote that
compact_threshonly affectslfs_fs_gc. Normal compactions still only occur when full.This depends on: #912