Enable ETAG based validation on every block download to provide higher consistency #1608
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vibhansa-msft
requested review from
souravgupta-msft,
ashruti-msft,
syeleti-msft and
jainakanksha-msft
as code owners
|
Later we should run some perf tests with data consistency on to inform users what % diff in performance, memory usage they can expect with this option on and off. |
ashruti-msft
approved these changes
Jan 20, 2025
jainakanksha-msft
approved these changes
Jan 20, 2025
…k with quotes while some apis dont:
syeleti-msft
approved these changes
Jan 28, 2025
ashruti-msft
approved these changes
Jan 29, 2025
vibhansa-msft
added a commit
that referenced
this pull request
Feb 3, 2025
* feat: support workload identity token (#1556) * feat: support workload identity token * Create block pool only once in child process (#1581) * create block pool in child only * Update golang.org/x/crypto to v0.31.0 (#1594) * Update golang.org/x/crypto to v0.31.0 * sync with main (#1603) * updated year in copyright message (#1601) * Use ListBlob for hns accounts (#1555) * Optimize HNS listing * Added statfs for block-cache (#1470) * Added statfs for block_cache * Add strong consistency check for data on disk (#1604) * Add strong consistency check for data on disk * bug in block cache open call (#1580) * current implementation of open file when opened in O_WRONLY will truncate the file to zero. This is incorrect behaviour. We don't see it in the normal scenario as write-back cache is on by default. Hence all the open calls with O_WRONLY will be redirected O_RDWR. To simulate this turn of the write-back cache and then open file in O_WRONLY. * Feature: Blob filter (#1595) * Integrating blob filter in azstorage * Serve getAttr call for destination file after the Copy finishes from the cache * Cleanup on start shall be set to cleanup temp cache (#1613) * Add Tests * Refactor the code and refresh the cache after copying the attributes * Automate blobfuse2 setup for new VM (#1575) added script for blobfuse setup and azsecpack setup in VM * * Update the Unit tests. * Refactor the Code * Update Changelog * do go fmt on src * Downgrade go version to 1.22.7 due to memory issues in 1.23 (#1619) * Enable ETAG based validation on every block download to provide higher consistency (#1608) * Make etag validation a defualt option * BUG#31069208: Fixed Prefix filtering from File Path (#1618) * Fixed the logic to filter out folder prefix from path * Added/Updated/Removed test case --------- Co-authored-by: weizhi <[email protected]> Co-authored-by: Sourav Gupta <[email protected]> Co-authored-by: Jan Jagusch <[email protected]> Co-authored-by: ashruti-msft <[email protected]> Co-authored-by: syeleti-msft <[email protected]> Co-authored-by: jainakanksha-msft <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
✅ What
In block-cache mode, every download is not validated for higher consistency. If a file is open or half-read and contents are modified on the container, subsequent reads will start fetching new content and provide inconsistent data to application. Adding an ETAG based validation will guard applications against such data corruption. When an 'open' file call comes ETAG from blob will be preserved by block-cache. Any subsequent block download will revalidate that ETAG is still the same. If ETAG has changed, then it means blob was modified on container after
openand in such case download and read operation will fail.🤔 Why
This will safeguard application from data corruption where blob is modified on container through other means.
👩🔬 How to validate if applicable
Open a file and then modify the file on container followed by a read operation on mounted path.
🔖 Related links
NA