Bug Fixes for Release 2.4.1 #1622
Conversation
syeleti-msft
commented
Feb 4, 2025
•
syeleti-msft
commented
- Update renameFileOptions in fuse2 handler
- Update the ETag for dst attribute while renaming the file
- Sanitize the ETag in datalake before adding it to the attribute
- Invalidate the attribute in attribute cache after doing a truncate operation on file. This is sort of quick hack done to avoid the complexity of truncate file in az storage component.
- Add atomic_o_trunc flag in fuse2 options that will allow the fuse library to always pass the O_TRUNC flag on open call. see the comments in the code for more details
- Send ETag as a struct field to the workitem, as directly retrieving etag from the handle inside the download method is not safe as it might cause a race. seems like Locking the handle to retrieve Etag inside the download method is also a bad idea, as we don't know the handle is already locked/unlocked.
- When O_TRUNC flag is passed to the open call in file cache and closed immediately without any writes on the data, the old data inside the file is not getting wiped out from the server. fixed this issue
- Remove usage of apt command in pipeline as it states that it doesn't have stable CLI interface to use in the scripts. Use always apt-get to install any packages.
* update attribute cache while doing a commit operation on file.
* Fix sanitizing the etag in datalake
…ibrary to always pass the O_TRUNC flag on open call.
syeleti-msft
requested review from
souravgupta-msft,
ashruti-msft and
jainakanksha-msft
as code owners
syeleti-msft
changed the title
syeleti-msft
force-pushed
the
syeleti/missingReleaseChanges
branch
from
ccea3cc to
09526c0
Compare
syeleti-msft
changed the title
syeleti-msft
changed the title
syeleti-msft
changed the title
| Dst: dstPath, | ||
| SrcAttr: srcAttr, | ||
| DstAttr: dstAttr, | ||
| }) |
There was a problem hiding this comment.
Update renameFileOptions in fuse2 handler
| // due to some changes that are required in az storage comp which | ||
| // were not necessarily required. Once they were done invalidation | ||
| // of the attribute can be removed. | ||
| ac.invalidatePath(options.Name) |
There was a problem hiding this comment.
Invalidate the attribute in attribute cache after doing a truncate operation on file. This is sort of quick hack done to avoid the complexity of truncate file in az storage component.
| @@ -400,7 +401,7 @@ func (dl *Datalake) GetAttr(name string) (blobAttr *internal.ObjAttr, err error) | |||
| Ctime: *prop.LastModified, | |||
| Crtime: *prop.LastModified, | |||
| Flags: internal.NewFileBitMap(), | |||
| ETag: (string)(*prop.ETag), | |||
There was a problem hiding this comment.
Sanitize the ETag in datalake before adding it to the attribute
| } | ||
|
|
||
| if copyStatus != nil && *copyStatus == blob.CopyStatusTypeSuccess { | ||
| modifyLMT(srcAttr, dstLMT) | ||
| modifyLMTandEtag(srcAttr, dstLMT, dstETag) |
There was a problem hiding this comment.
Update the ETag for dst attribute while renaming the file
| // Not checking the options since we don't allow user to configure this flag. | ||
| // This is the default behaviour for the fuse3 hence we don't pass this flag there. | ||
| // ref: https://github.com/libfuse/libfuse/blob/7f86f3be7148b15b71b63357813c66dd32177cf6/lib/fuse_lowlevel.c#L2161C2-L2161C16 | ||
| options += ",atomic_o_trunc" |
There was a problem hiding this comment.
Add atomic_o_trunc flag in fuse2 options that will allow the fuse library to always pass the O_TRUNC flag on open call. see the comments in the code for more details
| @@ -1055,8 +1061,7 @@ func (bc *BlockCache) download(item *workItem) { | |||
|
|
|||
| // Compare the ETAG value and fail download if blob has changed | |||
| if etag != "" { | |||
| etagVal, found := item.handle.GetValue("ETAG") | |||
There was a problem hiding this comment.
Send ETag as a struct field to the workitem, as directly retrieving etag from the handle inside the download method is not safe as it might cause a race. seems like Locking the handle to retrieve Etag inside the download method is also a bad idea, as we don't know the handle is already locked/unlocked.
| @@ -1042,6 +1042,9 @@ func (fc *FileCache) OpenFile(options internal.OpenFileOptions) (*handlemap.Hand | |||
| flock.Inc() | |||
|
|
|||
| handle := handlemap.NewHandle(options.Name) | |||
| if options.Flags&os.O_TRUNC != 0 { | |||
| handle.Flags.Set(handlemap.HandleFlagDirty) | |||
There was a problem hiding this comment.
When O_TRUNC flag is passed to the open call in file cache and closed immediately without any writes on the data, the old data inside the file is not getting wiped out from the server. fixed this issue
vibhansa-msft
changed the title
| @@ -938,12 +938,18 @@ func (bc *BlockCache) refreshBlock(handle *handlemap.Handle, index uint64, prefe | |||
|
|
|||
| // lineupDownload : Create a work item and schedule the download | |||
| func (bc *BlockCache) lineupDownload(handle *handlemap.Handle, block *Block, prefetch bool) { | |||
There was a problem hiding this comment.
Kidly add junit test case, where in case of lineing up we need to set Etag.
Co-authored-by: Sourav Gupta <[email protected]>
