Add get-page-ranges command for page blobs

[OddBloke]

Describe the bug

Given a sparse image.vhd:

$ ls -lah image.vhd 
-rw-r--r-- 1 ubuntu ubuntu 31G Jul  9 15:35 image.vhd
$ du -h image.vhd 
1015M	image.vhd

Running

az storage blob upload <snip> --type page --file image.vhd --name image.vhd

takes substantially longer than uploading a file that contains only the non-sparse parts of the image. This strongly suggests to me that (a) the upload isn't taking advantage of the file's sparseness, and (b) the file as stored in Azure won't be sparse (making operation on it more expensive in the future).

To Reproduce
Create a sparse file and upload it.

Expected behavior
The upload would take the amount of time that uploading the non-sparse part of the file takes, and would be stored as sparse in Azure.

Environment summary

pip installed on Ubuntu 18.10

[tjprescott]

Seems related to #5872.

[OddBloke]

Potentially, yes; in #5872 (comment) @zezha-msft said "We have the sparse file optimization for upload but not for download" which is why I filed this as a bug rather than a feature request.

(Some grepping through git history has no mention of "sparse" either here or in the storage repo, so I'm not sure what that comment is based upon to try and work out what problem I'm seeing.)

[tjprescott]

@zezha-msft for comment. cc/ @williexu

[zezha-msft]

Hi @OddBloke, thanks for reaching out!

The sparse file upload optimization for page blobs is contained in the following 2 commits: commit1 and commit2. In short, we read each chunk before uploading it to verify that it is not empty. We do store the file sparsely on the server side, but while uploading, reading the source file is still required. Thus, it might be almost impossible that uploading the sparse file takes the same amount of time as uploading only the non-sparse parts, because reading the source file could be a bottleneck. A fairer comparison would be uploading the sparse file with and without this optimization(current CLI vs previous CLI before optimization).

You can verify that the file is stored sparsely by performing a get page range.

[OddBloke]

Thanks, @zezha-msft, that's really helpful!

Is there a way to do the "get page range" from the CLI?

[williexu]

@OddBloke Not at the moment.
Would this be a useful utility for you?
If so, I can add a az storage blob get-page-ranges command as a new feature.

[OddBloke]

That would be useful for us, definitely!

…

On Mon, Jul 9, 2018 at 3:30 PM Willie Xu ***@***.***> wrote: @OddBloke <https://github.com/OddBloke> Not at the moment. Would this be a useful utility for you? If so, I can add a az storage blob get-page-ranges command as a new feature. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6754 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD1EL_mipTNsGAo6PkEzv_P5ipfbIvvks5uE69bgaJpZM4VIBlk> .