Introduce batch upload and download for blob #1428

troydai · 2016-11-23T09:02:12Z

Resolve issue #1432

10:21 $ az storage blob upload-batch -h

Command
    az storage blob upload-batch: Upload files to storage container as blobs.

Arguments
    --destination -d [Required]: The string represents the destination of this upload operation. The
                                 source can be the container URL or the container name. When the
                                 source is the container URL, the storage account name will parsed
                                 from the URL.
    --source -s      [Required]: The directory where the files to be uploaded.
    --dryrun                   : Show the summary of the operations to be taken instead of actually
                                 upload the file(s).
    --lease-id                 : Required if the blob has an active lease.
    --metadata                 : Metadata in space-separated key=value pairs. This overwrites any
                                 existing metadata.
    --pattern                  : The pattern is used for files globing. The supported patterns are
                                 '*', '?', '[seq]', and '[!seq]'.
    --timeout                  : Request timeout in seconds. Applies to each call to the service.
    --type -t                  : Defaults to 'page' for *.vhd files, or 'block' otherwise.  Allowed
                                 values: append, block, page.

Content Control Arguments
    --content-cache-control    : The cache control string.
    --content-disposition      : Conveys additional information about how to process the response
                                 payload, and can also be used to attach additional metadata.
    --content-encoding         : The content encoding type.
    --content-language         : The content language.
    --content-md5              : The content's MD5 hash.
    --content-type             : The content MIME type.
    --max-connections          : Default: 2.
    --maxsize-condition        : The max length in bytes permitted for an append blob.
    --validate-content         : Specifies that an MD5 hash shall be calculated for each chunk of
                                 the blob and verified by the service when the chunk has arrived.

Pre-condition Arguments
    --if-match                 : An ETag value, or the wildcard character (*). Specify this header
                                 to perform the operation only if the resource's ETag matches the
                                 value specified.
    --if-modified-since        : Alter only if modified since supplied UTC datetime
                                 (Y-m-d'T'H:M'Z').
    --if-none-match            : An ETag value, or the wildcard character (*). Specify this header
                                 to perform the operation only if the resource's ETag does not match
                                 the value specified. Specify the wildcard character (*) to perform
                                 the operation only if the resource does not exist, and fail the
                                 operation if it does exist.
    --if-unmodified-since      : Alter only if unmodified since supplied UTC datetime
                                 (Y-m-d'T'H:M'Z').

Storage Account Arguments
    --account-key              : Storage account key. Must be used in conjunction with storage
                                 account name. Environment variable: AZURE_STORAGE_KEY.
    --account-name             : Storage account name. Must be used in conjunction with either
                                 storage account key or a SAS token. Environment variable:
                                 AZURE_STORAGE_ACCOUNT.
    --connection-string        : Storage account connection string. Environment variable:
                                 AZURE_STORAGE_CONNECTION_STRING.
    --sas-token                : A Shared Access Signature (SAS). Must be used in conjunction with
                                 storage account name. Environment variable:
                                 AZURE_STORAGE_SAS_TOKEN.

Global Arguments
    --debug                    : Increase logging verbosity to show all debug logs.
    --help -h                  : Show this help message and exit.
    --output -o                : Output format.  Allowed values: json, jsonc, list, table, tsv.
                                 Default: json.
    --query                    : JMESPath query string. See http://jmespath.org/ for more
                                 information and examples.
    --verbose                  : Increase logging verbosity. Use --debug for full debug logs.

10:15 $ az storage blob download-batch -h

Command
    az storage blob download-batch: Download blobs in a container recursively.

Arguments
    --destination -d [Required]: The string represents the destination folder of this download
                                 operation. The folder must exist.
    --source -s      [Required]: The string represents the source of this download operation. The
                                 source can be the container URL or the container name. When the
                                 source is the container URL, the storage account name will parsed
                                 from the URL.
    --pattern                  : The pattern is used for files globing. The supported patterns are
                                 '*', '?', '[seq]', and '[!seq]'.

Storage Account Arguments
    --account-key              : Storage account key. Must be used in conjunction with storage
                                 account name. Environment variable: AZURE_STORAGE_KEY.
    --account-name             : Storage account name. Must be used in conjunction with either
                                 storage account key or a SAS token. Environment variable:
                                 AZURE_STORAGE_ACCOUNT.
    --connection-string        : Storage account connection string. Environment variable:
                                 AZURE_STORAGE_CONNECTION_STRING.
    --sas-token                : A Shared Access Signature (SAS). Must be used in conjunction with
                                 storage account name. Environment variable:
                                 AZURE_STORAGE_SAS_TOKEN.

Global Arguments
    --debug                    : Increase logging verbosity to show all debug logs.
    --help -h                  : Show this help message and exit.
    --output -o                : Output format.  Allowed values: json, jsonc, list, table, tsv.
                                 Default: json.
    --query                    : JMESPath query string. See http://jmespath.org/ for more
                                 information and examples.
    --verbose                  : Increase logging verbosity. Use --debug for full debug logs.

mention-bot · 2016-11-23T09:02:13Z

@troydai, thanks for your PR! By analyzing the history of the files in this pull request, we identified @tjprescott, @derekbekoe and @yugangw-msft to be potential reviewers.

tjprescott · 2016-11-23T17:47:19Z

Can you show the --help output for these updated commands?

derekbekoe · 2016-11-23T17:31:24Z

src/command_modules/azure-cli-storage/azure/cli/command_modules/storage/blob.py

+
+        result.append(client.get_blob_to_path(source_container_name, blob, dst))
+        if progress:
+            print('Copied %s' % blob)


Should use logger.warning here instead of printing to stdout

I'll remove the --progress option for now and draft a better solution later.

derekbekoe · 2016-11-23T17:34:36Z

src/command_modules/azure-cli-storage/azure/cli/command_modules/storage/blob.py

+    upload_action = _upload_blob if blob_type == 'block' or blob_type == 'page' else _append_blob
+
+    if dryrun:
+        print('upload action: from {} to {}'.format(source, destination))


Should use logger.warning here.

discussed with @derekbekoe offline, print to standard output is fine here since the command itself doesn't return anything.

derekbekoe · 2016-11-23T17:47:28Z

...es/azure-cli-storage/azure/cli/command_modules/storage/tests/integration_test_blob_upload.py

+        cli_main(command.split())
+
+        blobs = [b.name for b in self._blob_service.list_blobs(self._test_container_name)]
+        assert len(blobs) == 31


Where does this 31 come from?
Is it reproducible?

Similar with the other tests.
If they are going to be skipped by Travis CI, when/how do we run them?

They may be convert eventually to vcr test once all the features are completed.

I'll set up internal CI for long run integration test.

the result is stable and reproducible, because the sample is generated during test and controlled by the tests.

derekbekoe · 2016-11-23T17:49:02Z

requirements.txt

@@ -14,3 +14,4 @@ requests==2.9.1
 six==1.10.0
 tabulate==0.7.5
 vcrpy==1.7.4
+nose==1.3.7


Is nose used in this PR?
If not, does it need to be added here?

I'll remove it.

derekbekoe

Main thing to change is using logger.warning instead of print.

troydai · 2016-11-23T18:29:36Z

@derekbekoe @tjprescott PR is updated.

tjprescott

Mostly questions.

tjprescott · 2016-11-23T17:48:31Z

src/command_modules/azure-cli-storage/azure/cli/command_modules/storage/_params.py

+register_cli_argument('storage blob upload-batch', 'content_type', arg_group='Content Control')
+register_cli_argument('storage blob upload-batch', 'content_cache_control', arg_group='Content Control')
+register_cli_argument('storage blob upload-batch', 'content_language', arg_group='Content Control')
+register_cli_argument('storage blob upload-batch', 'max_connections', type=int, arg_group='Content Control')


max_connections pertains to concurrency. I don't think it would go in the Content Control group.

Also, we need logic in here to address #1105 for batch upload. (It should be in regular upload too, but not necessarily as part of this PR)

tjprescott · 2016-11-23T18:04:41Z

src/command_modules/azure-cli-storage/azure/cli/command_modules/storage/_params.py

@@ -255,7 +291,6 @@ def register_source_uri_arguments(scope):
 register_cli_argument('storage container delete', 'fail_not_exist', help='Throw an exception if the container does not exist.')

 register_cli_argument('storage container exists', 'blob_name', ignore_type)
-register_cli_argument('storage container exists', 'blob_name', ignore_type)


Need to make sure we include this in the list of breaking changes for this PR.

this is not breaking, I just remove duplicate code.

tjprescott · 2016-11-23T18:05:22Z

src/command_modules/azure-cli-storage/azure/cli/command_modules/storage/_validators.py

+    from azure.cli.command_modules.storage.storage_url_helpers import parse_url
+
+    # 1. quick check
+    if namespace.destination is None:


If destination/source are required parameters and not supplied, this will never be reached. Argparse will throw an error first.

cool, i'll remove this then.

tjprescott · 2016-11-23T18:06:53Z

src/command_modules/azure-cli-storage/azure/cli/command_modules/storage/_validators.py

+        raise ValueError('Source parameter is missing.')
+
+    if not os.path.exists(namespace.destination) or not os.path.isdir(namespace.destination):
+        raise ValueError('Destination folder {} does not exist'.format(namespace.source))


Why not create the directory? I would be more wary about a directory that already existed in which case files may be overwritten.

this is the choice i made to reduce the disk operation this command will execute so as to simply the logic.

tjprescott · 2016-11-23T18:08:47Z

src/command_modules/azure-cli-storage/azure/cli/command_modules/storage/_validators.py

+        raise ValueError('Destination folder {} does not exist'.format(namespace.source))
+
+    # 2. try to extract account name and container name from source string
+    storage_desc = parse_url(namespace.source)


I would give this a more descriptive name. At first I thought this was the generic urlparse library method. "parse_storage_url" or something like that.

will update.

tjprescott · 2016-11-23T18:30:14Z

src/command_modules/azure-cli-storage/azure/cli/command_modules/storage/blob.py

+        The pattern is used for files globing. The supported patterns are '*', '?', '[seq]',
+        and '[!seq]'.
+
+    :param bool dryrun:


Why does download not have dryrun?

because download dryrun is essentially list.

But can you list with patterns? Presumably I'd use dryrun to test my pattern before I actually move data.

tjprescott · 2016-11-23T18:31:37Z

src/command_modules/azure-cli-storage/azure/cli/command_modules/storage/storage_url_helpers.py

+from collections import namedtuple
+from azure.cli.core._profile import CLOUD
+
+StorageUrlDescription = namedtuple('StorageUrlDescription',


Where is SAS token?

not used here. I indent to do it in the future.

Tracking: #1434

tjprescott · 2016-11-23T18:32:40Z

src/command_modules/azure-cli-storage/azure/cli/command_modules/storage/storage_url_helpers.py

+    snapshot = None
+
+    if sys.version_info >= (3,):
+        from urllib.parse import urlparse # pylint: disable=no-name-in-module, import-error


Instead of this, try:
from six.moves.urllib.parse import urlparse

tjprescott · 2016-11-23T18:34:30Z

...d_modules/azure-cli-storage/azure/cli/command_modules/storage/tests/integration_test_base.py

+
+
+@skipIf(os.environ.get('Travis', 'false') == 'true', 'Integration tests are skipped in Travis CI')
+class StorageIntegrationTestBase(TestCase):


Why do we do this instead of a VCR test recording? Is this essentially a local "run live" test?

yes. i wrote these tests in a hurry, will be convert in the future.

tjprescott · 2016-11-23T18:35:30Z

requirements.txt

@@ -14,3 +14,4 @@ requests==2.9.1
 six==1.10.0
 tabulate==0.7.5
 vcrpy==1.7.4
+nose==1.3.7


What is this package for?

troydai · 2016-11-23T20:35:37Z

@tjprescott take another look?

troydai · 2016-11-23T21:24:42Z

@tjprescott dry run for download was just added.

tjprescott · 2016-11-23T21:26:34Z

Did we add logic to address #1105 for batch upload?

troydai · 2016-11-23T21:50:51Z

no, not specifically.

troydai · 2016-11-23T22:33:05Z

@tjprescott sign off?

* Azure/master: (39 commits) User should use no-cache so we build a fresh image (Azure#1455) Bump all modules to version 0.1.0b10 (Azure#1454) [Docs] Move around the order of install instructions. (Azure#1439) acs: *_install_cli mark cli as executable (Azure#1449) Fix resource list table. (Azure#1452) [Compute] VM NIC updates (Azure#1421) Introduce batch upload and download for blob (Azure#1428) Add auto-registration for resource providers. (Azure#1330) interpret the '@' syntax if something higher hasn't already done that. (Azure#1423) Aliasing plan argument with shorthand (Azure#1433) ad:fix one more place which still uses localtime for secret starttime (Azure#1429) Add table formatting for deployments and sort by timestmap. (Azure#1427) Add table formatting for resource group list (and add 'status') (Azure#1425) [Docs] Add shields specifying latest version and supported python versions (Azure#1420) Add new az storage blob copy start-batch command (Azure#1250) Component Discovery (Azure#1409) Add poison logic to prevent re-recording tests that need updating. (Azure#1412) [Storage] Fix storage table outputs and help text. (Azure#1402) [mention-bot] Attempt to fix config (Azure#1410) ad:use utc time on setting app's creds (Azure#1408) ...

azurecla added the cla-not-required label Nov 23, 2016

tjprescott self-assigned this Nov 23, 2016

derekbekoe reviewed Nov 23, 2016

View reviewed changes

derekbekoe suggested changes Nov 23, 2016

View reviewed changes

Introduce batch upload and download for blob

fc4f55f

tjprescott suggested changes Nov 23, 2016

View reviewed changes

derekbekoe approved these changes Nov 23, 2016

View reviewed changes

Review feedback 1

21184f0

troydai mentioned this pull request Nov 23, 2016

Accept sas token in the URL represent source and destination in blob upload-batch and download-batch command #1434

Closed

fix pylint error

2c01a14

troydai mentioned this pull request Nov 23, 2016

[Storage] Refine the logic of selecting blob type in batch-upload command #1436

Closed

review feedback 2: add dryrun to download-batch

2ae32e1

tjprescott approved these changes Nov 23, 2016

View reviewed changes

review feedback 3

dd9a64d

troydai merged commit 83788ee into Azure:master Nov 23, 2016

troydai deleted the azcopy4 branch November 23, 2016 23:22

xscript pushed a commit to xscript/azure-cli that referenced this pull request Nov 30, 2016

Introduce batch upload and download for blob (Azure#1428)

24d5899

haroldrandom added the cla-not-required label Oct 25, 2019



		@skipIf(os.environ.get('Travis', 'false') == 'true', 'Integration tests are skipped in Travis CI')
		class StorageIntegrationTestBase(TestCase):

Introduce batch upload and download for blob #1428

Introduce batch upload and download for blob #1428

Conversation

troydai commented Nov 23, 2016 • edited Loading

mention-bot commented Nov 23, 2016

tjprescott commented Nov 23, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

derekbekoe left a comment

Choose a reason for hiding this comment

troydai commented Nov 23, 2016

tjprescott left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

troydai commented Nov 23, 2016

troydai commented Nov 23, 2016

tjprescott commented Nov 23, 2016

troydai commented Nov 23, 2016

troydai commented Nov 23, 2016

troydai commented Nov 23, 2016 •

edited

Loading