From 938f17600f66c6c03523b68cfe26afe5b6c1b928 Mon Sep 17 00:00:00 2001 From: Ashish Kumar Date: Wed, 22 May 2019 11:31:45 -0400 Subject: [PATCH 01/11] Initial commit --- LICENSE | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 LICENSE diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..aa657f1 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2019 Ashish Kumar + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. From 262f1db6a6eb5c33034f9b9594060171bf7d8022 Mon Sep 17 00:00:00 2001 From: Ashish Kumar Date: Wed, 22 May 2019 11:38:53 -0400 Subject: [PATCH 02/11] New readme. --- README.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..128d081 --- /dev/null +++ b/README.md @@ -0,0 +1,13 @@ +# azure-blob-util +A lightweight utility to list, copy, and delete blobs from Azure storage + +## Introduction +Microsoft provides Azure Storage Explorer to perform various operations. It is a great tool, however, it is still an interactive tool. AzCopy is a yet another utility from Microsoft. AzCopy is a batch utility and runs very fast. Azure-blob-util is a hybrid of Azure Storage Explorer and AzCopy. + +Using this command line tool, you can list, delete, download, or copy blobs. The copy function copies blobs from one storage location to the other. + +## Requirements +Windows platform with .Net 4.6.1 or above installed. + + +If you just want to use the utility, download the distribution folder's contents. From a543e9b3f38852f7e47eb9c0820fd75a9c370ca2 Mon Sep 17 00:00:00 2001 From: Ashish Kumar Date: Wed, 22 May 2019 11:47:57 -0400 Subject: [PATCH 03/11] Update README.md --- README.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 128d081..4b984ce 100644 --- a/README.md +++ b/README.md @@ -9,5 +9,20 @@ Using this command line tool, you can list, delete, download, or copy blobs. Th ## Requirements Windows platform with .Net 4.6.1 or above installed. - +## Usage If you just want to use the utility, download the distribution folder's contents. +Run abu.exe from the command or powershell prompt. + +### Help +To get the command line options: +abu +abu -? + +### List blobs +Update the abu.exe.config file with the storage location you want to list. +Update the two keys, **storageConnectionString** and **storageContainer**. +Example: + + + + From 420083c7f44a814b449d6d7cc50dcec6341b692f Mon Sep 17 00:00:00 2001 From: Ashish Kumar Date: Wed, 22 May 2019 11:48:26 -0400 Subject: [PATCH 04/11] Update README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 4b984ce..e7abf1a 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,7 @@ abu -? Update the abu.exe.config file with the storage location you want to list. Update the two keys, **storageConnectionString** and **storageContainer**. Example: + From 94314fcb7140213f4daf92081b2c8c681070189a Mon Sep 17 00:00:00 2001 From: Ashish Kumar Date: Wed, 22 May 2019 11:50:31 -0400 Subject: [PATCH 05/11] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e7abf1a..f25815c 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ Run abu.exe from the command or powershell prompt. ### Help To get the command line options: -abu +abu abu -? ### List blobs From c942ea9f1caae33997091ae2750be41924e950a7 Mon Sep 17 00:00:00 2001 From: Ashish Kumar Date: Wed, 22 May 2019 11:53:15 -0400 Subject: [PATCH 06/11] Update README.md --- README.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index f25815c..60708d8 100644 --- a/README.md +++ b/README.md @@ -14,9 +14,9 @@ If you just want to use the utility, download the distribution folder's contents Run abu.exe from the command or powershell prompt. ### Help -To get the command line options: -abu -abu -? +To get the command line options: +**abu** +**abu -?** ### List blobs Update the abu.exe.config file with the storage location you want to list. @@ -27,3 +27,10 @@ Example: +The following commands will list all blobs in the specified container: +**abu -l** +**abu -list** + +The following commands will list all blobs in the specified container, which start with **temp**: +**abu -l temp** +**abu -list temp** From bce5d8467bc1a39553727d6096f7ec24e77112d2 Mon Sep 17 00:00:00 2001 From: Ashish Kumar Date: Wed, 22 May 2019 11:59:41 -0400 Subject: [PATCH 07/11] Update README.md --- README.md | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 60708d8..0e8cad7 100644 --- a/README.md +++ b/README.md @@ -32,5 +32,24 @@ The following commands will list all blobs in the specified container: **abu -list** The following commands will list all blobs in the specified container, which start with **temp**: -**abu -l temp** +**abu -l temp** **abu -list temp** + +Make a text file, mylist.txt and enter a few patterns. +**temp** +**chrome** + +The following commands will list all blobs in the specified container, which start with **temp** and **chrome**: +**abu -lf mylist.txt** +**abu -listfile mylist.txt** + +The following commands will list all blobs and their sizes in the specified container, which start with **temp** and **chrome**: +**abu -lfs mylist.txt** +**abu -listfilesize mylist.txt** + +You can dump the blob listing to another file by redirecting the output: +**abu -lf mylist.txt > myoutfile.txt** +This is particularly useful when you want to delete some blobs. + +### Delete blobs +Delete blobs option works on specific blobs From b7a94c32b85555780ebcec569d12f68a6472baaa Mon Sep 17 00:00:00 2001 From: Ashish Kumar Date: Wed, 22 May 2019 12:14:46 -0400 Subject: [PATCH 08/11] Update README.md --- README.md | 42 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 37 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 0e8cad7..c0f5635 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,9 @@ To get the command line options: **abu** **abu -?** +The utility works with pattern or patter files. Use the pattern files where multiple patterns are involved. The pattern file can be located in any folder. Make sure to specify the fully qualified pattern file name if it is not in the current working folder. +**Note that the Azure blobs are case sensitive.** + ### List blobs Update the abu.exe.config file with the storage location you want to list. Update the two keys, **storageConnectionString** and **storageContainer**. @@ -28,11 +31,11 @@ Example: The following commands will list all blobs in the specified container: -**abu -l** +**abu -l** or **abu -list** The following commands will list all blobs in the specified container, which start with **temp**: -**abu -l temp** +**abu -l temp** or **abu -list temp** Make a text file, mylist.txt and enter a few patterns. @@ -40,11 +43,11 @@ Make a text file, mylist.txt and enter a few patterns. **chrome** The following commands will list all blobs in the specified container, which start with **temp** and **chrome**: -**abu -lf mylist.txt** +**abu -lf mylist.txt** or **abu -listfile mylist.txt** The following commands will list all blobs and their sizes in the specified container, which start with **temp** and **chrome**: -**abu -lfs mylist.txt** +**abu -lfs mylist.txt** or **abu -listfilesize mylist.txt** You can dump the blob listing to another file by redirecting the output: @@ -52,4 +55,33 @@ You can dump the blob listing to another file by redirecting the output: This is particularly useful when you want to delete some blobs. ### Delete blobs -Delete blobs option works on specific blobs +Make sure the storage location is correct in the bu.exe.config file. Delete blobs option works on specific blobs. Make a list of blobs in some text file (mylist.txt) keeping one blob per line. +**temp/blob1.txt** +**chrome/folder1/blob2.data** + +To delete these blobs use one of the following commands: +**abu -d mylist.txt** or +**abu -delete mylist.txt** + +Delete blobs option does not work on patterns. You must supply an explicit list of blobs. + +### Download blobs +Make sure the storage location is correct in the bu.exe.config file. To download blobs using pattern use the following commands: +**abu -g temp** or +**abu -get temp** +A new guid folder would be created in the current working folder. All the blobs starting with **temp** would be copied into the guid folder. The blobs would be copied keeping the original folder structure in tact. + +To download multi-pattern blobs, make some text file, myblobs.txt. Enter the pattern or exact blob names per line. +**temp** +**chrome/folder1/blob2.data** +Use the following commands to download blobs which start with **temp** and **chrome/folder1/blob2.data**. +**abu -g myblobs.txt** or +**abu -get myblobs.txt** +Notice that this syntax is similar to the pattern syntax. The utility would use the file syntax if a file is found. If your intent was to download all blobs starting with myblobs.txt, then ensure that this file does not exist. + + + + + + + From 72669c41852de9bae8bdb56a696578b0d961beb5 Mon Sep 17 00:00:00 2001 From: Ashish Kumar Date: Wed, 22 May 2019 12:24:04 -0400 Subject: [PATCH 09/11] Update README.md --- README.md | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index c0f5635..5a6fde7 100644 --- a/README.md +++ b/README.md @@ -22,10 +22,9 @@ The utility works with pattern or patter files. Use the pattern files where mul **Note that the Azure blobs are case sensitive.** ### List blobs -Update the abu.exe.config file with the storage location you want to list. -Update the two keys, **storageConnectionString** and **storageContainer**. -Example: - +Update the abu.exe.config file with the storage location you want to list. +Update the two keys, **storageConnectionString** and **storageContainer**. +Example: @@ -69,19 +68,34 @@ Delete blobs option does not work on patterns. You must supply an explicit list Make sure the storage location is correct in the bu.exe.config file. To download blobs using pattern use the following commands: **abu -g temp** or **abu -get temp** + A new guid folder would be created in the current working folder. All the blobs starting with **temp** would be copied into the guid folder. The blobs would be copied keeping the original folder structure in tact. To download multi-pattern blobs, make some text file, myblobs.txt. Enter the pattern or exact blob names per line. **temp** **chrome/folder1/blob2.data** + Use the following commands to download blobs which start with **temp** and **chrome/folder1/blob2.data**. **abu -g myblobs.txt** or **abu -get myblobs.txt** -Notice that this syntax is similar to the pattern syntax. The utility would use the file syntax if a file is found. If your intent was to download all blobs starting with myblobs.txt, then ensure that this file does not exist. - - +Notice that this syntax is similar to the pattern syntax. The utility would use the file syntax if a file is found. If your intent was to download all blobs starting with myblobs.txt, then ensure that this file does not exist. +You can download the blobs in just one folder making their names flat. The slashes in the blob names would be converted to dashes. Use the following commands using a pattern or file name: +**abu -gf myblobs.txt** or +**abu -getflat myblobs.txt** +### Copy blobs +This feature copies blobs from one Azure storage location to the other. Update the abu.exe.config file with the source and target storage locations. +Update the four keys, **storageConnectionString**, **storageContainer**, **targetStorageConnectionString**, and **targetStorageContainer**. +Example: + + + + + +Make a list of blobs in some text file (mylist.txt.) use the following commands to copy blobs from source to target. +**abu -c mylist.txt** or +**abu -copy mylist.txt** From d2bd64e60560d68fe74a4b4642503f931b7eaa42 Mon Sep 17 00:00:00 2001 From: Ashish Kumar Date: Wed, 22 May 2019 12:24:40 -0400 Subject: [PATCH 10/11] Update README.md --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 5a6fde7..9ab4cfb 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,8 @@ The utility works with pattern or patter files. Use the pattern files where mul ### List blobs Update the abu.exe.config file with the storage location you want to list. Update the two keys, **storageConnectionString** and **storageContainer**. -Example: +Example: + @@ -88,7 +89,8 @@ You can download the blobs in just one folder making their names flat. The slas ### Copy blobs This feature copies blobs from one Azure storage location to the other. Update the abu.exe.config file with the source and target storage locations. Update the four keys, **storageConnectionString**, **storageContainer**, **targetStorageConnectionString**, and **targetStorageContainer**. -Example: +Example: + From ecbb234547857643315cde8f38148cf214ff13c0 Mon Sep 17 00:00:00 2001 From: Ashish Kumar Date: Wed, 22 May 2019 12:33:34 -0400 Subject: [PATCH 11/11] Update README.md --- README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 9ab4cfb..c1253ed 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ To get the command line options: **abu** **abu -?** -The utility works with pattern or patter files. Use the pattern files where multiple patterns are involved. The pattern file can be located in any folder. Make sure to specify the fully qualified pattern file name if it is not in the current working folder. +The utility works with pattern or pattern file. Use the pattern file when multiple patterns are involved. The pattern file can be located in any folder. Make sure to specify the fully qualified pattern file name if it is not in the current working folder. **Note that the Azure blobs are case sensitive.** ### List blobs @@ -55,15 +55,13 @@ You can dump the blob listing to another file by redirecting the output: This is particularly useful when you want to delete some blobs. ### Delete blobs -Make sure the storage location is correct in the bu.exe.config file. Delete blobs option works on specific blobs. Make a list of blobs in some text file (mylist.txt) keeping one blob per line. +Make sure the storage location is correct in the bu.exe.config file. Delete blobs option does not work on patterns. You must supply an explicit list of blobs. Make a list of blobs in some text file (mylist.txt) keeping one blob per line. **temp/blob1.txt** **chrome/folder1/blob2.data** To delete these blobs use one of the following commands: **abu -d mylist.txt** or -**abu -delete mylist.txt** - -Delete blobs option does not work on patterns. You must supply an explicit list of blobs. +**abu -delete mylist.txt** ### Download blobs Make sure the storage location is correct in the bu.exe.config file. To download blobs using pattern use the following commands: @@ -99,5 +97,7 @@ Example: Make a list of blobs in some text file (mylist.txt.) use the following commands to copy blobs from source to target. **abu -c mylist.txt** or -**abu -copy mylist.txt** +**abu -copy mylist.txt** +## Performance +For deleting, copying, and downloading operations you can get a better performance by increasing concurrency. This can be done by updaing the abu.exe.config file's **maxThreads** key. The default is 1. You have to find a happy balance for this number. Start with a lower number and keep increasing until the desired throughput is achieved. Remember, keeping this number too high does not necessarily mean, high performance.