Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Storage] support blob tier in upload/copy block blob #4

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions src/Storage/Storage.Management/ChangeLog.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@
- Additional information about change #1
-->
## Upcoming Release
* Support StandardBlobTier on Upload and Copy Block Blob
- Set-AzureStorageBlobContent
- Start-AzureStorageBlobCopy

## Version 1.5.0
* Change 2 parameters "-IndexDocument" and "-ErrorDocument404Path" from required to optional in cmdlet:
Expand Down
6 changes: 3 additions & 3 deletions src/Storage/Storage.Management/Storage.Management.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Azure.Management.Storage" Version="12.0.0" />
<PackageReference Include="Microsoft.Azure.Storage.Blob" Version="10.0.3" />
<PackageReference Include="Microsoft.Azure.Storage.File" Version="10.0.3" />
<PackageReference Include="Microsoft.Azure.Storage.Queue" Version="10.0.3" />
<PackageReference Include="Microsoft.Azure.Storage.Blob" Version="10.1.0" />
<PackageReference Include="Microsoft.Azure.Storage.File" Version="10.1.0" />
<PackageReference Include="Microsoft.Azure.Storage.Queue" Version="10.1.0" />
</ItemGroup>
<ItemGroup>
<Compile Include="$(StorageToolsPath)Adapters\ARM.Storage.6\ARMStorageProvider.cs" Link="Common\ARMStorageProvider.cs" />
Expand Down
35 changes: 26 additions & 9 deletions src/Storage/Storage.Management/help/Set-AzStorageBlobContent.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,9 @@ Uploads a local file to an Azure Storage blob.
### SendManual (Default)
```
Set-AzStorageBlobContent [-File] <String> [-Container] <String> [-Blob <String>] [-BlobType <String>]
[-Properties <Hashtable>] [-Metadata <Hashtable>] [-PremiumPageBlobTier <PremiumPageBlobTier>] [-Force]
[-AsJob] [-Context <IStorageContext>] [-ServerTimeoutPerRequest <Int32>] [-ClientTimeoutPerRequest <Int32>]
[-Properties <Hashtable>] [-Metadata <Hashtable>] [-PremiumPageBlobTier <PremiumPageBlobTier>]
[-StandardBlobTier <StandardBlobTier>] [-Force] [-AsJob] [-Context <IStorageContext>]
[-ServerTimeoutPerRequest <Int32>] [-ClientTimeoutPerRequest <Int32>]
[-DefaultProfile <IAzureContextContainer>] [-ConcurrentTaskCount <Int32>] [-WhatIf] [-Confirm]
[<CommonParameters>]
```
Expand All @@ -26,17 +27,17 @@ Set-AzStorageBlobContent [-File] <String> [-Container] <String> [-Blob <String>]
```
Set-AzStorageBlobContent [-File] <String> [-Blob <String>] -CloudBlobContainer <CloudBlobContainer>
[-BlobType <String>] [-Properties <Hashtable>] [-Metadata <Hashtable>]
[-PremiumPageBlobTier <PremiumPageBlobTier>] [-Force] [-AsJob] [-Context <IStorageContext>]
[-ServerTimeoutPerRequest <Int32>] [-ClientTimeoutPerRequest <Int32>]
[-PremiumPageBlobTier <PremiumPageBlobTier>] [-StandardBlobTier <StandardBlobTier>] [-Force] [-AsJob]
[-Context <IStorageContext>] [-ServerTimeoutPerRequest <Int32>] [-ClientTimeoutPerRequest <Int32>]
[-DefaultProfile <IAzureContextContainer>] [-ConcurrentTaskCount <Int32>] [-WhatIf] [-Confirm]
[<CommonParameters>]
```

### BlobPipeline
```
Set-AzStorageBlobContent [-File] <String> -CloudBlob <CloudBlob> [-BlobType <String>] [-Properties <Hashtable>]
[-Metadata <Hashtable>] [-PremiumPageBlobTier <PremiumPageBlobTier>] [-Force] [-AsJob]
[-Context <IStorageContext>] [-ServerTimeoutPerRequest <Int32>] [-ClientTimeoutPerRequest <Int32>]
[-Metadata <Hashtable>] [-PremiumPageBlobTier <PremiumPageBlobTier>] [-StandardBlobTier <StandardBlobTier>]
[-Force] [-AsJob] [-Context <IStorageContext>] [-ServerTimeoutPerRequest <Int32>] [-ClientTimeoutPerRequest <Int32>]
[-DefaultProfile <IAzureContextContainer>] [-ConcurrentTaskCount <Int32>] [-WhatIf] [-Confirm]
[<CommonParameters>]
```
Expand Down Expand Up @@ -90,12 +91,12 @@ The first command creates a hash table that contains metadata for a blob, and st
The second command uploads the file that is named ContosoPlanning to the container named ContosoUploads.
The blob includes the metadata stored in $Metadata, and has PremiumPageBlobTier as P10.

### Example 6: Upload a file to blob with specified blob properties
### Example 6: Upload a file to blob with specified blob properties, and set StandardBlobTier as Cool
```
PS C:\> Set-AzStorageBlobContent -File "ContosoPlanning" -Container "ContosoUploads" -Properties @{"ContentType" = "image/jpeg"; "ContentMD5" = "i727sP7HigloQDsqadNLHw=="}
PS C:\> Set-AzStorageBlobContent -File "ContosoPlanning" -Container "ContosoUploads" -Properties @{"ContentType" = "image/jpeg"; "ContentMD5" = "i727sP7HigloQDsqadNLHw=="} -StandardBlobTier Cool
```

This command uploads the file that is named ContosoPlanning to the container named ContosoUploads with specified blob properties.
This command uploads the file that is named ContosoPlanning to the container named ContosoUploads with specified blob properties, and sets StandardBlobTier as Cool

## PARAMETERS

Expand Down Expand Up @@ -372,6 +373,22 @@ Accept pipeline input: False
Accept wildcard characters: False
```

### -StandardBlobTier
Block Blob Tier

```yaml
Type: Microsoft.WindowsAzure.Storage.Blob.StandardBlobTier
Parameter Sets: (All)
Aliases:
Accepted values: Unknown, Hot, Cool, Archive

Required: False
Position: Named
Default value: None
Accept pipeline input: False
Accept wildcard characters: False
```

### -Confirm
Prompts you for confirmation before running the cmdlet.

Expand Down
51 changes: 37 additions & 14 deletions src/Storage/Storage.Management/help/Start-AzStorageBlobCopy.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,35 +16,35 @@ Starts to copy a blob.
### ContainerName (Default)
```
Start-AzStorageBlobCopy [-SrcBlob] <String> -SrcContainer <String> -DestContainer <String> [-DestBlob <String>]
[-PremiumPageBlobTier <PremiumPageBlobTier>] [-Context <IStorageContext>] [-DestContext <IStorageContext>]
[-Force] [-ServerTimeoutPerRequest <Int32>] [-ClientTimeoutPerRequest <Int32>]
[-DefaultProfile <IAzureContextContainer>] [-ConcurrentTaskCount <Int32>] [-WhatIf] [-Confirm]
[<CommonParameters>]
[-PremiumPageBlobTier <PremiumPageBlobTier>] [-StandardBlobTier <StandardBlobTier>]
[-Context <IStorageContext>] [-DestContext <IStorageContext>] [-Force] [-ServerTimeoutPerRequest <Int32>]
[-ClientTimeoutPerRequest <Int32>] [-DefaultProfile <IAzureContextContainer>] [-ConcurrentTaskCount <Int32>]
[-WhatIf] [-Confirm] [<CommonParameters>]
```

### BlobInstance
```
Start-AzStorageBlobCopy -CloudBlob <CloudBlob> -DestContainer <String> [-DestBlob <String>]
[-PremiumPageBlobTier <PremiumPageBlobTier>] [-Context <IStorageContext>] [-DestContext <IStorageContext>]
[-Force] [-ServerTimeoutPerRequest <Int32>] [-ClientTimeoutPerRequest <Int32>]
[-DefaultProfile <IAzureContextContainer>] [-ConcurrentTaskCount <Int32>] [-WhatIf] [-Confirm]
[<CommonParameters>]
[-PremiumPageBlobTier <PremiumPageBlobTier>] [-StandardBlobTier <StandardBlobTier>]
[-Context <IStorageContext>] [-DestContext <IStorageContext>] [-Force] [-ServerTimeoutPerRequest <Int32>]
[-ClientTimeoutPerRequest <Int32>] [-DefaultProfile <IAzureContextContainer>] [-ConcurrentTaskCount <Int32>]
[-WhatIf] [-Confirm] [<CommonParameters>]
```

### BlobInstanceToBlobInstance
```
Start-AzStorageBlobCopy -CloudBlob <CloudBlob> -DestCloudBlob <CloudBlob>
[-PremiumPageBlobTier <PremiumPageBlobTier>] [-Context <IStorageContext>] [-DestContext <IStorageContext>]
[-Force] [-ServerTimeoutPerRequest <Int32>] [-ClientTimeoutPerRequest <Int32>]
[-DefaultProfile <IAzureContextContainer>] [-ConcurrentTaskCount <Int32>] [-WhatIf] [-Confirm]
[<CommonParameters>]
[-PremiumPageBlobTier <PremiumPageBlobTier>] [-StandardBlobTier <StandardBlobTier>]
[-Context <IStorageContext>] [-DestContext <IStorageContext>] [-Force] [-ServerTimeoutPerRequest <Int32>]
[-ClientTimeoutPerRequest <Int32>] [-DefaultProfile <IAzureContextContainer>] [-ConcurrentTaskCount <Int32>]
[-WhatIf] [-Confirm] [<CommonParameters>]
```

### ContainerInstance
```
Start-AzStorageBlobCopy -CloudBlobContainer <CloudBlobContainer> [-SrcBlob] <String> -DestContainer <String>
[-DestBlob <String>] [-PremiumPageBlobTier <PremiumPageBlobTier>] [-Context <IStorageContext>]
[-DestContext <IStorageContext>] [-Force] [-ServerTimeoutPerRequest <Int32>]
[-DestBlob <String>] [-PremiumPageBlobTier <PremiumPageBlobTier>] [-StandardBlobTier <StandardBlobTier>]
[-Context <IStorageContext>] [-DestContext <IStorageContext>] [-Force] [-ServerTimeoutPerRequest <Int32>]
[-ClientTimeoutPerRequest <Int32>] [-DefaultProfile <IAzureContextContainer>] [-ConcurrentTaskCount <Int32>]
[-WhatIf] [-Confirm] [<CommonParameters>]
```
Expand Down Expand Up @@ -154,6 +154,13 @@ This command creates a context for the account named ContosoGeneral that uses th
The second command copies the file from the specified URI to the blob named ContosoPlanning in the container named ContosoArchive.
The command starts the copy operation in the context stored in $Context.

### Example 6: Copy a source block blob, to a destination block blob with StandardBlobTier as Archive
```
C:\PS>Start-AzStorageBlobCopy -SrcContainer "ContosoUploads" -SrcBlob "ContosoPlanning2015" -DestContainer "ContosoArchives" -DestBlob "ContosoPlanning2015Backup" -StandardBlobTier Archive
```

This command starts the copy operation of the block blob, to a destination block blob with StandardBlobTier as Archive.

## PARAMETERS

### -AbsoluteUri
Expand Down Expand Up @@ -510,6 +517,22 @@ Accept pipeline input: False
Accept wildcard characters: False
```

### -StandardBlobTier
Block Blob Tier

```yaml
Type: Microsoft.WindowsAzure.Storage.Blob.StandardBlobTier
Parameter Sets: ContainerName, BlobInstance, BlobInstanceToBlobInstance, ContainerInstance
Aliases:
Accepted values: Hot, Cool

Required: False
Position: Named
Default value: None
Accept pipeline input: False
Accept wildcard characters: False
```

### -Confirm
Prompts you for confirmation before running the cmdlet.

Expand Down
10 changes: 10 additions & 0 deletions src/Storage/Storage.Test/Service/MockStorageBlobManagement.cs
Original file line number Diff line number Diff line change
Expand Up @@ -716,6 +716,16 @@ public AccountProperties GetAccountProperties()
throw new NotImplementedException();
}

public Task SetStandardBlobTierAsync(CloudBlockBlob blob, AccessCondition accessCondition, StandardBlobTier tier, BlobRequestOptions options, OperationContext operationContext, CancellationToken cmdletCancellationToken)
{
throw new NotImplementedException();
}

public Task<string> StartCopyAsync(CloudBlockBlob blob, Uri source, StandardBlobTier standardBlobTier, AccessCondition sourceAccessCondition, AccessCondition destAccessCondition, BlobRequestOptions options, OperationContext operationContext, CancellationToken cancellationToken)
{
throw new NotImplementedException();
}

/// <summary>
/// The storage context
/// </summary>
Expand Down
6 changes: 3 additions & 3 deletions src/Storage/Storage.Test/Storage.Test.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,9 @@

<ItemGroup>
<PackageReference Include="Microsoft.Azure.Cosmos.Table" Version="0.10.1-preview" />
<PackageReference Include="Microsoft.Azure.Storage.Blob" Version="10.0.3" />
<PackageReference Include="Microsoft.Azure.Storage.File" Version="10.0.3" />
<PackageReference Include="Microsoft.Azure.Storage.Queue" Version="10.0.3" />
<PackageReference Include="Microsoft.Azure.Storage.Blob" Version="10.1.0" />
<PackageReference Include="Microsoft.Azure.Storage.File" Version="10.1.0" />
<PackageReference Include="Microsoft.Azure.Storage.Queue" Version="10.1.0" />
<PackageReference Include="MSTest.TestAdapter" Version="1.2.1" />
<PackageReference Include="MSTest.TestFramework" Version="1.2.1" />
</ItemGroup>
Expand Down
53 changes: 44 additions & 9 deletions src/Storage/Storage/Blob/Cmdlet/SetAzureStorageBlobContent.cs
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ public Hashtable Metadata

private Hashtable BlobMetadata = null;

[Parameter(HelpMessage = "Page Blob Tier", Mandatory = false)]
[Parameter(HelpMessage = "Premium Page Blob Tier", Mandatory = false)]
public PremiumPageBlobTier PremiumPageBlobTier
{
get
Expand All @@ -170,6 +170,22 @@ public PremiumPageBlobTier PremiumPageBlobTier

private PremiumPageBlobTier? pageBlobTier = null;

[Parameter(HelpMessage = "Block Blob Tier", Mandatory = false)]
[ValidateSet("Hot", "Cool", "Archive", IgnoreCase = true)]
public StandardBlobTier StandardBlobTier
{
get
{
return standardBlobTier.Value;
}

set
{
standardBlobTier = value;
}
}
private StandardBlobTier? standardBlobTier = null;

private BlobUploadRequestQueue UploadRequests = new BlobUploadRequestQueue();

/// <summary>
Expand Down Expand Up @@ -223,12 +239,12 @@ await DataMovementTransferHelper.DoTransfer(() =>
data.Record,
this.OutputStream).ConfigureAwait(false);

if (this.BlobProperties != null || this.BlobMetadata != null || this.pageBlobTier != null)
if (this.BlobProperties != null || this.BlobMetadata != null || this.pageBlobTier != null || this.standardBlobTier != null)
{
await Task.WhenAll(
this.SetBlobProperties(localChannel, blob, this.BlobProperties),
this.SetBlobMeta(localChannel, blob, this.BlobMetadata),
this.SetBlobTier(localChannel, blob, pageBlobTier)).ConfigureAwait(false);
this.SetBlobTier(localChannel, blob, pageBlobTier, standardBlobTier)).ConfigureAwait(false);
}

try
Expand Down Expand Up @@ -437,9 +453,9 @@ private async Task SetBlobMeta(IStorageBlobManagement localChannel, StorageBlob.
/// <param name="azureBlob">CloudBlob object</param>
/// <param name="blockBlobTier">Block Blob Tier</param>
/// <param name="pageBlobTier">Page Blob Tier</param>
private async Task SetBlobTier(IStorageBlobManagement localChannel, StorageBlob.CloudBlob blob, PremiumPageBlobTier? pageBlobTier)
private async Task SetBlobTier(IStorageBlobManagement localChannel, StorageBlob.CloudBlob blob, PremiumPageBlobTier? pageBlobTier = null, StandardBlobTier? standardBlobTier = null)
{
if (pageBlobTier == null)
if (pageBlobTier == null && standardBlobTier == null)
{
return;
}
Expand All @@ -451,6 +467,11 @@ private async Task SetBlobTier(IStorageBlobManagement localChannel, StorageBlob.
{
await Channel.SetPageBlobTierAsync((CloudPageBlob)blob, pageBlobTier.Value, requestOptions, OperationContext, CmdletCancellationToken).ConfigureAwait(false);
}
if (standardBlobTier != null)
{
AccessCondition accessCondition = null;
await Channel.SetStandardBlobTierAsync((CloudBlockBlob)blob, accessCondition, standardBlobTier.Value, requestOptions, OperationContext, CmdletCancellationToken).ConfigureAwait(false);
}
}

/// <summary>
Expand All @@ -467,12 +488,12 @@ protected override void OnTaskSuccessful(DataMovementUserData data)
AccessCondition accessCondition = null;
StorageBlob.BlobRequestOptions requestOptions = RequestOptions;

if (BlobProperties != null || BlobMetadata != null || this.pageBlobTier != null)
if (BlobProperties != null || BlobMetadata != null || this.pageBlobTier != null || this.standardBlobTier != null)
{
Task[] tasks = new Task[3];
tasks[0] = SetBlobProperties(localChannel, blob, BlobProperties);
tasks[1] = SetBlobMeta(localChannel, blob, BlobMetadata);
tasks[2] = SetBlobTier(localChannel, blob, pageBlobTier);
tasks[2] = SetBlobTier(localChannel, blob, pageBlobTier, standardBlobTier);
Task.WaitAll(tasks);
}

Expand Down Expand Up @@ -527,8 +548,22 @@ public override void ExecuteCmdlet()
DoBeginProcessing();
}

ValidateBlobTier(string.Equals(blobType, PageBlobType, StringComparison.InvariantCultureIgnoreCase)? StorageBlob.BlobType.PageBlob : StorageBlob.BlobType.Unspecified,
pageBlobTier);

// Validate the Blob Tier matches with blob Type
StorageBlob.BlobType type = StorageBlob.BlobType.BlockBlob;
if (string.Equals(blobType, BlockBlobType, StringComparison.InvariantCultureIgnoreCase))
{
type = StorageBlob.BlobType.BlockBlob;
}
else if (string.Equals(blobType, PageBlobType, StringComparison.InvariantCultureIgnoreCase))
{
type = StorageBlob.BlobType.PageBlob;
}
else
{
type = StorageBlob.BlobType.Unspecified;
}
ValidateBlobTier(type, pageBlobTier, standardBlobTier);

if (BlobProperties != null)
{
Expand Down
Loading