Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Misc Azure Fixes #1550

Merged
merged 1 commit into from
Mar 17, 2020
Merged

Misc Azure Fixes #1550

merged 1 commit into from
Mar 17, 2020

Conversation

joe-maley
Copy link
Contributor

@joe-maley joe-maley commented Mar 17, 2020

Change #1: Fix non-multipart uploads

This fixes an issue where we free the buffer before writing it in the
non-multipart upload path, which can be used for quicker small writes.

Adds an associated unit test.

Change #2: Missing error return path in blob flush

In the blob flush path for multipart uploads, the last flush of the write
cache may silently fail. This ensures the user receives a non-OK status
from flush_blob() in this scenario.

Change #3: Multipart uploads fail when they have > 10 chunks

In a the multipart upload (aka blocklist upload), each chunk is uploaded
a block with a string-id unique to the overall object (aka blob). There is
an undocumented requirement that all decoded string-ids must be of
equal length. Note that string-ids are b64 encoded/decoded in the REST
request.

Currently, the string-ids are generated from an incrementing integer. For
example, the id of the first chunk is "0", the second chunk is "1", the
tenth chunk is "9", and the eleventh chunk is "10". The eleventh chunk
has a string-id size of 2 while all the others have a size of 1. The
eleventh chunk (and all other future chunks) fail with the following
error message:

"The specified blob or block content is invalid"

This patches pads the string-ids to ensure they are all 5-characters long.
The maximum number of chunks is 50,000 so 5-characters is sufficient to
contain all chunk ids.

More info here:
https://gauravmantri.com/2013/05/18/windows-azure-blob-storage-dealing-with-the-specified-blob-or-block-content-is-invalid-error/

@joe-maley joe-maley requested a review from ihnorton March 17, 2020 16:12
Change #1: Fix non-multipart uploads

  This fixes an issue where we free the buffer before writing it in the
  non-multipart upload path, which can be used for quicker small writes.

  Adds an associated unit test.

Change #2: Missing error return path in blob flush

  In the blob flush path for multipart uploads, the last flush of the write
  cache may silently fail. This ensures the user receives a non-OK status
  from flush_blob() in this scenario.

Change #3: Multipart uploads fail when they have > 10 chunks

  In a the multipart upload (aka blocklist upload), each chunk is uploaded
  a block with a string-id unique to the overall object (aka blob). There is
  an undocumented requirement that all string-ids must be of equal length.

  Currently, the string-ids are generated from an incrementing integer. For
  example, the id of the first chunk is "0", the second chunk is "1", the
  tenth chunk is "9", and the eleventh chunk is "10". The eleventh chunk
  has a string-id size of 2 while all the others have a size of 1. The
  eleventh chunk (and all other future chunks) fail with the following
  error message:

    "The specified blob or block content is invalid"

  This patches pads the string-ids to ensure they are all 5-characters long.
  The maximum number of chunks is 50,000 so 5-characters is sufficient to
  contain all chunk ids.

  More info here:
  https://gauravmantri.com/2013/05/18/windows-azure-blob-storage-dealing-with-the-specified-blob-or-block-content-is-invalid-error/
@joe-maley joe-maley force-pushed the jpm/azure-misc-fixes branch from 653789e to f87b5ab Compare March 17, 2020 18:20
@joe-maley joe-maley merged commit 1b90cd4 into dev Mar 17, 2020
@joe-maley joe-maley deleted the jpm/azure-misc-fixes branch March 17, 2020 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants