[BUG] UnsupportedOperationException after calling BlobBatchClient.submitBatchWithResponse()

[lightningrob]

Describe the bug
After calling BlobBatchClient.submitBatchWithResponse() containing a batch of deletions, checking the response status throws the exception below. This does not always happen, maybe 20% of the time.

Exception or Stack Trace
java.lang.UnsupportedOperationException: Batch request has not been sent.
at com.azure.storage.blob.batch.BlobBatchOperationResponse.assertResponseReceived(BlobBatchOperationResponse.java:106)
at com.azure.storage.blob.batch.BlobBatchOperationResponse.getStatusCode(BlobBatchOperationResponse.java:43)
at com.github.ambry.cloud.azure.BatchBlobDeletionTest.purgeBlobs(BatchBlobDeletionTest.java:133)

To Reproduce
Run the attached main class (executes in a loop) until the exception occurs.

Code Snippet
BlobBatch blobBatch = blobBatchClient.getBlobBatch();
List<Response> responseList = new ArrayList<>();
for (String blobName : blobNames) {
responseList.add(blobBatch.deleteBlob(containerName, blobName));
}
int purgeCount = 0;
blobBatchClient.submitBatchWithResponse(blobBatch, false, Duration.ofHours(1), Context.NONE);
logger.debug("Batch submitted.");
for (int j = 0; j < responseList.size(); j++) {
Response response = responseList.get(j);

Expected behavior
Expect the status to be available by the time the submitBatch call returns.

Screenshots
N/A

Setup (please complete the following information):

• OS: MacOS 10.15.2
• IDE : IntelliJ
• Version of the Library used: azure-storage-blob-12.0.0.jar

Additional context
Need to supply program with system property azure.storage.connection.string pointing to your Azure storage account.
Note: The same program sometimes fails with a NullPointerException. (See #7256.)

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• [ x] Bug Description Added
• [ x] Repro Steps Added
• [ x] Setup information Added

BatchBlobDeletionTest.java.txt

[alzimmermsft]

@lightningrob It appears that occasionally a batch is missing an operation, this results in checking each operations status to fail when the operation that was dropped is checked.

I'm still investigating why an operation is dropped in the first place, I'll reply once that is determined.

[alzimmermsft]

The root of the issue is that the internal holder for the operations wasn't synchronized or thread-safe and was being used in a potentially threaded scenario. This led to operations being "dropped" occasionally.

[lightningrob]

The root of the issue is that the internal holder for the operations wasn't synchronized or thread-safe and was being used in a potentially threaded scenario. This led to operations being "dropped" occasionally.

I see. But would that explain seeing the issue in a single-threaded test?

[alzimmermsft]

Poor explanation on my behalf.

BlobBatch has an internal state of operations queued up to send when the batch is submitted, upon batch submission these operations need to be turned into the sub-requests which are sent in the batch request. Converting the operations into their sub-request representation leverages Reactor's flatMap which allows for multiple operations to be in-flight at once. This is where the issue of multiple-threads operating on the not thread-safe collection results in sub-request missing in the batch request, or in the case of #7256 a sub-request being null.

[alzimmermsft]

This issue should be resolved by the most recently release version of Azure Storage Blob Batch.

[alzimmermsft]

Resolved by PR #6882

[lightningrob]

@alzimmermsft I reran my test with release 12.2.0 and could not reproduce the issue after roughly 500 iterations. Thanks for the quick turnaround.