-
Notifications
You must be signed in to change notification settings - Fork 61
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix race that causes "lease id cannot be null or empty" illegal argument exception from storage #64
Conversation
exception from storage.
@JamesBirdsall, |
James, I thought this was throwing, no? #Closed Refers to: azure-eventhubs-eph/src/main/java/com/microsoft/azure/eventprocessorhost/AzureStorageCheckpointLeaseManager.java:394 in 50fb1db. [](commit_id = 50fb1db, deletion_comment = False) |
No, that message did not appear in the logs. Also, if that line threw, then the storage call which did throw and did appear in the logs would not have been made. |
} | ||
else | ||
{ | ||
newToken = leaseBlob.changeLease(newLeaseId, AccessCondition.generateLeaseCondition(lease.getToken())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lease.getToken() [](start = 92, length = 16)
Due to the same race, this can still throw NullRef, right? #Closed
getToken() does not fetch anything from Storage, it just returns the string currently stored in the Lease. That string does not change unless the instance is replaced or setToken() is called. In this case, all three calls to getToken() are guaranteed to return the same value. |
|
Fix race that causes "lease id cannot be null or empty" illegal argument exception from storage
Problem occurs in acquireLease and is a race condition that requires multiple instances of EventProcessorHost and storage calls to be interleaved in a particular order.
When this instance of EventProcessorHost scanned the lease blobs, this partition was unowned (token is empty) but between then and now, another instance of EPH has established a lease (getLeaseState() is LEASED). We normally enforce that we only steal the lease if it is still owned by the instance which owned it when we scanned, but we can't do that when we don't know who owns it. The safest thing to do is just fail the acquisition. If that means that one EPH instance gets more partitions than it should, rebalancing will take care of that quickly enough.
Description
This checklist is used to make sure that common guidelines for a pull request are followed.