Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Google Cloud races #5081

Merged
merged 3 commits into from
Aug 14, 2018
Merged

Fix Google Cloud races #5081

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3b68156
storage/gcs: fix race condition in releasing lock
sethvargo Aug 10, 2018
c7bb87e
storage/gcs: properly break out of loop during stop
sethvargo Aug 10, 2018
e7296a6
storage/spanner: properly break out of loop during stop
sethvargo Aug 10, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 32 additions & 6 deletions physical/gcs/gcs_ha.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,10 +180,30 @@ func (l *Lock) Unlock() error {
}
l.stopLock.Unlock()

// Delete
// Read the record value before deleting. This needs to be a CAS operation or
// else we might be deleting someone else's lock.
ctx := context.Background()
if err := l.backend.Delete(ctx, l.key); err != nil {
return err
r, err := l.get(ctx)
if err != nil {
return errwrap.Wrapf("failed to read lock for deletion: {{err}}", err)
}
if r != nil && r.Identity == l.identity {
ctx := context.Background()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This ctx call is unnecessary, ctx from above is already Background.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been bitten in the past during refactoring where it's unclear that a context is being used further down the method, so I'd prefer to keep the call to .get and the call to .Delete to use separate contexts that are explicit.

conds := storage.Conditions{
GenerationMatch: r.attrs.Generation,
MetagenerationMatch: r.attrs.Metageneration,
}

obj := l.backend.client.Bucket(l.backend.bucket).Object(l.key)
if err := obj.If(conds).Delete(ctx); err != nil {
// If the pre-condition failed, it means that someone else has already
// acquired the lock and we don't want to delete it.
if terr, ok := err.(*googleapi.Error); ok && terr.Code == 412 {
l.backend.logger.Debug("unlock: preconditions failed (lock already taken by someone else?)")
} else {
return errwrap.Wrapf("failed to delete lock: {{err}}", err)
}
}
}

// We are no longer holding the lock
Expand Down Expand Up @@ -254,20 +274,26 @@ func (l *Lock) watchLock() {
retries := 0
ticker := time.NewTicker(l.watchRetryInterval)

OUTER:
for {
// Check if the channel is already closed
select {
case <-l.stopCh:
break OUTER
default:
}

// Check if we've exceeded retries
if retries >= l.watchRetryMax-1 {
break
break OUTER
}

// Wait for the timer
<-ticker.C
select {
case <-ticker.C:
case <-l.stopCh:
break OUTER
}

// Attempt to read the key
r, err := l.get(context.Background())
Expand All @@ -278,7 +304,7 @@ func (l *Lock) watchLock() {

// Verify the identity is the same
if r == nil || r.Identity != l.identity {
break
break OUTER
}
}

Expand Down
12 changes: 9 additions & 3 deletions physical/spanner/spanner_ha.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -277,20 +277,26 @@ func (l *Lock) watchLock() {
retries := 0
ticker := time.NewTicker(l.watchRetryInterval)

OUTER:
for {
// Check if the channel is already closed
select {
case <-l.stopCh:
break OUTER
default:
}

// Check if we've exceeded retries
if retries >= l.watchRetryMax-1 {
break
break OUTER
}

// Wait for the timer
<-ticker.C
select {
case <-ticker.C:
case <-l.stopCh:
break OUTER
}

// Attempt to read the key
r, err := l.get(context.Background())
Expand All @@ -301,7 +307,7 @@ func (l *Lock) watchLock() {

// Verify the identity is the same
if r == nil || r.Identity != l.identity {
break
break OUTER
}
}

Expand Down