Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql: refresh table leases asynchronously #17227

Closed
vivekmenezes opened this issue Jul 26, 2017 · 1 comment · Fixed by #18606 or #18859
Closed

sql: refresh table leases asynchronously #17227

vivekmenezes opened this issue Jul 26, 2017 · 1 comment · Fixed by #18606 or #18859
Assignees
@lgo @vivekmenezes
Milestone
2.0

Comments

@vivekmenezes
Copy link
Contributor

When a table lease is about to expire we refresh the lease in the context of the request trying to use a table descriptor. It will be better to issue a request to acquire a new lease, and use the old (but not yet expired) table lease for the request. This is likely to show up one day as an increased tail latency.
@vivekmenezes vivekmenezes added this to the 1.2 milestone Jul 26, 2017
@vivekmenezes vivekmenezes self-assigned this Jul 26, 2017
@vivekmenezes vivekmenezes mentioned this issue Jul 26, 2017
Merged
@vivekmenezes
Copy link
Contributor Author

Actually this problem is a lot worse. Once the lease is about to expire the first request to want a table descriptor and note that the lease is about to expire will initiate the process of acquiring a lease. The problem is all subsequent requests wanting the table descriptor will also get blocked waiting for the new lease being acquired. We should not block these requests, they should all use the old lease while the new lease is being acquired.

@vivekmenezes vivekmenezes mentioned this issue Sep 19, 2017
Closed
@lgo lgo mentioned this issue Sep 19, 2017
Merged
lgo added a commit to lgo/cockroach that referenced this issue Sep 19, 2017
@lgo
sql: Refresh table leases asynchronously on access
45910e0 
Previously if we would only refresh a table lease if it would be expired for the
transaction timestamp. Now we will refresh the lease asynchronously on access if
it is about to expire in order to prevent transactions from blocking on lease
acquisition.

Fixes cockroachdb#17227.
lgo added a commit to lgo/cockroach that referenced this issue Sep 19, 2017
@lgo
sql: Refresh table leases asynchronously on access
9c767a3 
Previously if we would only refresh a table lease if it would be expired for the
transaction timestamp. Now we will refresh the lease asynchronously on access if
it is about to expire in order to prevent transactions from blocking on lease
acquisition.

Fixes cockroachdb#17227.
@lgo lgo mentioned this issue Sep 21, 2017
Closed
lgo added a commit to lgo/cockroach that referenced this issue Sep 25, 2017
@lgo
sql: Refresh table leases asynchronously on access
9effa57 
Previously if we would only refresh a table lease if it would be expired for the
transaction timestamp. Now we will refresh the lease asynchronously on access if
it is about to expire in order to prevent transactions from blocking on lease
acquisition.

Fixes cockroachdb#17227.
lgo added a commit to lgo/cockroach that referenced this issue Sep 26, 2017
@lgo
sql: Refresh table leases asynchronously on access
5dda53f 
Previously if we would only refresh a table lease if it would be expired for the
transaction timestamp. Now we will refresh the lease asynchronously on access if
it is about to expire in order to prevent transactions from blocking on lease
acquisition.

Fixes cockroachdb#17227.
lgo added a commit to lgo/cockroach that referenced this issue Sep 27, 2017
@lgo
sql: Refresh table leases asynchronously on access
ec9f73b 
Previously if we would only refresh a table lease if it would be expired for the
transaction timestamp. Now we will refresh the lease asynchronously on access if
it is about to expire in order to prevent transactions from blocking on lease
acquisition.

Fixes cockroachdb#17227.
lgo added a commit to lgo/cockroach that referenced this issue Sep 27, 2017
@lgo
sql: Refresh table leases asynchronously on access
24bb4fe 
Previously if we would only refresh a table lease if it would be expired for the
transaction timestamp. Now we will refresh the lease asynchronously on access if
it is about to expire in order to prevent transactions from blocking on lease
acquisition.

Fixes cockroachdb#17227.
lgo added a commit to lgo/cockroach that referenced this issue Sep 28, 2017
@lgo
Revert "sql: Refresh table leases asynchronously on access"
80e31f4 
This reverts commit ec9f73b.
The commit was attempting to fix a lease deadline bug, cockroachdb#18684,
and close cockroachdb#17227 by not blocking on table descriptor lease acquisition.
The fix prevents the first issue, but does not actually resolve the
second issue. There was also a flaky test added which will be fixed with the
revert, cockroachdb#18827.

Even after this commits revert, cockroachdb#18804 fixes the first bug for SERIALIZABLE
transactions.

A replacement patch is in-progress at cockroachdb#18844.
lgo added a commit to lgo/cockroach that referenced this issue Oct 2, 2017
@lgo
sql: Table lease refresh done asynchronously
02e1911 
This commit completes changes TableDescriptor lease acquisition to
acquire a new lease before the current lease expires in order to prevent
routines from fully blocking when there are no unexpired leases.

When we asynchronously attempt to acquire a lease we create a goroutine
which only lives for the length of the attempt. If it fails, the same
process will repeat to attempt to get a lease until the current lease
expires and routines block on the current acquisition. Afterwards, the
error will become user-facing if no lease is acquired yet.

Closes cockroachdb#17227.
@andreimatei andreimatei reopened this Oct 3, 2017
@lgo lgo mentioned this issue Oct 3, 2017
Merged
lgo added a commit to lgo/cockroach that referenced this issue Oct 3, 2017
@lgo
sql: Table lease refresh done asynchronously
29e73ce 
This commit completes changes TableDescriptor lease acquisition to
acquire a new lease before the current lease expires in order to prevent
routines from fully blocking when there are no unexpired leases.

Using singleflight, singleflight.DoChan can be used to do a non-blocking
equivilant of singleflight.Do. This will also make any calls to
singleflight.Do to not call the function twice, accomplishing what we
need to do for async renewals.

If renewal continously fails, the error will become user-facing if no
lease is acquired yet by the time a lease is expired.

Closes cockroachdb#17227.
lgo added a commit to lgo/cockroach that referenced this issue Oct 18, 2017
@lgo
sql: Add refreshing for table descriptor leases
b9ec6a8 
This commit changes TableDescriptor lease acquisition to acquire a new
lease before the current lease expires. This prevents routines from
fully blocking when there are no valid leases.

If renewal continously fails, the error will become user-facing if no
lease is acquired yet by the time a lease is expired.

Closes cockroachdb#17227.
lgo added a commit to lgo/cockroach that referenced this issue Oct 20, 2017
@lgo
sql: Add refreshing for table descriptor leases
7905053 
This commit changes TableDescriptor lease acquisition to acquire a new
lease before the current lease expires. This prevents routines from
fully blocking when there are no valid leases.

If renewal continously fails, the error will become user-facing if no
lease is acquired yet by the time a lease is expired. During quiescing
we also need to ensure all renewal timers are stopped.

Closes cockroachdb#17227.
lgo added a commit to lgo/cockroach that referenced this issue Oct 23, 2017
@lgo
sql: Add background renewal for table descriptor leases
165da70 
This commit changes TableDescriptor lease acquisition to acquire a new
lease if the current lease is about to expire. This prevents routines
from blocking when there are no valid leases on a frequently accessed
table.

If renewal continously fails, the error will become user-facing if no
lease is acquired yet by the time a lease is expired.

Closes cockroachdb#17227.
lgo added a commit to lgo/cockroach that referenced this issue Oct 23, 2017
@lgo
sql: Add background renewal for table descriptor leases
4652a1d 
This commit changes TableDescriptor lease acquisition to acquire a new
lease if the current lease is about to expire. This prevents routines
from blocking when there are no valid leases on a frequently accessed
table.

If renewal continously fails, the error will become user-facing if no
lease is acquired yet by the time a lease is expired.

Closes cockroachdb#17227.
lgo added a commit to lgo/cockroach that referenced this issue Oct 24, 2017
@lgo
sql: Add background renewal for table descriptor leases
c6a0511 
This commit changes TableDescriptor lease acquisition to acquire a new
lease if the current lease is about to expire. This prevents routines
from blocking when there are no valid leases on a frequently accessed
table.

If renewal continously fails, the error will become user-facing if no
lease is acquired yet by the time a lease is expired.

Closes cockroachdb#17227.
lgo added a commit to lgo/cockroach that referenced this issue Oct 24, 2017
@lgo
sql: Add background renewal for table descriptor leases
e0bfa70 
This commit changes TableDescriptor lease acquisition to acquire a new
lease if the current lease is about to expire. This prevents routines
from blocking when there are no valid leases on a frequently accessed
table.

If renewal continously fails, the error will become user-facing if no
lease is acquired yet by the time a lease is expired.

Closes cockroachdb#17227.
lgo added a commit to lgo/cockroach that referenced this issue Oct 24, 2017
@lgo
sql: Add background renewal for table descriptor leases
d7b7942 
This commit changes TableDescriptor lease acquisition to acquire a new
lease if the current lease is about to expire. This prevents routines
from blocking when there are no valid leases on a frequently accessed
table.

If renewal continously fails, the error will become user-facing if no
lease is acquired yet by the time a lease is expired.

Closes cockroachdb#17227.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lgo lgo

@vivekmenezes vivekmenezes

Labels
None yet
Projects
None yet
Milestone
2.0
3 participants
@andreimatei @lgo @vivekmenezes