Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't auto-release Redlock on instance destruction #274

Merged
merged 2 commits into from
Nov 23, 2020
Merged

Conversation

brainix
Copy link
Owner

@brainix brainix commented Nov 23, 2020

This PR effectively reverts this one: #272

When I deployed this to production, I started seeing these tracebacks in
the logs:

rajiv.shah@RajivsMacBookPro spool % heroku run python3 manage.py renumber_popularity
Running python3 manage.py renumber_popularity on ⬢ spool... up, run.4291 (Hobby)
[2020-11-23 00:46:42,717] INFO in stooges: renumber_popularity() held redlock:popularity for 38 ms
Exception ignored in: <function Redlock.__del__ at 0x7f5aa68db8b0>
Traceback (most recent call last):
  File "/app/.heroku/python/lib/python3.9/site-packages/pottery/redlock.py", line 472, in __del__
    self.__release()
  File "/app/.heroku/python/lib/python3.9/site-packages/pottery/redlock.py", line 409, in release
    futures.add(executor.submit(self.__release_master, master))
  File "/app/.heroku/python/lib/python3.9/concurrent/futures/thread.py", line 163, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown

On second thought, the Redlock's timeout should be enough to preserve
liveness; we don't need to also auto-release Redlock on instance
destruction.

This PR effectively reverts this one: #272

When I deployed this to production, I started seeing these tracebacks in
the logs:

    rajiv.shah@RajivsMacBookPro spool % heroku run python3 manage.py renumber_popularity
    Running python3 manage.py renumber_popularity on ⬢ spool... up, run.4291 (Hobby)
    [2020-11-23 00:46:42,717] INFO in stooges: renumber_popularity() held redlock:popularity for 38 ms
    Exception ignored in: <function Redlock.__del__ at 0x7f5aa68db8b0>
    Traceback (most recent call last):
      File "/app/.heroku/python/lib/python3.9/site-packages/pottery/redlock.py", line 472, in __del__
        self.__release()
      File "/app/.heroku/python/lib/python3.9/site-packages/pottery/redlock.py", line 409, in release
        futures.add(executor.submit(self.__release_master, master))
      File "/app/.heroku/python/lib/python3.9/concurrent/futures/thread.py", line 163, in submit
        raise RuntimeError('cannot schedule new futures after '
    RuntimeError: cannot schedule new futures after interpreter shutdown

On second thought, the Redlock's timeout should be enough to preserve
liveness; we don't need to also auto-release Redlock on instance
destruction.
@brainix brainix merged commit 40ea08a into master Nov 23, 2020
@brainix brainix deleted the redlock-del branch November 23, 2020 00:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant