Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing test: duplicates in Worker.ready #6096

Open
crusaderky opened this issue Apr 8, 2022 · 0 comments
Open

Missing test: duplicates in Worker.ready #6096

crusaderky opened this issue Apr 8, 2022 · 0 comments
Labels
tests Unit tests and/or continuous integration

Comments

@crusaderky
Copy link
Collaborator

Follow-up from #6062

  1. A task is assigned to a Worker
  2. The task transitions to ready, thus is pushed into the Worker.ready heap, or to constrained, thus is pushed into the Worker.constrained heap
  3. Before it reaches computing state, it is stolen. As there's no way to pop from the middle of a heap, the task stays there.
  4. The task is assigned to the worker again before Worker._ensure_computing could reach its position in the heap
  5. You end up with a duplicate key in the heap.

There's nothing wrong with having a duplicate per se, and Worker._ensure_computing gracefully deals with it; however the use case is far from trivial. At the moment of writing, replacing the graceful handling with a hard assertion of no duplicates only causes very mild flakiness in the most demanding stress tests.

Write a unit test that deterministically triggers the use case both with and without resource constraints.

CC @fjetter

@crusaderky crusaderky added the tests Unit tests and/or continuous integration label Apr 8, 2022
@crusaderky crusaderky changed the title Missing test: task is stolen and then readded to a worker Missing test: duplicates in Worker.ready Apr 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tests Unit tests and/or continuous integration
Projects
None yet
Development

No branches or pull requests

1 participant