Thread-safety improvements. Add tests with multiple threads. CI improvements. #122
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The test failures are coming from replacing I think it's best to be conservative with this update so I'm going to revert that commit. |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #122 +/- ##
==========================================
- Coverage 79.29% 79.19% -0.10%
==========================================
Files 10 10
Lines 1922 1923 +1
==========================================
- Hits 1524 1523 -1
- Misses 398 400 +2 ☔ View full report in Codecov by Sentry. |
JamesWrigley
force-pushed
the
threadsafety
branch
from
2be833c to
877be4f
Compare
|
As discussed offline, added back the commits replacing |
|
Again discussed on slack. We decided to revert to |
IanButterworth
changed the title
This was referenced Jan 24, 2025
This should fix an exception seen in CI from the lingering timeout task:
```
Test Summary: | Pass Total Time
Deserialization error recovery and include() | 11 11 3.9s
From worker 4: Unhandled Task ERROR: EOFError: read end of file
From worker 4: Stacktrace:
From worker 4: [1] wait
From worker 4: @ .\asyncevent.jl:159 [inlined]
From worker 4: [2] sleep(sec::Float64)
From worker 4: @ Base .\asyncevent.jl:265
From worker 4: [3] (::DistributedNext.var"#34#37"{DistributedNext.Worker, Float64})()
From worker 4: @ DistributedNext D:\a\DistributedNext.jl\DistributedNext.jl\src\cluster.jl:213
```
JamesWrigley
force-pushed
the
threadsafety
branch
from
a866f6b to
c09b034
Compare
IanButterworth
force-pushed
the
threadsafety
branch
2 times, most recently
from
f6f1d38 to
190a6b6
Compare
IanButterworth
changed the title
IanButterworth
force-pushed
the
threadsafety
branch
2 times, most recently
from
e7212a9 to
dcd89eb
Compare
IanButterworth
changed the title
IanButterworth
force-pushed
the
threadsafety
branch
7 times, most recently
from
ddb8ad4 to
836bed1
Compare
IanButterworth
force-pushed
the
threadsafety
branch
9 times, most recently
from
3d6f794 to
54575e2
Compare
IanButterworth
force-pushed
the
threadsafety
branch
from
54575e2 to
e28698f
Compare
| @@ -274,7 +274,7 @@ end | |||
| const any_gc_flag = Threads.Condition() | |||
| function start_gc_msgs_task() | |||
| errormonitor( | |||
| Threads.@spawn begin | |||
| @async begin | |||
There was a problem hiding this comment.
This change fixed the race (#124). It seems to have always been a @spawn hence why I didn't initially convert it to @async.
IanButterworth
force-pushed
the
threadsafety
branch
from
e28698f to
ca2bd3c
Compare
IanButterworth
approved these changes
Jan 26, 2025
JamesWrigley
commented
Jan 26, 2025
This cuts the running time down from ~1m to ~10s. It shouldn't be necessary to re-create the workers for every iteration of the inner loop.
IanButterworth
force-pushed
the
threadsafety
branch
from
f189b1c to
8494bbd
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Co-authored by @IanButterworth
@spawnto@asyncto get tests passing. Moving back to@spawnwill take more work, but it should be low priority because the main thread, where Distributed usually runs, will already bestickyanyway, so the gain is minor.fbf8c12(#122) is necessary to avoid each worker in those tests re-precompiling any stdlibs they use. Speeds up macos runs by ~1 minute97fc237(#122) adds a sigusr1/siginfo profile before killing a worker that won't shut down, to aid debuggingFixes #124
TODO
@async)CC @jpsamaroo