JIT caching updates #886
JIT caching updates #886
Conversation
|
For some reason, with the cache warming enabled, the tests seem to fail more consistently. Great. I'll postpone merging this until I know why. |
sk1p
force-pushed
the
jit-caching-warmup
branch
2 times, most recently
from
eecc69b to
af96616
Compare
This should now behave better, as we should only have a single jit compilation when re-using the dask worker and scheduler. |
Codecov Report
@@ Coverage Diff @@
## master #886 +/- ##
==========================================
+ Coverage 68.78% 68.94% +0.16%
==========================================
Files 267 267
Lines 11711 11749 +38
Branches 1592 1597 +5
==========================================
+ Hits 8055 8100 +45
+ Misses 3355 3351 -4
+ Partials 301 298 -3
Continue to review full report at Codecov.
|
|
/azp run libertem.libertem-data |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
Tests are still not stable, but this time I can reproduce the issue with Python 3.6 locally. |
|
Tracked down to tornadoweb/tornado#2034 dask/distributed#4356, proposed fix at tornadoweb/tornado#2963 |
|
Wow, nice catch! 🚀 |
|
So, do we have to wait for upstream or can we circumvent this in LiberTEM? |
This one is hard to circumvent, as the problem is caused by a centrally used utility function. I think it could be possible to make the failure less likely, by just taking a different code path. I'll need to try a few things, and I'll come back to this PR. |
sk1p
force-pushed
the
jit-caching-warmup
branch
from
af96616 to
1e14208
Compare
|
Looks like that worked, both Mac and Win Py36 build finished. |
|
/azp run libertem.libertem-data |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
Nice to see that it goes around that issue! 🎉 Is the low test coverage an artifact from not collecting coverage from Dask, or could we still add test cases that improve coverage? |
This also adds a new `run_each_worker` method on `JobExecutor`
Also return the results of each function run
It's possible that this is less likely to trigger the problem with distributed.utils.All / tornado.gen.WaitItertor
Use an inline executor to collect coverage
sk1p
force-pushed
the
jit-caching-warmup
branch
from
796557b to
b03e2fb
Compare
|
/azp run libertem.libertem-data |
|
Azure Pipelines successfully started running 1 pipeline(s). |
It was, but now I've added a test case using the |
|
Awesome, merging! |
Btw. this was recently merged, so hopefully will be fixed for good on the next release of tornado. |
Fixes #798 once numba properly caches functions with w/ closures over other functions. Independent from this, we can decide if we want to add a work-around to make these functions cacheable.
Contributor Checklist:
changes were made to the GUI)