-
Notifications
You must be signed in to change notification settings - Fork 303
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ability to use dask distributed when generating animation videos #609
Merged
djhoese
merged 9 commits into
pytroll:master
from
djhoese:feature-distributed-animations
Feb 9, 2019
Merged
Add ability to use dask distributed when generating animation videos #609
djhoese
merged 9 commits into
pytroll:master
from
djhoese:feature-distributed-animations
Feb 9, 2019
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
stickler-ci
reviewed
Feb 6, 2019
Codecov Report
@@ Coverage Diff @@
## master #609 +/- ##
==========================================
- Coverage 78.14% 78.13% -0.01%
==========================================
Files 136 136
Lines 19555 19615 +60
==========================================
+ Hits 15281 15327 +46
- Misses 4274 4288 +14
Continue to review full report at Codecov.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is a rewrite of the animation saving logic in the MultiScene. It removes the "cascaded" dask graph that I was using before because I discovered this was not multiprocess safe (opening a new ffmpeg connection on every process/node). I now give the option to use distributed to upload frames to be computed and write them as they complete. My hope is that some users with higher performance systems (or clusters) can take advantage of their systems specs by specifying
batch_size
and computing multiple frames at a time.If a user does not have distributed installed then a basic one-frame-at-a-time computation will be done using normal dask computations.
git diff origin/master -- "*py" | flake8 --diff
I should note that this is documented in the function but not emphasized in the sphinx docs. I'll consider adding that soon.