Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to use dask distributed when generating animation videos #609

Merged
merged 9 commits into from
Feb 9, 2019

Conversation

djhoese
Copy link
Member

@djhoese djhoese commented Feb 6, 2019

This PR is a rewrite of the animation saving logic in the MultiScene. It removes the "cascaded" dask graph that I was using before because I discovered this was not multiprocess safe (opening a new ffmpeg connection on every process/node). I now give the option to use distributed to upload frames to be computed and write them as they complete. My hope is that some users with higher performance systems (or clusters) can take advantage of their systems specs by specifying batch_size and computing multiple frames at a time.

If a user does not have distributed installed then a basic one-frame-at-a-time computation will be done using normal dask computations.

  • Tests added
  • Tests passed
  • Passes git diff origin/master -- "*py" | flake8 --diff
  • Fully documented

I should note that this is documented in the function but not emphasized in the sphinx docs. I'll consider adding that soon.

@djhoese djhoese added enhancement code enhancements, features, improvements optimization labels Feb 6, 2019
@djhoese djhoese self-assigned this Feb 6, 2019
@codecov
Copy link

codecov bot commented Feb 7, 2019

Codecov Report

Merging #609 into master will decrease coverage by <.01%.
The diff coverage is 82.22%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #609      +/-   ##
==========================================
- Coverage   78.14%   78.13%   -0.01%     
==========================================
  Files         136      136              
  Lines       19555    19615      +60     
==========================================
+ Hits        15281    15327      +46     
- Misses       4274     4288      +14
Impacted Files Coverage Δ
satpy/tests/test_multiscene.py 96.49% <100%> (+0.62%) ⬆️
satpy/multiscene.py 80.08% <74.19%> (-3.41%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7c264ee...24205cf. Read the comment docs.

@coveralls
Copy link

coveralls commented Feb 7, 2019

Coverage Status

Coverage decreased (-0.005%) to 78.139% when pulling 24205cf on djhoese:feature-distributed-animations into 7c264ee on pytroll:master.

@djhoese djhoese merged commit 71d215c into pytroll:master Feb 9, 2019
@djhoese djhoese deleted the feature-distributed-animations branch February 9, 2019 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement code enhancements, features, improvements optimization
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants