Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add distributed support for StreamingDataset #18850

Merged
merged 65 commits into from
Oct 25, 2023
Merged

Add distributed support for StreamingDataset #18850

merged 65 commits into from
Oct 25, 2023

Conversation

tchaton
Copy link
Contributor

@tchaton tchaton commented Oct 24, 2023 •
edited by github-actions bot
Loading

What does this PR do?

This PR adds distributed support for the StreamingDataset with several shuffling strategy.

Fixes #<issue_number>

Before submitting
  • Was this discussed/agreed via a GitHub issue? (not for typos and docs)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests? (not for typos and docs)
  • Did you verify new and existing tests pass locally with your changes?
  • Did you list all the breaking changes introduced by this pull request?
  • Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist
  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified

📚 Documentation preview 📚: https://pytorch-lightning--18850.org.readthedocs.build/en/18850/

cc @Borda

tchaton and others added 30 commits October 18, 2023 00:24
@tchaton
update
44a942c
@tchaton
update
f95e9f7
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
593edef 
for more information, see https://pre-commit.ci
@tchaton
update
4ba6ecb
@tchaton
Merge branch 'fix_download' of https://github.com/Lightning-AI/lightning
c2de5e7 
 into fix_download
@tchaton
update
1d296c9
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
6ac552a 
for more information, see https://pre-commit.ci
@tchaton
update
5eff54a
@tchaton
Merge branch 'fix_download' of https://github.com/Lightning-AI/lightning
ece8e25 
 into fix_download
@tchaton
update
3a186d7
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
d8e8630 
for more information, see https://pre-commit.ci
update
fe36a09
update
a3c7aa9
update
4bcb86b
update
047d92a
update
e9acb32
update
8e93747
update
9f5009b
update
5918c54
update
962d8b3
Merge branch 'master' into fix_download
3569257
update
2d4ecc8
update
9f91479
update
502bae7
@tchaton
update
b9ee9f4
@tchaton
update
670c0bb
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
9d15107 
for more information, see https://pre-commit.ci
@tchaton
update
5d949b6
@tchaton
Merge branch 'fix_download' of https://github.com/Lightning-AI/lightning
1988c38 
 into fix_download
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
8acd994 
for more information, see https://pre-commit.ci
@github-actions github-actions bot added the dependencies Pull requests that update a dependency file label Oct 24, 2023
@github-actions github-actions bot removed app (removed) Generic label for Lightning App package dependencies Pull requests that update a dependency file labels Oct 24, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Oct 24, 2023 •
edited
Loading

⚡ Required checks status: All passing 🟢

Groups summary

🟢 lightning_data: CPU workflow
Check ID Status
data-cpu (macOS-11, lightning, 3.10, 2.1) success ✅
data-cpu (ubuntu-20.04, lightning, 3.10, 2.1) success ✅
data-cpu (windows-2022, lightning, 3.10, 2.1) success ✅

These checks are required after the changes to src/lightning/data/streaming/cache.py, src/lightning/data/streaming/dataset.py, src/lightning/data/streaming/dataset_optimizer.py, src/lightning/data/streaming/item_loader.py, src/lightning/data/streaming/reader.py, src/lightning/data/streaming/sampler.py, src/lightning/data/streaming/shuffle.py, tests/tests_data/streaming/test_cache.py, tests/tests_data/streaming/test_dataset.py, tests/tests_data/streaming/test_sampler.py.

🟢 mypy
Check ID Status
mypy success ✅

These checks are required after the changes to src/lightning/data/streaming/cache.py, src/lightning/data/streaming/dataset.py, src/lightning/data/streaming/dataset_optimizer.py, src/lightning/data/streaming/item_loader.py, src/lightning/data/streaming/reader.py, src/lightning/data/streaming/sampler.py, src/lightning/data/streaming/shuffle.py.

🟢 install
Check ID Status
install-pkg (ubuntu-22.04, app, 3.8) success ✅
install-pkg (ubuntu-22.04, app, 3.11) success ✅
install-pkg (ubuntu-22.04, fabric, 3.8) success ✅
install-pkg (ubuntu-22.04, fabric, 3.11) success ✅
install-pkg (ubuntu-22.04, pytorch, 3.8) success ✅
install-pkg (ubuntu-22.04, pytorch, 3.11) success ✅
install-pkg (ubuntu-22.04, lightning, 3.8) success ✅
install-pkg (ubuntu-22.04, lightning, 3.11) success ✅
install-pkg (ubuntu-22.04, notset, 3.8) success ✅
install-pkg (ubuntu-22.04, notset, 3.11) success ✅
install-pkg (macOS-12, app, 3.8) success ✅
install-pkg (macOS-12, app, 3.11) success ✅
install-pkg (macOS-12, fabric, 3.8) success ✅
install-pkg (macOS-12, fabric, 3.11) success ✅
install-pkg (macOS-12, pytorch, 3.8) success ✅
install-pkg (macOS-12, pytorch, 3.11) success ✅
install-pkg (macOS-12, lightning, 3.8) success ✅
install-pkg (macOS-12, lightning, 3.11) success ✅
install-pkg (macOS-12, notset, 3.8) success ✅
install-pkg (macOS-12, notset, 3.11) success ✅
install-pkg (windows-2022, app, 3.8) success ✅
install-pkg (windows-2022, app, 3.11) success ✅
install-pkg (windows-2022, fabric, 3.8) success ✅
install-pkg (windows-2022, fabric, 3.11) success ✅
install-pkg (windows-2022, pytorch, 3.8) success ✅
install-pkg (windows-2022, pytorch, 3.11) success ✅
install-pkg (windows-2022, lightning, 3.8) success ✅
install-pkg (windows-2022, lightning, 3.11) success ✅
install-pkg (windows-2022, notset, 3.8) success ✅
install-pkg (windows-2022, notset, 3.11) success ✅

These checks are required after the changes to src/lightning/data/streaming/cache.py, src/lightning/data/streaming/dataset.py, src/lightning/data/streaming/dataset_optimizer.py, src/lightning/data/streaming/item_loader.py, src/lightning/data/streaming/reader.py, src/lightning/data/streaming/sampler.py, src/lightning/data/streaming/shuffle.py.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

@mergify mergify bot removed the has conflicts label Oct 24, 2023
@tchaton tchaton changed the title Add distributed support Add distributed support for StreamingDataset Oct 24, 2023
lantiga
lantiga approved these changes Oct 24, 2023
View reviewed changes
Copy link
Collaborator

@lantiga lantiga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved for expediting, but comments need to be addressed (here on in a subsequent PR)

src/lightning/data/streaming/dataset.py Show resolved Hide resolved
src/lightning/data/streaming/shuffle.py Outdated Show resolved Hide resolved
src/lightning/data/streaming/shuffle.py Outdated Show resolved Hide resolved
src/lightning/data/streaming/shuffle.py Outdated Show resolved Hide resolved
tests/tests_data/streaming/test_dataset.py Show resolved Hide resolved
tests/tests_data/streaming/test_dataset.py Outdated Show resolved Hide resolved
@mergify mergify bot added the ready PRs ready to be merged label Oct 24, 2023
lantiga
lantiga approved these changes Oct 24, 2023
View reviewed changes
Copy link
Collaborator

@lantiga lantiga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

src/lightning/data/streaming/shuffle.py Outdated Show resolved Hide resolved
src/lightning/data/streaming/shuffle.py Outdated Show resolved Hide resolved
src/lightning/data/streaming/shuffle.py Outdated Show resolved Hide resolved
src/lightning/data/streaming/shuffle.py Outdated Show resolved Hide resolved
@tchaton tchaton merged commit 8748258 into master Oct 25, 2023
@tchaton tchaton deleted the add_distributed_support branch October 25, 2023 02:18
tsenst pushed a commit to tsenst/lightning that referenced this pull request Oct 26, 2023
@tchaton @lantiga
@pre-commit-ci @Borda @tsenst
Add distributed support for StreamingDataset (Lightning-AI#18850)
a2269a4 
Co-authored-by: Luca Antiga <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: thomas <[email protected]>
Co-authored-by: Jirka Borovec <[email protected]>
Borda pushed a commit that referenced this pull request Nov 2, 2023
@tchaton @Borda
Add distributed support for StreamingDataset (#18850)
6a0ca88 
Co-authored-by: Luca Antiga <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: thomas <[email protected]>
Co-authored-by: Jirka Borovec <[email protected]>
(cherry picked from commit 8748258)
Borda pushed a commit that referenced this pull request Nov 2, 2023
@tchaton @Borda
Add distributed support for StreamingDataset (#18850)
7044ba6 
Co-authored-by: Luca Antiga <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: thomas <[email protected]>
Co-authored-by: Jirka Borovec <[email protected]>
(cherry picked from commit 8748258)
@awaelchli awaelchli added this to the 2.1.x milestone Nov 3, 2023
lantiga pushed a commit that referenced this pull request Nov 6, 2023
@tchaton @lantiga
Add distributed support for StreamingDataset (#18850)
ce82dc0 
Co-authored-by: Luca Antiga <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: thomas <[email protected]>
Co-authored-by: Jirka Borovec <[email protected]>
(cherry picked from commit 8748258)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mestir mestir mestir left review comments

@lantiga lantiga lantiga approved these changes

@awaelchli awaelchli awaelchli approved these changes

@ethanwharris ethanwharris Awaiting requested review from ethanwharris ethanwharris is a code owner

@nohalon nohalon Awaiting requested review from nohalon

@justusschock justusschock Awaiting requested review from justusschock justusschock is a code owner

@Borda Borda Awaiting requested review from Borda Borda is a code owner

Assignees
No one assigned
Labels
ready PRs ready to be merged
Projects
None yet
Milestone
2.1.x
Development

Successfully merging this pull request may close these issues.
5 participants
@tchaton @lantiga @awaelchli @mestir @Borda