Data Processor: Resolve several bugs found while publishing a Studio #19309

tchaton · 2024-01-18T10:37:46Z

What does this PR do?

This PR resolves some issues:

upstream the resolver from lightning-cloud to ease the development of its functionalities.
adds support for partial function so the published Studio is as clean as possible
tiny fixes there and there

Fixes #<issue_number>

Before submitting

Was this discussed/agreed via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

cc @Borda

gitguardian · 2024-01-18T10:37:53Z

⚠️ GitGuardian has uncovered 4 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	Secret	Commit	Filename
5685611	Generic High Entropy Secret	`c10c882`	tests/tests_data/streaming/test_resolver.py	View secret
5685611	Generic High Entropy Secret	`c10c882`	tests/tests_data/streaming/test_resolver.py	View secret
5685611	Generic High Entropy Secret	`c10c882`	tests/tests_data/streaming/test_resolver.py	View secret
5685611	Generic High Entropy Secret	`c10c882`	tests/tests_data/streaming/test_resolver.py	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secret safely. Learn here the best practices.
Revoke and rotate this secret.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!}

for more information, see https://pre-commit.ci

…lightning into some_fixes

for more information, see https://pre-commit.ci

…lightning into some_fixes

github-actions · 2024-01-18T16:01:40Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 lightning_data: CPU workflow

Check ID	Status
data-cpu (macOS-11, lightning, 3.10, 2.1)	success	✅
data-cpu (ubuntu-20.04, lightning, 3.10, 2.1)	success	✅
data-cpu (windows-2022, lightning, 3.10, 2.1)	success	✅

These checks are required after the changes to src/lightning/data/__init__.py, src/lightning/data/streaming/cache.py, src/lightning/data/streaming/combined.py, src/lightning/data/streaming/constants.py, src/lightning/data/streaming/data_processor.py, src/lightning/data/streaming/dataset.py, src/lightning/data/streaming/functions.py, src/lightning/data/streaming/resolver.py, tests/tests_data/streaming/test_data_processor.py, tests/tests_data/streaming/test_dataset.py, tests/tests_data/streaming/test_functions.py, tests/tests_data/streaming/test_reader.py, tests/tests_data/streaming/test_resolver.py.

🟢 lightning_app: Tests workflow

Check ID	Status
app-pytest (macOS-11, lightning, 3.8, latest)	success	✅
app-pytest (macOS-11, lightning, 3.8, oldest)	success	✅
app-pytest (macOS-11, app, 3.9, latest)	success	✅
app-pytest (macOS-12, app, 3.11, latest)	success	✅
app-pytest (ubuntu-20.04, lightning, 3.8, latest)	success	✅
app-pytest (ubuntu-20.04, lightning, 3.8, oldest)	success	✅
app-pytest (ubuntu-20.04, app, 3.9, latest)	success	✅
app-pytest (ubuntu-22.04, app, 3.11, latest)	success	✅
app-pytest (windows-2022, lightning, 3.8, latest)	success	✅
app-pytest (windows-2022, lightning, 3.8, oldest)	success	✅
app-pytest (windows-2022, app, 3.8, latest)	success	✅
app-pytest (windows-2022, app, 3.11, latest)	success	✅

These checks are required after the changes to requirements/app/app.txt.

🟢 lightning_app: Examples

Check ID	Status
app-examples (macOS-11, lightning, 3.9, latest)	success	✅
app-examples (macOS-11, lightning, 3.9, oldest)	success	✅
app-examples (macOS-11, app, 3.9, latest)	success	✅
app-examples (ubuntu-20.04, lightning, 3.9, latest)	success	✅
app-examples (ubuntu-20.04, lightning, 3.9, oldest)	success	✅
app-examples (ubuntu-20.04, app, 3.9, latest)	success	✅
app-examples (windows-2022, lightning, 3.9, latest)	success	✅
app-examples (windows-2022, lightning, 3.9, oldest)	success	✅
app-examples (windows-2022, app, 3.9, latest)	success	✅

These checks are required after the changes to requirements/app/app.txt.

🟢 lightning_app: Docs

Check ID	Status
docs-make (app, doctest)	success	✅
docs-make (app, html)	success	✅

These checks are required after the changes to requirements/app/app.txt.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to requirements/app/app.txt, src/lightning/data/__init__.py, src/lightning/data/streaming/cache.py, src/lightning/data/streaming/combined.py, src/lightning/data/streaming/constants.py, src/lightning/data/streaming/data_processor.py, src/lightning/data/streaming/dataset.py, src/lightning/data/streaming/functions.py, src/lightning/data/streaming/resolver.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.11)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.11)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.11)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.11)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.11)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.11)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.11)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.11)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.11)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.11)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.11)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.11)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.11)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.11)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.11)	success	✅

These checks are required after the changes to src/lightning/data/__init__.py, src/lightning/data/streaming/cache.py, src/lightning/data/streaming/combined.py, src/lightning/data/streaming/constants.py, src/lightning/data/streaming/data_processor.py, src/lightning/data/streaming/dataset.py, src/lightning/data/streaming/functions.py, src/lightning/data/streaming/resolver.py, requirements/app/app.txt.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

codecov · 2024-01-18T16:43:36Z

Codecov Report

Merging #19309 (3893894) into master (93c1ab0) will decrease coverage by 32%.
The diff coverage is 85%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #19309      +/-   ##
==========================================
- Coverage      84%      51%     -32%     
==========================================
  Files         445      440       -5     
  Lines       37332    37404      +72     
==========================================
- Hits        31182    19135   -12047     
- Misses       6150    18269   +12119

src/lightning/data/streaming/resolver.py

src/lightning/data/streaming/data_processor.py

src/lightning/data/streaming/functions.py

…19309)

thomas added 13 commits January 17, 2024 19:50

update

5170a68

update

c7cb17d

update

5bce9fe

update

0e4820f

update

da9ff24

update

b2c572e

update

f2c289e

update

b7821ac

update

da23815

update

c10c882

update

b3d818f

update

e95b00a

update

2a4a9e0

github-actions bot added app (removed) Generic label for Lightning App package dependencies Pull requests that update a dependency file data (external) litdata package labels Jan 18, 2024

tchaton changed the title ~~Data Processor: Some fixes to enable~~ Data Processor: Enable partial functions Jan 18, 2024

thomas and others added 12 commits January 18, 2024 10:44

update

a2445b9

update

31022eb

update

31106bc

update

9c87bf5

update

c64694f

update

2f55818

update

a53ce0f

[pre-commit.ci] auto fixes from pre-commit.com hooks

52d56d5

for more information, see https://pre-commit.ci

update

53c8a69

Merge branch 'some_fixes' of https://github.com/Lightning-AI/pytorch-…

01ec165

…lightning into some_fixes

update

b00f525

[pre-commit.ci] auto fixes from pre-commit.com hooks

c40bba6

for more information, see https://pre-commit.ci

pre-commit-ci bot and others added 9 commits January 18, 2024 14:48

[pre-commit.ci] auto fixes from pre-commit.com hooks

27d08b4

for more information, see https://pre-commit.ci

update

4caf578

Merge branch 'some_fixes' of https://github.com/Lightning-AI/pytorch-…

93ce91e

…lightning into some_fixes

update

ec16224

update

57bc1e9

update

7c0d590

update

d5b7fb6

update

77c75c2

update

ab93049

tchaton changed the title ~~Data Processor: Enable partial functions~~ Data Processor: Resolve several bugs found while publishing a Studio Jan 18, 2024

tchaton marked this pull request as ready for review January 18, 2024 16:01

tchaton requested review from lantiga, Borda and awaelchli as code owners January 18, 2024 16:01

update

356b55d

ethanwharris approved these changes Jan 18, 2024

View reviewed changes

justusschock approved these changes Jan 18, 2024

View reviewed changes

src/lightning/data/streaming/resolver.py Show resolved Hide resolved

src/lightning/data/streaming/resolver.py Outdated Show resolved Hide resolved

src/lightning/data/streaming/resolver.py Show resolved Hide resolved

src/lightning/data/streaming/resolver.py Show resolved Hide resolved

mergify bot added the ready PRs ready to be merged label Jan 18, 2024

awaelchli approved these changes Jan 18, 2024

View reviewed changes

src/lightning/data/streaming/data_processor.py Show resolved Hide resolved

src/lightning/data/streaming/data_processor.py Show resolved Hide resolved

src/lightning/data/streaming/functions.py Show resolved Hide resolved

thomas added 4 commits January 18, 2024 19:16

update

cf9ff92

update

760bf37

update

be4b0e6

update

3893894

tchaton merged commit 97d71ab into master Jan 18, 2024
88 checks passed

tchaton deleted the some_fixes branch January 18, 2024 20:46

awaelchli added this to the 2.1.x milestone Jan 30, 2024

awaelchli added a commit that referenced this pull request Jan 30, 2024

Data Processor: Resolve several bugs found while publishing a Studio (#…

9525827

…19309)

lexierule pushed a commit that referenced this pull request Jan 31, 2024

Data Processor: Resolve several bugs found while publishing a Studio (#…

25353e2

…19309)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Data Processor: Resolve several bugs found while publishing a Studio #19309

Data Processor: Resolve several bugs found while publishing a Studio #19309

tchaton commented Jan 18, 2024 •

edited by github-actions bot

Loading

gitguardian bot commented Jan 18, 2024 •

edited

Loading

github-actions bot commented Jan 18, 2024 •

edited

Loading

codecov bot commented Jan 18, 2024 •

edited

Loading

Data Processor: Resolve several bugs found while publishing a Studio #19309

Data Processor: Resolve several bugs found while publishing a Studio #19309

Conversation

tchaton commented Jan 18, 2024 • edited by github-actions bot Loading

What does this PR do?

PR review

gitguardian bot commented Jan 18, 2024 • edited Loading

⚠️ GitGuardian has uncovered 4 secrets following the scan of your pull request.

github-actions bot commented Jan 18, 2024 • edited Loading

⚡ Required checks status: All passing 🟢

Groups summary

codecov bot commented Jan 18, 2024 • edited Loading

Codecov Report

tchaton commented Jan 18, 2024 •

edited by github-actions bot

Loading

gitguardian bot commented Jan 18, 2024 •

edited

Loading

github-actions bot commented Jan 18, 2024 •

edited

Loading

codecov bot commented Jan 18, 2024 •

edited

Loading