Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set limits for fetcher.done #18441

Merged
merged 85 commits into from
Sep 7, 2023
Merged
Changes from 1 commit
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
4cd141d
wip
awaelchli Aug 30, 2023
4f92b94
wip
awaelchli Aug 30, 2023
06fcccc
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 30, 2023
67aac4d
fix
awaelchli Aug 30, 2023
49ce20f
fixes
awaelchli Aug 30, 2023
36a00c1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 30, 2023
817eb18
update
awaelchli Aug 30, 2023
3083632
fix
awaelchli Aug 30, 2023
e7cb210
implement len for other modes
awaelchli Aug 30, 2023
c4d3961
fix
awaelchli Aug 31, 2023
7a35014
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 31, 2023
8cf3513
set limits in training loop
awaelchli Aug 31, 2023
4f50461
Merge remote-tracking branch 'origin/dataloader-iter/via-loader-lengt…
awaelchli Aug 31, 2023
fa53bcd
convert the limits
awaelchli Aug 31, 2023
5a62929
Merge branch 'master' into dataloader-iter/via-loader-length
awaelchli Aug 31, 2023
eaa2be6
None check
awaelchli Aug 31, 2023
2fd6783
fix passing limits
awaelchli Aug 31, 2023
63e58e8
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 31, 2023
36d3a3a
fix test
awaelchli Aug 31, 2023
c339972
Merge remote-tracking branch 'origin/dataloader-iter/via-loader-lengt…
awaelchli Aug 31, 2023
a1bbe0c
construction site
awaelchli Sep 2, 2023
f3f8418
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 2, 2023
5c69255
fixes
awaelchli Sep 2, 2023
5779edc
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 2, 2023
281f6db
update combined loader test
awaelchli Sep 4, 2023
337c47b
rage commit a;dfja;efj awoiefj aiowpefjaweoifj apweiosfj pwaeofjawepo…
awaelchli Sep 4, 2023
37c104d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 4, 2023
8c77e59
rage
awaelchli Sep 4, 2023
c10e609
fix
awaelchli Sep 4, 2023
457a395
magic mock
awaelchli Sep 4, 2023
2957fd8
update error check
awaelchli Sep 4, 2023
3f8e2e1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 4, 2023
2705bdc
clean up
awaelchli Sep 4, 2023
c3304ef
Merge remote-tracking branch 'origin/dataloader-iter/via-loader-lengt…
awaelchli Sep 4, 2023
a0cd393
simplify len calculation
awaelchli Sep 4, 2023
34b95da
update
awaelchli Sep 4, 2023
5dfa069
doctests
awaelchli Sep 4, 2023
7870fef
update
awaelchli Sep 4, 2023
ec9a62f
update
awaelchli Sep 4, 2023
57eabba
fix
awaelchli Sep 4, 2023
cf2d4ed
fix test
awaelchli Sep 4, 2023
8335505
clean up
awaelchli Sep 4, 2023
60b8dd4
refactor
awaelchli Sep 4, 2023
e97380f
mypy
awaelchli Sep 4, 2023
683ca49
fix test
awaelchli Sep 4, 2023
f3055bb
generalize test
awaelchli Sep 4, 2023
a4636e3
limits input validation in mode iterator
awaelchli Sep 4, 2023
9db298a
extend test
awaelchli Sep 4, 2023
b522c90
extend test
awaelchli Sep 4, 2023
f0e3e8e
update
awaelchli Sep 4, 2023
95f05f9
test edge cases when limit is 0 or iterable is empty
awaelchli Sep 4, 2023
9abc96c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 4, 2023
a1a1942
update test
awaelchli Sep 4, 2023
66863b1
annoying mypy
awaelchli Sep 4, 2023
38a03f5
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 4, 2023
a5db7a4
update
awaelchli Sep 4, 2023
408b1e4
Update src/lightning/pytorch/utilities/combined_loader.py
awaelchli Sep 5, 2023
16f817a
update type of max_batches and add changelog
awaelchli Sep 5, 2023
eb825bd
review
awaelchli Sep 5, 2023
36ebb8e
try without setting trainer.training=True
awaelchli Sep 5, 2023
eb67256
review
awaelchli Sep 5, 2023
ffe1f15
determine is_last_batch in case of dataloader_iter
awaelchli Sep 5, 2023
1d460b9
add test
awaelchli Sep 5, 2023
52c4041
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 5, 2023
dace1f8
allow limits larger than the iterable sizes
awaelchli Sep 5, 2023
85dbf22
Merge remote-tracking branch 'origin/dataloader-iter/via-loader-lengt…
awaelchli Sep 5, 2023
4dd975e
reset bug_report_model.py
awaelchli Sep 5, 2023
9d443b9
mypy
awaelchli Sep 5, 2023
dfbf8ef
todo
awaelchli Sep 5, 2023
8eb80c8
clean up
awaelchli Sep 5, 2023
2d97dbb
resolve todo. it is needed, lots of test fail otherwise
awaelchli Sep 5, 2023
157ec6d
raise StopIteration when length is reached
awaelchli Sep 5, 2023
98a13c4
fix test
awaelchli Sep 5, 2023
dd56105
Merge branch 'master' into dataloader-iter/via-loader-length
carmocca Sep 5, 2023
9159a31
Fix bad merge
carmocca Sep 5, 2023
7361f95
raise error if iter() not called before len()
awaelchli Sep 6, 2023
64b9deb
simplify len computation
awaelchli Sep 6, 2023
af7dcc7
move tests to test_fetchers.py
awaelchli Sep 6, 2023
94a8be7
move the special stopping condition to the _DataFetcherWrapper
awaelchli Sep 6, 2023
72ce38d
revert the sum(limits) = 0 change
awaelchli Sep 6, 2023
5627bf4
fix test for on_batch_start indices with dataloader_iter
awaelchli Sep 6, 2023
b63bcb1
try to skip additional iter() call in first epoch
awaelchli Sep 6, 2023
ccdc469
remove a comment
awaelchli Sep 6, 2023
190c480
check if data_fetcher's iterator exists
awaelchli Sep 6, 2023
aa2b2b7
update guard for iter() decision
awaelchli Sep 6, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
try without setting trainer.training=True
  • Loading branch information
awaelchli committed Sep 5, 2023
commit 36ebb8e8a9a48696d5fc6a7a47d2aadf55e6e230
3 changes: 0 additions & 3 deletions src/lightning/pytorch/loops/fit_loop.py
Original file line number Diff line number Diff line change
Expand Up @@ -253,10 +253,7 @@ def setup_data(self) -> None:

combined_loader.limits = limits

trainer.training = True
self._data_fetcher = _select_data_fetcher(trainer)
trainer.training = False

self._data_fetcher.setup(combined_loader)
iter(self._data_fetcher) # creates the iterator inside the fetcher
max_batches = sized_len(combined_loader)
Expand Down