EpochIterator, TensorFlowTrainer : refactoring and fixes
#20396
Conversation
Tensorflow trainer's train and test functions refactoring Fix keras-team#20394 Fix keras-team#20390 Fix keras-team#20344
keras/src/trainers/epoch_iterator.py
Outdated
| @property | ||
| def num_batches(self): | ||
| if self.steps_per_epoch: | ||
| return self.steps_per_epoch | ||
| # Either copied from the data_adapter, or | ||
| # inferred at the end of an iteration. | ||
| return self._num_batches | ||
|
|
||
|
|
||
| class EpochIterator(_EpochIterator): |
There was a problem hiding this comment.
Couldn't the two classes be merged into one?
There was a problem hiding this comment.
Yes, I merged them.
keras/src/trainers/epoch_iterator.py
Outdated
| class EpochIterator(_EpochIterator): | ||
| def __next__(self): | ||
| buffer = [] | ||
| step, iterator = super().__next__() |
There was a problem hiding this comment.
Just need to call next(self._epoch_iterator)
| @@ -75,55 +75,86 @@ def __init__( | |||
| def _get_iterator(self): | |||
| return self.data_adapter.get_numpy_iterator() | |||
|
|
|||
| def enumerate_epoch(self): | |||
There was a problem hiding this comment.
This method must be kept for backwards compatibility (for users who have a custom fit())
|
It's fairly confusing to have backend-specific iterators subclass |
Restored `enumerate_epoch`
I removed |
google-ml-butler
bot
added
kokoro:force-run
ready to pull
google-ml-butler
bot
removed
awaiting review
ready to pull
Fixes #20394
Fixes #20390
Fixes #20344
TensorFlowTrainer:iterator.get_next_as_optional(), which enablessteps_per_executionto exceed the previous limit of 32 while preventingtf.errors.OutOfRangeErrorEpochIterator:steps_per_epochandsteps_per_execution