You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If we stick to torchdata for datasets v2, every object on the datapipe graph has to be serializable for multiprocessing. IIRC, this is only a requirement if forking from the main process is not possible, but that is the case on Windows.
Our classification references have the ability to cache the dataset which includes the transformations:
So far we don't have a unified test framework for our prototype transforms and thus it is not easy to add tests for all transforms. Note that we don't need tests for the functional kernels and dispatchers, since they are already tested for JIT scriptability and that protocol is a lot stricter than pickling.
This is important for two reasons:
torchdata
for datasets v2, every object on the datapipe graph has to be serializable for multiprocessing. IIRC, this is only a requirement if forking from the main process is not possible, but that is the case on Windows.vision/references/video_classification/train.py
Line 189 in 6e203b4
So far we don't have a unified test framework for our prototype transforms and thus it is not easy to add tests for all transforms. Note that we don't need tests for the functional kernels and dispatchers, since they are already tested for JIT scriptability and that protocol is a lot stricter than pickling.
cc @vfdev-5 @datumbox @bjuncek
The text was updated successfully, but these errors were encountered: