Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

not all prototype transforms are serializable #6728

Open
pmeier opened this issue Oct 10, 2022 · 0 comments
Open

not all prototype transforms are serializable #6728

pmeier opened this issue Oct 10, 2022 · 0 comments

Comments

@pmeier
Copy link
Collaborator

pmeier commented Oct 10, 2022

This is important for two reasons:

  1. If we stick to torchdata for datasets v2, every object on the datapipe graph has to be serializable for multiprocessing. IIRC, this is only a requirement if forking from the main process is not possible, but that is the case on Windows.
  2. Our classification references have the ability to cache the dataset which includes the transformations:

utils.save_on_master((dataset, traindir), cache_path)

So far we don't have a unified test framework for our prototype transforms and thus it is not easy to add tests for all transforms. Note that we don't need tests for the functional kernels and dispatchers, since they are already tested for JIT scriptability and that protocol is a lot stricter than pickling.

cc @vfdev-5 @datumbox @bjuncek

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant