[PyOV] Add torchvision to OpenVINO preprocessing converter

[p-wysocki]

Details:

• Tools for converting a torchvision.transforms object into OpenVINO preprocessing

• Based on @AlexKoff88's POC

• List of supported torchvision transforms:

  • Compose
  • ToTensor
  • ConvertImageDtype
  • Normalize
  • Resize
  • CenterCrop
  • Pad
  • Grayscale

• Bindings for PILLOW interpolation modes

• Example use case:

preprocess_pipeline = torchvision.transforms.Compose(
    [
        torchvision.transforms.Resize(256, interpolation=transforms.InterpolationMode.NEAREST),
        torchvision.transforms.CenterCrop((216, 218)),
        torchvision.transforms.Pad((2, 3, 4, 5), fill=3),
        torchvision.transforms.ToTensor(),
        torchvision.transforms.ConvertImageDtype(torch.float32),
        torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    ]
)

torch_model = SimpleConvnet(input_channels=3)

torch.onnx.export(torch_model, torch.randn(1, 3, 224, 224), "test_convnet.onnx", verbose=False, input_names=["input"], output_names=["output"])
core = Core()
ov_model = core.read_model(model="test_convnet.onnx")

test_input = np.random.randint(255, size=(260, 260, 3), dtype=np.uint16)
ov_model = PreprocessConverter.from_torchvision(
    model=ov_model, transform=preprocess_pipeline, input_example=Image.fromarray(test_input.astype("uint8"), "RGB")
)
ov_model = core.compile_model(ov_model, "CPU")
ov_input = np.expand_dims(test_input, axis=0)
output = ov_model.output(0)
ov_result = ov_model(ov_input)[output]

Tickets:

• 111306, 90100

TODO

• assert that the tests are running correctly in CI
• add test cases for torch.nn.Sequential object instead of torchvision.transforms.Compose torch.nn.Sequential does not work with PIL objects https://pytorch.org/vision/stable/transforms.html#transforms-scriptability
• add tests for PILLOW interpolation modes bindings

[rkazants]

Also, I think it makes sense to add it into PyTorch layer tests scope

[p-wysocki]

These tests do not match the definition of layer tests. If there's a better place we can put preprocessing tests I'll move them somewhere else, but I don't think they belong to openvino/tests/layer_tests.

[ilya-lavrenov]

The problem we have here is that different categories of tests (read as "folder with python code") require the same frameworks.
And it results to the fact we have requirements files, where we have a lot of frameworks like ONNX and PT. We already have a ticket (and PR) to move ONNX tests from python folder to folder closer to ONNX FE. Similar with PR tests - what are we testing here? Python API ? or integration with PT? I suppose the later one, so here I aligned with @rkazants that such tests are closer to PT tests, which allows to group tests by frameworks:

• (No frameworks) - OpenVINO Python API, which does not require heavy dependencies
• (PT) - PT Layer tests, PT decoder, preprocessing tests
• (ONNX) - ONNX tests
• ...

Alternatively, we need to create tox.ini files where developers can test only Python API or only PT preprocessing or only interaction of ONNX and OpenVINO. Currently, we have a mix of everything in one place - OpenVINO Python API, ONNX tests and newly introduced PT preprocessing, also we have different decoders in openvino.frontends package. And it's hard to test everything together.

[jiwaszki]

One remark.