[LibTorch] Expected Tensor but got None with inception v3

[kbegiedza]

Description

When I run simple grpc demo client with pyTorch model loaded I get an error.

[StatusCode.INTERNAL] PyTorch execute failure: isTensor() INTERNAL ASSERT FAILED at "/opt/tritonserver/include/torch/ATen/core/ivalue_inl.h":137, please report a bug to PyTorch. Expected Tensor but got None
Exception raised from toTensor at /opt/tritonserver/include/torch/ATen/core/ivalue_inl.h:137 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6c (0x7f5c183ea6cc in /opt/tritonserver/backends/pytorch/libc10.so)

Triton Information
nvidia/tritonserver:20.12-py3

To Reproduce
Export pre-trained model:

model = torch.hub.load('pytorch/vision:v0.6.0', 'inception_v3', pretrained=True)
model.eval()

script = torch.jit.script(model)
script.save('/output/inception_v3.pt')

Add to triton with config:
config.pbtxt:

name: "inception_v3"
instance_group [
  {
    count: 1
    kind: KIND_GPU
  }
]
input {
    name: "INPUT__0"
    data_type: TYPE_FP32
    dims: -1
    dims: 3
    dims: 299
    dims: 299
}
output {
    name: "OUTPUT__0"
    data_type: TYPE_FP32
    dims: -1
    dims: 1000
}

inputs = []
outputs = []
inputs.append(grpcclient.InferInput('INPUT__0', [1, 3, 299, 299], "FP32"))

input0_data = np.random.rand(1, 3, 299, 299)
input0_data_fp32 = input0_data.astype(np.float32)

inputs[0].set_data_from_numpy(input0_data_fp32)

outputs.append(grpcclient.InferRequestedOutput('OUTPUT__0'))

[deadeyegoodwin]

Are you able to run your torchscript model using the pytorch python API with the same input data?

[kbegiedza]

@deadeyegoodwin With pyTorch API with python I can easily run this without any errors.

Code:

import torch
import numpy as np
import torchvision

model = torch.hub.load('pytorch/vision:v0.6.0',
                       'inception_v3', pretrained=True)
model.eval()

input0_data = np.random.rand(1, 3, 299, 299)
input0_data_fp32 = input0_data.astype(np.float32)
input_data = torch.from_numpy(input0_data_fp32)

out = model.forward(input_data)

print('Model {0}'.format(out.size()))

ts_model = torch.jit.script(model)
ts_model.eval()

out = ts_model.forward(input_data)

print('JIT Model {0}'.format(out[0].size()))

Outputs:

Model torch.Size([1, 1000])
JIT Model torch.Size([1, 1000])

[CoderHam]

@Ursanon If you look at the server logs you will see:
[W inception.py:35] Warning: Scripted Inception3 always returns Inception3 Tuple (function )

Source looks akin to https://github.com/pytorch/vision/blob/master/torchvision/models/inception.py#L208

Note how there is a condition for scripting that causes this. I would recommend tracing the model instead. 

Output from scripted model: ts_model(input_data)

InceptionOutputs(logits=tensor([[-8.8582e-01, 5.6576e-01, 1.1851e+00, 3.5941e-01, 7.1092e-01,
-4.8846e-01, 2.0933e-01, 1.6255e+00, 1.1349e+00, 1.1491e+00,
1.2937e+00, 2.3032e+00, 2.9147e+00, 1.5196e+00, 1.7663e+00,
1.9466e+00, 2.5229e+00, 9.3673e-01, 2.4246e+00, 1.5150e+00,
5.7333e-01, 5.6608e+00, 3.8614e+00, 3.0505e+00, 2.4129e+00,
...
grad_fn=<AddBackward0>), aux_logits=None)

Output from pytorch native model: model(input_data)

tensor([[-8.8582e-01, 5.6576e-01, 1.1851e+00, 3.5941e-01, 7.1092e-01,
-4.8846e-01, 2.0933e-01, 1.6255e+00, 1.1349e+00, 1.1491e+00,
1.2937e+00, 2.3032e+00, 2.9147e+00, 1.5196e+00, 1.7663e+00,
1.9466e+00, 2.5229e+00, 9.3673e-01, 2.4246e+00, 1.5150e+00,
5.7333e-01, 5.6608e+00, 3.8614e+00, 3.0505e+00, 2.4129e+00,
...
grad_fn=<AddmmBackward>)

[CoderHam]

@Ursanon I tested tracing instead of scripting and it worked fine.

traced_model = torch.jit.trace(model, (input_data, ))
traced_model.eval()

traced_model(input_data)
traced_model.save('inception_v3.pt')

I replaced the inception_v3.pt file and the models runs in Tritonserver as expected.

[kbegiedza]

Thanks for your work @CoderHam, will check tracing.