Using ONNX for inference in haystack

[laifuchicago]

To Author:
After I converted my model to onnx, how can I use haystack Elasticsearch to do the inference? Or currently haystack does not support?

Jonathan Sung

[tholor]

Hey @laifuchicago ,

not yet, but we are on it :)
Should be available within the next sprint.

[tholor]

Depends on deepset-ai/FARM#539

[tanaysoni]

Hi @laifuchicago, with #438, you can use convert & use ONNX models for BERT, RoBERTa & XLM-RoBERTa.

Here's a sample code snippet to convert RoBERTa to ONNX format and create a FARMReader instance:

from haystack.reader.farm import FARMReader
from pathlib import Path

onnx_model_path = Path("roberta-onnx-model")
FARMReader.convert_to_onnx(model_name="deepset/roberta-base-squad2", output_path=onnx_model_path)

reader = FARMReader(onnx_model_path)