SQL or in-memory document stores for dense retriever (DPR)

[venuraja79]

Question
Hello team - Thanks a lot for this repository. please suggest if it's feasible to extend sql/ in-memory document stores to work with DPR retriever?

Additional context
The update_embeddings method doesn't seem to be implemented. If it's of any value, please suggest. This can be useful for easier/small size deployments. Thank you!

[tanaysoni]

Hi @venuraja79, you can use the DPR Retriever with a FAISS, InMemory, or Elasticsearch document store.

I would suggest using the FAISS document store for DPR. It uses SQL under-the-hood to store the documents and the document embeddings are indexed in a FAISS Index.

The update_embeddings() method is implemented for FAISS and Elasticsearch document stores.

To get started, you can follow the tutorial 6(colab version).

[venuraja79]

Thank you @tanaysoni, this helps! Would you be interested if I make a pull request if I implemented it for in-memory store?

[tanaysoni]

That'd be highly appreciated, @venuraja79! I'd suggest making an early work-in-progress pull request. Let me know if there's anything you need to discuss before getting started.

[venuraja79]

Sure, Thank you!

[venuraja79]

Thank you @tanaysoni. I'll close this thread.