Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

faiss document store should save it's configuration alongside the index #1366

Closed
mathislucka opened this issue Aug 23, 2021 · 3 comments
Closed
Assignees

Comments

@mathislucka
Copy link
Member

mathislucka commented Aug 23, 2021

Is your feature request related to a problem? Please describe.
As discussed in #1337 (comment) the FAISS document store should save it's configuration when the index is stored on disk. Otherwise, an end-user will have to store the config somewhere else or they'll have to remember the configuration. Also, the load method currently does not take any configuration options which means that indexes loaded from disk will always use the default configuration.

Describe the solution you'd like
When calling the save method a json (?) file is saved which contains the current configuration of the index. The load method will automatically look for a config-file which is located in the same directory as the index file. If a config file is missing, the index will be initialized with the default config (thus keeping backwards compatibility).

Describe alternatives you've considered
None

@julian-risch
Copy link
Member

Hi @mathislucka thank you for creating this issue based on the discussion in #1337 We will discuss internally whether we can tackle this issue in one of our next sprints. If you or somebody else are/is interested in contributing here, feel free to create a work-in-progress pull request. 🙂

@tholor
Copy link
Member

tholor commented Sep 15, 2021

One more thing we should tackle here: When loading the DocumentStore we should assert that the number of docs in SQL match the number of docs in FAISS

@ZanSara
Copy link
Contributor

ZanSara commented Sep 20, 2021

Hey @mathislucka, we improved the save/load API of FAISSDocumentStore to use a small configuration file in #1366. It also implements the check mentioned by @tholor. Check it out! I'm closing the issue now, please re-open if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants