Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add per-database configuration parameters #10

Closed
daverigby opened this issue Apr 29, 2024 · 2 comments
Closed

Add per-database configuration parameters #10

daverigby opened this issue Apr 29, 2024 · 2 comments
Labels
enhancement New feature or request

Comments

@daverigby
Copy link
Collaborator

Different databases require different configuration parameters, of different types:

  • Mandatory parameters - e.g. username/password for authentication or api_key, hostname or index name to identify index
  • Optional / variable parameters - e.g. pgvector's HNSW index takes M and ef_construction to control index speed vs recall tradeoff.

We need to pass these options through when that database is selected, in a scalable manner (we don't really want to show the union of all options across all databases). We also want to set sensible default values for them - e.g. pgvector should default to settings to connect to the local Docker container.

This also has some impact in how results are collected and reported - if we want to perform benchmark runs with a range of values for ef_construction for example, how should those results be reported?

@daverigby daverigby added the enhancement New feature or request label Apr 30, 2024
@daverigby
Copy link
Collaborator Author

Pydantic has a related package - https://github.com/pydantic/pydantic-settings - which handles much of this and integrates with Pydantic's models (which we already use in VSB). However it doesn't have support for overriding on the command line, which is a significant omission IMO.

There is an open Issue and linked PR to add command-line support - see pydantic/pydantic-settings#209 - but as of writing is not yet merged...

@daverigby
Copy link
Collaborator Author

This has mostly been implemented via #82 - each database has it's own arguments in an option group, and when specifying a given database, the arguments required for it are checked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant