Check that required arguments for a given database are specified

[daverigby]

Different databases require different arguments - for example:

• pinecone requires that --pinecone_api_key and --pinecone_index_name are specified.
• pgvector requires that --pgvector_username and --pgvector_password are specified.

Currently this is not validated as part of parsing the command-line options, and will only be detected later on when we attempt to initialise the given database, which isn't necessarily clear to the user. We should add explicit checking so the user gets a message as soon as possible.

[daverigby]

Possible solutions as suggested by Chat-GPT:

To implement arguments which depend on other arguments using Python's argparse library, you can create custom validation logic after parsing the arguments. This validation logic will check the values of specific arguments and enforce the dependencies accordingly. Here’s how you can do it:

1. Define all the arguments, including those that may be dependent on the values of other arguments.
2. Parse the arguments using argparse.
3. Implement a validation function to check the dependencies and raise an error if the dependencies are not met.

Here’s a complete example demonstrating this approach:

import argparse
import sys

def validate_args(args):
    if args.database == "pinecone":
        if not args.pinecone_api_key:
            sys.exit("Error: --pinecone_api_key must be specified when --database is 'pinecone'.")
    elif args.database == "pgvector":
        if not args.pgvector_user or not args.pgvector_password:
            sys.exit("Error: --pgvector_user and --pgvector_password must be specified when --database is 'pgvector'.")
    else:
        sys.exit("Error: Unsupported database specified.")

def main():
    parser = argparse.ArgumentParser(description="Database argument parser example.")
    
    parser.add_argument('--database', choices=['pinecone', 'pgvector'], required=True, help="Specify the database to use.")
    parser.add_argument('--pinecone_api_key', help="API key for Pinecone database.")
    parser.add_argument('--pgvector_user', help="User for pgvector database.")
    parser.add_argument('--pgvector_password', help="Password for pgvector database.")
    
    args = parser.parse_args()
    
    validate_args(args)
    
    # Continue with the rest of your program logic
    print(f"Using database: {args.database}")
    if args.database == "pinecone":
        print(f"Pinecone API Key: {args.pinecone_api_key}")
    elif args.database == "pgvector":
        print(f"pgvector User: {args.pgvector_user}")
        print(f"pgvector Password: {args.pgvector_password}")

if __name__ == "__main__":
    main()

Explanation

1. Argument Definitions:

   • --database is a required argument that can have the values pinecone or pgvector.
   • --pinecone_api_key is an optional argument specific to the pinecone database.
   • --pgvector_user and --pgvector_password are optional arguments specific to the pgvector database.

2. Parsing Arguments:

   • args = parser.parse_args() parses the command-line arguments.

3. Validation Function:

   • validate_args(args) checks the value of --database and ensures that the necessary dependent arguments are provided.
   • If the conditions are not met, sys.exit is called with an error message, which terminates the program.

4. Usage Example:

   • If the user runs the script with --database pinecone, they must also provide --pinecone_api_key.
   • If the user runs the script with --database pgvector, they must provide both --pgvector_user and --pgvector_password.

This approach allows you to handle complex argument dependencies and provide clear error messages to the user if the required arguments are not specified correctly.

Q: Does Python have any libraries which can simplify implementing this?

Yes, Python has several libraries that can simplify the implementation of argument dependencies, such as argparse-helpers and voluptuous. These libraries provide higher-level abstractions for defining and validating command-line arguments. Below is an example using the argparse-helpers library.

Using argparse-helpers

The argparse-helpers library provides convenient utilities to simplify argument parsing and validation. You can install it via pip:

pip install argparse-helpers

Here’s an example that demonstrates how to use argparse-helpers to enforce argument dependencies:

import argparse
from argparse_helpers import conditional_required, group

def main():
    parser = argparse.ArgumentParser(description="Database argument parser example.")
    
    # Define the main argument group
    with group(parser, required=True) as g:
        g.add_argument('--database', choices=['pinecone', 'pgvector'], help="Specify the database to use.")
    
    # Define conditional required arguments based on the --database value
    conditional_required(parser, '--pinecone_api_key', when={'database': 'pinecone'}, help="API key for Pinecone database.")
    conditional_required(parser, '--pgvector_user', when={'database': 'pgvector'}, help="User for pgvector database.")
    conditional_required(parser, '--pgvector_password', when={'database': 'pgvector'}, help="Password for pgvector database.")

    args = parser.parse_args()
    
    # Continue with the rest of your program logic
    print(f"Using database: {args.database}")
    if args.database == "pinecone":
        print(f"Pinecone API Key: {args.pinecone_api_key}")
    elif args.database == "pgvector":
        print(f"pgvector User: {args.pgvector_user}")
        print(f"pgvector Password: {args.pgvector_password}")

if __name__ == "__main__":
    main()

Explanation

1. Import Libraries:

   • argparse is used for basic argument parsing.
   • argparse-helpers provides the conditional_required function to simplify conditional argument requirements.

2. Define Main Argument Group:

   • The group context manager is used to define a required group of arguments. In this case, only --database is required.

3. Define Conditional Arguments:

   • conditional_required is used to specify that certain arguments are required based on the value of --database.
   • when={'database': 'pinecone'} indicates that --pinecone_api_key is required if --database is pinecone.
   • Similarly, when={'database': 'pgvector'} indicates that --pgvector_user and --pgvector_password are required if --database is pgvector.

4. Parsing Arguments:

   • args = parser.parse_args() parses the command-line arguments and enforces the conditional requirements.

5. Program Logic:

   • The program logic follows, using the parsed arguments.

Other Libraries

voluptuous

voluptuous is a library for data validation and can also be used to validate command-line arguments after they are parsed by argparse.

Example Using voluptuous:

import argparse
import sys
from voluptuous import Schema, Required, Invalid

def validate_args(args):
    schema = Schema({
        Required('database'): str,
        'pinecone_api_key': str,
        'pgvector_user': str,
        'pgvector_password': str,
    })

    try:
        validated = schema(dict(args._get_kwargs()))
    except Invalid as e:
        sys.exit(f"Validation error: {e}")

    if validated['database'] == 'pinecone':
        if not validated.get('pinecone_api_key'):
            sys.exit("Error: --pinecone_api_key must be specified when --database is 'pinecone'.")
    elif validated['database'] == 'pgvector':
        if not validated.get('pgvector_user') or not validated.get('pgvector_password'):
            sys.exit("Error: --pgvector_user and --pgvector_password must be specified when --database is 'pgvector'.")
    else:
        sys.exit("Error: Unsupported database specified.")

def main():
    parser = argparse.ArgumentParser(description="Database argument parser example.")
    
    parser.add_argument('--database', choices=['pinecone', 'pgvector'], required=True, help="Specify the database to use.")
    parser.add_argument('--pinecone_api_key', help="API key for Pinecone database.")
    parser.add_argument('--pgvector_user', help="User for pgvector database.")
    parser.add_argument('--pgvector_password', help="Password for pgvector database.")
    
    args = parser.parse_args()
    
    validate_args(args)
    
    # Continue with the rest of your program logic
    print(f"Using database: {args.database}")
    if args.database == "pinecone":
        print(f"Pinecone API Key: {args.pinecone_api_key}")
    elif args.database == "pgvector":
        print(f"pgvector User: {args.pgvector_user}")
        print(f"pgvector Password: {args.pgvector_password}")

if __name__ == "__main__":
    main()

Conclusion

Using libraries like argparse-helpers or voluptuous can simplify the implementation of argument dependencies in Python, making your code cleaner and easier to maintain. These libraries provide higher-level abstractions for defining and validating command-line arguments, reducing boilerplate and potential errors.

[daverigby]

While ChatGPT's suggestion of argparse-helpers looks promising, I cannot find any mention of such module - and indeed pip doesn't have it as an available module. Not sure where ChatGPT got this from...