dir-assistant

Chat with your current directory's files using a local or API LLM.

Summary

dir-assistant is a CLI python application available through pip that recursively indexes all text files in the current working directory so you can chat with them using a local or API LLM. By "chat with them", it is meant that their contents will automatically be included in the prompts sent to the LLM, with the most contextually relevant files included first. dir-assistant is designed primarily for use as a coding aid and automation tool.

Features

• Includes an interactive chat mode and a single prompt non-interactive mode.
• When enabled, it will automatically make file updates and commit to git.
• Local platform support for CPU (OpenBLAS), Cuda, ROCm, Metal, Vulkan, and SYCL.
• API support for all major LLM APIs. More info in the LiteLLM Docs.
• Uses a unique method for finding the most important files to include when submitting your prompt to an LLM called CGRAG (Contextually Guided Retrieval-Augmented Generation). You can read this blog post for more information about how it works.

Table of Contents

1. New Features
   1. Notable Upstream News
2. Quickstart
   1. Quickstart with Local Default Model
   2. Quickstart with API Model
3. Install
4. Embedding Model Configuration
5. Optional: Select A Hardware Platform
6. API Configuration
7. Local LLM Model Download
8. Running
   1. Automated file update and git commit
   2. Additional directories
   3. Ignoring files
   4. Overriding Configurations with Environment Variables
9. Upgrading
10. Additional Help
11. Contributors
12. Acknowledgements
13. Limitations
14. Todos
15. Additional Credits

New Features

• Automatically override configs by using matching environment variables
• Run a single prompt and quit with the new -s CLI option
• Persistent prompt history across sessions

Notable Upstream News

This section is dedicated to changes in libraries which can impact users of dir-assistant.

llama-cpp-python

• KV cache quants now available for most models. This enables reduced memory consumption per context token.
• Improved flash attention implementation for ROCM. This drastically reduces VRAM usage for large contexts on AMD cards.

These changes allow a 32B model with 128k context to comfortably run on all GPUs with at least 20GB of VRAM if enabled.

Quickstart

In this section are recipes to run dir-assistant in basic capacity to get you started quickly.

Quickstart with Local Default Model

To get started locally, you can download a default llm model. Default configuration with this model requires 8GB of memory on most hardware. You will be able to adjust the configuration to fit higher or lower memory requirements. To run via CPU:

pip install dir-assistant
dir-assistant models download-embed
dir-assistant models download-llm
cd directory/to/chat/with
dir-assistant

To run with hardware acceleration, use the platform subcommand:

...
dir-assistant platform cuda
cd directory/to/chat/with
dir-assistant

See which platforms are supported using -h:

dir-assistant platform -h

For Ubuntu 24.04

pip3 has been replaced with pipx starting in Ubuntu 24.04.

pipx install dir-assistant
...
dir-assistant platform cuda --pipx

Quickstart with API Model

To get started using an API model, you can use Google Gemini 1.5 Flash, which is currently free. To begin, you need to sign up for Google AI Studio and create an API key. After you create your API key, enter the following commands:

pip install dir-assistant
dir-assistant models download-embed
dir-assistant setkey GEMINI_API_KEY xxxxxYOURAPIKEYHERExxxxx
cd directory/to/chat/with
dir-assistant

You can optionally hardware-accelerate your local embedding model so indexing is quicker:

...
dir-assistant platform cuda
cd directory/to/chat/with
dir-assistant

See which platforms are supported using -h:

dir-assistant platform -h

For Ubuntu 24.04

pip3 has been replaced with pipx starting in Ubuntu 24.04.

pipx install dir-assistant
...
dir-assistant platform cuda --pipx

Install

Install with pip:

pip install dir-assistant

The default configuration for dir-assistant is API-mode. If you download an LLM model with download-llm, local-mode will automatically be set. To change from API-mode to local-mode, set the ACTIVE_MODEL_IS_LOCAL setting.

For Ubuntu 24.04

pip3 has been replaced with pipx starting in Ubuntu 24.04.

pipx install dir-assistant

Embedding Model Configuration

You must use an embedding model regardless of whether you are running an LLM via local or API mode, but you can also choose whether the embedding model is local or API using the ACTIVE_EMBED_IS_LOCAL setting. Generally local embedding will be faster, but API will be higher quality. To start, it is recommended to use a local model. You can download a good default embedding model with:

dir-assistant models download-embed

If you would like to use another embedding model, open the models directory with:

dir-assistant models

Note: The embedding model will be hardware accelerated after using the platform subcommand. To disable hardware acceleration, change n_gpu_layers = -1 to n_gpu_layers = 0 in the config.

Optional: Select A Hardware Platform

By default dir-assistant is installed with CPU-only compute support. It will work properly without this step, but if you would like to hardware accelerate dir-assistant, use the command below to compile llama-cpp-python with your hardware's support.

dir-assistant platform cuda

Available options: cpu, cuda, rocm, metal, vulkan, sycl

Note: The embedding model and the local llm model will be run with acceleration after selecting a platform. To disable hardware acceleration change n_gpu_layers = -1 to n_gpu_layers = 0 in the config.

For Ubuntu 24.04

pip3 has been replaced with pipx starting in Ubuntu 24.04.

dir-assistant platform cuda --pipx

For Platform Install Issues

System dependencies may be required for the platform command and are outside the scope of these instructions.

If you have any issues building llama-cpp-python, the project's install instructions may offer more info: https://github.com/abetlen/llama-cpp-python

API Configuration

If you wish to use an API LLM, you will need to configure it. To configure which LLM API dir-assistant uses, you must edit LITELLM_MODEL and the appropriate API key in your configuration. To open your configuration file, enter:

dir-assistant config open

Once editing the file, change:

[DIR_ASSISTANT]
LITELLM_MODEL = "gemini/gemini-1.5-flash-latest"
LITELLM_CONTEXT_SIZE = 500000
...
[DIR_ASSISTANT.LITELLM_API_KEYS]
GEMINI_API_KEY = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

LiteLLM supports all major LLM APIs, including APIs hosted locally. View the available options in the LiteLLM providers list.

There is a convenience subcommand for modifying and adding API keys:

dir-assistant setkey GEMINI_API_KEY xxxxxYOURAPIKEYHERExxxxx

However, in most cases you will need to modify other options when changing APIs.

Local LLM Model Download

If you want to use a local LLM, you can download a low requirements default model with:

dir-assistant models download-llm

Note: The local LLM model will be hardware accelerated after using the platform subcommand. To disable hardware acceleration, change n_gpu_layers = -1 to n_gpu_layers = 0 in the config.

Configuring A Custom Local Model

If you would like to use a custom local LLM model, download a GGUF model and place it in your models directory. Huggingface has numerous GGUF models to choose from. The models directory can be opened in a file browser using this command:

dir-assistant models

After putting your gguf in the models directory, you must configure dir-assistant to use it:

dir-assistant config open

Edit the following setting:

[DIR_ASSISTANT]
LLM_MODEL = "Mistral-Nemo-Instruct-2407.Q6_K.gguf"

Llama.cpp Options

Llama.cpp provides a large number of options to customize how your local model is run. Most of these options are exposed via llama-cpp-python. You can configure them with the [DIR_ASSISTANT.LLAMA_CPP_OPTIONS], [DIR_ASSISTANT.LLAMA_CPP_EMBED_OPTIONS], and [DIR_ASSISTANT.LLAMA_CPP_COMPLETION_OPTIONS] sections in the config file.

The options available for llama-cpp-python are documented in the Llama constructor documentation.

What the options do is also documented in the llama.cpp CLI documentation.

The most important llama-cpp-python options are related to tuning the LLM to your system's VRAM:

• Setting n_ctx lower will reduce the amount of VRAM required to run, but will decrease the amount of file text that can be included when running a prompt.
• CONTEXT_FILE_RATIO sets the proportion of prompt history to file text to be included when sent to the LLM. Higher ratios mean more file text and less prompt history. More file text generally improves comprehension.
• If your llm n_ctx times CONTEXT_FILE_RATIO is smaller than your embed n_ctx, your file text chunks have the potential to be larger than your llm context, and thus will not be included. To ensure all files can be included, make sure your embed context is smaller than n_ctx times CONTEXT_FILE_RATIO.
• Larger embed n_ctx will chunk your files into larger sizes, which allows LLMs to understand them more easily.
• n_batch must be smaller than the n_ctx of a model, but setting it higher will probably improve performance.

For other tips about tuning Llama.cpp, explore their documentation and do some google searches.

Running

dir-assistant

Running dir-assistant will scan all files recursively in your current directory. The most relevant files will automatically be sent to the LLM when you enter a prompt.

dir-assistant is shorthand for dir-assistant start. All arguments below are applicable for both.

Options for Running

The following arguments are available while running dir-assistant:

• -i --ignore: A list of space-separated filepaths to ignore
• -d --dirs: A list of space-separated directories to work on (your current directory will always be used)
• -s --single-prompt: Run a single prompt and output the final answer
• -v --verbose: Show debug information during execution

Example usage:

# Run a single prompt and exit
dir-assistant -s "What does this codebase do?"

# Show debug information
dir-assistant -v

# Ignore specific files and add additional directories
dir-assistant -i ".log" ".tmp" -d "../other-project"

Automated file update and git commit

The COMMIT_TO_GIT feature allows dir-assistant to make changes directly to your files and commit the changes to git during the chat. By default, this feature is disabled, but after enabling it, the assistant will suggest file changes and ask whether to apply the changes. If confirmed, it stages the changes and creates a git commit with the prompt message as the commit message.

To enable the COMMIT_TO_GIT feature, update the configuration:

dir-assistant config open

Change or add the following setting:

[DIR_ASSISTANT]
...
COMMIT_TO_GIT = true

Once enabled, the assistant will handle the Git commit process as part of its workflow. To undo a commit, type undo in the prompt.

Additional directories

You can include files from outside your current directory to include in your dir-assistant session:

dir-assistant -d /path/to/dir1 ../dir2

Ignoring files

You can ignore files when starting up so they will not be included in the assistant's context:

dir-assistant -i file.txt file2.txt

There is also a global ignore list in the config file. To configure it first open the config file:

dir-assistant config open

Then edit the setting:

[DIR_ASSISTANT]
...
GLOBAL_IGNORES = [
    ...
    "file.txt"
]

Overriding Configurations with Environment Variables

Any configuration setting can be overridden using environment variables. The environment variable name should match the configuration key name:

# Override the model path
export DIR_ASSISTANT__LLM_MODEL="mistral-7b-instruct.Q4_K_M.gguf"

# Enable git commits
export DIR_ASSISTANT__COMMIT_TO_GIT=true

# Change context ratio
export DIR_ASSISTANT__CONTEXT_FILE_RATIO=0.7

# Change llama.cpp embedding options
export DIR_ASSISTANT__LLAMA_CPP_EMBED_OPTIONS__n_ctx=2048

# Example setting multiple env vars inline with the command
DIR_ASSISTANT__COMMIT_TO_GIT=true DIR_ASSISTANT__CONTEXT_FILE_RATIO=0.7 dir-assistant

This allows multiple config profiles for your custom use cases.

# Run with different models
DIR_ASSISTANT__LLM_MODEL="model1.gguf" dir-assistant -s "What does this codebase do?"
DIR_ASSISTANT__LLM_MODEL="model2.gguf" dir-assistant -s "What does this codebase do?"

# Test with different context ratios
DIR_ASSISTANT__CONTEXT_FILE_RATIO=0.8 dir-assistant

Upgrading

Some version upgrades may have incompatibility issues in the embedding index cache. Use this command to delete the index cache so it may be regenerated:

dir-assistant clear

Additional Help

Use the -h argument with any command or subcommand to view more information. If your problem is beyond the scope of the helptext, please report a github issue.

Contributors

We appreciate contributions from the community! For a list of contributors and how you can contribute, please see CONTRIBUTORS.md.

Acknowledgements

• Local LLMs are run via the fantastic llama-cpp-python package
• API LLMS are run using the also fantastic LiteLLM package

Limitations

• Only tested on Ubuntu 22.04, Ubuntu 24.04, and OSX. Please let us know if you run it successfully on other platforms by submitting an issue.
• Dir-assistant only detects and reads text files at this time.

Todos

• API LLMs
• RAG
• File caching (improve startup time)
• CGRAG (Contextually-Guided Retrieval-Augmented Generation)
• Multi-line input
• File watching (automatically reindex changed files)
• Single-step pip install
• Model download
• Commit to git
• API Embedding models
• Immediate mode for better compatibility with custom script automations
• Web search
• Daemon mode for API-based use

Additional Credits

Special thanks to Blazed.deals for sponsoring this project.