Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Assistive chatbot v2: decompose questions and evaluate Guru card retrieval #20

Merged
merged 2 commits into from
May 2, 2024

Conversation

yoomlam
Copy link
Collaborator

@yoomlam yoomlam commented Apr 30, 2024

Ticket

Resolves https://navalabs.atlassian.net/browse/DST-184: Build pipeline from question-transformer LLM to retriever

Changes

Build pipeline from question-transformer LLM to retriever, including retrieval evaluation

  • original question → transformed question(s) → retriever → retrieval results; compare against desired Guru cards
  • In a sense, the LLM is a tool-using or function-calling LLM, where the tool is the retriever, and the tool input is a question.

Testing

Testing instructions and expected behavior:

  1. pip install -r requirements.txt
  2. Update .env depending on the LLMs you want to test
  3. Run LLM_MODEL_NAME="gpt-4-turbo" RETRIEVE_K=4 ./decompose-questions.py

@yoomlam yoomlam force-pushed the yl/v2-decompose-question branch from 0114796 to e6c28f9 Compare April 30, 2024 19:37
Comment on lines -26 to -31
llm = ollama_client(_llm_model_name, settings={"temperature": 0.1})


EMBEDDINGS = {
"all-MiniLM-L6-v2": {
"func": SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2"),
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved these into functions to reduce load time. Lazy-load them (i.e., only when needed).

@yoomlam yoomlam requested review from ccheng26 and KevinJBoyer April 30, 2024 19:40
@yoomlam yoomlam merged commit 6ddcfe8 into main May 2, 2024
2 checks passed
@yoomlam yoomlam deleted the yl/v2-decompose-question branch May 2, 2024 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants