Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Update OpenAIAnswerGenerator defaults and with learnings from PromptNode #4038

Merged
merged 22 commits into from
Feb 11, 2023

Conversation

sjrl
Copy link
Contributor

@sjrl sjrl commented Feb 2, 2023

Related Issues

  • fixes N/A

Proposed Changes:

Breaking Change: Update to Default Values

  • Updated the default values for presence_penalty and frequency_penalty to the recommended values by OpenAI to avoid repetitive text.

Reasonable values for the penalty coefficients are around 0.1 to 1 if the aim is to just reduce repetitive samples somewhat.

  • Previously we had the values set to -2 (now both are 0.1), which greatly increased the chance of repetitive text being generated and is what caused the occurrence of answers ending with repetitive commas or periods.
  • Updated the default model to text-davinci-003 since it returns much higher quality of results than the text-curie-001 model
  • Update the max_tokens to 50 since davinci-003 has a much bigger max seq length of 4000 tokens and the generated answers are often quite verbose

Additional Features (non-breaking):

Additionally, given some of the learnings from the PromptNode I:

  • Printed the full prompt to logger.debug.
  • Allow users to modify the instruction_prompt at initialization (with the default instruction prompt being the same) using a PromptTemplate. This allows for more flexibility which could be useful. Prior, we found that during internal usage we would often inject additional instructions into the query to get the output to change.

How did you test it?

Existing tests

Notes for the reviewer

Checklist

  • Add additional documentation and examples in Haystack docs to help users understand how to use this node.
  • I have read the contributors guidelines and the code of conduct
  • I have updated the related issue with new insights and changes
  • I added tests that demonstrate the correct behavior of the change
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
  • I documented my code
  • I ran pre-commit hooks and fixed any issue

@sjrl sjrl requested a review from a team as a code owner February 2, 2023 12:34
@sjrl sjrl requested review from silvanocerza and tholor and removed request for a team February 2, 2023 12:34
@sjrl sjrl self-assigned this Feb 2, 2023
Copy link
Member

@tholor tholor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes all sense. I would only switch the way users can modify the prompt to be more consistent across haystack

haystack/nodes/answer_generator/openai.py Outdated Show resolved Hide resolved
haystack/nodes/answer_generator/openai.py Show resolved Hide resolved
@sjrl sjrl added breaking change type:feature New feature or request topic:pipeline topic:predictions format of predictions, score, probability ... labels Feb 3, 2023
@sjrl sjrl requested a review from tholor February 3, 2023 15:44
@sjrl sjrl requested a review from tholor February 7, 2023 07:39
@sjrl sjrl requested a review from tholor February 8, 2023 08:17
@sjrl sjrl requested a review from agnieszka-m February 8, 2023 08:44
@github-actions github-actions bot added the type:documentation Improvements on the docs label Feb 10, 2023
Copy link
Contributor

@agnieszka-m agnieszka-m left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I committed the changes here directly

@sjrl sjrl merged commit 75ef959 into main Feb 11, 2023
@sjrl sjrl deleted the answer_gen/update branch February 11, 2023 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking change topic:predictions format of predictions, score, probability ... topic:tests type:documentation Improvements on the docs type:feature New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants