[ML-OB] ragas evaluation integration #27217
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 3 commits
January 18, 2025 10:56
Preview links (active after the
|
added 3 commits
January 21, 2025 17:14
lievan
changed the title
added 3 commits
January 21, 2025 19:06
|
Added DOCS-9960 to track review |
github-actions
bot
added
Images
lievan
commented
Feb 11, 2025
lievan
commented
Feb 11, 2025
lievan
commented
Feb 11, 2025
Co-authored-by: lievan <[email protected]>
barieom
approved these changes
Feb 11, 2025
Co-authored-by: Barry Eom <[email protected]>
|
Hold for merge until ddtrace 3.0.0 is released |
jhgilbert
approved these changes
Feb 11, 2025
|
|
||
| 3. Run the script with the Ragas Faithfulness evaluation enabled: | ||
| ```bash | ||
| DD_LLMOBS_EVALUATORS=ragas_faithfulness DD_ENV=dev DD_API_KEY=<YOUR-DD-API-KEY> DD_SITE=datadoghq.com python quickstart.py |
There was a problem hiding this comment.
It seems like you're intentionally hyphenating your placeholders to distinguish them from Bash vars etc., so I won't make suggestions on those.
Co-authored-by: Jen Gilbert <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do? What is the motivation?
Introduce LLM Observability's integration with the RAGAS evaluation framework
Existing LLM integrations are all tracing related - by turning one of them on, a user gets automatic tracing if they use a LLM library we integration with
Our ragas integration is the first "evaluations" integration - we're aiming to provide a similar experience by allowing users to get evaluations, out of the box, from the ragas open source framework simply by flipping on an environment variable
Merge instructions
Merge readiness:
Merge queue is enabled in this repo. To have it automatically merged after it receives the required reviews, create the PR (from a branch that follows the
<yourname>/descriptionnaming convention) and then add the following PR comment:Additional notes