Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve question generation efficiency In Response Relevancy metrics #1810

Merged

Conversation

hundredeuk2
Copy link
Contributor

Description: This update refactors the _ascore method to use asyncio.gather for parallel execution of question generation tasks. Key improvements include:

Issue : When ResponseRelevancy evaluation loops based on the strictness variable (default 3), which is taking longer than expected

Performance Enhancement: By processing all tasks concurrently, the overall execution time is significantly reduced, especially for high strictness values.
Code Simplification: The refactored code is more concise and aligns better with Python's asynchronous programming patterns.
Scalability: The new implementation handles a larger number of tasks without blocking, making it more suitable for use cases requiring high concurrency.
Backward Compatibility: All existing functionality remains intact, with no changes required to external interfaces.

This change improves the efficiency and maintainability of the codebase, aligning with best practices for asynchronous programming in Python.

Result after resolution : Average 1 run of Example code based on tutorial code (10 runs total) reduced to 4.35s => 2.82s

@dosubot dosubot bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Jan 6, 2025
@hundredeuk2
Copy link
Contributor Author

Speed difference when experimenting with the entire dataset during the tutorial

which is

from datasets import load_dataset
dataset = load_dataset(
    "explodinggradients/amnesty_qa",
    "english_v3",
    trust_remote_code=True
)

before fix the code

25.063s

delta_times = [25.36874794960022,
 29.800117015838623,
 23.23536705970764,
 26.765387058258057,
 24.31095004081726,
 25.598613023757935,
 25.597922801971436,
 24.84323501586914,
 22.47859835624695,
 22.639552116394043]

after fix the code

23.32s

delta_times = [21.71244788169861,
 20.64484715461731,
 21.470897912979126,
 29.34908390045166,
 20.96466302871704,
 23.604390144348145,
 19.539119720458984,
 21.615919828414917,
 26.599979162216187,
 27.745948314666748]

@jjmachan
Copy link
Member

jjmachan commented Jan 7, 2025

hey @hundredeuk2 thank you so much for improving the codebase we really appreciate that - merging this now

@jjmachan jjmachan merged commit 4e732cf into explodinggradients:main Jan 7, 2025
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size:S This PR changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants