Adds audio querying to MultimodalQ&A gateway #974
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: okhleif-IL <[email protected]> * added in audio dict creation Signed-off-by: okhleif-IL <[email protected]> * separated audio from prompt Signed-off-by: okhleif-IL <[email protected]> * added ASR endpoint Signed-off-by: okhleif-IL <[email protected]> * removed ASR endpoints from mm embedding Signed-off-by: okhleif-IL <[email protected]> * edited return logic, fixed function call Signed-off-by: okhleif-IL <[email protected]> * added megaservice to elif Signed-off-by: okhleif-IL <[email protected]> * reworked helper func Signed-off-by: okhleif-IL <[email protected]> * Append audio to prompt Signed-off-by: okhleif-IL <[email protected]> * Reworked handle messages, added metadata Signed-off-by: okhleif-IL <[email protected]> * Moved dictionary logic to right place Signed-off-by: okhleif-IL <[email protected]> * changed logic to rely on message len Signed-off-by: okhleif-IL <[email protected]> * list --> empty str Signed-off-by: okhleif-IL <[email protected]> --------- Signed-off-by: Melanie Buehler <[email protected]> Signed-off-by: okhleif-IL <[email protected]> Signed-off-by: dmsuehir <[email protected]>
for more information, see https://pre-commit.ci
Signed-off-by: okhleif-IL <[email protected]>
1 task
Signed-off-by: okhleif-IL <[email protected]>
Fixed role bug where enumeration was wrong
for more information, see https://pre-commit.ci
Codecov ReportAttention: Patch coverage is
|
ashahba
reviewed
Dec 5, 2024
Signed-off-by: Melanie Buehler <[email protected]>
Signed-off-by: Melanie Buehler <[email protected]>
Signed-off-by: Melanie Buehler <[email protected]>
Adds unit test coverage for audio query
for more information, see https://pre-commit.ci
Signed-off-by: Melanie Buehler <[email protected]>
Fix port number placement
mkbhanda
approved these changes
Dec 6, 2024
| else: | ||
| return prompt | ||
|
|
||
| def convert_audio_to_text(self, audio): | ||
| # translate audio to text by passing in dictionary to ASR |
There was a problem hiding this comment.
comment quirky! dictionary is a data type here but can get mixed with the English word dictionary (word meanings)
| else: | ||
| input_dict = {"byte_str": audio[0]} | ||
|
|
||
| response = requests.post(self.asr_endpoint, data=json.dumps(input_dict), proxies={"http": None}) |
There was a problem hiding this comment.
should proxies be read from some environment variable for a more general solution?
There was a problem hiding this comment.
Why this is setting proxies in the first place, shouldn't those be set well before this point?
eero-t
reviewed
Dec 9, 2024
| import unittest | ||
| from typing import Union | ||
|
|
||
| import requests | ||
| from fastapi import Request | ||
|
|
||
| os.environ["ASR_SERVICE_PORT"] = "8086" |
There was a problem hiding this comment.
Why this overrides environment, instead of taking the value from environment?
4 tasks
|
Labeling this as |
|
Looking at new changes in |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds ASR endpoint, speech audio processing, prompt construction, and return of decoded audio in response metadata. This goes with GenAIExamples PR: opea-project/GenAIExamples#1225.
Issues
Part of the MultimodalQnA Audio & Image Enhancements RFC
Type of change
Dependencies
N/A
Tests
Automated tests were added to GenAIExamples.