Fix milvus dataprep ingest files failure #1299
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: lvliang-intel <[email protected]>
lvliang-intel
requested review from
XinyuYe-Intel and
letonghan
as code owners
letonghan
approved these changes
Feb 18, 2025
Spycsh
approved these changes
Feb 19, 2025
cwlacewe
pushed a commit
to cwlacewe/GenAIComps
that referenced
this pull request
Feb 21, 2025
Signed-off-by: lvliang-intel <[email protected]> Co-authored-by: Letong Han <[email protected]> Signed-off-by: Lacewell, Chaunte W <[email protected]>
aMahanna
added a commit
to arangoml/GenAIComps1.2
that referenced
this pull request
Feb 26, 2025
commit ad8f517 Author: Dina Suehiro Jones <[email protected]> Date: Wed Feb 26 11:35:04 2025 -0800 Dataprep Multimodal Redis README fixes (opea-project#1330) Signed-off-by: Dina Suehiro Jones <[email protected]> commit c70f868 Author: Ervin Castelino <[email protected]> Date: Tue Feb 25 07:42:42 2025 +0000 Update README.md (opea-project#1253) Signed-off-by: Ervin <[email protected]> commit 589587a Author: ZePan110 <[email protected]> Date: Mon Feb 24 17:54:51 2025 +0800 Fix docker image security issues (opea-project#1321) Signed-off-by: ZePan110 <[email protected]> commit f5699e4 Author: Brijesh Thummar <[email protected]> Date: Mon Feb 24 12:00:19 2025 +0530 [Doc] vLLM - typo in README.md (opea-project#1302) Fix Typo in README Signed-off-by: [email protected] <[email protected]> commit 364ccad Author: Jonathan Minkin <[email protected]> Date: Sun Feb 23 19:27:31 2025 -0800 Add support for string message in Bedrock textgen (opea-project#1291) * Add support for string message in bedrock, update README * Add test for string message in test script Signed-off-by: Jonathan Minkin <[email protected]> commit 625aec9 Author: Daniel De León <111013930+daniel-de-leon-user293@users.noreply.github.com> Date: Fri Feb 21 13:20:58 2025 -0800 Add native support for toxicity detection guardrail microservice (opea-project#1258) * add opea native support for toxic-prompt-roberta * add test script back * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add comp name env variable * set default port to 9090 Signed-off-by: Daniel Deleon <[email protected]> * add service to compose Signed-off-by: Daniel Deleon <[email protected]> * removed debug print Signed-off-by: Daniel Deleon <[email protected]> * remove triton version because habana updated Signed-off-by: Daniel Deleon <[email protected]> * add locust results Signed-off-by: Daniel Deleon <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * skip warmup for halluc test Signed-off-by: Daniel Deleon <[email protected]> --------- Signed-off-by: Daniel Deleon <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Liang Lv <[email protected]> Co-authored-by: Abolfazl Shahbazi <[email protected]> commit 4352636 Author: ZePan110 <[email protected]> Date: Fri Feb 21 17:14:08 2025 +0800 Fix trivy issue in Dockerfile (opea-project#1304) Signed-off-by: ZePan110 <[email protected]> commit 135ef91 Author: rbrugaro <[email protected]> Date: Thu Feb 20 15:29:39 2025 -0800 Change neo4j Bolt default PORT from 7687 to $NEO4J_PORT2 (opea-project#1292) * Change neo4j Bolt default PORT from 7687 to -configured the port in neo4j compose.yaml to use variable value -made all corresponding changes in neo4j dataprep and retriever components and test scripts to use env variable instead of default port value. Signed-off-by: rbrugaro <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * missing positional arg in milvus dataprep Signed-off-by: rbrugaro <[email protected]> * remove redundance in stop_docker Signed-off-by: rbrugaro <[email protected]> * resolve retriever to neo4j connectivity issue bad URL Signed-off-by: rbrugaro <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * set neo4j ports to neo4j defaults and fix environment variables in READMEs Signed-off-by: rbrugaro <[email protected]> --------- Signed-off-by: rbrugaro <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Liang Lv <[email protected]> commit a4f6af1 Author: Letong Han <[email protected]> Date: Thu Feb 20 13:38:01 2025 +0800 Refine dataprep test scripts (opea-project#1305) * Refine dataprep Milvus CI Signed-off-by: letonghan <[email protected]> commit 2102a8e Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu Feb 20 10:46:19 2025 +0800 Bump transformers (opea-project#1278) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Liang Lv <[email protected]> commit a033c05 Author: Liang Lv <[email protected]> Date: Wed Feb 19 14:19:02 2025 +0800 Fix milvus dataprep ingest files failure (opea-project#1299) Signed-off-by: lvliang-intel <[email protected]> Co-authored-by: Letong Han <[email protected]> commit 022d052 Author: lkk <[email protected]> Date: Wed Feb 19 09:50:59 2025 +0800 fix agent message format. (opea-project#1297) 1. set default session_id for react_langchain strategy, because the langchain version upgrade. 2. fix request message format commit 7727235 Author: Liang Lv <[email protected]> Date: Tue Feb 18 20:55:20 2025 +0800 Refine CLIP embedding microservice by leveraging the third-party CLIP (opea-project#1298) * Refine CLI embedding microservice using dependency Signed-off-by: lvliang-intel <[email protected]> commit a353f99 Author: Spycsh <[email protected]> Date: Mon Feb 17 11:35:38 2025 +0800 Fix telemetry connection issue when disabling telemetry (opea-project#1290) * Fix telemetry connection issue when disabling telemetry - use ENABLE_OPEA_TELEMETRY to control whether to enable open telemetry, default false - fix the issue that logs always show telemetry connection error with each request when telemetry is disabled - ban the above error propagation to microservices when telemetry is disabled Signed-off-by: Spycsh <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix ut failure where required the flag to be on * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Spycsh <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> commit 7c2e7f6 Author: xiguiw <[email protected]> Date: Sat Feb 15 11:25:15 2025 +0800 update vLLM CPU to latest tag (opea-project#1285) Get the latest vLLM stable version. Signed-off-by: Wang, Xigui <[email protected]> commit c3c8497 Author: Letong Han <[email protected]> Date: Fri Feb 14 22:29:38 2025 +0800 Fix Qdrant retriever RAG issue. (opea-project#1289) * Fix Qdrant retriever no retrieved result issue. Signed-off-by: letonghan <[email protected]> commit 47f68a4 Author: Letong Han <[email protected]> Date: Fri Feb 14 20:29:27 2025 +0800 Fix the retriever issue of Milvus (opea-project#1286) * Fix the retriever issue of Milvus DB that data can not be retrieved after ingested using dataprep. Signed-off-by: letonghan <[email protected]> --------- Signed-off-by: letonghan <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> commit 0e3f8ab Author: minmin-intel <[email protected]> Date: Thu Feb 13 20:24:02 2025 -0800 Improve multi-turn capability for agent (opea-project#1248) * first code for multi-turn Signed-off-by: minmin-intel <[email protected]> * test redispersistence Signed-off-by: minmin-intel <[email protected]> * integrate persistent store in react llama Signed-off-by: minmin-intel <[email protected]> * test multi-turn Signed-off-by: minmin-intel <[email protected]> * multiturn for assistants api and chatcompletion api Signed-off-by: minmin-intel <[email protected]> * update readme and ut script Signed-off-by: minmin-intel <[email protected]> * update readme and ut scripts Signed-off-by: minmin-intel <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix bug Signed-off-by: minmin-intel <[email protected]> * change memory type naming Signed-off-by: minmin-intel <[email protected]> * fix with_memory as str Signed-off-by: minmin-intel <[email protected]> --------- Signed-off-by: minmin-intel <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> commit 4a90692 Author: rbrugaro <[email protected]> Date: Thu Feb 13 18:12:25 2025 -0800 Bug Fix neo4j dataprep ingest error handling and skip_ingestion argument passing (opea-project#1288) * Fix dataprpe ingest error handling and skip_ingestion argument passing in dataprep neo4j integration Signed-off-by: rbrugaro <[email protected]> commit d1dfd0e Author: Spycsh <[email protected]> Date: Thu Feb 13 22:39:47 2025 +0800 Align mongo related chathistory/feedbackmanagement/promptregistry image names with examples (opea-project#1284) Align mongo related chathistory/feedbackmanagement/promptregistry image names with examples Signed-off-by: Spycsh <[email protected]> Co-authored-by: Liang Lv <[email protected]> commit bef501c Author: Liang Lv <[email protected]> Date: Thu Feb 13 21:18:58 2025 +0800 Fix VDMS retrieval issue (opea-project#1252) * Fix VDMS retrieval issue Signed-off-by: lvliang-intel <[email protected]> commit 23b2be2 Author: ZePan110 <[email protected]> Date: Thu Feb 13 16:07:14 2025 +0800 Fix Build latest images on push event workflow (opea-project#1282) Signed-off-by: ZePan110 <[email protected]> commit f8e6216 Author: Spycsh <[email protected]> Date: Wed Feb 12 15:45:14 2025 +0800 fix metric id issue when init multiple Orchestrator instance (opea-project#1280) Signed-off-by: Spycsh <[email protected]> commit d3906ce Author: chen, suyue <[email protected]> Date: Wed Feb 12 14:56:55 2025 +0800 update default service list (opea-project#1276) Signed-off-by: chensuyue <[email protected]> commit 17b9672 Author: XinyaoWa <[email protected]> Date: Wed Feb 12 13:53:31 2025 +0800 Fix langchain and huggingface version to avoid bug in FaqGen and DocSum, remove vllm hpu triton version fix (opea-project#1275) * Fix langchain and huggingface version to avoid bug Signed-off-by: Xinyao Wang <[email protected]> commit b777db7 Author: Letong Han <[email protected]> Date: Mon Feb 10 16:00:55 2025 +0800 Fix Dataprep Ingest Data Issue. (opea-project#1271) * Fix Dataprep Ingest Data Issue. Trace: 1. The update of `langchain_huggingface.HuggingFaceEndpointEmbeddings` caused the wrong size of embedding vectors. 2. Wrong size vectors are wrongly saved into Redis database, and the indices are not created correctly. 3. The retriever can not retrieve data from Redis using index due to the reasons above. 4. Then the RAG seems `not work`, for the file uploaded can not be retrieved from database. Solution: Replace all of the `langchain_huggingface.HuggingFaceEndpointEmbeddings` to `langchain_community.embeddings.HuggingFaceInferenceAPIEmbeddings`, and modify related READMEs and scirpts. Related issue: - opea-project/GenAIExamples#1473 - opea-project/GenAIExamples#1482 --------- Signed-off-by: letonghan <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> commit 0df374b Author: Daniel De León <111013930+daniel-de-leon-user293@users.noreply.github.com> Date: Sun Feb 9 22:01:58 2025 -0800 Update docs for LLamaGuard & WildGuard Microservice (opea-project#1259) * working README for CLI and compose Signed-off-by: Daniel Deleon <[email protected]> * update for direct python execution Signed-off-by: Daniel Deleon <[email protected]> * fix formatting Signed-off-by: Daniel Deleon <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * bring back depends_on condition Signed-off-by: Daniel Deleon <[email protected]> --------- Signed-off-by: Daniel Deleon <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abolfazl Shahbazi <[email protected]> commit fb86b5e Author: Louie Tsai <[email protected]> Date: Sat Feb 8 00:58:33 2025 -0800 Add Deepseek model into validated model table and add required Gaudi cards for LLM microservice (opea-project#1267) * Update README.md for Deepseek support and numbers of required gaudi cards Signed-off-by: Tsai, Louie <[email protected]> * Update README.md Signed-off-by: Tsai, Louie <[email protected]> --------- Signed-off-by: Tsai, Louie <[email protected]> commit ecb7f7b Author: Spycsh <[email protected]> Date: Fri Feb 7 16:58:22 2025 +0800 Fix web-retrievers hub client and tei endpoint issue (opea-project#1270) * fix web-retrievers hub client and tei endpoint issue Signed-off-by: Spycsh <[email protected]> commit 5baada8 Author: ZePan110 <[email protected]> Date: Thu Feb 6 15:03:00 2025 +0800 Fix CD test issue. (opea-project#1263) 1.Fix template name in README 2.Fix invalid release name Signed-off-by: ZePan110 <[email protected]> commit fa01f46 Author: minmin-intel <[email protected]> Date: Wed Feb 5 13:57:57 2025 -0800 fix tei embedding and tei reranking bug (opea-project#1256) Signed-off-by: minmin-intel <[email protected]> Co-authored-by: Abolfazl Shahbazi <[email protected]> commit 4ede405 Author: Eero Tamminen <[email protected]> Date: Wed Feb 5 22:04:50 2025 +0200 Create token metrics only when they are available (opea-project#1092) * Create token metrics only when they are available This avoids generation of useless token/request histogram metrics for services that use Orchestrator class, but never call its token processing functionality. (Helps in differentiating frontend megaservice metrics from backend megaservice ones, especially when multiple OPEA applications run in the same cluster.) Also change Orchestrator CI test workaround to use unique prefix for each metric instance, instead of metrics being (singleton) class variables. Signed-off-by: Eero Tamminen <[email protected]> * Add locking for latency metric creation / method change As that that could be called from multiple request handling threads. Signed-off-by: Eero Tamminen <[email protected]> --------- Signed-off-by: Eero Tamminen <[email protected]> Co-authored-by: Malini Bhandaru <[email protected]>
dtrawins
pushed a commit
to dtrawins/GenAIComps
that referenced
this pull request
Mar 5, 2025
Signed-off-by: lvliang-intel <[email protected]> Co-authored-by: Letong Han <[email protected]> Signed-off-by: Dariusz Trawinski <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fix milvus dataprep ingest files failure. This issue is introduced by PR #1288
Issues
n/aType of change
List the type of change like below. Please delete options that are not relevant.
Dependencies
None
Tests
Local test and CI test.