Skip to content

Commit

Permalink
Add a new section to change LLM model such as deepseek based on valid…
Browse files Browse the repository at this point in the history
…ated model table in LLM microservice (opea-project#1501)

Signed-off-by: Tsai, Louie <[email protected]>
Co-authored-by: Wang, Kai Lawrence <[email protected]>
Co-authored-by: xiguiw <[email protected]>
Signed-off-by: Chingis Yundunov <[email protected]>
  • Loading branch information
3 people authored and Chingis Yundunov committed Mar 4, 2025
1 parent cce50fc commit 90a06cb
Show file tree
Hide file tree
Showing 2 changed files with 31 additions and 0 deletions.
12 changes: 12 additions & 0 deletions ChatQnA/docker_compose/intel/cpu/xeon/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,22 @@ To set up environment variables for deploying ChatQnA services, follow these ste
```

3. Set up other environment variables:

```bash
source ./set_env.sh
```

4. Change Model for LLM serving

By default, Meta-Llama-3-8B-Instruct is used for LLM serving, the default model can be changed to other validated LLM models.
Please pick a [validated llm models](https://github.com/opea-project/GenAIComps/tree/main/comps/llms/src/text-generation#validated-llm-models) from the table.
To change the default model defined in set_env.sh, overwrite it by exporting LLM_MODEL_ID to the new model or by modifying set_env.sh, and then repeat step 3.
For example, change to Llama-2-7b-chat-hf using the following command.

```bash
export LLM_MODEL_ID="meta-llama/Llama-2-7b-chat-hf"
```

## Quick Start: 2.Run Docker Compose

```bash
Expand Down
19 changes: 19 additions & 0 deletions ChatQnA/docker_compose/intel/hpu/gaudi/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,25 @@ To set up environment variables for deploying ChatQnA services, follow these ste
source ./set_env.sh
```

4. Change Model for LLM serving

By default, Meta-Llama-3-8B-Instruct is used for LLM serving, the default model can be changed to other validated LLM models.
Please pick a [validated llm models](https://github.com/opea-project/GenAIComps/tree/main/comps/llms/src/text-generation#validated-llm-models) from the table.
To change the default model defined in set_env.sh, overwrite it by exporting LLM_MODEL_ID to the new model or by modifying set_env.sh, and then repeat step 3.
For example, change to DeepSeek-R1-Distill-Qwen-32B using the following command.

```bash
export LLM_MODEL_ID="deepseek-ai/DeepSeek-R1-Distill-Qwen-32B"
```

Please also check [required gaudi cards for different models](https://github.com/opea-project/GenAIComps/tree/main/comps/llms/src/text-generation#system-requirements-for-llm-models) for new models.
It might be necessary to increase the number of Gaudi cards for the model by exporting NUM_CARDS to the new model or by modifying set_env.sh, and then repeating step 3. For example, increase the number of Gaudi cards for DeepSeek-R1-
Distill-Qwen-32B using the following command:

```bash
export NUM_CARDS=4
```

## Quick Start: 2.Run Docker Compose

```bash
Expand Down

0 comments on commit 90a06cb

Please sign in to comment.