Skip to content

Commit

Permalink
docs: simplify, link to an e2e example
Browse files Browse the repository at this point in the history
Signed-off-by: mudler <[email protected]>
  • Loading branch information
mudler committed Apr 24, 2023
1 parent 9b914fb commit 3f769bf
Showing 1 changed file with 2 additions and 23 deletions.
25 changes: 2 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -320,32 +320,11 @@ To run local models, it is possible to use OpenAI compatible APIs, for instance

<details>

To run local inference, you need to download the models first, for instance you can find `ggml` compatible models in [huggingface.com](https://huggingface.co/models?search=ggml).
To run local inference, you need to download the models first, for instance you can find `ggml` compatible models in [huggingface.com](https://huggingface.co/models?search=ggml) (for example vicuna, alpaca and koala).

### Start the API server

To start the API server, follow the instruction in [LocalAI](https://github.com/go-skynet/LocalAI#usage):

```
git clone https://github.com/go-skynet/LocalAI
cd LocalAI
# copy your models to models/
cp your-model models/
# (optional) Edit the .env file to set the number of concurrent threads used for inference
# echo "THREADS=14" > .env
# start with docker-compose
docker compose up -d --build
# Check that the API is accessible at localhost:8080
curl http://localhost:8080/v1/models
# {"object":"list","data":[{"id":"your-model","object":"model"}]}
```

In order to use a local model, you might probably need to set a prompt template. This depends on the model being used. Create a file next your model ending by `.tmpl`, see some of the [templates examples in LocalAI](https://github.com/go-skynet/LocalAI/tree/master/prompt-templates).
To start the API server, follow the instruction in [LocalAI](https://github.com/go-skynet/LocalAI#example-use-gpt4all-j-model).

### Run k8sgpt

Expand Down

0 comments on commit 3f769bf

Please sign in to comment.