Aiming to containerize the UndreamAI Server from here and prepare it for serverless deployments.
Setup:
- Download a .gguf of your desired model into /models
You can find quantized Meta-Llama-3-8B models here.
-
Set environment variables in .env
-
docker-compose up
Images with the models can be found here
To do:
- create default image for all linux architectures
- add all flags as env variables
- create and test image for serverless deployments