parallel inference for multiple requests

[ooaykac]

What should I do to enable multiple users to ask questions to the language model simultaneously and receive responses? Does llama.cpp support parallel inference for concurrent operations?

How can we ensure that requests made to the language model are processed and inferred in parallel, rather than sequentially, to serve multiple users simultaneously?

[lukaLLM]

+1

[bandoti]

This discussion on batch processing may provide some guidance.