Frontend Improvements

[zhuohan123]

1. Current implementation of the FastAPI+asyncio+ray combination seems slow
2. Merge Hao’s throughput profiling code.
3. Make the frontend looks like OpenAI’s API.

[merrymercy]

For the OpenAI API server, you can learn from FastChat’s implementation (https://github.com/lm-sys/FastChat/blob/main/docs/openai_api.md), which supports all major OpenAI features such as completion, chat, and embedding. It can work with existing apps (e.g., LangChain) without code modifications

Another option is to directly import FastChat and extend the existing the cacheflow integration in FastChat.

[WoosukKwon]

@merrymercy Thanks for the suggestion! @zhuohan123 is working on the first direction.

[zhuohan123]

For the OpenAI API server, you can learn from FastChat’s implementation (https://github.com/lm-sys/FastChat/blob/main/docs/openai_api.md), which supports all major OpenAI features such as completion, chat, and embedding. It can work with existing apps (e.g., LangChain) without code modifications

Another option is to directly import FastChat and extend the existing the cacheflow integration in FastChat.

Thanks for the suggestion! We implemented a OpenAI API Server in #116 following fastchat's implementation. We currently implement the completion API. In the future, we are planning to import fastchat to implement the chat completion API.