vLLM server capabilities

[QwertyJack]

Thank you for your excellent work!

I've noticed that you've implemented a vLLM-based server with some modifications, but I have a few questions:

1. Why did you omit the served-model parameter? It's quite useful for loading a model from local storage while maintaining the same name.
2. Do you think including utility endpoints like /metrics and /health would be beneficial?

Thanks in advance.

[jeffreymeetkai]

Thanks for pointing this out. I've enabled --served-model-name and you should be able to load a local model while accepting requests with another name (e.g.: official model names like meetkai/functionary-small-v3.2).

For question 2, we do not find the need to implement yet as it seem to involve long lines of code to implement as seen from here, and there is also not much demand for it. We also use a monkey-patched AsyncLLMEngine here when grammar sampling is enabled. I'm not sure if monkey-patching other parts of vLLM is needed to get /health to work with this too.

[QwertyJack]

I see.
Btw, I found an easy way to integrate those two endpoints. PTAL

[nskumz]

@QwertyJack , could you please share the process to get the vLLM endpoint /metrics from this model

[QwertyJack]

@QwertyJack , could you please share the process to get the vLLM endpoint /metrics from this model

After #263 is merged, vLLM style endpoints GET /metrics and GET /health will be available after the service starts.

Using the latest main branch and following the README should ensure it works smoothly.