[Frontend] Support suffix in completions API (fill-in-the-middle, FIM) #9522
Conversation
|
Planning to simplify this, would be better for it to work more like chat templates than tool parsers. |
|
This is exciting! Looking forward to review |
Handle model-specific FIM encoding rules in a similar way to how we're handling different tool parsers.
njhill
force-pushed
the
fim
branch
2 times, most recently
from
0b1e762 to
37135ae
Compare
njhill
force-pushed
the
fim
branch
2 times, most recently
from
c4886c3 to
4a2e7e9
Compare
|
FYI @patrickvonplaten if you'd like to check the Mistral part. Is there a small Mistral model on the HF hub that we could use for this in tests? |
|
I guess codestral might be too big? https://huggingface.co/mistralai/Codestral-22B-v0.1 The 7B-v0.3 should also work reasonably well for FIM though: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3 |
|
Thanks @patrickvonplaten, yes I was hoping for something smaller than codestral 22B ... will try with the 7B thanks! |
# Conflicts: # vllm/entrypoints/openai/api_server.py # vllm/entrypoints/openai/serving_completion.py # vllm/entrypoints/openai/serving_embedding.py # vllm/entrypoints/openai/serving_engine.py
|
This pull request has merge conflicts that must be resolved before it can be |
|
Hi, this would be a great addition to continue.dev. |
|
Hi, following this as well. Thanks! |
|
@thomasbs17 @gadikar-deshaw apologies, I've been diverted from this by other priorities but will try to resurrect it when I can. Any help would be welcome - apart from bringing the existing changes up to date I think the only thing remaining was adding tests. |
Sounds good, thanks! I'll try to look into adding the tests |
|
Hi, when this PR can be merged into main? |
mgoin
changed the title
|
Thank you @thomasbs17! I'll try to look at this today. |
Co-authored-by: Thomas Bouamoud <[email protected]>
Signed-off-by: Nick Hill <[email protected]>
|
Thanks @thomasbs17! I have merged your main-merge, and another commit to fix the linting errors. |
Signed-off-by: Nick Hill <[email protected]>
For code models that have been trained with an infilling task.
Handle model-specific FIM encoding rules in a similar way to how we're handling different tool parsers.
For example, can be enabled for for Codestral by adding command line arg
--fim mistral(as well as--tokenizer-mode mistralin this case).