[Frontend] New allowed_token_ids decoding request parameter
#6753
Conversation
This PR adds a new LogitsProcessor for constraining decoded tokens to a fixed set of token ids. This can be useful for some classification use cases. It is exposed via a new allowed_token_ids parameter in the OpenAI completion API (wouldn't be applicable to chat use cases).
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge). To run full CI, you can do one of these:
🚀 |
|
This reminds me of #5986... |
|
@DarkLight1337 it's a bit different in that this would be used in conjunction with a limited set of specific class tokens and would probably only be used with max_tokens=1, it's not intended for text generation. #5986 is for excluding certain words from generated text output. |
|
Thanks @DarkLight1337, I'll add a test too. |
|
Could the same result be achieved with high logit_bias on the token ids? |
|
@mgoin we thought about this but it's not really reliable since we don't know what the logits distribution looks like .. what value do we use in a general case to ensure that the tokens of interest all end up higher than any other token, while not overflowing |
github-actions
bot
added
the
ready
* upstream/main: (66 commits) [Bugfix] Fix PaliGemma MMP (vllm-project#6930) [TPU] Fix greedy decoding (vllm-project#6933) [Kernel] Tuned int8 kernels for Ada Lovelace (vllm-project#6848) [Kernel] Fix marlin divide-by-zero warnings (vllm-project#6904) [ci] GHA workflow to remove ready label upon "/notready" comment (vllm-project#6921) [Kernel] Remove unused variables in awq/gemm_kernels.cu (vllm-project#6908) [Frontend] New `allowed_token_ids` decoding request parameter (vllm-project#6753) [Bugfix] Allow vllm to still work if triton is not installed. (vllm-project#6786) [TPU] Support tensor parallelism in async llm engine (vllm-project#6891) [Kernel] Fix deprecation function warnings squeezellm quant_cuda_kernel (vllm-project#6901) [Core] Reduce unnecessary compute when logprobs=None (vllm-project#6532) [Kernel] Tuned FP8 Kernels for Ada Lovelace (vllm-project#6677) [Model] Initialize support for InternVL2 series models (vllm-project#6514) [Misc] Pass cutlass_fp8_supported correctly in fbgemm_fp8 (vllm-project#6871) Add Nemotron to PP_SUPPORTED_MODELS (vllm-project#6863) [Kernel] Increase precision of GPTQ/AWQ Marlin kernel (vllm-project#6795) [TPU] Reduce compilation time & Upgrade PyTorch XLA version (vllm-project#6856) [Docs] Add RunLLM chat widget (vllm-project#6857) [Model] Initial support for BLIP-2 (vllm-project#5920) [CI/Build][Doc] Update CI and Doc for VLM example changes (vllm-project#6860) ...
|
Why isn’t chat case applicable? |
…roject#6753) Signed-off-by: Alvant <[email protected]>
|
How can this be used with chat completions? |
This PR adds a new
LogitsProcessorfor constraining decoded tokens to a fixed set of token ids. This is needed for some of our classification use cases.It is exposed via a new
allowed_token_idsparameter in the OpenAI completion API (wouldn't be applicable to chat use cases).