模型非流式回答的内容不完整，而设置stream=True就能完整输出，设置max_tokens无效

[aimi0914]

[danielhanchen]

@aimi0914 Apologies I had to translate that:

The content of the model's non-streaming answer is incomplete, but setting stream=True can complete the output. Setting max_tokens is invalid #310

I can see you set max_tokens to 4096, but it just stops generating? Very weird. Tbh I'm not really sure how to debug this, since this seems like an application you're creating

[aimi0914]

@aimi0914 Apologies I had to translate that:

The content of the model's non-streaming answer is incomplete, but setting stream=True can complete the output. Setting max_tokens is invalid #310

I can see you set max_tokens to 4096, but it just stops generating? Very weird. Tbh I'm not really sure how to debug this, since this seems like an application you're creating

My code is as follows, simple call with openai library. Qwen1.5-14B-Chat. The model was deployed using fastchat

[aimi0914]

@aimi0914 Apologies I had to translate that:

The content of the model's non-streaming answer is incomplete, but setting stream=True can complete the output. Setting max_tokens is invalid #310

I can see you set max_tokens to 4096, but it just stops generating? Very weird. Tbh I'm not really sure how to debug this, since this seems like an application you're creating

Replacing the model with baichuan and testing with the same code is fine. So I suspect it's the qwen model

[aimi0914]

vllm-project/vllm#3034 (comment)
the same question

[danielhanchen]

@aimi0914 Oh interesting! Hmm I wonder why the Qwen model doesn't work well