[feat] vLLM generation deterministic option/flag

[PeterSH6]

Hi vllm maintainers,

Thanks for the awesome project!

I'm wondering is there a deterministic option/flag to let the model generate identical results in different runs with the same prompts? (Also support random and beam search sampler, not only greedy sampler)
Does it enough to get deterministic behavior by setting the following random state?
I'm not sure what other factors will violate the determinism.

torch.manual_seed(seed)
np.random.seed(seed)
random.seed(seed)
torch.use_deterministic_algorithms(True)

CC: @WoosukKwon @zhuohan123 @Yard1

[simon-mo]

This is already supported through sampling params and OpenAI compatible API as of v0.3.2

[simon-mo]

Closed by #2514

[PeterSH6]

It seems that the latest version support per-request seed. But it may still have indeterminacy

When using torch.use_deterministic_algorithms(True), the pytorch will get error as the cumsum() operation in https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/sampler.py#L205, does not have deterministic cuda kernel.

Therefore, the current version may not be 'really deterministic', is it possible to bypass this operation?
@simon-mo

[simon-mo]

Good point. It seems it is still unresolved on the PyTorch side: pytorch/pytorch#75240

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[AntreasAntonio]

Any updates on this? It seems that I can have deterministic outputs when I am using say A100 to A100 tests, but as soon as I change my GPU the outputs change (but remain consistent for a given GPU).

[njaramish]

@AntreasAntonio I do not think you should expect deterministic results on different GPUs, as they may have different drivers, different versions of CUDA/cuDNN/cuBLAS, and different kernels for executing the same operation, which can cause small differences in floating point computations. These small differences can accumulate over a long sequence, eventually causing a prediction of token X instead of token Y, at which point your generation will take a completely different path).