Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow greedy decoding with 0.0 temperature #182

Merged
merged 1 commit into from
Dec 2, 2024
Merged

Allow greedy decoding with 0.0 temperature #182

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions src/vllm_tgis_adapter/grpc/grpc_server.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -630,11 +630,13 @@ async def _validate_and_convert_params(
)

random_sampling_params: dict[str, Any]
if greedy:
temperature = sampling.temperature if sampling.HasField("temperature") else 1.0
if greedy or temperature == 0.0:
# 0.0 temperature is equivalent to greedy decoding
random_sampling_params = {"temperature": 0.0}
else:
random_sampling_params = {
"temperature": with_default(sampling.temperature, 1.0),
"temperature": temperature,
"top_k": with_default(sampling.top_k, -1),
"top_p": with_default(sampling.top_p, 1.0),
"seed": sampling.seed if sampling.HasField("seed") else None,
Expand Down
5 changes: 3 additions & 2 deletions src/vllm_tgis_adapter/grpc/pb/generation.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,8 +132,9 @@ message DecodingParameters {


message SamplingParameters {
// Default (0.0) means disabled (equivalent to 1.0)
float temperature = 1;
// Unset will default to 1.0
// 0.0 is equivalent to greedy decoding
optional float temperature = 1;
// Default (0) means disabled
uint32 top_k = 2;
// Default (0) means disabled (equivalent to 1.0)
Expand Down
3 changes: 0 additions & 3 deletions src/vllm_tgis_adapter/grpc/validation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ class TGISValidationError(str, Enum):
https://github.ibm.com/ai-foundation/fmaas-inference-server/blob/main/router/src/validation.rs#L238-L271
"""

Temperature = "temperature must be >= 0.05"
TopP = "top_p must be > 0.0 and <= 1.0"
TopK = "top_k must be strictly positive"
TypicalP = "typical_p must be <= 1.0"
Expand Down Expand Up @@ -139,8 +138,6 @@ def validate_params( # noqa: C901
)
):
TGISValidationError.SampleParametersGreedy.error()
if sampling.temperature and sampling.temperature < 0.05:
TGISValidationError.Temperature.error()
if sampling.top_k < 0:
TGISValidationError.TopK.error()
if not (0 <= sampling.top_p <= 1):
Expand Down