Promptly respond to timeout requests under reject policy #333
Conversation
kthui
force-pushed
the
jacky-dynamic-batch-timeout
branch
from
558f59c to
0c968ec
Compare
| // Recapture the outer most lock to keep making progress. | ||
| lock.lock(); | ||
| } | ||
| WaitForPayloadSlotAvailable(&lock, default_wait_microseconds); |
There was a problem hiding this comment.
default_wait_microseconds is a fixed value, wouldn't there still be delay of 0.5 seconds? Or that is acceptable for rejecting requests?
There was a problem hiding this comment.
I think 0.5s should be fine for now. What they were complaining about is timeout of 2s in the scenario, when a model executes for 10s, so timeout requests are waiting for 8 extra seconds to be returned.
We can always adjust it upon their feedback. @kthui, what do you think?
There was a problem hiding this comment.
Yes. I think we can have them try this version first, we can always reduce the delay interval if they ask. I have filed an enhancement ticket to remove the need of a manually set interval.
There was a problem hiding this comment.
wouldn't there still be delay of 0.5 seconds? Or that is acceptable for rejecting requests?
I think it is acceptable, since they mentioned "[some large model] could last for tens of seconds" and it becomes a concern, which 0.5 seconds is significantly less than tens of seconds.
Related PR: triton-inference-server/server#6938
Currently, the dynamic batch scheduler only rejects timed-out requests when a payload slot is available and a new batch of requests is formed and submitted for inference. The payload slot availability is based on the completion of the previous batched inference request, which can take a long time depending on the model. Thus, the rejection of timed-out requests can be significantly delayed.
This change adds the ability for the dynamic batch scheduler to reject timed-out requests while waiting for the availability of a payload slot.