Allow non-decoupled model to send response and FINAL flag separately #6017
Conversation
src/http_server.cc
Outdated
|
|
||
| // Defer sending the response until FINAL flag is seen or | ||
| // there is error | ||
| if ((err == nullptr) && (flags & TRITONSERVER_RESPONSE_COMPLETE_FINAL) == 0) { |
There was a problem hiding this comment.
what would happen if there is an error in when calling TRITONBACKEND_ResponseSend()? Would the error be returned to the client and there would be no information sent to the client on TRITONBACKEND_ResponseFactorySendFlags?
There was a problem hiding this comment.
I made the change to not send error until FINAL flag is seen, otherwise there can be issue on the backend trying to call the response send function again after the userp has been released. Currently the only assumption is that the backend will not call the function again once final flag is sent.
GuanLuo
force-pushed
the
gluo-response
branch
2 times, most recently
from
bcfa971 to
5f21ff1
Compare
|
@GuanLuo do we still plan to implement this to help unify decoupled/non-decoupled logic in backends like RIVA/TRTLLM etc? |
rmccorm4
added
the
investigating
GuanLuo
force-pushed
the
gluo-response
branch
2 times, most recently
from
2f787bd to
d9831d9
Compare
src/grpc/infer_handler.cc
Outdated
| } | ||
|
|
||
| state->context_->EraseInflightState(state); | ||
|
|
||
| #ifdef TRITON_ENABLE_TRACING |
There was a problem hiding this comment.
I have a feeling that INFER_RESPONSE_COMPLETE multiple capture issue will be happening here, since deferring the response send is happening later.
…with the meaning.
|
@rmccorm4 , I've created a ticket for this: DLIS-6308 |
Follow up on triton-inference-server/core#229
For custom backend, one may call send response in the following style, even for "non-decoupled" model
This yields single response of the request from the client's perspective and would expect a successful inference through HTTP / non-streaming GRPC as it is "non-decoupled". The previous implementation is restricted that it simply assume decoupled use case when response complete callback is invoked multiple time. The change is to relax the restriction and collapse the above into single response to the client.