Add support for request rescheduling

[krishung5]

Testing: triton-inference-server/server#6509

[Tabrizian]

The code changes look good to me. I was wondering do you think there could be a memory leak in non-decoupled case with these changes? For non-decoupled we create N responses in the beginning:

python_backend/src/python_be.cc

Line 1380 in 60a9091

auto err = TRITONBACKEND_ResponseNew(&response, requests[i]);

How is the None responses going to be treated by server? I didn't see any flags adjustments for non-decoupled mode for responses.

[krishung5]

The code changes look good to me. I was wondering do you think there could be a memory leak in non-decoupled case with these changes? For non-decoupled we create N responses in the beginning:

python_backend/src/python_be.cc

Line 1380 in 60a9091

auto err = TRITONBACKEND_ResponseNew(&response, requests[i]);

How is the None responses going to be treated by server? I didn't see any flags adjustments for non-decoupled mode for responses.

The backend will clean up the response if the associated request is rescheduled. Please see here:

python_backend/src/python_be.cc

Lines 1577 to 1585 in 8b01823

	if (pb_infer_requests[r]->ReleaseFlags() ==
	TRITONSERVER_REQUEST_RELEASE_RESCHEDULE) {
	// For rescheduled requests, we do not need to send a response.
	LOG_IF_ERROR(
	TRITONBACKEND_ResponseDelete((*responses)[r]),
	"failed to delete response");
	(*responses)[r] = nullptr;
	continue;
	}