Fix the pending_request_count to release the failed requests

[tanmayv25]

Before

After sending a 136 requests that will fail in Enqueue:

curl -s localhost:8002/metrics | grep pending
# HELP nv_inference_pending_request_count Instantaneous number of pending requests awaiting execution per-model.
# TYPE nv_inference_pending_request_count gauge
nv_inference_pending_request_count{model="test_model",version="1"} 136

After fix

After sending a 136 requests that will fail in Enqueue:

curl -s localhost:8002/metrics | grep pending
# HELP nv_inference_pending_request_count Instantaneous number of pending requests awaiting execution per-model.
# TYPE nv_inference_pending_request_count gauge
nv_inference_pending_request_count{model="test_model",version="1"} 0

I couldn't find any tests for nv_inference_pending_request_count metrics. Need to add testing of these features to avoid regression. Will open another ticket to enhance the testing.

[rmccorm4]

Should the request release callback be called internally on failure? Calling request release should set the state and update accordingly I see in our contract that we don't take ownership of the request if InferAsync -> Run -> Enqueue fails, so it's expected that the user would release the request.

[rmccorm4]

As mentioned offline, I don't love the idea of setting the state to RELEASED when the request hasn't actually had it's release callbacks called. The state should represent that it has actually been released. Is there another way?

[tanmayv25]

Ideally, the request should be set to pending state iff the Enqueue call was a success. However, there is an asynchronous interaction between the enqueue thread and and the backend threads that schedules the requests and moves it out from the queue.
We should not invoke callbacks for the failed requests as the failed requests are owned by the caller.

[tanmayv25]

Added a new state for tracking the failed requests. Discussed offline that the logic might need some clean-up.

[rmccorm4]

Also, existing tests can be found here: https://github.com/triton-inference-server/server/blob/main/qa/L0_metrics/metrics_queue_size_test.py

Feel free to rename test script from queue size to pending request count or something if it makes it easier to find