test: Tests for Metrics API enhancement to include error counters

[indrajit96]

What does the PR do?

Currently We don't have any mechanism to count any failures that happen in triton core before the request gets to the backend.
Enhances existing metrics API to incorporate failure and failure counts in inference requests.
This PR adds new test

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

• build
• ci
• docs
• feat
• fix
• perf
• refactor
• revert
• style
• test

Related PRs:

triton-inference-server/core#377

Where should the reviewer start?

RespondIfError function call in core

Test plan:

Test for REJECTED
Modified qa/L0_model_queue/model_queue_test.py to capture rejections in Ensemble Model and Non Ensemble Model in dynamic batcher.
custom_zero_1_float32 model expected to have 4 request rejects

Test for OTHER
Modified qa/L0_model_queue/model_queue_test.py to capture OTHER errors in Ensemble Model.
ensemble_zero_1_float32 expects to have 2 request rejected reported as OTHER.

Test for CANCELED
Modified qa/L0_request_cancellation/scheduler_test.py to capture various cancellations.
_test_sequence_batch_scheduler_queued_request_cancellation test cancel 2 requests, expected to be reported as 2 in test_direct_sequence_batch_scheduler_request_cancellation() and test_sequence_batch_scheduler_backlog_request_cancellation()

Test for BACKEND
Modified qa/L0_backend_python/lifecycle/lifecycle_test.py which generates errors from backend inside test_infer_pymodel_error by using wrong model name.
Expect the error count to be 1.

Untested :

• REJECTED request which gets triggered from SequenceBatchScheduler()
• OTHER uncategorized failures for individual (non-ensemble) models
• Follow-up ticket for further testing: TODO @indrajit96

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• closes Jira issue: DLIS-5530

[rmccorm4]

Code LGTM overall, will save further nitpicks since this is planned to be cherry-picked. Let's get some docs additions in metrics.md with this PR to explain this feature so that the docs update can also be cherry-picked together with the feature. We can then point to the new docs in the release notes.

[indrajit96]

Newly added docs