test: Fix requested output deleting extra outputs

[kthui]

What does the PR do?

Enhance Python backend requested outputs testing when extra outputs from the model are deleted.

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

• build
• ci
• docs
• feat
• fix
• perf
• refactor
• revert
• style
• test

Related PRs:

triton-inference-server/python_backend#390

Where should the reviewer start?

Start with the change on the related PR, and then look at the new test case on this PR.

Test plan:

N/A

• CI Pipeline ID: See related PR.

Caveats:

N/A

Background

N/A

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

N/A

[kthui]

New test failing before the fix:

io_test.py F                                                             [100%]

=================================== FAILURES ===================================
______________ IOTest.test_requested_output_decoupled_prior_crash ______________

self = <io_test.IOTest testMethod=test_requested_output_decoupled_prior_crash>

    def test_requested_output_decoupled_prior_crash(self):
        model_name = "llm"
        prompt = "test"
    
        text_input_data = np.array([[prompt]]).astype(object)
        inputs = [grpcclient.InferInput("text_input", text_input_data.shape, "BYTES")]
        inputs[-1].set_data_from_numpy(text_input_data)
    
        requested_outputs = [grpcclient.InferRequestedOutput("text_output")]
    
        user_data = UserData()
        with grpcclient.InferenceServerClient(f"{_tritonserver_ipaddr}:8001") as client:
            client.start_stream(callback=partial(callback, user_data))
            client.async_stream_infer(model_name=model_name, inputs=inputs, outputs=requested_outputs)
            client.stop_stream()
    
        outputs = ""
        while not user_data._completed_requests.empty():
            result = user_data._completed_requests.get(block=False)
            if isinstance(result, InferenceServerException):
>               raise result
E               tritonclient.utils.InferenceServerException: Stub process 'llm_0_0' is not healthy.

io_test.py:283: InferenceServerException
- generated xml file: /opt/tritonserver/qa/L0_backend_python/io/test_requested_output_decoupled_prior_crash.report.xml -
=========================== short test summary info ============================
FAILED io_test.py::IOTest::test_requested_output_decoupled_prior_crash - trit...
============================== 1 failed in 1.25s ===============================

Note: To trigger the crash, the model must be sufficiently complicated, for instance, a decoupled string identity model cannot trigger the crash.

[kthui]

With the Clean up on the test model, the fault is still reproducible without the fix:

=================================== FAILURES ===================================
______________ IOTest.test_requested_output_decoupled_prior_crash ______________
...
>               raise result
E               tritonclient.utils.InferenceServerException: Stub process 'llm_0_0' is not healthy.

io_test.py:285: InferenceServerException
...
============================== 1 failed in 1.26s ===============================

and the test passed with the fix.