[LLC] [Core] Resolve design concerns about pipeline

[annelo-msft]

Continuing discussion started in this PR: #24122 (comment)

Was: Resolve design concerns about response error classification - context on this below

In addition, we would like to use this work on the pipeline design to resolve the issue that we are throwing when the HttpPipeline public constructor is called AND a policy is added to the call via RequestOptions.

[KrzysztofCwalina]

Here is some context:

I have to say something still feels hacky about our error logic in the pipeline. It seems wrong that Response.IsError needs to be done with a custom policy. Error classification is such a fundamental thing that I think it should be done regardless of what policies are in the pipeline.

It seems wrong that the classifier is set on the pipeline. It should be a parameter to the pipeline.Send method. The method would set it on the message. Message would be a fundamental object that flows through the system and Response is just a "view" over a message, i.e. Response has a pointer to message not the other way.

If the pipeline was designed this way, the error feature would be trivial to implement: a) message has internal IsError property that simply calls the classifier stored in message. b) response, having a pointer to message, would just call _message.IsError.

This is just an example of a feature that would get easier with this architecture. I feel like this would help with many other problems we ran into in the past.

Can we still refactor the system to be like that?

[annelo-msft]

More relevant context about alternative designs:

We chose the pipeline policy approach because there is a concern about IsError not being set correctly as the Response flows through the pipeline:
#24122 (comment)

An alternate approach (setting IsError in SendAsync) is illustrated here: 243a191#diff-0a59c43e2383e8557538b07995896820037da860dd0a133824970499a259b58eR73

We needed to modify the definition of SendAsync, and we had some concerns regarding potential perf impact here:
a780216

Next steps here include having a conversation around the tradeoffs of setting IsError at the beginning of the Response's lifetime vs. right before it exits the pipeline.

[annelo-msft]

I have to say something still feels hacky about our error logic in the pipeline. It seems wrong that Response.IsError needs to be done with a custom policy. Error classification is such a fundamental thing that I think it should be done regardless of what policies are in the pipeline.

This now happens in the transport policy.

[annelo-msft]

Initial issue is addressed, and creating #25565 to track remaining.