Record a metric of proxy errors

[olix0r]

The proxy handles errors by synthesizing responses; however, depending
on where such an error occurs, it may not be recorded by a metrics
layer.

This change introduces a request_errors_total counter that records
the number/kind of errors encountered. This is immediately helpful for
debugging and diagnostics, but could also be helpful to view in the UI.
We may want to revisit this metric, especialy its labels, before relying
on it more prominently. But in the meantime it's extraordinarily useful
as it is.

[hawkw]

seems very straightforward — lgtm!

[kleimkuhler]

I have a question about the implementation of error-metrics. It's not a blocker on merging, but once that's resolved, everything else looks good to me.

[kleimkuhler]

What are your thoughts on changing these structs to:

pub struct RecordErrorLayer<L, K: Hash + Eq> {
    label: L,
    registry: Registry<K>,
}

pub struct RecordError<L, K: Hash + Eq, S> {
    label: L,
    registry: Registry<K>,
    inner: S,
}

When reading through this change, this change is what I expected to see and how I thought about it when understanding the rest of the impls.

[kleimkuhler]

Alternatively, this could be similar to the Registry in #428 where:

type Registry<L> = Arc<Mutex<IndexMap<L, Arc<Metrics>>>>;

[olix0r]

Eh, the Registry type is really there as a public handle for getting layers and reporting the results... It's a bit confusing for Registry to produce layers but those layers just hold Registries?

This might all be clearer if I split the types up into separate files?

[kleimkuhler]

Yea it helps, but I think what I'm reacting to is it seems like each *-metrics crate has it's own way of implementing what it's concept of a Registry is and they are just different enough that I feel like I'm reviewing new impls.

Reviewing this change, the other open #428 metrics change, and the previous http-metrics crate change all have a similar purpose but don't really make their metrics recording services the same way.

I'm okay merging this as is, but it may just be something to be aware of that as more metrics are added, see if it makes sense to introduce a reusable "metrics registry".