CA: implement prioritization/rate limitting

[howardjohn]

Depends on #298

Currently we have 3 sources of CA calls:

1. On demand: we have a request incoming and need a cert (right now!).
2. Background refresh: we have a cert and its near expiration, we need to refresh. By default this is 24hr lifetime, start refresh at 12hr
3. Prewarming. New workload (or ztunnel just started), we want to preload the cert to reduce latency on first call (avoid "cold starts")

In order of important, this likely looks like: On Demand >>>> Background refresh when really close to expiration > Prewarming == Background refresh. Could be simplified to just "on demand is top priority".

Additionally, CA requests are expensive. 255 concurrent requests to 1 istiod would almost certainly overwhelm it (in CPU cost); other CAs may have different constraints. We are also not the only client. We need a sensible strategy that trades off not killing the CA with getting all the certs when we want.

At the very least, we should have a way to prioritize on demand requests. This could be something simple like adding some delay in prewarming, or have some priority queue.

[bleggett]

Dumb q - do we (or do we intend to) support the SDS protocol so that ztunnel could play nice with a compatible replacement workload CA/SDS, like the rest of Istio does today?

[howardjohn]

I started #251 to define these integration interfaces. I hadn't put SDS in there but its plausible.

I will say in general the idea has been to have a few common integration points. For example, instead of 10 telemetry providers just use OTEL, which itself supports the kitchen sink of providers.

There is less of a standard around CAs compared to telemetry though.

[keithmattix]

I've seen some semblance of this in the code; has this been implemented?

[howardjohn]

yeah I think this is done. thanks!