add pageserver SLO for startup performance: tenant load & time-to-active

[problame]

We should have a pageserver-level SLO for the time it takes until all tenants of the pageserver have reached state "Active" or "Broken".

This can be broken down into two metrics:

1. one gauge metric that is 1 exactly while the tenant loads initiated by tenant::mgr::init are going on
2. a global histogram that tracks time-to-active

• we already log this in feat: log how long tenant activation takes #4080 , just need to add the histogram

What to do with the metrics:

• We can then multiply the histogram with the gauge and alert on outliers.
• Also, we can alert on the contiguous 1-time of the gauge not exceeding a threshold

Related:

• pageserver: expose prometheus metrics for startup time #4893

[vadim2404]

Does this metric depend on tenant size or any other thing? Because for SLO, it makes sense to remove the "noise" first.

[problame]

Does this metric depend on tenant size or any other thing?

Suspected bottlenecks right now:

• get remote index_part.json's
  • network latency is dominant here
  • concurrency limiter says hello as well
• building the layer maps
  • this is CPU-bound

Regardless, I think we should aspire to something like 5 seconds after restart, all tenants are "Active" or "Broken".

I think this is achievable.

Because for SLO, it makes sense to remove the "noise" first.

Obviously, we won't add alerts which we know we'll break.
We'll add the metric, create a dashboard, measure, understand, fix first.

[vadim2404]

Regardless, I think we should aspire to something like 5 seconds after restart, all tenants are "Active" or "Broken".

It sounds relevant

[problame]

Relevant to what? To your

Does this metric depend on tenant size or any other thing? Because for SLO, it makes sense to remove the "noise" first.

or generally relevant?

[vadim2404]

generally, to start with it (about the SLO)

[koivunej]

This is related to #4025.

[koivunej]

I am eager to see the distribution of these activations, then I can comment more on if that makes sense as an SLO.

[problame]

Edited the description to include alerting on contiguous 1-time of the gauge.

[koivunej]

It's a really slow CI day and I am eager to test unrelated code in staging. Might as well hack these two because I created the initial load time watching already in e879d6c.

Also, we can alert on the contiguous 1-time of the gauge not exceeding a threshold

Can this be implemented in promql?

Later remembered: Tenant activations which happen as a result of creation will be instant, because there is no other load. I at least wouldn't want them on the same histogram because then it will say "some large percentile is very fast", even if on restart init activations would take 130s.

[koivunej]

Closed wrong issue.

[jcsp]

I went ahead and added the metrics for this in #4893