Exemplar API: For now in sidecar and querier

[yeya24]

As mentioned in today's public meeting, it would be good if we can support exemplars in Thanos.

Blocked by the upstream WIP pr.

As there is no design to persist exemplars right now, we can start from supporting Thanos sidecar to query the /api/v1/query_exemplars API from Prometheus.

Describe the solution you'd like

1. Add ExemplarQuerier interface and /api/v1/query_exemplars API to thanos querier.
2. Add one more gRPC API to query exemplars.
3. Implement this gRPC API in the proxy store, which performs fanout exemplar queries.
4. Implement this gRPC API in Thanos sidecar, which simply calls the same HTTP API from Prometheus

[goku321]

I can work on this if that's okay with some guidance.

[bwplotka]

@goku321 go for it. We are happy to guide you 🤗

Feel free to propose this on our new Contributor Office Hours if you want to meet & collab ❤️ (will be announced wider today): https://docs.google.com/document/d/137XnxfOT2p1NcNUq6NWZjwmtlSdA6Wyti86Pd6cyQhs/edit#heading=h.cifspg8gw2m5

[bwplotka]

Added some detailed steps here: #3345 (comment)

Those also applies to Exemplars (: @goku321

[goku321]

Just a heads up. work is in progress, will create a PR soon.

[bwplotka]

Amazing @goku321 ❤️ Can't wait!

[yeya24]

Added some detailed steps here: #3345 (comment)

Those also applies to Exemplars (: @goku321

The current design is that exemplars are only in memory, so only Sidecar supports it now. I feel it is possible that exemplars can be supported via remote write so the TSDB store might need to support this, too. If we need to persist it in the future, then store gateway might need to have this as well.

If we choose the proxy way like rules API, what's the benefit of this over adding it to the general store gRPC API?

[bwplotka]

store gRPC API?

What exactly do you mean? Adding another method? Adding another field to Series response (which might make less sense as there is literally Examplars API (another endpoint).)?

If it's another method that it's not a big difference for gRPC if it's part of same API or different.

via remote write so the TSDB store might need to support this, too.

While definitely, we want to push it via remote write some day (or something along those lines), the current plan is to don't extend TSDB format with this. In that case it will be stored in different way so might be the case that some Thanos component will expose Exemplars API but not store API.

I believe we could start as a separate API/method as this allows us flexibility. Nothing bad in store gateway exposing both Examplars API and Series API, no? (:

[yeya24]

I believe we could start as a separate API/method as this allows us flexibility. Nothing bad in store gateway exposing both Examplars API and Series API, no? (:

Yes, that makes sense.

[Debanitrkl]

@bwplotka I would like to work on this, also previously I have worked with gRPC. Could help me with where I should begin with I was going through the contributor office hours but I think it's not updated since few days.

[goku321]

Hi @Debanitrkl I know I'm taking a bit of time on this but I am very close to finish this one 😅

[idoqo]

Hi @bwplotka, since @goku321 is already working on this, is the related LFX mentorship task (Thanos: gRPC Exemplars API) still valid?