jsonnet: set GOMAXPROCS on ingesters

[dimitarvdimitrov]

What this PR does

We've been experimenting with GOMAXPROCS on ingesters at GL for the past few months. The test included all production cells and was comparing ingester-zone-a and zone-b (with GOMAXPROCS) to zone-c (without GOMAXPROCS)

This is the summary of results:

• Push & Query outlier pods are significantly lower with GOMAXPROCS. This makes the cell more resilient to bad nodes and to inefficient NUMA node assignment; this was the goal of the experiment
• QueryStream performance under sudden load is worse
• CPU usage is generally slightly lower. Reduced available cores via GOMAXPROCS explains this
• heap might appear slightly increased. However, reduced CPU time (due to fewer cores, see point above) for garbage collection can explain why peak heap can be left to rise before GC kicks in. This is generally safe and should not lead to OOMs for example

To me GOMAXPROCS is a tradeoff between lower stable-state resource usage and resilience to outlier infrastructure and being able to handle sudden usage peaks without increasing latency. With the help of query-scheduler scheduling improvements (prioritizing queries that wouldn't touch slowed-down components) and autoscaling (more frequent ingester vertical autoscaling, querier autoscaling) I think we can accept the increased QueryStream latency.

Which issue(s) this PR fixes or relates to

Fixes #

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[narqo]

Wouldn't this potentially set it to more than what is available for the host/container? E.g. if a VM has only 4 cores, and there is only one container running on the host (kinda line in AWS fargate), we set maxprocs to 7?

[dimitarvdimitrov]

very good point 🙂 And Peter already thought about that

• Clamp GOMAXPROCS when higher than runtime.NumCPU #8201

[narqo]

Oh, I didn't know that 👍

Maybe it's worth it to link to clamping PR from the comment, to make sure this doesn't trigger anyone later

Suggested change

	// double the requests, but with headroom of at least 3 and at most 6 CPU
	// too little headroom can lead to throttling
	// too much headroom can lead to setting the limit above the available CPU cores
	requests + std.max(3, std.min(6, requests)),
	// double the requests, but with headroom of at least 3 and at most 6 CPU
	// too little headroom can lead to throttling
	// too much headroom can lead to setting the limit above the available CPU cores
	// if the value exceeds the number of available CPU cores on the host, mimir caps the maxprocs on application start (ref grafana/mimir#8201)
	requests + std.max(3, std.min(6, requests)),

[narqo]

🔥 works for me, although I'd expand the "headroom" comment.

Also, CI seems to be unhappy.

[dimitarvdimitrov]

🔥 works for me, although I'd expand the "headroom" comment.

i blindly copied from the internal jsonnet, but I realized it's wrong. Corrected now