Lower PriorityClass for Flux controller Pods

[balonik]

As a follow up for #3800 where Flux controller Pods got priorityClass=system-cluster-critical I would like to suggest the default to be lowered.

system-cluster-critical seems to be too high for a GitOps controller, which is not really "that" important for running workloads on clusters.
In low resources scenarios it is possible that more important workloads are evicted, like:

• Ingress Controller
• Admission controllers
• Public Cloud controllers (LoadBalancers, Storage, ...)
• Workloads behind Mutating webhooks
• and more

I don't really care if Flux is running if my Ingress Controller is down :)

I know I can change it in Flux, but I think the default should be a reasonable value based on general importance of the tool.
I know I can increase or set priorityClass=system-cluster-critical for other Pods, but at what point I would need to run all of them with such high priority?

If there will be an agreement to lower the priority I can work on adjusting the Helm Chart accordingly.

[stefanprodan]

I can work on adjusting the Helm Chart accordingly.

We don't offer a Helm chart for Flux under the FluxCD org.

I don't really care if Flux is running if my Ingress Controller is down :)

I do 😄

Feel free to change this on your cluster, I'm personal not for undoing it. Others maintainers may have a different opinion on this, you may want to join the dev meetings and convince them.

[kingdonb]

It is important that Flux runs else GitOps cannot happen, if we can support the users with docs about how to adjust the priorityClass for their workloads as well as Flux I'm for making that change in the docs. Do you have a patch for this or do you need help creating one @balonik - perhaps we can convert this issue into a docs one, WDYT?