[Helm] Ingester rollout-group collides with Mimir

[lindeskar]

Describe the bug
The loki Helm chart with deploymentMode: Distributed and zoneAwareReplication enabled (default) generates Ingester StatefulSets with labels for rollout-operator. Ex.:

loki/production/helm/loki/templates/ingester/statefulset-ingester-zone-a.yaml

Lines 61 to 62 in 9c96d26

	name: ingester-zone-a
	rollout-group: ingester

The mimir-distributed Helm chart generates StatefulSets with the same label values. Ex. mimir-ingester-zone-a-0:

name: ingester-zone-a
rollout-group: ingester

Deploying the two charts to the same Namespace means rollout-operator will select both Mimir and Loki StatefulSets and get confused about the rollout status. For me one of the Mimir Ingester Pods is constantly being recreated.

To Reproduce
Steps to reproduce the behavior:

1. Deploy mimir-distributed chart with default values
2. Deploy loki chart with distributed-values.yaml

Expected behavior
rollout-operator handles Mimir and Loki Ingesters as separate rollout-groups.

Environment:

• Infrastructure: Kubernetes
• Deployment tool: Helm
• Chart version: 6.6.2

Screenshots, Promtail config, or terminal output
From rollout-operator Pod:

level=info ts=2024-06-07T08:17:49.880277442Z msg="StatefulSet status is reporting all pods ready, but the rollout operator has found some not-Ready pods" statefulset=loki-ingester-zone-a not_ready_pods=mimir-ingester-zone-a-0

[lindeskar]

My suggested fix; use loki.ingesterFullname in the rollout-group label: #13170

[Sadzeih]

I'm having the same issue, would it be possible for a maintainer to review the PR @lindeskar opened?

[nvmforero]

I am also facing this issue. I don't see a workaround since ingester anti-affinity rules are ignored with zoneAwareReplication enabled. Disabling rollout_operator also does not remove the rollout-group: ingester labels from Loki ingester pods.

Edit: Workaround was to use kustomize and change the rollout-group label to loki-ingester for all relevant loki resources.

[cydergoth]

This just cost us a few days and a lot of panic

[slim-bean]

We are discussing this more internally, but generally we don't recommend running different databases in the same namespace. Mainly because we don't do it this way so we don't know where else problems like this might occur (service name collisions? memberlist joining between clusters?!)

[cydergoth]

Unfortunately that doesn't work for us as our security and deployment models assume a namespace per functional area/tenant not per service and it would be hard to change as some of our clusters already have 100s of namespaces.

…

On Wed, Nov 27, 2024, 1:41 PM Ed Welch ***@***.***> wrote: We are discussing this more internally, but generally we don't recommend running different databases in the same namespace. Mainly because we don't do it this way so we don't know where else problems like this might occur (service name collisions? memberlist joining between clusters?!) — Reply to this email directly, view it on GitHub <#13168 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPWAWXDDLNVSZIDDGCB6B32CYN6BAVCNFSM6AAAAABI6KRLOWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBUGY2TKNRSGU> . You are receiving this because you commented.Message ID: ***@***.***>