Rollout clobbers pre-existing ALB annotations

[alkalinecoffee]

Summary

We have pre-existing annotations in place for our ALB listener rules, but argo appears to be clobbering those annotations so they never actually take effect.

Our service requires the use of sticky sessions. For enabling stickiness on targets, this is done by first setting this ingress annotation:

alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=60

But since both the canary and stable target groups are routed by the same listener rule, we need to also enable target group stickiness at the rule level by adding setting the TargetGroupStickinessConfig in the following annotation (as described here):

alb.ingress.kubernetes.io/actions.xvp-ping-api-dev-stable:
  "{\"ForwardConfig\":{\"TargetGroupStickinessConfig\":{\"DurationSeconds\":300,\"Enabled\":true},\"TargetGroups\":[{\"ServiceName\":\"xvp-ping-api-dev-canary\",\"ServicePort\":8080,\"Weight\":50},{\"ServiceName\":\"xvp-ping-api-dev-stable\",\"ServicePort\":8080,\"Weight\":50}]},\"Type\":\"forward\"}",

Without specifying the TargetGroupStickinessConfig in our annotation, we see the following error from the aws-load-balancer-controller when it tries to update the ALB rules:

InvalidLoadBalancerAction: You must enable group stickiness on a rule if you enabled target stickiness on one of its target groups

After some debugging, we can see in the controller logs that its trying to push the following payload to the ELB API:

{
  "type": "forward",
  "forwardConfig": {
    "targetGroups": [
      {
        "targetGroupARN": { "$ref": "#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/xvp-ping-api-dev/xvp-ping-api-dev-alb-xvp-ping-api-dev-stable:8080/status/targetGroupARN" },
        "weight": 100
      },
      {
        "targetGroupARN": { "$ref": "#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/xvp-ping-api-dev/xvp-ping-api-dev-alb-xvp-ping-api-dev-canary:8080/status/targetGroupARN" },
        "weight": 0
      }
    ]
  }
}

We can see in the above output that the TargetGroupStickinessConfig configuration we set in our annotation is missing. As it stands now, we're unable to enable sticky sessions with argo.

If I'm understanding the code here correctly, any pre-existing annotations are not considered when building the listener rule's forwardConfig. This value is explicitly set with target group values and nothing else.

Should this code be updated to more gracefully handle annotations with pre-existing values so they are not lost? Or perhaps a sort of template may be provided to use as a basis for building the ingress annotation?

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.