ALB Invalid Listener takes down entire gateway resource

[andreasthuen]

Hi, we are using Azure Application Gateway for Containers as a load balancer for the workloads running in our AKS clusters.

We are running multiple different endpoints/domains in the ALB and they all terminate in the Gateway.
Each workload uses its own certificate and is set up as listeners in the Gateway resource:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: dev-alb-gateway
  namespace: alb-system
  annotations:
    alb.networking.azure.io/alb-namespace: alb-system
    alb.networking.azure.io/alb-name: dev-alb
spec:
  gatewayClassName: azure-alb-external
  listeners:
  - name: https-listener1
    port: 443
    protocol: HTTPS
    hostname: domain1.contoso.com
    allowedRoutes:
      namespaces:
        from: All
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        group: ""
        name: domain1-tls-certificate
        namespace: app-1-ns
  - name: https-listener2
    port: 443
    protocol: HTTPS
    hostname: domain2.contoso.com
    allowedRoutes:
      namespaces:
        from: All
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        group: ""
        name: domain2-tls-certificate
        namespace: app-2-ns

We are using a ReferenceGrant to allow the ALB to use k8s secrets from a different namespace, and this works as fine.

However, if f.ex. one of the k8s secrets does not exist in the referenced namespace, all listeners will stop working, not just the one with the invalid certificate.

We would of course expect the misconfigured listener to stop working, but we rather have it that not all other listeners also stopped working.

We are running the ALB helm chart version: 1.3.7 and our AKS clusters are version: 1.31.2

[andreasthuen]

We have now also tested this using ALB controller helm chart version 1.3.9 and the issue still occurs.

[JackStromberg]

Acklowedging the ask and we are currently investigating.

Thank you,
Jack

[AndersRunningen]

any updates on this @JackStromberg?

If you need anyone to test something we can always do some testing in our test-environment.

Anders

[andreasthuen]

Any news here @JackStromberg ?

[JackStromberg]

If you need complete isolation between different configurations, we'd recommend separate deployments via different Gateway resources. We evaluated this internally and there are tradeoffs in partial reconciliation on a given resource. At this time, this won't be something that is changed.

[ozen]

So if we want Azure ALB to be robust, we need to create 1 gateway per listener?
And this will incur more cost for us, right?

[JackStromberg]

You'd have a unique frontend per gateway resource. You do not need additional AGC or association resources.