nginx-ingress-controller/GCP - Inconsistent X-Forwarded-For

[roobert]

Hi, when using the nginx-ingress-controller on GCP I'm getting inconsistent X-Forwarded-For IP, I either see the real client IP (which is what I'd expect) or a local 10.0.0.0/8 address.

My test setup is as follows:

image: nginx-ingress-controller:0.9.0-beta.8

echoserver configuration:

Ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: echoserver
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
    - secretName: echoserver-tls
      hosts:
        - echoserver.example.com
  rules:
    - host: echoserver.example.com
      http:
        paths:
        - path: /
          backend:
            serviceName: echoserver
            servicePort: 80

Service:

apiVersion: v1
kind: Service
metadata:
  name: echoserver
  annotations:
    acme/certificate: echoserver.example.com
    acme/secretName: echoserver-tls
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: echoserver

Deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: echoserver
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: echoserver
    spec:
      containers:
      - image: gcr.io/google_containers/echoserver:1.0
        imagePullPolicy: Always
        name: echoserver
        ports:
        - containerPort: 8080

nginx-ingress-controller configuration:

nginx-configmap settings:

  proxy-connect-timeout: "15"
  proxy-read-timeout: "600"
  proxy-send-timeout: "600"
  hsts-include-subdomains: "false"
  body-size: "64m"
  server-name-hash-bucket-size: "256"
  enable-vts-status: "true"
  large-client-header-buffers: "8 32k"
  use-proxy-protocol: "true"

Deployed server block:

   server {
        server_name echoserver.example.com;
        listen 80 proxy_protocol;
        listen [::]:80 proxy_protocol;
        set $proxy_upstream_name "-";

        listen 442 proxy_protocol ssl http2;
        listen [::]:442 proxy_protocol  ssl http2;
        # PEM sha: eeb3b614f807f9b4898e2251a696a17b990e4a87
        ssl_certificate                         /ingress-controller/ssl/default-echoserver-tls.pem;
        ssl_certificate_key                     /ingress-controller/ssl/default-echoserver-tls.pem;

        more_set_headers                        "Strict-Transport-Security: max-age=15724800;";

        vhost_traffic_status_filter_by_set_key $geoip_country_code country::$server_name;

        # enforce ssl on server side
        if ($pass_access_scheme = http) {
            return 301 https://$best_http_host$request_uri;
        }
        location / {
            set $proxy_upstream_name "default-echoserver-80";
            port_in_redirect off;

            client_max_body_size                    "1m";

            proxy_set_header Host                   $best_http_host;

            # Pass the extracted client certificate to the backend

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Real-IP              $the_real_ip;
            proxy_set_header X-Forwarded-For        $the_x_forwarded_for;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            proxy_set_header X-Original-URI         $request_uri;
            proxy_set_header X-Scheme               $pass_access_scheme;

            proxy_set_header X-Forwarded-Host       $best_http_host;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers

            proxy_connect_timeout                   15s;
            proxy_send_timeout                      600s;
            proxy_read_timeout                      600s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            proxy_pass http://default-echoserver-80;
        }

    }

Here's my test:

> while sleep 0.5; do curl https://echoserver.example.com/ -s | grep X-Forwarded; echo '=====' ; done

=====
X-Forwarded-For=10.195.0.4
X-Forwarded-Host=echoserver.example.com
X-Forwarded-Port=443
X-Forwarded-Proto=https
=====
X-Forwarded-For=10.195.0.8
X-Forwarded-Host=echoserver.example.com
X-Forwarded-Port=443
X-Forwarded-Proto=https
=====
X-Forwarded-For=10.195.0.8
X-Forwarded-Host=echoserver.example.com
X-Forwarded-Port=443
X-Forwarded-Proto=https
=====
X-Forwarded-For=80.x.x.x
X-Forwarded-Host=echoserver.example.com
X-Forwarded-Port=443
X-Forwarded-Proto=https
=====
X-Forwarded-For=10.195.0.2
X-Forwarded-Host=echoserver.example.com
X-Forwarded-Port=443
X-Forwarded-Proto=https

I think this must be due to the following where I think the scheme must be being incorrectly determined for some requests?

    map $pass_access_scheme $the_x_forwarded_for {
       default           $remote_addr;
       https             $proxy_protocol_addr;
    }

Any help appreciated.

[roobert]

Fixed this by setting the external-traffic annotation on the nginx-ingress-controller service:

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress-controller
  namespace: system-ingress
  annotations:
    service.beta.kubernetes.io/external-traffic: OnlyLocal
spec:
  type: LoadBalancer
  loadBalancerIP: x.x.x.x
  ports:
  - port: 80
    name: http
  - port: 443
    name: https
  selector:
    app: nginx-ingress-controller

[aledbf]

@roobert please update the image to quay.io/aledbf/nginx-ingress-controller:0.148
Fix in #890