Forward Source IP to Pods behind ingress-nginx

[page-fault-in-nonpaged-area]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Neither. Bug Perhaps?

NGINX Ingress controller version:
quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0

Found here as per the installation guide at time of writing.

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4", GitCommit:"793658f2d7ca7f064d2bdf606519f9fe1229c381", GitTreeState:"clean", BuildDate:"2017-08-17T17:03:51Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean", BuildDate:"2018-01-04T11:40:06Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

This is currently how my environment is set up:

Internet -> HAProxy vm -> K8S cluster (1 master, 2 worker)

Ingress-nginx is setup according to the bare metal guide. The cluster uses flannel for networking. There is nothing else on it besides a go image that reads client source ip. This image replaces default-backend.

• Cloud provider or hardware configuration: OVH Public Cloud vm's
• OS (e.g. from /etc/os-release): Ubuntu 16.04 LTS
• Kernel (e.g. uname -a):

Linux m1 4.4.0-97-generic #120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Linux w1 4.4.0-97-generic #120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Linux w2 4.4.0-97-generic #120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

• Install tools: kubeadm

What happened:

I have a vm that sits in front of the cluster. Currently it is running HAProxy but NGINX produces the same results (nginx with use-proxy-protocol: "true"). My end goal is to allow the pods associated with the default backend to be able to read the actual source client source IP.

Here's a sample log of what happens when use-proxy-protocol is turned on or off:

10.244.0.0 - [10.244.0.0] - - [10/Jan/2018:23:06:42 +0000] "GET /platform/ping HTTP/1.1" 200 16 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7" 367 0.002 [upstream-default-backend] 10.244.3.101:80 16 0.002 200
10.244.0.0 - [10.244.0.0] - - [10/Jan/2018:23:06:59 +0000] "GET /platform/ping HTTP/1.1" 200 16 "-" "curl/7.54.0" 91 0.074 [upstream-default-backend] 10.244.3.101:80 16 0.074 200
10.244.0.0 - [10.244.0.0] - - [10/Jan/2018:23:09:51 +0000] "PROXY TCP4 127.0.0.1 127.0.0.1 43088 80" 400 173 "-" "-" 0 0.001 [] - - - -
10.244.0.0 - [10.244.0.0] - - [10/Jan/2018:23:09:59 +0000] "PROXY TCP4 127.0.0.1 127.0.0.1 43092 80" 400 173 "-" "-" 0 0.001 [] - - - -
10.244.0.0 - [10.244.0.0] - - [10/Jan/2018:23:10:09 +0000] "PROXY TCP4 127.0.0.1 127.0.0.1 43096 80" 400 173 "-" "-" 0 0.002 [] - - - -
I0110 23:11:42.050971       5 controller.go:211] backend reload required
I0110 23:11:42.054732       5 event.go:218] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", UID:"7539f546-f599-11e7-bee6-fa163e2f1153", APIVersion:"v1", ResourceVersion:"127044", FieldPath:""}): type: 'Normal' reason: 'UPDATE' ConfigMap ingress-nginx/nginx-configuration
I0110 23:11:42.138901       5 controller.go:220] ingress backend successfully reloaded...
127.0.0.1 - [127.0.0.1] - - [10/Jan/2018:23:11:56 +0000] "GET /platform/ping HTTP/1.1" 200 16 "-" "curl/7.47.0" 86 0.003 [upstream-default-backend] 10.244.3.101:80 16 0.003 200
142.xx.xxx.xx - [142.xx.xxx.xx] - - [10/Jan/2018:23:15:50 +0000] "GET / HTTP/1.1" 500 21 "-" "curl/7.47.0" 78 0.020 [upstream-default-backend] 10.244.3.101:80 21 0.020 500
142.xx.xxx.xx - [142.xx.xxx.xx] - - [10/Jan/2018:23:16:02 +0000] "GET /platform/bitcoin HTTP/1.1" 200 45 "-" "curl/7.47.0" 94 0.165 [upstream-default-backend] 10.244.3.101:80 45 0.165 200
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:16:16 +0000] "GET / HTTP/1.1" 500 21 "-" "curl/7.54.0" 78 0.002 [upstream-default-backend] 10.244.3.101:80 21 0.002 500
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:16:30 +0000] "GET /platform/bitcoin HTTP/1.1" 200 45 "-" "curl/7.54.0" 94 0.002 [upstream-default-backend] 10.244.3.101:80 45 0.002 200
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:16:43 +0000] "GET /platform/bitcoin HTTP/1.1" 200 45 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7" 370 0.049 [upstream-default-backend] 10.244.3.101:80 45 0.049 200
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:16:44 +0000] "GET /favicon.ico HTTP/1.1" 404 9 "http://142.xx.xxx.xx/platform/bitcoin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7" 324 0.013 [upstream-default-backend] 10.244.3.101:80 9 0.013 404
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:17:04 +0000] "GET /platform/bitcoin HTTP/1.1" 200 45 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7" 370 0.002 [upstream-default-backend] 10.244.3.101:80 45 0.002 200
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:17:07 +0000] "GET /platform/ping HTTP/1.1" 200 16 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7" 367 0.002 [upstream-default-backend] 10.244.3.101:80 16 0.002 200
216.249.49.20 - [216.249.49.20] - - [10/Jan/2018:23:17:56 +0000] "GET /platform/ping HTTP/1.1" 200 16 "-" "curl/7.54.0" 91 0.002 [upstream-default-backend] 10.244.3.101:80 16 0.002 200
Logs from 1/10/18 10:17 PM to 1/10/18 11:17 PM UTC

142.xx.xxx.xx is the IP of the HAProxy vm

216.249.49.20 is an external IP coming from the university. As you can see, the ingress pod can read external IP's passed from HAProxy with use-proxy-protocol: "true" Just fine. This also works with NGINX (on vm outside cluster) with use-proxy-protocol: "false".

But when I curl the address of HAProxy vm, I get:

demonfuse@Williams-MacBook-Pro ~/N/K/NGINX> curl 142.xx.xxx.xx/platform/ping
pong2 10.244.2.6                

10.244.2.6 is the IP of the ingress pod. I think if I am correct ingress-nginx at this point has the real source IP.

Is there a way to add forward real IP and forward headers to pods behind ingress-nginx via configmaps? From what I can tell here it much of it should be set by default. If not what should I do?

What you expected to happen:
Pods in default backend should read real client source IP. Not the internal IP of the node that the ingress pod sits on.

How to reproduce it (as minimally and precisely as possible):

1. Install ingress-nginx on brand new cluster
2. Redirect traffic from HAProxy / external load balancer to ingress-nginx
3. Go script:

import (
    "github.com/kataras/iris"
    "github.com/kataras/iris/context"
    ...
)

func main() {
	app := iris.New()
        app.Get("/platform/ping", func(ctx context.Context) {
		fmt.Println("connected with " + ctx.RemoteAddr() + "!")
		ctx.WriteString("pong2 " + ctx.RemoteAddr())
	})

        ...

        app.Run(iris.Addr(":80"), iris.WithoutServerError(iris.ErrServerClosed))
}

[aledbf]

@Ascendance to preserve the source IP address in that environment you need to enable proxy protocol (send-proxy in the server) in haproxy and use-proxy-protocol: "true" in the ingress controller configuration configmap. With that change nginx will receive the real IP address.

Just in case ctx.RemoteAddr() will never receive the real IP address, you need to use the X-Forwarded-For header.

[page-fault-in-nonpaged-area]

My HAProxy is already configured for send_proxy as follows:

global
   maxconn 4096
   log 127.0.0.1 local0 notice
   maxconn 2000
   user haproxy
   group haproxy

defaults
   log   global
   mode   http
   retries   3
   option redispatch
   maxconn   2000
   timeout connect 5000
   timeout client  50000
   timeout server  50000

frontend TestServerTest
    bind 142.44.247.248:80
    mode tcp
    default_backend TestServernodes

backend TestServernodes
    mode tcp
    server TestServer01 142.xx.xxx.xx:80 send-proxy

NGINX can see the real external IP of proxied request as per the logs:

216.249.49.41 - [216.249.49.41] - - [11/Jan/2018:20:10:18 +0000] "GET /platform/ping HTTP/1.1" 200 16 "-" "curl/7.54.0" 91 0.301 [upstream-default-backend] 10.244.3.101:80 16 0.301 200
216.249.49.41 - [216.249.49.41] - - [11/Jan/2018:20:10:19 +0000] "GET /platform/ping HTTP/1.1" 200 16 "-" "curl/7.54.0" 91 0.095 [upstream-default-backend] 10.244.3.101:80 16 0.095 200

My ingress configmap is set as follows:

apiVersion: v1
data:
  proxy-set-headers: "ingress-nginx/custom-headers"
  use-proxy-protocol: "true"
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app: ingress-nginx

And my custom header is set as follows:

apiVersion: v1
data:
  X-Forwarded-For: "142.xx.xxx.xx"
kind: ConfigMap
metadata:
  name: custom-headers
  namespace: ingress-nginx

I have tried a combination of IP's for X-Forwarded-For including the IP of the HAProxy vm, the internal IP of the ingress pod, and the external IP associated with the ingress-nginx service.

For all cases curling the HAProxy vm still returns internal IP of 10.244.2.6

@aledbf What should I put for X-Forwarded-For ?

[page-fault-in-nonpaged-area]

I fixed it! The problem was was within the iris web framework.

[naisanza]

@Ascendance how did you fix it?

[gitnik]

I am having the same issue. Default install on a cluster running Kubernetes 1.10 and the source IP is not being passed through.

Here's what I get when I run a simple print_r($_SERVER) on the php:7.2-apache image:

Array
(
    [HTTP_HOST] => foo.bar.ooo
    [HTTP_CONNECTION] => close
    [HTTP_X_REAL_IP] => 10.244.2.0
    [HTTP_X_FORWARDED_FOR] => 10.244.2.0
    [HTTP_X_FORWARDED_HOST] => foo.bar.ooo
    [HTTP_X_FORWARDED_PORT] => 443
    [HTTP_X_FORWARDED_PROTO] => https
    [HTTP_X_ORIGINAL_URI] => /
    [HTTP_X_SCHEME] => https
    [HTTP_PRAGMA] => no-cache
    [HTTP_CACHE_CONTROL] => no-cache
    [HTTP_UPGRADE_INSECURE_REQUESTS] => 1
    [HTTP_DNT] => 1
    [HTTP_USER_AGENT] => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
    [HTTP_ACCEPT] => text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
    [HTTP_ACCEPT_ENCODING] => gzip, deflate, br
    [HTTP_ACCEPT_LANGUAGE] => en,de;q=0.9,es;q=0.8,en-DE;q=0.7,en-US;q=0.6
    [PATH] => /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    [SERVER_SIGNATURE] => <address>Apache/2.4.25 (Debian) Server at foo.bar.ooo Port 80</address>

    [SERVER_SOFTWARE] => Apache/2.4.25 (Debian)
    [SERVER_NAME] => foo.bar.ooo
    [SERVER_ADDR] => 10.244.3.55
    [SERVER_PORT] => 80
    [REMOTE_ADDR] => 10.244.3.244
    [DOCUMENT_ROOT] => /var/www/html
    [REQUEST_SCHEME] => http
    [CONTEXT_PREFIX] => 
    [CONTEXT_DOCUMENT_ROOT] => /var/www/html
    [SERVER_ADMIN] => webmaster@localhost
    [SCRIPT_FILENAME] => /var/www/html/index.php
    [REMOTE_PORT] => 50560
    [GATEWAY_INTERFACE] => CGI/1.1
    [SERVER_PROTOCOL] => HTTP/1.1
    [REQUEST_METHOD] => GET
    [QUERY_STRING] => 
    [REQUEST_URI] => /
    [SCRIPT_NAME] => /index.php
    [PHP_SELF] => /index.php
    [REQUEST_TIME_FLOAT] => 1530804810.715
    [REQUEST_TIME] => 1530804810
    [argv] => Array
        (
        )

    [argc] => 0
)

As you can see there's no "source" IP being passed in any way. Anything else I cloud try? Is there some more info I can provide?

[page-fault-in-nonpaged-area]

Not sure in PHP but in go and iris the framework decided to use fast source IP lookup(TL;DR whoever is in front of it) instead of actually inspecting the real IP attached to class ctx headers. You might want to look at how PHP handles REMOTE_ADDR.

With go and Iris I was able to get the real IP. So I’m sure it’s somewhere with PHP.