Custom-Headers annotation not working with 1.10.1 (changes are visible in git tag for 1.10.1)

[BernhardGruen]

What happened:

According to the tag controller-1.10.1 the file nginx.tmpl should support the Custom-Headers functionality implemented in #9742. But it does not.

Link to nginx.tmpl in tag controller-1.10.1: https://github.com/kubernetes/ingress-nginx/blob/controller-v1.10.1/rootfs/etc/nginx/template/nginx.tmpl

When comparing with the release-1.10 branch I see that the support is not included there: https://github.com/kubernetes/ingress-nginx/blob/release-1.10/rootfs/etc/nginx/template/nginx.tmpl

Therefore it seems either the tagging process was not working correctly (i.e. tagged based on main) or branch release-1.10 was not updated and was then used as base for the image build process.

What you expected to happen:

I expect that I am able to set custom headers. But as the function is not available in the file nginx.tmpl it basically can't work at all.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

1.10.1

Kubernetes version (use kubectl version):

1.28.6

Environment:

does not matter in that case, I believe.

How to reproduce this issue:

Add global-allowed-response-headers: Content-Type,X-no-meaning to the config map of the ingress controller.

Add an annotation nginx.ingress.kubernetes.io/custom-headers: custom-headers-configmap to an ingress.

Create said configmap in the same namespace (using Content-Type: text/json as data).

Test if the header has changed from text/html to text/json.

Anything else we need to know:

I believe the error happened while tagging controller-1.10.1 as it seems to be based of main instead of branch release-1.10.
The release notes of the controller (https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.10.1) also do not reference #9742.

Maybe all this only happened because the documentation https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-headers already contains an annotation that will only be present with the next minor release 1.11.0. But I really don't know.

[longwuyuan]

/triage accepted
/priority important-soon

Yes true.

Line-391 in main

ingress-nginx/rootfs/etc/nginx/template/nginx.tmpl

Line 391 in 9b4b914

# Custom headers for response

seems different from same line-391 in branch release-1-10

ingress-nginx/rootfs/etc/nginx/template/nginx.tmpl

Line 391 in 4bf5b5f

{{ if or $cfg.DisableAccessLog $cfg.DisableHTTPAccessLog }}

@Gacko maybe the change was not cherry-picked. cc @strongjz @rikatz @tao12345666333

[longwuyuan]

@cgroschupp can you confirm test this please

[adrianmoisey]

According to the tag controller-1.10.1 the file nginx.tmpl should support the Custom-Headers functionality implemented in #9742. But it does not.

I'm curious about this.

I've looked at the release for 1.10.1, and it doesn't seem to imply that #9742 is bundled with it.

Specifically if I visit https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.10.1, I don't see the characters "9742" on the page, nor do I see "annotation"

What are you seeing that makes you say that the feature is in 1.10.1?

[BernhardGruen]

According to that controller tag the controller image should contain that feature.
https://github.com/kubernetes/ingress-nginx/blob/controller-v1.10.1/rootfs/etc/nginx/template/nginx.tmpl

It seems the controller was not tagged based on release-1.10 branch but instead on main.

And also the documentation mentions custom headers. Therefore I thought that should work.

[longwuyuan]

/assign

Will check and follow up. Maybe the cherrypicking to the branch did not happen

[adrianmoisey]

According to that controller tag the controller image should contain that feature. https://github.com/kubernetes/ingress-nginx/blob/controller-v1.10.1/rootfs/etc/nginx/template/nginx.tmpl

Oh right, I thought you meant that the tag's description (well, release) contains the feature.
Yeah, something is up with the 1.10.1 tag and branch, it looks like they diverged for some reason.

[adrianmoisey]

The history of rootfs/etc/nginx/template/nginx.tmpl for https://github.com/kubernetes/ingress-nginx/blob/release-1.10/rootfs/etc/nginx/template/nginx.tmpl vs https://github.com/kubernetes/ingress-nginx/blob/controller-v1.10.1/rootfs/etc/nginx/template/nginx.tmpl is definitely different.

[codypperson]

I also experienced a problem related to skew between the nginx.tmpl on GitHub vs the container image. I typically grab the nginx.tmpl from the tag on GitHub when updating my manifest to deploy, but after deploying to my staging environment the controller was unable to start due to templating errors: can't evaluate field GeoIP2AutoReloadMinutes in type config.Configuration. I then figured out there was a large drift in the nginx.tmpl from GitHub, vs the nginx.tmpl from the actual container image, which is what I ended up using.

Here is the difference between the two:

cperson@THINKPAD:~/Downloads/ingress-nginx-v1.10.1$ curl -sSL https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.10.1/rootfs/etc/nginx/template/nginx.tmpl > nginx_from_github_tag_v1.10.1.tmpl
cperson@THINKPAD:~/Downloads/ingress-nginx-v1.10.1$ docker run --rm registry.k8s.io/ingress-nginx/controller:v1.10.1 cat /etc/nginx/template/nginx.tmpl > nginx_from_image_tag_v1.10.1.tmpl
cperson@THINKPAD:~/Downloads/ingress-nginx-v1.10.1$ diff nginx_from_github_tag_v1.10.1.tmpl nginx_from_image_tag_v1.10.1.tmpl 
175,177d174
<         {{ if (gt $cfg.GeoIP2AutoReloadMinutes 0) }}
<         auto_reload {{ $cfg.GeoIP2AutoReloadMinutes }}m;
<         {{ end }}
189,191d185
<         {{ if (gt $cfg.GeoIP2AutoReloadMinutes 0) }}
<         auto_reload {{ $cfg.GeoIP2AutoReloadMinutes }}m;
<         {{ end }}
203,205d196
<         {{ if (gt $cfg.GeoIP2AutoReloadMinutes 0) }}
<         auto_reload {{ $cfg.GeoIP2AutoReloadMinutes }}m;
<         {{ end }}
229,231d219
<         {{ if (gt $cfg.GeoIP2AutoReloadMinutes 0) }}
<         auto_reload {{ $cfg.GeoIP2AutoReloadMinutes }}m;
<         {{ end }}
255,257d242
<         {{ if (gt $cfg.GeoIP2AutoReloadMinutes 0) }}
<         auto_reload {{ $cfg.GeoIP2AutoReloadMinutes }}m;
<         {{ end }}
265,267d249
<         {{ if (gt $cfg.GeoIP2AutoReloadMinutes 0) }}
<         auto_reload {{ $cfg.GeoIP2AutoReloadMinutes }}m;
<         {{ end }}
275,277d256
<         {{ if (gt $cfg.GeoIP2AutoReloadMinutes 0) }}
<         auto_reload {{ $cfg.GeoIP2AutoReloadMinutes }}m;
<         {{ end }}
292,294d270
<         {{ if (gt $cfg.GeoIP2AutoReloadMinutes 0) }}
<         auto_reload {{ $cfg.GeoIP2AutoReloadMinutes }}m;
<         {{ end }}
334,337d309
<     {{ if gt $cfg.GRPCBufferSizeKb 0 }}
<     grpc_buffer_size {{ $cfg.GRPCBufferSizeKb }}k;
<     {{ end }}
< 
1490,1496d1461
<             {{ end }}
< 
<             {{ if $location.CustomHeaders }}
<             # Custom Response Headers
<             {{ range $k, $v := $location.CustomHeaders.Headers }}
<             more_set_headers {{ printf "%s: %s" $k $v | escapeLiteralDollar | quote }};
<             {{ end }}
cperson@THINKPAD:~/Downloads/ingress-nginx-v1.10.1$ 

[gorge511]

We have the same issue with not matching template from 1.10.1 git tag template with 1.10.1 build.

Due to some specific configuration we need custom template in our deployment, so I updated the docker image to version 1.10.1 and updated the template accordingly from the version here on GitHub tag 1.10.1 and now the ingress controller is not able to start because of missing variables to the template.

I0513 12:11:35.449450       8 controller.go:190] "Configuration changes detected, backend reload required"
E0513 12:11:35.452153       8 controller.go:205] Unexpected failure reloading the backend:
template: nginx.tmpl:334:17: executing "nginx.tmpl" at <$cfg.GRPCBufferSizeKb>: can't evaluate field GRPCBufferSizeKb in type config.Configuration
E0513 12:11:35.452208       8 queue.go:131] "requeuing" err="template: nginx.tmpl:334:17: executing \"nginx.tmpl\" at <$cfg.GRPCBufferSizeKb>: can't evaluate field GRPCBufferSizeKb in type config.Configuration" key="luft-system/kube-state-metrics-cgpjj"
I0513 12:11:35.452339       8 event.go:364] Event(v1.ObjectReference{Kind:"Pod", Namespace:"luft-system", Name:"ingress-nginx-controller-7c4986467-v8w6c", UID:"b38b633e-482b-4609-a327-eb451cc0c1c7", APIVersion:"v1", ResourceVersion:"531509399", FieldPath:""}): type: 'Warning' reason: 'RELOAD' Error reloading NGINX: template: nginx.tmpl:334:17: executing "nginx.tmpl" at <$cfg.GRPCBufferSizeKb>: can't evaluate field GRPCBufferSizeKb in type config.Configuration
I0513 12:11:38.686242       8 sigterm.go:36] "Received SIGTERM, shutting down"

I'm going to update our custom template accordingly to the version baked in the image and we should be fine for now. However, it would be nice to fix this discrepancy so others don't have the same problem.

[Gacko]

As already pointed out here, this issue is caused by the Git tag being set on the wrong branch. The release-1.10 branch the image was built and the release was forged from does not contain this feature, as we are not releasing new features but only patches on an existing minor release.

/close

[k8s-ci-robot]

@Gacko: Closing this issue.

In response to this:

As already pointed out here, this issue is caused by the Git tag being set on the wrong branch. The release-1.10 branch the image was built and the release was forged from does not contain this feature, as we are not releasing new features but only patches on an existing minor release.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[BernhardGruen]

@Gacko I fully understand that this feature was not planned to be integrated into 1.10 as I also wrote at the end of my original message. But still there is a bug in the tagging process that can and will eventually lead to confusion.
Also the official documentation https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-headers already mentions custom-headers. Maybe the documentation should be based on the current stable branch instead of an unreleased branch.

[Gacko]

I think the tagging issue only affects v1.10.1, we had some issues with forging that release.

Regarding the docs: It has been like this ever since. To fix this, we would need to have docs per tag, atm docs are being published directly from main branch. @longwuyuan, can you come up with a solution for that?

[satyamz]

@BernhardGruen Was this feature eventually included? I am trying it with 1.11.3 and it isn't working for me.

[pierluigilenoci]

@satyamz This feature was released in version 1.11.0 of the controller, so it should be present.
https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v1.11.0
It is possible that it does not work in your case due to some bug, but it is undoubtedly present.
So, I think you should open a new issue explaining how to reproduce the error so that someone can correct this possible malfunction.

[satyamz]

@pierluigilenoci thanks. Yes, I found it in v1.11.0. However, its still not working as expected. I am checking my configs to see if this is really a bug on my side.

[satyamz]

I am getting following in the logs:

E1103 11:29:05.234098       7 annotations.go:216] "error reading Ingress annotation" err="location denied, reason: unable to find configMap \"custom-headers-map\"" name="CustomHeaders" ingress="default/sample-ingress"

I checked clusterrole and role they do have permission for getting configMap. Also, config itself exists in the default namespace.

[rikatz]

Is the namespace of the configmap the same of ingress resource?

[satyamz]

@rikatz Yes. Both configMap and ingress are in the same namespace.

[satyamz]

I filed a bug here: #12287 I included all information there. Please let me know if I can help you with more information on this.

[longwuyuan]

I don't see there is a bug here.
I got the same error message about location denied.
Then I spec'd the cm name prefixed with the namespace like "default/$cmname" and i got a response code of 200

% helm -n ingress-nginx ls
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS      CHART                   APP VERSION
ingress-nginx   ingress-nginx   2               2024-11-04 12:04:04.995992157 +0530 IST deployed    ingress-nginx-4.11.3    1.11.3     
[~] 
% k describe cm customheaders 
Name:         customheaders
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
X-Using-Nginx-Controller:
----
true
Name:
----
Satyam
X-Different-Name:
----
true

BinaryData
====

Events:  <none>
[~] 
% k describe ingress httpd 
Name:             httpd
Labels:           <none>
Namespace:        default
Address:          192.168.58.2
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host                Path  Backends
  ----                ----  --------
  httpd.mydomain.com  
                      /   httpd:80 (10.244.0.63:80)
Annotations:          nginx.ingress.kubernetes.io/custom-headers: default/customheaders
Events:
  Type    Reason  Age                   From                      Message
  ----    ------  ----                  ----                      -------
  Normal  Sync    115s (x3 over 5m36s)  nginx-ingress-controller  Scheduled for sync
[~] 
% curl httpd.mydomain.com --resolve httpd.mydomain.com:80:`minikube ip` -i
HTTP/1.1 200 OK
Date: Mon, 04 Nov 2024 07:10:24 GMT
Content-Type: text/html
Content-Length: 45
Connection: keep-alive
Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT
ETag: "2d-432a5e4a73a80"
Accept-Ranges: bytes
Name: Satyam
X-Different-Name: true
X-Using-Nginx-Controller: true

<html><body><h1>It works!</h1></body></html>

LOGS
% 
I1104 07:00:26.566752       7 nginx.go:337] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"                [25/63]
I1104 07:00:26.567020       7 controller.go:193] "Configuration changes detected, backend reload required"                                                                                    
I1104 07:00:26.568500       7 status.go:85] "New leader elected" identity="ingress-nginx-controller-7d978dd4c-qbzrg"                                                                          
I1104 07:00:26.595566       7 controller.go:213] "Backend successfully reloaded"                                                                                                              
I1104 07:00:26.595642       7 controller.go:224] "Initial sync, sleeping for 1 second"                                                                                                        
I1104 07:00:26.595678       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7d978dd4c-kjhtj", UID:"7975097b-b2c6-45a9-9a9c-aee4
5f8ed10f", APIVersion:"v1", ResourceVersion:"167040", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration                                 
I1104 07:01:21.199965       7 leaderelection.go:268] successfully acquired lease ingress-nginx/ingress-nginx-leader                                                                           
I1104 07:01:21.200015       7 status.go:85] "New leader elected" identity="ingress-nginx-controller-7d978dd4c-kjhtj"                                                                          
E1104 07:01:24.676070       7 annotations.go:219] "error reading Ingress annotation" err="location denied, reason: unable to find configMap \"default/customheaders\"" name="CustomHeaders" in
gress="default/httpd"                                                                                                                                                                         
I1104 07:01:24.691561       7 admission.go:149] processed ingress via admission controller {testedIngressLength:2 testedIngressTime:0.015s renderingIngressLength:2 renderingIngressTime:0.001
s admissionTime:0.016s testedConfigurationSize:20.0kB}                                                                                                                                        
I1104 07:01:24.691591       7 main.go:107] "successfully validated configuration, accepting" ingress="default/httpd"                                                                          
I1104 07:01:24.697160       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"httpd", UID:"8211227b-e7c7-4495-a6ea-b05552c037ac", APIVersion:"networking.k8s
.io/v1", ResourceVersion:"167125", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync                                                                                           
E1104 07:01:24.697227       7 annotations.go:219] "error reading Ingress annotation" err="location denied, reason: unable to find configMap \"default/customheaders\"" name="CustomHeaders" in
gress="default/httpd"                                                                                                                                                                         
I1104 07:01:24.697483       7 controller.go:193] "Configuration changes detected, backend reload required"                                                                                    
I1104 07:01:24.725666       7 controller.go:213] "Backend successfully reloaded"                                                                                                              
I1104 07:01:24.725816       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7d978dd4c-kjhtj", UID:"7975097b-b2c6-45a9-9a9c-aee4
5f8ed10f", APIVersion:"v1", ResourceVersion:"167040", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration                                 
I1104 07:03:01.198258       7 controller.go:193] "Configuration changes detected, backend reload required"                                                                                    
I1104 07:03:01.225498       7 controller.go:213] "Backend successfully reloaded"                                                                                                              
I1104 07:03:01.225642       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7d978dd4c-kjhtj", UID:"7975097b-b2c6-45a9-9a9c-aee4
5f8ed10f", APIVersion:"v1", ResourceVersion:"167040", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration                                 
E1104 07:03:21.219327       7 annotations.go:219] "error reading Ingress annotation" err="location denied, reason: unable to find configMap \"customheaders\"" name="CustomHeaders" ingress="d
efault/httpd"                                                                                                                                                                                
I1104 07:03:21.234697       7 admission.go:149] processed ingress via admission controller {testedIngressLength:2 testedIngressTime:0.015s renderingIngressLength:2 renderingIngressTime:0s ad
missionTime:0.015s testedConfigurationSize:19.9kB}                                                                                                                                           
I1104 07:03:21.234725       7 main.go:107] "successfully validated configuration, accepting" ingress="default/httpd"                                                                          
I1104 07:03:21.238090       7 store.go:440] "Found valid IngressClass" ingress="default/httpd" ingressclass="nginx"                                                                           
E1104 07:03:21.238159       7 annotations.go:219] "error reading Ingress annotation" err="location denied, reason: unable to find configMap \"customheaders\"" name="CustomHeaders" ingress="d
efault/httpd"                                                                                                                                                                                
I1104 07:03:21.238188       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"httpd", UID:"49fa78f3-9240-446e-9a3b-490f89717750", APIVersion:"networking.k8s
.io/v1", ResourceVersion:"167241", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync                                                                                           
I1104 07:03:21.238470       7 controller.go:193] "Configuration changes detected, backend reload required"                                                                                    
I1104 07:03:21.270151       7 controller.go:213] "Backend successfully reloaded"                                                                                                             
I1104 07:03:21.270422       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7d978dd4c-kjhtj", UID:"7975097b-b2c6-45a9-9a9c-aee4
5f8ed10f", APIVersion:"v1", ResourceVersion:"167040", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration                                 
I1104 07:04:21.203640       7 status.go:304] "updating Ingress status" namespace="default" ingress="httpd" currentValue=null newValue=[{"ip":"192.168.58.2"}]                                 
I1104 07:04:21.208163       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"httpd", UID:"49fa78f3-9240-446e-9a3b-490f89717750", APIVersion:"networking.k8s
.io/v1", ResourceVersion:"167300", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync                                                                                           
E1104 07:04:21.208326       7 annotations.go:219] "error reading Ingress annotation" err="location denied, reason: unable to find configMap \"customheaders\"" name="CustomHeaders" ingress="d
efault/httpd"                                                                                                                                                                                
I1104 07:04:21.208704       7 controller.go:193] "Configuration changes detected, backend reload required"                                                                                    
I1104 07:04:21.236874       7 controller.go:213] "Backend successfully reloaded"                                                                                                             
I1104 07:04:21.237065       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7d978dd4c-kjhtj", UID:"7975097b-b2c6-45a9-9a9c-aee4
5f8ed10f", APIVersion:"v1", ResourceVersion:"167040", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration                                 
I1104 07:07:02.652708       7 admission.go:149] processed ingress via admission controller {testedIngressLength:2 testedIngressTime:0.017s renderingIngressLength:2 renderingIngressTime:0s ad
missionTime:0.017s testedConfigurationSize:22.1kB}                                             
I1104 07:07:02.652733       7 main.go:107] "successfully validated configuration, accepting" ingress="default/httpd"
I1104 07:07:02.656567       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"httpd", UID:"49fa78f3-9240-446e-9a3b-490f89717750", APIVersion:"networking.k8s
.io/v1", ResourceVersion:"167455", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1104 07:07:02.656841       7 controller.go:193] "Configuration changes detected, backend reload required"
I1104 07:07:02.684153       7 controller.go:213] "Backend successfully reloaded"               
I1104 07:07:02.684311       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7d978dd4c-kjhtj", UID:"7975097b-b2c6-45a9-9a9c-aee4
5f8ed10f", APIVersion:"v1", ResourceVersion:"167040", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
192.168.58.1 - - [04/Nov/2024:07:07:38 +0000] "GET / HTTP/1.1" 200 45 "-" "curl/7.88.1" 82 0.001 [default-httpd-80] [] 10.244.0.63:80 45 0.001 200 d4ceeb6e7508ab18b9bbdb7aef72c9d2
192.168.58.1 - - [04/Nov/2024:07:09:44 +0000] "GET / HTTP/1.1" 200 45 "-" "curl/7.88.1" 82 0.000 [default-httpd-80] [] 10.244.0.63:80 45 0.000 200 7750fde5babfc5f3dfef31f8047b381f
192.168.58.1 - - [04/Nov/2024:07:09:59 +0000] "GET / HTTP/1.1" 200 45 "-" "curl/7.88.1" 82 0.001 [default-httpd-80] [] 10.244.0.63:80 45 0.001 200 9e933118e104d67a7510d14cbbced3e2