Skip to content

VirtualServer prevents cert-manager from creating a certificate #2069

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jannekem opened this issue Oct 11, 2021 · 15 comments
Closed

VirtualServer prevents cert-manager from creating a certificate #2069

jannekem opened this issue Oct 11, 2021 · 15 comments
Labels
stale Pull requests/issues with no activity

Comments

@jannekem
Copy link

jannekem commented Oct 11, 2021 •
edited
Loading

I'm having an issue trying to get cert-manager and VirtualServer play nicely with each other after upgrading both to the latest version. Apparently this configuration has worked before. (I have changed the actual domain here.)

My certificate definition is something like this:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: preview-bc-dev-cert
  namespace: ingress
spec:
  secretName: preview-bc-dev-cert
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  issuerRef:
    name: letsencrypt-http01-certs
  dnsNames:
  - preview-bc-dev.devops.web.mydomain.com

And the VirtualServer deployment looks something like this:

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  annotations:
    certmanager.k8s.io/issuer: letsencrypt-http01-certs
    meta.helm.sh/release-name: preview-bc-dev
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2020-12-02T14:43:47Z"
  generation: 9
  labels:
    app.kubernetes.io/managed-by: Helm
  name: preview-bc-dev
  namespace: ingress
  resourceVersion: "127791902"
  uid: 0d04f389-3b30-486e-857f-759de4796bb7
spec:
  host: preview-bc-dev.devops.web.mydomain.com
  routes:
  - location-snippets: "\nlocation /.well-known/acme-challenge/ {\n    auth_basic
      off;\n}\nauth_basic           \"Test Environment Authentication\";\nauth_basic_user_file
      /etc/secret-volume/.htpasswd; \n\nproxy_set_header l5d-dst-override preview-bc-dev.preview-bc-dev.svc.cluster.local:80;\n"
    path: /
    route: preview-bc-dev/preview-bc-dev
  tls:
    redirect:
      enable: true
    secret: preview-bc-dev-cert

The location snippet contains configuration for basic authentication which is used in the dev environment, though it is disabled for the /.well-known/acme-challenge/ path, and I'm not prompted for a password if I try to access that. The certificate requests fail with a 404 error, the cert-manager pod logs have this message:

E1008 12:32:07.029685       1 sync.go:185] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="preview-bc-dev.devops.web.mydomain.com" "resource_kind"="Challenge" "resource_name"="preview-bc-dev-cert-zjzv2-2682410378-712319758" "resource_namespace"="ingress" "resource_version"="v1" "type"="HTTP-01"

The ingress controller logs look like this:

10.0.195.86 - - [11/Oct/2021:06:08:05 +0000] "GET /.well-known/acme-challenge/cU9X7h-YJj_X_Fp4VZIYp4S9NKHQwT_X_xuX-0o3elw HTTP/1.1" 301 169 "-" "cert-manager/v1.5.4 (clean)" "-"
2021/10/11 06:08:05 [error] 508#508: *762887 open() "/etc/nginx/html/.well-known/acme-challenge/cU9X7h-YJj_X_Fp4VZIYp4S9NKHQwT_X_xuX-0o3elw" failed 
(2: No such file or directory), client: 10.0.195.86, server: preview-bc-dev.devops.web.mydomain.com, request: "GET /.well-known/acme-challenge/cU9X7h-YJj_X_Fp4VZIYp4S9NKHQwT_X_xuX-0o3elw HTTP/1.1", host: "preview-bc-dev.devops.web.mydomain.com", referrer: "http://preview-bc-dev.devops.web.mydomain.com/.well-known/acme-challenge/cU9X7h-YJj_X_Fp4VZIYp4S9NKHQwT_X_xuX-0o3elw"
10.0.195.86 - - [11/Oct/2021:06:08:05 +0000] "GET /.well-known/acme-challenge/cU9X7h-YJj_X_Fp4VZIYp4S9NKHQwT_X_xuX-0o3elw HTTP/1.1" 404 153 "preview-bc-dev.devops.web.mydomain.com/.well-known/acme-challenge/cU9X7h-YJj_X_Fp4VZIYp4S9NKHQwT_X_xuX-0o3elw" "cert-manager/v1.5.4 (clean)" "-"

cert-manager creates Ingress objects for the challenges

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
  creationTimestamp: "2021-10-08T12:30:30Z"
  generateName: cm-acme-http-solver-
  generation: 1
  labels:
    acme.cert-manager.io/http-domain: "3464825950"
    acme.cert-manager.io/http-token: "813370860"
    acme.cert-manager.io/http01-solver: "true"
  name: cm-acme-http-solver-zkzx7
  namespace: ingress
  ownerReferences:
  - apiVersion: acme.cert-manager.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: Challenge
    name: preview-bc-dev-cert-xls48-66762260-1129632840
    uid: 6952a48c-3a75-4492-8bb8-0d84fe8e3e9d
  resourceVersion: "127867795"
  uid: 7d305da7-beca-4a4f-a615-61d1bfc4a6b9
spec:
  ingressClassName: nginx
  rules:
  - host: preview-bc-dev.devops.web.mydomain.com
    http:
      paths:
      - backend:
          service:
            name: cm-acme-http-solver-955z5
            port:
              number: 8089
        path: /.well-known/acme-challenge/0cEhJNEV4pICEDt3UdDYeaqt7G_ztaIHJ3nVjDTRc20
        pathType: ImplementationSpecific
status:
  loadBalancer: {}

If I remove the VirtualServer the certificate request resolves immediately. Then I can create the VirtualServer again and use the updated certificate but I don't want to do this every two months for all environments.

Environment
AKS cluster running Kubernetes 1.21.2
cert-manager v1.5.4
NGINX Ingress Controller v2.0.0
Using NGINX
@github-actions
Copy link

Hi @jannekem thanks for reporting!

Be sure to check out the docs while you wait for a human to take a look at this 🙂

Cheers!

@jannekem
Copy link
Author

I tried to overcome this issue by configuring the route path with a regular expression that doesn't match anything starting with /.well-known/acme-challenge/ but it seems that this is not possible.

The project documentation points to the nginx location block docs for more information about the path field. Therefore it was a bit surprising that the solution from this Stack Overflow answer didn't work. Apparently this project is using Go regex which doesn't support lookarounds.

@jannekem
Copy link
Author

It seems I forgot to include the VirtualServerRoute in the first message:

apiVersion: k8s.nginx.org/v1
kind: VirtualServerRoute
metadata:
  annotations:
    meta.helm.sh/release-name: preview-bc-dev
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2020-12-02T14:43:47Z"
  generation: 6
  labels:
    app.kubernetes.io/managed-by: Helm
  name: preview-bc-dev
  namespace: preview-bc-dev
  resourceVersion: "128606746"
  uid: 80249329-82d4-4a3d-ab6a-86b20c0dec23
spec:
  host: preview-bc-dev.devops.web.mydomain.com
  subroutes:
  - action:
      pass: preview-bc-dev
    path: /
  - action:
      pass: preview-bc-dev
    location-snippets: "proxy_set_header l5d-dst-override preview-bc-dev.preview-bc-dev.svc.cluster.local:80;
      \     \n\nauth_basic           \"Test Environment Authentication\";\nauth_basic_user_file
      /etc/secret-volume/.htpasswd;\n"
    path: /phones
  upstreams:
  - name: preview-bc-dev
    port: 80
    service: cache-service
status:
  externalEndpoints:
  - ip: <redacted>
    ports: '[80,443]'
  message: 'Configuration for preview-bc-dev/preview-bc-dev was added or updated '
  reason: AddedOrUpdated
  referencedBy: ingress/preview-bc-dev
  state: Valid

Is anyone successfully running cert-manager with a similar configuration?

@gamunu
Copy link

gamunu commented Nov 25, 2021

@jannekem Have you tried the acme.cert-manager.io/http01-edit-in-place: "true" https://cert-manager.io/docs/usage/ingress/ In the past I had similar issues and this resolved it.

@jannekem
Copy link
Author

jannekem commented Dec 8, 2021

Hmm. That documentation talks about Ingress resources. Can you use that annotation with a VirtualServer too?

@brianehlert
Copy link
Collaborator

In our short list is to make cert-manager VirtualServer aware.

@jannekem
Copy link
Author

jannekem commented Dec 9, 2021

That's great news! Is there an issue that I can follow somewhere?

@brianehlert
Copy link
Collaborator

I was thinking I would link to this one.

@github-actions
Copy link

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the stale Pull requests/issues with no activity label Mar 10, 2022
@github-actions
Copy link

This issue was closed because it has been stalled for 10 days with no activity.

@brianehlert brianehlert removed the stale Pull requests/issues with no activity label Mar 21, 2022
@brianehlert brianehlert reopened this Mar 21, 2022
@lucacome
Copy link

Initial support for VirtualServer was added in https://github.com/nginxinc/kubernetes-ingress/releases/tag/v2.2.0

@github-actions
Copy link

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the stale Pull requests/issues with no activity label Jul 21, 2022
@jannekem
Copy link
Author

jannekem commented Aug 1, 2022

I see this issue was closed as completed but the initial cert-manager support introduced in v2.2.0 didn't include ACME so my problem still persists. Is ACME support somewhere on the roadmap?

@brianehlert
Copy link
Collaborator

ACME challenge for VirtualServer was completed with the 2.3 release.

@jannekem
Copy link
Author

jannekem commented Aug 2, 2022

Ah, I managed to miss that release since the PR wasn't linked to this issue. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stale Pull requests/issues with no activity
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@lucacome @brianehlert @jannekem @gamunu