Seeing "failed to find any PEM data in certificate input" in consul connect injector pod

[jacobmammoliti]

Overview of the Issue

I have deployed a Consul 1.9.3 cluster on my K3s RaspberryPi cluster and am noticing a constant crashloopbackoff when I enable the connectInjector. I have posted the logs below.

Adding note: I have run the same deployment against a GKE cluster without any issues.

Reproduction Steps

Deploy Consul on a K3s RaspberryPi cluster with the terraform code here: https://github.com/arctiqjacob/blizzard-dc/blob/main/k3s/terraform/consul.tf

Logs

consul-k8s logs

kubectl logs consul-connect-injector-webhook-deployment-64f6fd4859-ks4tw  -n consul --previous
Listening on ":8080"...
Error loading TLS keypair: tls: failed to find any PEM data in certificate input
2021/03/10 22:57:59 http: TLS handshake error from 10.42.2.1:53774: No certificate available.
terminated received, shutting down
2021/03/10 22:57:59 http: TLS handshake error from 10.42.2.1:53776: No certificate available.
Error listening: http: Server closed
E0310 22:57:59.395255       1 controller.go:124] error syncing cache
E0310 22:57:59.395758       1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Pod: Get "https://10.43.0.1:443/api/v1/pods?labelSelector=consul.hashicorp.com%2Fconnect-inject-status": context canceled
2021-03-10T22:57:59.395Z [ERROR] cleanupResource: unable to get nodes: error="Get "https://10.43.0.1:443/api/v1/nodes": context canceled"
2021-03-10T22:57:59.396Z [INFO]  cleanupResource: received stop signal, shutting down
2021-03-10T22:57:59.396Z [ERROR] healthCheckResource: unable to get pods: err="Get "https://10.43.0.1:443/api/v1/pods?labelSelector=consul.hashicorp.com%2Fconnect-inject-status": context canceled"
E0310 22:57:59.492412       1 controller.go:124] error syncing cache
2021-03-10T22:57:59.396Z [INFO]  healthCheckResource: received stop signal, shutting down

get pods

kubectl get pods -n consul         
NAME                                                          READY   STATUS             RESTARTS   AGE
consul-server-1                                               1/1     Running            0          16h
consul-server-2                                               1/1     Running            0          16h
consul-nmpmb                                                  1/1     Running            0          16h
consul-zmb56                                                  1/1     Running            0          16h
consul-r5dzv                                                    1/1     Running           0          16h
consul-server-0                                               1/1     Running            0          16h
consul-connect-injector-webhook-deployment-64f6fd4859-ks4tw   0/1     CrashLoopBackOff   370        15h

Expected behavior

Expecting this to run without any issues.

Environment details

If not already included, please provide the following:

• consul-k8s version: 0.24.0
• jsiebens/envoy-arm64 version: 1.13.3

[ndhanushkodi]

Hey @jacobmammoliti this is an intermittent error due to signaling logic in the connectInjector. This should be fixed as part of an upcoming refactor to use the same certificate manager across components. In the meanwhile this error should go away on its own after some restarts.

[jacobmammoliti]

@ndhanushkodi do you have a rough timeline of when that refactor is coming? I've found even after many crashes, it never becomes healthy. In the get pods I posted above, the restart count got up to 370.

[ndhanushkodi]

@jacobmammoliti We are expecting to finish the refactor and have a release by mid-late April!

[lkysow]

I think this might be the kube liveness check killing the pod. See hashicorp/consul-helm#885

[jacobmammoliti]

@ilyavinnik bingo.. for now I went and modified the probe directly on the deployment and it is stable now 👍

[lkysow]

This fix will be in our next release.