invalid bearer token, service account token has expired

[pharthiphan]

Environmental Info:
K3s Version: v1.23.6+k3s1

Node(s) CPU architecture, OS, and Version: Linux dev3-kv-02 5.13.0-52-generic #59~20.04.1-Ubuntu SMP Fri Jun 17 21:11:05 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration: 3 servers, 16 agents

Describe the bug:
Multiple pods from different namespace are in the ContainerCreating state including calico-kube-controllers and coredns in the kube-system namespace

$ kubectl get pod -A | grep ContainerCreating
cdi                cdi-apiserver-cdb4566f6-vq2zx                            0/1     ContainerCreating          0               6h39m
cdi                cdi-deployment-6868d685d7-kjmdb                          0/1     ContainerCreating          0               6h39m
cdi                cdi-operator-5756f45b77-js2c9                            0/1     ContainerCreating          0               6h39m
cdi                cdi-uploadproxy-77d7fd6c8b-h94mj                         0/1     ContainerCreating          0               6h39m
kube-system        calico-kube-controllers-54965c7ccb-rvksv                 0/1     ContainerCreating          0               5h39m
kube-system        coredns-d76bd69b-4dtbl                                   0/1     ContainerCreating          0               5h39m
kube-system        kube-sriov-cni-ds-arm64-5j69h                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-5ljzh                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-5mmfz                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-5zkvh                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-8z5zc                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-bpk9q                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-gqks9                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-hsx9k                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-jrlpb                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-krt9n                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-m42j4                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-nnrw2                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-nqshk                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-q2stf                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-r8b9d                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-vn2gj                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-vstrn                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-wdbcj                            0/1     ContainerCreating          0               5h38m
kube-system        kube-sriov-cni-ds-arm64-zxgrl                            0/1     ContainerCreating          0               5h38m
kube-system        metrics-server-7cd5fcb6b7-zbkfj                          0/1     ContainerCreating          0               5h38m
kubevirt           virt-api-6574dcd954-bc9kx                                0/1     ContainerCreating          0               6h39m
kubevirt           virt-api-6574dcd954-mcrlw                                0/1     ContainerCreating          0               6h39m
kubevirt           virt-controller-74dc5677b-2fnfh                          0/1     ContainerCreating          0               6h39m
kubevirt           virt-controller-74dc5677b-kqpv2                          0/1     ContainerCreating          0               6h39m
kubevirt           virt-handler-24dvx                                       0/1     ContainerCreating          0               6h39m
kubevirt           virt-handler-2ld2r                                       0/1     ContainerCreating          0               6h39m
kubevirt           virt-handler-4v97l                                       0/1     ContainerCreating          0               6h39m
kubevirt           virt-handler-6cb6m                                       0/1     ContainerCreating          0               6h39m
kubevirt           virt-handler-778pn                                       0/1     ContainerCreating          0               6h39m
kubevirt           virt-handler-8ggsp                                       0/1     ContainerCreating          0               6h39m
kubevirt           virt-handler-986z2                                       0/1     ContainerCreating          0               6h39m
kubevirt           virt-handler-dplkn                                       0/1     ContainerCreating          0               6h39m
kubevirt           virt-handler-gjb9t                                       0/1     ContainerCreating          0               6h39m
kubevirt           virt-handler-hq5fg                                       0/1     ContainerCreating          0               6h39m
kubevirt           virt-handler-jh6f2                                       0/1     ContainerCreating          0               6h39m
kubevirt           virt-handler-khspp                                       0/1     ContainerCreating          0               6h39m
kubevirt           virt-handler-r9gr5                                       0/1     ContainerCreating          0               6h39m
kubevirt           virt-handler-szc9z                                       0/1     ContainerCreating          0               6h39m
kubevirt           virt-handler-vhqn6                                       0/1     ContainerCreating          0               6h39m
kubevirt           virt-handler-vtxvl                                       0/1     ContainerCreating          0               6h39m
kubevirt           virt-handler-wqhrc                                       0/1     ContainerCreating          0               6h39m
kubevirt           virt-handler-xvqfp                                       0/1     ContainerCreating          0               6h39m
kubevirt           virt-handler-zpbhw                                       0/1     ContainerCreating          0               6h39m
kubevirt           virt-operator-5f9c5b5fd7-5hwhb                           0/1     ContainerCreating          0               6h39m
kubevirt           virt-operator-5f9c5b5fd7-ntggq                           0/1     ContainerCreating          0               6h39m

kubectl describe pod throws the following error for all the pods stuck in the ContainerCreating state

Warning FailedCreatePodSandBox 55s (x1755 over 6h24m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "986af2c3af9e173a6f4084fcf73795bccf3b996c98f5a80b9f0a04a554cb8a21": plugin type="multus" name="multus-cni-network" failed (add): [cdi/cdi-apiserver-cdb4566f6-vq2zx/d82b198a-de67-4463-8e76-884e022fdc99:k8s-pod-network]: error adding container to network "k8s-pod-network": plugin type="calico" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized

Journal logs flooded with the following error
Nov 24 13:00:43 dev3-kv-02 k3s[2648839]: E1124 13:00:43.020907 2648839 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has expired]"

Steps To Reproduce:
Started to see all of sudden, I think Certificates for renewed

• Installed K3s:

Expected behavior:
kube-system pods to be up and running

Actual behavior:
some of them like calico and DNS pods are in the ContainerCreating state

Additional context / logs:
Tried to restart the k3s service on servers using systemctl restart k3s and on agents using systemctl restart k3s-node but didn't help

[brandond]

I've never seen this on a cluster that doesn't have something wrong with the datastore. Are you using etcd or external SQL? Are there any errors on the server nodes? Are all nodes in the cluster running the same version of k3s? Can you replicate this on the latest 1.23 patch release?

[pharthiphan]

Yes, using embedded etcd

# ETCDCTL_ENDPOINTS='https://10.25.56.2:2379,https://10.25.56.12:2379,https://10.25.56.245:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint status --write-out=table
+---------------------------+------------------+---------+---------+-----------+-----------+------------+
|         ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+---------------------------+------------------+---------+---------+-----------+-----------+------------+
|   https://10.25.56.2:2379 | 740e022e49b6e8d5 |   3.5.4 |   32 MB |     false |         8 |   17181880 |
|  https://10.25.56.12:2379 | 4aae74a834425a16 |   3.5.4 |   32 MB |      true |         8 |   17181881 |
| https://10.25.56.245:2379 | a0274eddce9c4f35 |   3.5.4 |   32 MB |     false |         8 |   17181881 |
+---------------------------+------------------+---------+---------+-----------+-----------+------------+

# ETCDCTL_ENDPOINTS='https://10.25.56.2:2379,https://10.25.56.12:2379,https://10.25.56.245:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint health --write-out=table
https://10.25.56.12:2379 is healthy: successfully committed proposal: took = 3.019958ms
https://10.25.56.245:2379 is healthy: successfully committed proposal: took = 1.909959ms
https://10.25.56.2:2379 is healthy: successfully committed proposal: took = 1.757678ms

I can only see the following errors on k3s journal

error getting ClusterInformation: connection is unauthorized: Unauthorized
Nov 25 02:12:12 dev3-kv-01 k3s[3488546]: E1125 02:12:12.413543 3488546 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, service account token has expired]"

Yes all server and agent nodes are running the same version v1.23.6+k3s1

[brandond]

Is time set correct on all the nodes in the cluster? Are you using ntp or something else to keep them in sync?

[pharthiphan]

yes all the nodes time is in sync

10.25.56.[2,12-21,24,26-29,245] (17)
---------------
Fri Nov 25 04:37:41 UTC 2022
---------------
10.25.56.25
---------------
Fri Nov 25 04:37:42 UTC 2022

[brandond]

This sounds a lot like rancher/rke2#3425 (comment), but that is in rke2 which packages Canal (which includes Calico) as a supported CNI. Since you've disabled the packaged k3s default Flannel CNI and deployed Calico in its place, it would be on you to verify its configuration and ensure that the token is being renewed.