can you expand the docs for K3S_TOKEN security

[abe-winter]

Is your feature request related to a problem? Please describe.
It's not clear from the k3s docs whether K3S_TOKEN (recommended here) is a two-way validation. I.e., is it just a way for the new agent to authenticate to the cluster? Or does it also allow the new agent to validate that the server is who it appears to be?

Describe the solution you'd like

Can you please add a line in the docs saying if:

• K3S_TOKEN allows the client to validate the server
• whether this is safe to use on untrusted connections, or should only be used on a private network

Describe alternatives you've considered
(digging through code and old issues)

Additional context

• from clientaccess/token.go, it looks like the client validates the server's CA bundle, but I don't know exactly what that means in terms of trusted connections
• rke's certs docs, possibly relevant?
• jvns blog on kube CAs is a good overview, but also makes it sound like there are many approaches here
• cert rotation docs in k3s wiki
• seems like token-auth-file is recommended against in the k3s security self-assessment? (not sure if that's the same as passing K3S_TOKEN). I think that's the same as this CIS rule
• discussion of token lifetime here Documentation on node token lifetime and scope? #2645

[brandond]

This is a reasonable ask, but the short answer is that the token can be in one of two formats:

1. Just the bare token: simply the server or agent password required to join the cluster. This is the format that should be used when creating a new cluster with the --token or --agent-token flag.
2. K10 format: A packed format that includes the fixed string K10, the hash of the cluster CA, a username, and password. When this format is used, the cluster's CA certificate hash is validated against the hash included in the token. This is the format that should be used when joining additional servers or agents to the cluster, and can be retrieved from the node-token file on a server.

[abe-winter]

Thank you!

When connecting with K10 format, is the sequence:

1. new node connects to cluster via TLS, using the server cert
2. the new node uses the K10 hash to validate the current TLS connection
3. new node logs in with user/password from K10

(I.e. does validation happen before the user/password are transmitted)

[brandond]

Everything uses TLS.

1. Node connects to the server (without authentication, and without validating the TLS connection's certificate chain) and retrieves the cluster CA certificate.
2. Node verifies that the CA certificate hash matches the hash in the token.
3. Node verifies that the certificate used by the TLS connection to the server was issued by the cluster CA certificate.
4. Node sends credentials to download bootstrap configuration.

Step 2 is omitted if the hash is not included in the token; any cluster CA sent by the remote node is accepted. If you want to validate the identity of the cluster you're joining, you should always use the K10 format.

[abe-winter]

Thanks, that makes total sense

[sourcehawk]

This is a reasonable ask, but the short answer is that the token can be in one of two formats:

1. Just the bare token: simply the server or agent password required to join the cluster. This is the format that should be used when creating a new cluster with the --token or --agent-token flag.

Does this mean that when I install the first server with the --token flag, I can just use the same --token flag on all nodes to join the cluster and don't need to retrieve a node token from the first installation?

[brandond]

Yes, you can absolutely use that short token format on all your nodes. You'll be missing the CA validation that comes from including the K10 prefix, but that cannot be known ahead of time as the CA certs are generated during initial server startup.