Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add kubeadm style bootstrap token secret support #6663

Merged
merged 5 commits into from
Feb 7, 2023

Conversation

brandond
Copy link
Member

@brandond brandond commented Dec 20, 2022

Proposed Changes

  • Add k3s token create - same functionality as kubeadm token create
    Can be used to create time-limited join tokens that are automatically cleaned up by the controller-manager when the TTL expires. Tokens are for joining agents only. Servers must know the legacy join/encryption token.
  • Allow agents to join using bootstrap tokens or existing kubelet certs (via NodeAuth admission plugin)

Types of Changes

enhancement

Verification

  • use k3s token create to create a join token
    • join an agent to the cluster with the token
  • use k3s token list to list the token
  • use k3s token delete to delete the token
  • use k3s token create --ttl 5m to create a time-limited token
    • note that it is removed when the TTL is up
    • note that nodes joined to the cluster using this token can restart successfully
    • note that nodes cannot rejoin the cluster with the expired token if their node object is deleted from the cluster

Testing

Covered by CI

Linked Issues

User-Facing Change

K3s now supports `kubeadm` style join tokens. `k3s token create` now creates join token secrets, optionally with a limited TTL.
K3s agents joined with an expired or deleted token stay in the cluster using existing client certificates via the NodeAuthorization admission plugin, unless their Node object is deleted from the cluster.

Further Comments

@brandond brandond requested a review from a team as a code owner December 20, 2022 03:23
@brandond brandond changed the title Bootstrap token Add kubeadm style bootstrap token secret support Dec 20, 2022
@brandond brandond force-pushed the bootstrap_token branch 3 times, most recently from 1f2d40d to 7db8022 Compare December 20, 2022 21:45
@brandond
Copy link
Member Author

brandond commented Dec 23, 2022

Outstanding issues:

  • Can't delete tokens by secret name - just full token
  • Token delete message missing trailing newline
  • No way to get full token after create command is run - how does upstream handle this? appears to be intentional
  • Kubelet certificate auth continues to work after token is deleted - does not check for existence of node (need to hook into node authz?)
Auth debug logs
Dec 23 01:00:15 systemd-node-1 k3s[267]: time="2022-12-23T01:00:15Z" level=debug msg="Authenticated request from 127.0.0.1:34092 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:15 systemd-node-1 k3s[267]: time="2022-12-23T01:00:15Z" level=debug msg="Authenticated request from 127.0.0.1:34096 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:15 systemd-node-1 k3s[267]: time="2022-12-23T01:00:15Z" level=debug msg="Authenticated request from 127.0.0.1:34108 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:15 systemd-node-1 k3s[267]: time="2022-12-23T01:00:15Z" level=debug msg="Authenticated request from 127.0.0.1:34118 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:15 systemd-node-1 k3s[267]: time="2022-12-23T01:00:15Z" level=debug msg="Authenticated request from 127.0.0.1:34124 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:15 systemd-node-1 k3s[267]: time="2022-12-23T01:00:15Z" level=debug msg="Authenticated request from 127.0.0.1:34134 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:15 systemd-node-1 k3s[267]: time="2022-12-23T01:00:15Z" level=debug msg="Authenticated request from 127.0.0.1:34144 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"

Dec 23 01:00:17 systemd-node-1 k3s[267]: time="2022-12-23T01:00:17Z" level=debug msg="Authenticated request from 127.0.0.1:34154 with user=system:node:systemd-node-1 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:17 systemd-node-1 k3s[267]: time="2022-12-23T01:00:17Z" level=debug msg="Authenticated request from 127.0.0.1:34206 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"

Dec 23 01:00:22 systemd-node-1 k3s[267]: time="2022-12-23T01:00:22Z" level=debug msg="Authenticated request from 127.0.0.1:53970 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:22 systemd-node-1 k3s[267]: time="2022-12-23T01:00:22Z" level=debug msg="Authenticated request from 127.0.0.1:53978 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"

Dec 23 01:00:24 systemd-node-1 k3s[267]: time="2022-12-23T01:00:24Z" level=debug msg="Authenticated request from 172.17.0.4:50486 with user=system:node:systemd-node-1 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"

Dec 23 01:00:30 systemd-node-1 k3s[267]: time="2022-12-23T01:00:30Z" level=debug msg="Authenticated request from 172.17.0.5:49510 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:30 systemd-node-1 k3s[267]: time="2022-12-23T01:00:30Z" level=debug msg="Authenticated request from 172.17.0.5:49526 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:30 systemd-node-1 k3s[267]: time="2022-12-23T01:00:30Z" level=debug msg="Authenticated request from 172.17.0.5:49530 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:30 systemd-node-1 k3s[267]: time="2022-12-23T01:00:30Z" level=debug msg="Authenticated request from 172.17.0.5:49540 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:30 systemd-node-1 k3s[267]: time="2022-12-23T01:00:30Z" level=debug msg="Authenticated request from 172.17.0.5:49548 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:31 systemd-node-1 k3s[267]: time="2022-12-23T01:00:31Z" level=debug msg="Authenticated request from 172.17.0.5:49550 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:31 systemd-node-1 k3s[267]: time="2022-12-23T01:00:31Z" level=debug msg="Authenticated request from 172.17.0.5:49552 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:32 systemd-node-1 k3s[267]: time="2022-12-23T01:00:32Z" level=debug msg="Authenticated request from 172.17.0.5:51544 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:32 systemd-node-1 k3s[267]: time="2022-12-23T01:00:32Z" level=debug msg="Authenticated request from 172.17.0.5:51554 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:32 systemd-node-1 k3s[267]: time="2022-12-23T01:00:32Z" level=debug msg="Authenticated request from 172.17.0.5:51612 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:32 systemd-node-1 k3s[267]: time="2022-12-23T01:00:32Z" level=debug msg="Authenticated request from 172.17.0.5:51616 with user=node groups=[k3s:agent system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"

Dec 23 01:00:52 systemd-node-1 k3s[267]: time="2022-12-23T01:00:52Z" level=debug msg="Authenticated request from 127.0.0.1:36774 with user=system:apiserver groups=[system:masters system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:52 systemd-node-1 k3s[267]: time="2022-12-23T01:00:52Z" level=debug msg="Authenticated request from 127.0.0.1:36804 with user=system:apiserver groups=[system:masters system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:52 systemd-node-1 k3s[267]: time="2022-12-23T01:00:52Z" level=debug msg="Authenticated request from 127.0.0.1:36790 with user=system:apiserver groups=[system:masters system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:52 systemd-node-1 k3s[267]: time="2022-12-23T01:00:52Z" level=debug msg="Authenticated request from 127.0.0.1:36758 with user=system:apiserver groups=[system:masters system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:52 systemd-node-1 k3s[267]: time="2022-12-23T01:00:52Z" level=debug msg="Authenticated request from 127.0.0.1:36820 with user=system:apiserver groups=[system:masters system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:53 systemd-node-1 k3s[267]: time="2022-12-23T01:00:53Z" level=debug msg="Authenticated request from 127.0.0.1:36824 with user=system:apiserver groups=[system:masters system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:00:53 systemd-node-1 k3s[267]: time="2022-12-23T01:00:53Z" level=debug msg="Authenticated request from 127.0.0.1:36822 with user=system:apiserver groups=[system:masters system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"

Dec 23 01:02:54 systemd-node-1 k3s[267]: time="2022-12-23T01:02:54Z" level=debug msg="Authenticated request from 172.17.0.5:52688 with user=system:bootstrap:wo58li groups=[system:bootstrappers system:bootstrappers:k3s:default-node-token system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:02:54 systemd-node-1 k3s[267]: time="2022-12-23T01:02:54Z" level=debug msg="Authenticated request from 172.17.0.5:52690 with user=system:bootstrap:wo58li groups=[system:bootstrappers system:bootstrappers:k3s:default-node-token system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:02:54 systemd-node-1 k3s[267]: time="2022-12-23T01:02:54Z" level=debug msg="Authenticated request from 172.17.0.5:52698 with user=system:bootstrap:wo58li groups=[system:bootstrappers system:bootstrappers:k3s:default-node-token system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:02:54 systemd-node-1 k3s[267]: time="2022-12-23T01:02:54Z" level=debug msg="Authenticated request from 172.17.0.5:52704 with user=system:bootstrap:wo58li groups=[system:bootstrappers system:bootstrappers:k3s:default-node-token system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:02:54 systemd-node-1 k3s[267]: time="2022-12-23T01:02:54Z" level=debug msg="Authenticated request from 172.17.0.5:52716 with user=system:bootstrap:wo58li groups=[system:bootstrappers system:bootstrappers:k3s:default-node-token system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:02:54 systemd-node-1 k3s[267]: time="2022-12-23T01:02:54Z" level=debug msg="Authenticated request from 172.17.0.5:52732 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:02:54 systemd-node-1 k3s[267]: time="2022-12-23T01:02:54Z" level=debug msg="Authenticated request from 172.17.0.5:52744 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"

Dec 23 01:02:57 systemd-node-1 k3s[267]: time="2022-12-23T01:02:57Z" level=debug msg="Authenticated request from 172.17.0.5:52790 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:02:57 systemd-node-1 k3s[267]: time="2022-12-23T01:02:57Z" level=debug msg="Authenticated request from 172.17.0.5:52794 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:02:57 systemd-node-1 k3s[267]: time="2022-12-23T01:02:57Z" level=debug msg="Authenticated request from 172.17.0.5:52864 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:02:57 systemd-node-1 k3s[267]: time="2022-12-23T01:02:57Z" level=debug msg="Authenticated request from 172.17.0.5:52872 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"

Dec 23 01:05:32 systemd-node-1 k3s[267]: time="2022-12-23T01:05:32Z" level=debug msg="Authenticated request from 127.0.0.1:47502 with user=system:apiserver groups=[system:masters system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"

Dec 23 01:06:12 systemd-node-1 k3s[267]: time="2022-12-23T01:06:12Z" level=debug msg="Authenticated request from 172.17.0.5:38580 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:06:12 systemd-node-1 k3s[267]: time="2022-12-23T01:06:12Z" level=debug msg="Authenticated request from 172.17.0.5:38582 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:06:12 systemd-node-1 k3s[267]: time="2022-12-23T01:06:12Z" level=debug msg="Authenticated request from 172.17.0.5:38590 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:06:12 systemd-node-1 k3s[267]: time="2022-12-23T01:06:12Z" level=debug msg="Authenticated request from 172.17.0.5:38606 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:06:12 systemd-node-1 k3s[267]: time="2022-12-23T01:06:12Z" level=debug msg="Authenticated request from 172.17.0.5:38612 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:06:12 systemd-node-1 k3s[267]: time="2022-12-23T01:06:12Z" level=debug msg="Authenticated request from 172.17.0.5:38624 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:06:12 systemd-node-1 k3s[267]: time="2022-12-23T01:06:12Z" level=debug msg="Authenticated request from 172.17.0.5:38638 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:06:13 systemd-node-1 k3s[267]: time="2022-12-23T01:06:13Z" level=debug msg="Authenticated request from 172.17.0.5:38688 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:06:13 systemd-node-1 k3s[267]: time="2022-12-23T01:06:13Z" level=debug msg="Authenticated request from 172.17.0.5:38702 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:06:13 systemd-node-1 k3s[267]: time="2022-12-23T01:06:13Z" level=debug msg="Authenticated request from 172.17.0.5:38764 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:06:13 systemd-node-1 k3s[267]: time="2022-12-23T01:06:13Z" level=debug msg="Authenticated request from 172.17.0.5:38778 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"

Dec 23 01:09:46 systemd-node-1 k3s[267]: time="2022-12-23T01:09:46Z" level=debug msg="Authenticated request from 172.17.0.5:53614 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:09:46 systemd-node-1 k3s[267]: time="2022-12-23T01:09:46Z" level=debug msg="Authenticated request from 172.17.0.5:53622 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:09:46 systemd-node-1 k3s[267]: time="2022-12-23T01:09:46Z" level=debug msg="Authenticated request from 172.17.0.5:53634 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:09:46 systemd-node-1 k3s[267]: time="2022-12-23T01:09:46Z" level=debug msg="Authenticated request from 172.17.0.5:53638 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:09:46 systemd-node-1 k3s[267]: time="2022-12-23T01:09:46Z" level=debug msg="Authenticated request from 172.17.0.5:53646 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:09:46 systemd-node-1 k3s[267]: time="2022-12-23T01:09:46Z" level=debug msg="Authenticated request from 172.17.0.5:53654 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:09:46 systemd-node-1 k3s[267]: time="2022-12-23T01:09:46Z" level=debug msg="Authenticated request from 172.17.0.5:53658 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:09:47 systemd-node-1 k3s[267]: time="2022-12-23T01:09:47Z" level=debug msg="Authenticated request from 172.17.0.5:53692 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:09:47 systemd-node-1 k3s[267]: time="2022-12-23T01:09:47Z" level=debug msg="Authenticated request from 172.17.0.5:53712 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:09:48 systemd-node-1 k3s[267]: time="2022-12-23T01:09:48Z" level=debug msg="Authenticated request from 172.17.0.5:53772 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"
Dec 23 01:09:48 systemd-node-1 k3s[267]: time="2022-12-23T01:09:48Z" level=debug msg="Authenticated request from 172.17.0.5:53806 with user=system:node:systemd-node-2 groups=[system:nodes system:authenticated] audiences=authenticator.Audiences(nil) extra=map[string][]string(nil)"

@brandond brandond force-pushed the bootstrap_token branch 2 times, most recently from 38c51d8 to ca92ad7 Compare December 24, 2022 19:03
@brandond
Copy link
Member Author

brandond commented Dec 27, 2022

With latest changes node auth is blocked when the node doesn't exist - but the messages leave something to be desired.

On the server the log just says level=error msg="nodes \"systemd-node-2\" not found.

On the agent it says level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"

EDIT: This has been addressed.

@brandond brandond force-pushed the bootstrap_token branch 2 times, most recently from 99868f6 to 8a84320 Compare January 13, 2023 17:45
@larssb
Copy link

larssb commented Jan 14, 2023

This is really really awesome. Thanks! Looking forward to this.

Copy link
Member

@dereknola dereknola left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have reviewed the new commands and CLI and it looks good, I am waiting for #6615 to be merged and for this PR to be rebased before formal approval.

@brandond
Copy link
Member Author

brandond commented Feb 6, 2023

Will rebase once the CA rotation PR is merged

Allow bootstrapping with kubeadm bootstrap token strings or existing
Kubelet certs. This allows agents to join the cluster using kubeadm
bootstrap tokens, as created with the `k3s token create` command.

When the token expires or is deleted, agents can successfully restart by
authenticating with their kubelet certificate via node authentication.
If the token is gone and the node is deleted from the cluster, node auth
will fail and they will be prevented from rejoining the cluster until
provided with a valid token.

Servers still must be bootstrapped with the static cluster token, as
they will need to know it to decrypt the bootstrap data.

Signed-off-by: Brad Davidson <[email protected]>
Signed-off-by: Brad Davidson <[email protected]>
Signed-off-by: Brad Davidson <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants