Generation of certificates and keys for etcd gated if etcd is disabled. #6998

bartossh · 2023-02-22T12:13:15Z

Problem:
When support for etcd was added in 3957142, the generation of certificates and keys for etcd was not gated behind the use of managed etcd. Keys are generated and distributed across servers even if managed etcd is not enabled.

Solution:
Allow generation of certificates and keys only if managed etc is enabled. Check config.DisableETCD flag.

Proposed Changes

Gate generating ETCD Certificates by checking config.DisableETCD flag.

Types of Changes

Bugfix

Verification

When running k3s server provide --disable-etcd flag

Testing

There are no unit tests present.

Linked Issues

Related issue

User-Facing Change

No user-facing changes in that PR.

Further Comments

Trivial change.

brandond · 2023-02-22T19:20:10Z

Thanks for the PR!

Have you checked to see what happens if you later enable etcd by restarting K3s with --cluster-init added to the args, and then attempt to join a second server to the cluster? Does it handle that properly?

bartossh · 2023-02-23T08:05:06Z

Thanks for the suggestion, I haven't checked that scenario but I am on my way...

bartossh · 2023-02-26T09:12:20Z

TEST: (tested on Ubuntu 20.04 focal inside docker containers ubuntu:latest)

I've created a docker network and run the k3s servers in two separate containers:

Build k3s for development:

SKIP_VALIDATE=true make

Network:

λ ~/git/k8s/k3s/ 3404/shouldd_not_perform_etcd_setup_if_etcd_is_not_enabled docker network inspect k3s-network
[
    {
        "Name": "k3s-network",
        "Id": "14e9b03b2323fbbb40db6c9e5b75c16a7f8a739dd4a1c7011e59c9f57c3a6c7d",
        "Created": "2023-02-26T09:21:39.110735057+01:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "b18418c2a74f143d0996e5a158ebc42afbce2a5c7c19dc88848e41491d8a56e8": {
                "Name": "wonderful_golick",
                "EndpointID": "39bb4fdccb6bfb5af7c5e1ee5c29471dcac63c69e5c91b3c857db32183c15baf",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            },
            "c8523bf03e01e6e93bc1cb80601c2fd8e5985cc5abc641bf6f8bad9f05fa3533": {
                "Name": "exciting_lichterman",
                "EndpointID": "fe420aeef760f5bc10731b8d9c102bafedee26c4c0d97fd97ac2644f9bc27458",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

Running the first container:

docker run -it --volume ${PWD}:/k3s --network k3s-network --publish 6443:6443 ubuntu:latest /bin/bash

Run the second container:

docker run -it --volume ${PWD}:/k3s --network k3s-network --publish 6446:6443 ubuntu:latest /bin/bash

SCENARIO 1: run k3s in the first container with etcd only and run k3s in the second container with --disable-etcd connecting to the first container etcd on port 6443

The First container:

./bin/k3s server --cluster-init --disable-apiserver --disable-controller-manager --disable-scheduler

The Second container:

./bin/k3s server --cluster-init --disable-etcd --server https://172.18.0.2:6443 -t K1062ecd389f0149f7c55d19a23253e6811f77badf341a61046153ab68ddb7071::server:6be8b4b37e6e76a8f3803e701fd36af8

Connection succeeded.

SCENARIO 2: restart k3s in the second container with --cluster-init.

./bin/k3s server --cluster-init

Restarts and generates servers etcd certs. Creating k3s server succeeded.

brandond · 2023-02-27T19:44:33Z

What you'd want to test is:

start 1st server without --cluster-init (no etcd)
stop 1st server, start it with --cluster-init (convert to etcd)
add 2nd server pointing at first server

I am a little concerned that if the etcd CA certs don't exist when the datastore is initialized, they won't get put into it and the second server will fail to join.

Problem: When support for etcd was added in 3957142, generation of certificates and keys for etcd was not gated behind use of managed etcd. Keys are generated and distributed across servers even if managed etcd is not enabled. Solution: Allow generation of certificates and keys only if managed etc is enabled. Check config.DisableETCD flag. Signed-off-by: Bartossh <[email protected]> refactor

bartossh · 2023-03-02T09:07:50Z

I'm not sure that I followed exactly what you meant.

first server: ./bin/k3s server without --cluster-init
first server: ./bin/k3s server --cluster-init --disable-apiserver --disable-controller-manager --disable-scheduler - only etcd
second server: ./bin/k3s server --cluster-init --disable-etcd --server https://172.25.0.2:6443 -t

INFO[0000] Starting k3s v1.26.1+k3s-232cc1e5-dirty (232cc1e5)
INFO[0000] Managed etcd cluster not yet initialized
INFO[0000] Reconciling bootstrap data between datastore and disk
INFO[0000] certificate CN=system:admin,O=system:masters signed by CN=k3s-client-ca@1677747240: notBefore=2023-03-02 08:54:00 +0000 UTC notAfter=2024-03-01 09:01:34 +0000 UTC
INFO[0000] certificate CN=system:kube-controller-manager signed by CN=k3s-client-ca@1677747240: notBefore=2023-03-02 08:54:00 +0000 UTC notAfter=2024-03-01 09:01:34 +0000 UTC
INFO[0000] certificate CN=system:kube-scheduler signed by CN=k3s-client-ca@1677747240: notBefore=2023-03-02 08:54:00 +0000 UTC notAfter=2024-03-01 09:01:34 +0000 UTC
INFO[0000] certificate CN=system:apiserver,O=system:masters signed by CN=k3s-client-ca@1677747240: notBefore=2023-03-02 08:54:00 +0000 UTC notAfter=2024-03-01 09:01:34 +0000 UTC
INFO[0000] certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1677747240: notBefore=2023-03-02 08:54:00 +0000 UTC notAfter=2024-03-01 09:01:34 +0000 UTC
INFO[0000] certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1677747240: notBefore=2023-03-02 08:54:00 +0000 UTC notAfter=2024-03-01 09:01:34 +0000 UTC
INFO[0000] certificate CN=k3s-cloud-controller-manager signed by CN=k3s-client-ca@1677747240: notBefore=2023-03-02 08:54:00 +0000 UTC notAfter=2024-03-01 09:01:34 +0000 UTC
INFO[0000] certificate CN=kube-apiserver signed by CN=k3s-server-ca@1677747240: notBefore=2023-03-02 08:54:00 +0000 UTC notAfter=2024-03-01 09:01:34 +0000 UTC
INFO[0000] certificate CN=system:auth-proxy signed by CN=k3s-request-header-ca@1677747240: notBefore=2023-03-02 08:54:00 +0000 UTC notAfter=2024-03-01 09:01:34 +0000 UTC
INFO[0000] certificate CN=etcd-client signed by CN=etcd-server-ca@1677747240: notBefore=2023-03-02 08:54:00 +0000 UTC notAfter=2024-03-01 09:01:34 +0000 UTC
INFO[0000] certificate CN=etcd-peer signed by CN=etcd-peer-ca@1677747240: notBefore=2023-03-02 08:54:00 +0000 UTC notAfter=2024-03-01 09:01:34 +0000 UTC
INFO[0000] certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1677747240: notBefore=2023-03-02 08:54:00 +0000 UTC notAfter=2024-03-01 09:01:34 +0000 UTC
WARN[0000] dynamiclistener [::]:6443: no cached certificate available for preload - deferring certificate load until storage initialization or first client request
INFO[0000] Active TLS secret / (ver=) (count 10): map[listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-172.25.0.3:172.25.0.3 listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-b017174339f9:b017174339f9 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/fingerprint:SHA1=AFD207A83BF28EECF1B1CB433D99C030BCEE3371]
INFO[0000] Running load balancer k3s-etcd-server-load-balancer 127.0.0.1:2379 -> [172.25.0.2:2379]
WARN[0001] Failed to remove this node from etcd members
INFO[0001] Tunnel server egress proxy mode: agent
INFO[0001] Tunnel server egress proxy waiting for runtime core to become available

dereknola · 2023-03-08T22:42:45Z

For step 3, you have conflicting --cluster-init --disable-etcd flags. That is not how you join other servers to a cluster.

Steps should be:

first server: ./bin/k3s server without --cluster-init (db is sqlite)
first server: ./bin/k3s server --cluster-init (db is converted to etcd)
second server: ./bin/k3s server --server https://172.25.0.2:6443/ (second server joins first and uses etcd)

bartossh · 2023-03-15T11:32:52Z

@dereknola thanks for the correction. I did a test according to your suggestion and the behaviour is the same with and without changes from this PR.
The second server connects and the etcd is started.
If there are any other tests you would like me to perform or something I should add as confirmation, please let me know.

…d. (k3s-io#6998) Problem: When support for etcd was added in 3957142, generation of certificates and keys for etcd was not gated behind use of managed etcd. Keys are generated and distributed across servers even if managed etcd is not enabled. Solution: Allow generation of certificates and keys only if managed etc is enabled. Check config.DisableETCD flag. Signed-off-by: Bartossh <[email protected]>

…d. (k3s-io#6998) Problem: When support for etcd was added in 3957142, generation of certificates and keys for etcd was not gated behind use of managed etcd. Keys are generated and distributed across servers even if managed etcd is not enabled. Solution: Allow generation of certificates and keys only if managed etc is enabled. Check config.DisableETCD flag. Signed-off-by: Bartossh <[email protected]> Signed-off-by: Derek Nola <[email protected]>

bartossh requested a review from a team as a code owner February 22, 2023 12:13

bartossh force-pushed the 3404/shouldd_not_perform_etcd_setup_if_etcd_is_not_enabled branch from 232cc1e to f3d21fd Compare February 26, 2023 08:58

bartossh mentioned this pull request Feb 26, 2023

K3s should not perform etcd setup when managed etcd is not enabled #3404

Closed

bartossh force-pushed the 3404/shouldd_not_perform_etcd_setup_if_etcd_is_not_enabled branch from f3d21fd to 38a6e2b Compare February 28, 2023 07:40

dereknola approved these changes Apr 10, 2023

View reviewed changes

tashima42 mentioned this pull request Jul 5, 2023

Update pull request template user-facing changes instructions #7878

Closed

briandowns approved these changes Jul 10, 2023

View reviewed changes

galal-hussein approved these changes Jul 10, 2023

View reviewed changes

dereknola merged commit 3461739 into k3s-io:master Jul 11, 2023

cguertin14 mentioned this pull request Aug 1, 2023

Minor: k3s update from v1.27.3+k3s1 to v1.27.4+k3s1 cguertin14/k3s-ansible-ha#23

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generation of certificates and keys for etcd gated if etcd is disabled. #6998

Generation of certificates and keys for etcd gated if etcd is disabled. #6998

bartossh commented Feb 22, 2023

brandond commented Feb 22, 2023

bartossh commented Feb 23, 2023

bartossh commented Feb 26, 2023 •

edited

Loading

brandond commented Feb 27, 2023 •

edited

Loading

bartossh commented Mar 2, 2023 •

edited

Loading

dereknola commented Mar 8, 2023

bartossh commented Mar 15, 2023 •

edited

Loading

Generation of certificates and keys for etcd gated if etcd is disabled. #6998

Generation of certificates and keys for etcd gated if etcd is disabled. #6998

Conversation

bartossh commented Feb 22, 2023

Proposed Changes

Types of Changes

Verification

Testing

Linked Issues

User-Facing Change

Further Comments

brandond commented Feb 22, 2023

bartossh commented Feb 23, 2023

bartossh commented Feb 26, 2023 • edited Loading

brandond commented Feb 27, 2023 • edited Loading

bartossh commented Mar 2, 2023 • edited Loading

dereknola commented Mar 8, 2023

bartossh commented Mar 15, 2023 • edited Loading

bartossh commented Feb 26, 2023 •

edited

Loading

brandond commented Feb 27, 2023 •

edited

Loading

bartossh commented Mar 2, 2023 •

edited

Loading

bartossh commented Mar 15, 2023 •

edited

Loading