Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support GKE / change the default controller namespace #99

Closed
wants to merge 8 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ script:
minikube status
while ! kubectl cluster-info; do sleep 3; done
kubectl create -f $INT_SSC_CONF
kubectl rollout status deployment/sealed-secrets-controller -n kube-system -w
kubectl rollout status deployment/sealed-secrets-controller -n sealed-secrets -w
make integrationtest CONTROLLER_IMAGE=$CONTROLLER_IMAGE
fi

Expand Down
66 changes: 63 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,11 @@ release.
**See additional TPR->CRD migration section below if updating an
existing Sealed Secrets installation from Kubernetes <= 1.7**

**See additional namespace migration section below if updating from a
version prior to 0.8.0, the controller was previously installed into
`kube-system` namespace by default but to support managed services like
GKE the default was changed to `sealed-secrets`.**

```sh
$ release=$(curl --silent "https://api.github.com/repos/bitnami-labs/sealed-secrets/releases/latest" | sed -n 's/.*"tag_name": *"\([^"]*\)".*/\1/p')

Expand All @@ -34,14 +39,14 @@ $ kubectl create -f https://github.com/bitnami-labs/sealed-secrets/releases/down
# Install SealedSecret CRD (for k8s >= 1.7)
$ kubectl create -f https://github.com/bitnami-labs/sealed-secrets/releases/download/$release/sealedsecret-crd.yaml

# Install server-side controller into kube-system namespace (by default)
# Install server-side controller into `sealed-secrets` namespace (by default)
$ kubectl create -f https://github.com/bitnami-labs/sealed-secrets/releases/download/$release/controller.yaml
```

After the `SealedSecret` resource is created with either
`sealedsecret-tpr.yaml` or `sealedsecret-crd.yaml` (depending on
Kubernetes version), `controller.yaml` will install the controller
into `kube-system` namespace, create a service account and necessary
into `sealed-secrets` namespace, create a service account and necessary
RBAC roles.

After a few moments, the controller will start, generate a key pair,
Expand Down Expand Up @@ -162,6 +167,52 @@ the generic documentation for more information on the process.
$ kubectl scale --replicas=1 -n kube-system deployment/sealed-secrets-controller
```

### Migration from SealedSecret <0.8.0 to 0.8.0+

In versions prior to 0.8.0 the controller was installed into
`kube-system` namespace by default but to support managed services like
GKE the default was changed to `sealed-secrets`.
[Issue ref](https://github.com/bitnami-labs/sealed-secrets/issues/90)

To migrate, follow this procedure:

1. Backup your existing encryption private key:
```sh
$ kubectl get secret -n sealed-secrets sealed-secrets-key -o yaml >master.key
```

2. Prepare to load your encryption private key into the new namespace:
```sh
$ sed -i 's/kube-system/sealed-secrets/g' master.key
```
Or on Mac OS:
```sh
$ sed -i '' 's/kube-system/sealed-secrets/g' master.key
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really, osx doesn't have a "normal" sed? Perhaps we can do sed <master.key >moved.key or something that works on both platforms..

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, annoying indeed. I'll see if I can come up with a generic command.

```

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should create the secret here, before installing the new controller. That way we don't need to restart the controller post-install.

Something like:

3. Load the existing master secret key into the new namespace:
    ```sh
    $ kubectl create -f moved.key
    ```

What I actually think we should do is combine these 3 steps into one, since I think it's easier to see what's going on as a single step (please disagree if you don't think so):

1. Copy your existing master key to the new namespace:
    ```sh
    $ kubectl create namespace sealed-secrets
    $ kubectl get secret -n sealed-secrets sealed-secrets-key -o yaml |
       sed 's/kube-system/sealed-secrets/' |
       kubectl create -f -
    ```
2. Install the new controller.
3. Delete the old kube-system controller.
4. Verify unsealing new/modified keys is working as expected, then delete the old kube-system master key.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anguslees will having two controllers in different namespaces create any race conditions on a live cluster? Hopefully no since then that means we can simplify/reorder these.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've gone ahead and updated this, can always re-order 3+4 if there's any possibility for race conditions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have the possibility of 2 controllers running simultaneously for a brief period during a "normal" controller upgrade (same namespace/Deployment upgrade). This should be fine (at worst we decrypt a SealedSecret twice).

3. Install the latest sealed-secrets controller into the `sealed-secrets` namespace (the new default) and wait a minute or so for it to boot
```sh
$ release=$(curl --silent "https://api.github.com/repos/bitnami-labs/sealed-secrets/releases/latest" | sed -n 's/.*"tag_name": *"\([^"]*\)".*/\1/p')
$ kubectl create -f https://github.com/bitnami-labs/sealed-secrets/releases/download/$release/controller.yaml
```

4. Scale down the new controller and then override the secret it just created with your existing one you just prepared in steps 1/2:
```sh
$ kubectl scale deployment sealed-secrets-controller -n sealed-secrets --replicas=0
$ kubectl replace secret -n sealed-secrets sealed-secrets-key master.key
```

5. Delete the old controller and encryption private key from the `kube-system` namespace:
```sh
$ kubectl delete deployment sealed-secrets-controller -n kube-system
$ kubectl delete secret sealed-secrets-key -n kube-system
```

6. Finally, scale back up the new controller:
```sh
$ kubectl scale deployment sealed-secrets-controller -n sealed-secrets --replicas=1
```

## Usage

```sh
Expand Down Expand Up @@ -236,11 +287,20 @@ No, the private key is only stored in the Secret managed by the controller (unle

If you do want to make a backup of the encryption private key, it's easy to do from an account with suitable access and:

`kubectl get secret -n sealed-secrets sealed-secrets-key -o yaml >master.key`

Or prior to 0.8.0 use:

`kubectl get secret -n kube-system sealed-secrets-key -o yaml >master.key`

NOTE: This is the controller's public + private key and should be kept omg-safe!

To restore from a backup after some disaster, just put that secret back before starting the controller - or if the controller was already started, replace the newly-created secret and restart the controller:

`kubectl replace secret -n sealed-secrets sealed-secrets-key master.key`
`kubectl delete pod -n sealed-secrets -l name=sealed-secrets-controller`

Or prior to 0.8.0 use:

`kubectl replace secret -n kube-system sealed-secrets-key master.key`
`kubectl delete pod -n kube-system -l name=sealed-secrets-controller`
`kubectl delete pod -n kube-system -l name=sealed-secrets-controller`
2 changes: 1 addition & 1 deletion cmd/kubeseal/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ import (
var (
// TODO: Verify k8s server signature against cert in kube client config.
certFile = flag.String("cert", "", "Certificate / public key to use for encryption. Overrides --controller-*")
controllerNs = flag.String("controller-namespace", metav1.NamespaceSystem, "Namespace of sealed-secrets controller.")
controllerNs = flag.String("controller-namespace", "sealed-secrets", "Namespace of sealed-secrets controller.")
controllerName = flag.String("controller-name", "sealed-secrets-controller", "Name of sealed-secrets controller.")
outputFormat = flag.String("format", "json", "Output format for sealed secret. Either json or yaml")
dumpCert = flag.Bool("fetch-cert", false, "Write certificate to stdout. Useful for later use with --cert")
Expand Down
2 changes: 1 addition & 1 deletion controller-norbac.jsonnet
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ local trim = function(str) (
str
);

local namespace = "kube-system";
local namespace = "sealed-secrets";
local controllerImage = std.extVar("CONTROLLER_IMAGE");

// This is a bit odd: Downgrade to apps/v1beta1 so we can continue
Expand Down
2 changes: 1 addition & 1 deletion integration/controller_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ func getData(s *v1.Secret) map[string][]byte {
}

func fetchKeys(c corev1.SecretsGetter) (*rsa.PrivateKey, []*x509.Certificate, error) {
s, err := c.Secrets("kube-system").Get("sealed-secrets-key", metav1.GetOptions{})
s, err := c.Secrets("sealed-secrets").Get("sealed-secrets-key", metav1.GetOptions{})
if err != nil {
return nil, nil, err
}
Expand Down