# Introduction to Sealed Secrets Checkout the [Sealed Secrets GitHub Repo](https://github.com/bitnami-labs/sealed-secrets)
There are a number of use-cases where this is a really great concept.
1) GitOps - Storing your YAML manifests in Git and using GitOps tools to sync the manifests to your clusters (For example Flux and ArgoCD!) 2) Giving a team access to secrets without revealing the secret material. developer: "I want to confirm my deployed secret value is X in the cluster"
developer can compare `sealedSecret` YAML in Git, with the `sealedSecret` in the cluster and confirm the value is the same.
## Create a kubernetes cluster In this guide we we'll need a Kubernetes cluster for testing. Let's create one using [kind](https://kind.sigs.k8s.io/)
``` kind create cluster --name sealedsecrets --image kindest/node:v1.23.5 ``` See cluster up and running: ``` kubectl get nodes NAME STATUS ROLES AGE VERSION sealedsecrets-control-plane Ready control-plane,master 2m12s v1.23.5 ``` ## Run a container to work in ### run Alpine Linux: ``` docker run -it --rm -v ${HOME}:/root/ -v ${PWD}:/work -w /work --net host alpine sh ``` ### install kubectl ``` apk add --no-cache curl curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl chmod +x ./kubectl mv ./kubectl /usr/local/bin/kubectl ``` ### install helm ``` curl -o /tmp/helm.tar.gz -LO https://get.helm.sh/helm-v3.10.1-linux-amd64.tar.gz tar -C /tmp/ -zxvf /tmp/helm.tar.gz mv /tmp/linux-amd64/helm /usr/local/bin/helm chmod +x /usr/local/bin/helm ``` ### test cluster access: ``` /work # kubectl get nodes NAME STATUS ROLES AGE VERSION sealedsecrets-control-plane Ready control-plane,master 3m26s v1.23.5 ``` ## Install Sealed Secret Controller ### download the YAML In this demo we'll use version [0.19.1](https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.19.1/controller.yaml) of the sealed secrets controller downloaded from the [Github releases](https://github.com/bitnami-labs/sealed-secrets/releases) page ``` curl -L -o ./kubernetes/secrets/sealed-secrets/controller-v0.19.1.yaml https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.19.1/controller.yaml ``` ### install using Helm You can also install the controller using `helm` ``` helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets helm search repo sealed-secrets --versions helm template sealed-secrets --version 2.7.0 -n kube-system sealed-secrets/sealed-secrets \ > ./kubernetes/secrets/sealed-secrets/controller-helm-v0.19.1.yaml ``` With `helm template` we can explore the YAML and then replace the `helm template` with `helm install` to install the chart ### install using YAML manifest ``` kubectl apply -f kubernetes/secrets/sealed-secrets/controller-v0.19.1.yaml ``` ### Check the installation The controller deploys to the `kube-system` namespace by default. ``` kubectl -n kube-system get pods ``` Check the logs of the sealed secret controller ``` kubectl -n kube-system logs -l name=sealed-secrets-controller --tail -1 ``` From the logs we can see that it writes the encryption key its going to use as a kubernetes secret
Example log: ``` 2022/11/05 21:38:20 New key written to kube-system/sealed-secrets-keymwzn9 ``` ## Encryption keys ``` kubectl -n kube-system get secrets kubectl -n kube-system get secret sealed-secrets-keygxlvg -o yaml ``` ## Download KubeSeal The same way we downloaded the sealed secrets controller from the [GitHub releases](https://github.com/bitnami-labs/sealed-secrets/releases) page, we'll want to download kubeseal from the assets section ``` curl -L -o /tmp/kubeseal.tar.gz \ https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.19.1/kubeseal-0.19.1-linux-amd64.tar.gz tar -xzf /tmp/kubeseal.tar.gz -C /tmp/ chmod +x /tmp/kubeseal mv /tmp/kubeseal /usr/local/bin/ ``` We can now run `kubeseal --help` ## Sealing a basic Kubernetes Secret Looks at our existing Kubernetes secret YAML ``` cat kubernetes/secrets/secret.yaml ``` If you run `kubeseal` you will see it pause and expect input from `stdin`.
You can paste your secret YAML and press CTRL+D to terminate `stdin`.
You will notice it writes a `sealedSecret` to `stdout`.
We can then automate this using `|` characters.
Create a sealed secret using `stdin` : ``` cat kubernetes/secrets/secret.yaml | kubeseal -o yaml > kubernetes/secrets/sealed-secrets/sealed-secret.yaml ``` Create a sealed secret using a YAML file: ``` kubeseal -f kubernetes/secrets/secret.yaml -o yaml > kubernetes/secrets/sealed-secrets/sealed-secret.yaml ``` Deploy the sealed secret ``` kubectl apply -f kubernetes/secrets/sealed-secrets/sealed-secret.yaml ``` Now few seconds later, see the secret ``` kubectl -n default get secret NAME TYPE DATA AGE mysecret Opaque 1 25s ``` ## How the encryption key is managed By default the controller generates a key as we saw earlier and stores it in a Kubernetes secret.
By default, the controller will generate a new active key every 30 days. It keeps old keys so it can decrypt previous encrypted sealed secrets and will use the active key with new encryption.
It's important to keep these keys secured.
When the controller starts it consumes all the secrets and will start using them
This means we can backup these keys in a Vault and use them to migrate our clusters if we wanted to.
We can also override the renewal period to increase or decrease the value. `0` turns it off
To showcase this I can set `--key-renew-period=` to 5min to watch how it works.
``` apk add nano export KUBE_EDITOR=nano ``` Set the flag on the command like so to add a new key every 5 min for testing: ``` spec: containers: - command: - controller - --key-renew-period=5m kubectl edit deployment/sealed-secrets-controller --namespace=kube-system ``` You should see a new key created under secrets in the `kube-system` namespace ``` kubectl -n kube-system get secrets ``` ## Backup your encryption keys To get your keys out for backup purpose, it's as simple as grabbing a secret by label using `kubectl` : ``` kubectl get secret -n kube-system \ -l sealedsecrets.bitnami.com/sealed-secrets-key \ -o yaml \ > kubernetes/secrets/sealed-secrets/sealed-secret-keys.key ``` This can be used when migrating from one cluster to another, or simply for keeping backups.
## Migrate your encryption keys to a new cluster To test this, lets delete our cluster and recreate it.
``` kind delete cluster --name sealedsecrets kind create cluster --name sealedsecrets --image kindest/node:v1.23.5 # check the cluster kubectl get nodes # redeploy sealed-secrets controller kubectl apply -f kubernetes/secrets/sealed-secrets/controller-v0.19.1.yaml kubectl -n kube-system get pods ``` ### restore our encryption keys ``` kubectl apply -f kubernetes/secrets/sealed-secrets/sealed-secret-keys.key ``` ### apply our old sealed secret ``` kubectl apply -f kubernetes/secrets/sealed-secrets/sealed-secret.yaml ``` ### see sealed secret status To troubleshoot the secret, you can use the popular `kubectl describe` command.
Note that we're unable to decrypt the secret.
Why is that ?
We'll this is because the encryption key secrets are read when the controller starts.
So we will need to restart the controller to that it can read ingest the encryption keys: ``` kubectl delete pod -n kube-system -l name=sealed-secrets-controller ``` ## Re-encrypting secrets with the latest key We can also use `kubeseal --re-encrypt` to encrypt a secret again.
Let's say we want to encrypt with the latest key.
This will re-encrypt the sealed secret without having to pull the actual secret to the client
``` cat ./kubernetes/secrets/sealed-secrets/sealed-secret.yaml \ | kubeseal --re-encrypt -o yaml ``` I can then save this to override the original old local sealed secret file: ``` cat ./kubernetes/secrets/sealed-secrets/sealed-secret.yaml \ | kubeseal --re-encrypt -o yaml \ > tmp.yaml && mv tmp.yaml ./kubernetes/secrets/sealed-secrets/sealed-secret.yaml ```