sealed secret prototyping

2025-06-06 17:01:30 +00:00 · 2022-11-08 19:44:56 +11:00 · 2022-11-08 19:44:56 +11:00 · ee246f3e9f
commit ee246f3e9f
parent 84890e84c3
3 changed files with 51 additions and 296 deletions
--- a/kubernetes/secrets/sealed-secrets/README.md
+++ b/kubernetes/secrets/sealed-secrets/README.md
@ -2,6 +2,16 @@

 Checkout the [Sealed Secrets GitHub Repo](https://github.com/bitnami-labs/sealed-secrets) </br>

+There are a number of use-cases where this is a really great concept. </br>
+
+1) GitOps - Storing your YAML manifests in Git and using GitOps tools to sync the manifests to your clusters (For example Flux and ArgoCD!)
+
+2) Giving a team access to secrets without revealing the secret material.
+
+developer: "I want to confirm my deployed secret value is X in the cluster" </br>
+
+developer can compare `sealedSecret` YAML in Git, with the `sealedSecret` in the cluster and confirm the value is the same. </br>
+
 ## Create a kubernetes cluster

 In this guide we we'll need a Kubernetes cluster for testing. Let's create one using [kind](https://kind.sigs.k8s.io/) </br>
@ -135,7 +145,7 @@ cat kubernetes/secrets/secret.yaml

 If you run `kubeseal` you will see it pause and expect input from `stdin`. </br>
 You can paste your secret YAML and press CTRL+D to terminate `stdin`. </br>
-You will notive it writes a `sealedSecret` to `stdout`. </br>
+You will notice it writes a `sealedSecret` to `stdout`. </br>
 We can then automate this using `|` characters. </br>

 Create a sealed secret using `stdin` :
@ -154,7 +164,6 @@ Deploy the sealed secret

 ```
 kubectl apply -f kubernetes/secrets/sealed-secrets/sealed-secret.yaml
-sealedsecret.bitnami.com/mysecret created
 ```

 Now few seconds later, see the secret
@ -167,15 +176,22 @@ mysecret              Opaque                                1      25s

 ## How the encryption key is managed

-TODO:  How the encryption key is managed and stored 
+By default the controller generates a key as we saw earlier and stores it in a Kubernetes secret. </br>
+By default, the controller will generate a new active key every 30 days.
+It keeps old keys so it can decrypt previous encrypted sealed secrets and will use the active key with new encryption. </br>

-TODO:  Set a duration to test `--key-renew-period=<value>`
+It's important to keep these keys secured. </br>
+When the controller starts it consumes all the secrets and will start using them </br>
+This means we can backup these keys in a Vault and use them to migrate our clusters if we wanted to. </br>
+
+We can also override the renewal period to increase or decrease the value. `0` turns it off </br>
+
+To showcase this I can set `--key-renew-period=<value>` to 5min to watch how it works. </br>

 ```
 apk add nano
 export KUBE_EDITOR=nano
 ```
-
 Set the flag on the command like so to add a new key every 5 min for testing:

 ```
@ -194,11 +210,9 @@ You should see a new key created under secrets in the `kube-system` namespace
 kubectl -n kube-system get secrets
 ```

-
 ## Backup your encryption keys

-TODO: * backup encryption keys 
-TODO:  migrating kubernetes clusters
+To get your keys out for backup purpose, it's as simple as grabbing a secret by label using `kubectl` :

 ```
 kubectl get secret -n kube-system \
@ -206,10 +220,11 @@ kubectl get secret -n kube-system \
  -o yaml \
  >  kubernetes/secrets/sealed-secrets/sealed-secret-keys.key
 ```
+This can be used when migrating from one cluster to another, or simply for keeping backups. </br>

 ## Migrate your encryption keys to a new cluster 

-Delete & Deploy a new Kubernetes cluster
+To test this, lets delete our cluster and recreate it. </br>

 ```
 kind delete cluster --name sealedsecrets
@ -231,22 +246,40 @@ kubectl -n kube-system get pods
 kubectl apply -f kubernetes/secrets/sealed-secrets/sealed-secret-keys.key
 ```

-restart the controller:
-```
-kubectl delete pod -n kube-system -l name=sealed-secrets-controller
-```
-
 ### apply our old sealed secret

 ```
 kubectl apply -f kubernetes/secrets/sealed-secrets/sealed-secret.yaml
 ```

-TODO:  Encrypted sealed secrets across namespaces  https://github.com/bitnami-labs/sealed-secrets#scopes
+### see sealed secret status
+
+To troubleshoot the secret, you can use the popular `kubectl describe` command. </br>
+Note that we're unable to decrypt the secret. </br>
+Why is that ? </br>
+
+We'll this is because the encryption key secrets are read when the controller starts. </br> 
+So we will need to restart the controller to that it can read ingest the encryption keys:
+
+```
+kubectl delete pod -n kube-system -l name=sealed-secrets-controller
+```

 ## Re-encrypting secrets with the latest key

+We can also use `kubeseal --re-encrypt` to encrypt a secret again. </br>
+Let's say we want to encrypt with the latest key. </br>
+This will re-encrypt the sealed secret without having to pull the actual secret to the client </br>
+
 ```
-kubeseal --re-encrypt <my_sealed_secret.json >tmp.json \
-  && mv tmp.json my_sealed_secret.json
+cat ./kubernetes/secrets/sealed-secrets/sealed-secret.yaml \
+| kubeseal --re-encrypt -o yaml
+```
+
+I can then save this to override the original old local sealed secret file:
+
+```
+cat ./kubernetes/secrets/sealed-secrets/sealed-secret.yaml \
+| kubeseal --re-encrypt -o yaml \
+> tmp.yaml && mv tmp.yaml ./kubernetes/secrets/sealed-secrets/sealed-secret.yaml
 ```
--- a/kubernetes/secrets/sealed-secrets/controller-helm-v0.19.1.yaml
+++ b/kubernetes/secrets/sealed-secrets/controller-helm-v0.19.1.yaml
@ -1,278 +0,0 @@
---
-# Source: sealed-secrets/templates/service-account.yaml
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: sealed-secrets
-  namespace: kube-system
-  labels:
-    app.kubernetes.io/name: sealed-secrets
-    helm.sh/chart: sealed-secrets-2.7.0
-    app.kubernetes.io/instance: sealed-secrets
-    app.kubernetes.io/managed-by: Helm
-    app.kubernetes.io/version: v0.19.1
---
-# Source: sealed-secrets/templates/cluster-role.yaml
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
-  name: secrets-unsealer
-  labels:
-    app.kubernetes.io/name: sealed-secrets
-    helm.sh/chart: sealed-secrets-2.7.0
-    app.kubernetes.io/instance: sealed-secrets
-    app.kubernetes.io/managed-by: Helm
-    app.kubernetes.io/version: v0.19.1
-rules:
-  - apiGroups:
-      - bitnami.com
-    resources:
-      - sealedsecrets
-    verbs:
-      - get
-      - list
-      - watch
-  - apiGroups:
-      - bitnami.com
-    resources:
-      - sealedsecrets/status
-    verbs:
-      - update
-  - apiGroups:
-      - ""
-    resources:
-      - secrets
-    verbs:
-      - get
-      - list
-      - create
-      - update
-      - delete
-      - watch
-  - apiGroups:
-      - ""
-    resources:
-      - events
-    verbs:
-      - create
-      - patch
---
-# Source: sealed-secrets/templates/cluster-role-binding.yaml
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
-  name: sealed-secrets
-  labels:
-    app.kubernetes.io/name: sealed-secrets
-    helm.sh/chart: sealed-secrets-2.7.0
-    app.kubernetes.io/instance: sealed-secrets
-    app.kubernetes.io/managed-by: Helm
-    app.kubernetes.io/version: v0.19.1
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: secrets-unsealer
-subjects:
-  - apiGroup: ""
-    kind: ServiceAccount
-    name: sealed-secrets
-    namespace: kube-system
---
-# Source: sealed-secrets/templates/role.yaml
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: sealed-secrets-key-admin
-  namespace: kube-system
-  labels:
-    app.kubernetes.io/name: sealed-secrets
-    helm.sh/chart: sealed-secrets-2.7.0
-    app.kubernetes.io/instance: sealed-secrets
-    app.kubernetes.io/managed-by: Helm
-    app.kubernetes.io/version: v0.19.1
-rules:
-  - apiGroups:
-      - ""
-    resourceNames:
-      - sealed-secrets-key
-    resources:
-      - secrets
-    verbs:
-      - get
-  - apiGroups:
-      - ""
-    resources:
-      - secrets
-    verbs:
-      - create
-      - list
---
-# Source: sealed-secrets/templates/role.yaml
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: sealed-secrets-service-proxier
-  namespace: kube-system
-  labels:
-    app.kubernetes.io/name: sealed-secrets
-    helm.sh/chart: sealed-secrets-2.7.0
-    app.kubernetes.io/instance: sealed-secrets
-    app.kubernetes.io/managed-by: Helm
-    app.kubernetes.io/version: v0.19.1
-rules:
-  - apiGroups:
-      - ""
-    resourceNames:
-      - sealed-secrets
-    resources:
-      - services
-    verbs:
-      - get
-  - apiGroups:
-      - ""
-    resourceNames:
-      - 'http:sealed-secrets:'
-      - 'http:sealed-secrets:http'
-      - sealed-secrets
-    resources:
-      - services/proxy
-    verbs:
-      - create
-      - get
---
-# Source: sealed-secrets/templates/role-binding.yaml
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: sealed-secrets-key-admin
-  namespace: kube-system
-  labels:
-    app.kubernetes.io/name: sealed-secrets
-    helm.sh/chart: sealed-secrets-2.7.0
-    app.kubernetes.io/instance: sealed-secrets
-    app.kubernetes.io/managed-by: Helm
-    app.kubernetes.io/version: v0.19.1
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: Role
-  name: sealed-secrets-key-admin
-subjects:
-  - apiGroup: ""
-    kind: ServiceAccount
-    name: sealed-secrets
-    namespace: kube-system
---
-# Source: sealed-secrets/templates/role-binding.yaml
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: sealed-secrets-service-proxier
-  namespace: kube-system
-  labels:
-    app.kubernetes.io/name: sealed-secrets
-    helm.sh/chart: sealed-secrets-2.7.0
-    app.kubernetes.io/instance: sealed-secrets
-    app.kubernetes.io/managed-by: Helm
-    app.kubernetes.io/version: v0.19.1
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: Role
-  name: sealed-secrets-service-proxier
-subjects:
- apiGroup: rbac.authorization.k8s.io
-  kind: Group
-  name: system:authenticated
---
-# Source: sealed-secrets/templates/service.yaml
-apiVersion: v1
-kind: Service
-metadata:
-  name: sealed-secrets
-  namespace: kube-system
-  labels:
-    app.kubernetes.io/name: sealed-secrets
-    helm.sh/chart: sealed-secrets-2.7.0
-    app.kubernetes.io/instance: sealed-secrets
-    app.kubernetes.io/managed-by: Helm
-    app.kubernetes.io/version: v0.19.1
-spec:
-  type: ClusterIP
-  ports:
-    - name: http
-      port: 8080
-      targetPort: http
-      nodePort: null
-  selector:
-    app.kubernetes.io/name: sealed-secrets
-    app.kubernetes.io/instance: sealed-secrets
---
-# Source: sealed-secrets/templates/deployment.yaml
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: sealed-secrets
-  namespace: kube-system
-  labels:
-    app.kubernetes.io/name: sealed-secrets
-    helm.sh/chart: sealed-secrets-2.7.0
-    app.kubernetes.io/instance: sealed-secrets
-    app.kubernetes.io/managed-by: Helm
-    app.kubernetes.io/version: v0.19.1
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: sealed-secrets
-      app.kubernetes.io/instance: sealed-secrets
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: sealed-secrets
-        app.kubernetes.io/instance: sealed-secrets
-    spec:
-      securityContext:
-        fsGroup: 65534
-      serviceAccountName: sealed-secrets
-      containers:
-        - name: controller
-          command:
-            - controller
-          args:
-            - --update-status
-            - --key-prefix
-            - "sealed-secrets-key"
-          image: docker.io/bitnami/sealed-secrets-controller:v0.19.1
-          imagePullPolicy: IfNotPresent
-          ports:
-            - containerPort: 8080
-              name: http
-          livenessProbe:
-            failureThreshold: 3
-            initialDelaySeconds: 0
-            periodSeconds: 10
-            successThreshold: 1
-            timeoutSeconds: 1
-            httpGet:
-              path: /healthz
-              port: http
-          readinessProbe:
-            failureThreshold: 3
-            initialDelaySeconds: 0
-            periodSeconds: 10
-            successThreshold: 1
-            timeoutSeconds: 1
-            httpGet:
-              path: /healthz
-              port: http
-          resources:
-            limits: {}
-            requests: {}
-          securityContext:
-            readOnlyRootFilesystem: true
-            runAsNonRoot: true
-            runAsUser: 1001
-          volumeMounts:
-            - mountPath: /tmp
-              name: tmp
-      volumes:
-        - name: tmp
-          emptyDir: {}
--- a/kubernetes/secrets/sealed-secrets/sealed-secret.yaml
+++ b/kubernetes/secrets/sealed-secrets/sealed-secret.yaml
@ -6,7 +6,7 @@ metadata:
  namespace: default
 spec:
  encryptedData:
-    secret.json: AgBic1vT0+Y9WcUY3GqOlX+Zjstl7IDEY/U/XOykbRXGtUo6kgAG7Bkq/CyoXqDaNKD+4UhRpd9LZUw/6pvMpAjqsgSQw0ZBaALivxyZ7xSJ3/CpfsLlatWQoVWsnIoY8s0VW0eKhKPq0t30sNJSghH9yni3dgvBI3cyS1Xe2s9A7m/eUqUJqa82UE5y1UuRTRNiSUPmAVCaaW1md6K17DB9V7TE4mQn23IYF/xxSjjEhYFwsra2d7AXvSJiY63mHKRltQ6NxzJGA7xpvJTPV2MY1YIz8wsyKk8gQ2WZnnJrNGCoRlkE3hw2ZUFJGrivlWogcVYDL8EW8ngEEGdrhAbbTwzV17GDVt4rqgk6ag37yXL5rc01YD6BVTDGLglb59sRtRNkHAFr8GZmawVosswjRRDNHhzemgmpI2sOAn/ftvM5bjBNStnLBXAvOrh6sKDAiC2hERYegSjc/dwa9sdKI095lOduNXk55HorlyfiJ6bSV7h9Qc/A6P/whjL1YkZ+3iZA2b2DLEOWPSbaRhUYv4vfwkk3Abis3olXfI1dKwRec039AibhSeE/8FcrAik21XScWXT/QEH80yq6fzOUTMQRXLRzx7XRHfawEt6Oz5WV8mmR/XwueYx6Hw/SG8hxx6UUQqPaS66xVJgyL0g2EwonjLEtNQc+5/xrhaWAqyyRoeCB+2da5ZSUYWVhYdw99XNeg9HQ5mQpdPjMBW0/83gG2ZIr8XPgoo3OalYD5vwL4Q8drw==
+    secret.json: AgBGAxpwLeMnCxvuGAY0tipvPVexoqucs3qKUNwD8MNy9PXw+SwLnUzMGSCX1Q/I8/EvNcDOY7vkenHts4Rzexva6m0MNMn60223QEm1394HRQQS5xELnCyAsjpSeHuN4TizxmWU/O8IBPvAMYINFapA0Hc3VADArKaToECyJJxAxBmUFUuQTVdWS8E/yX1ZMibpwYdPQt0PK7i/8eA6Qln7KfvNxp4KaWz4enL9uScIztXLfeHaDgE7CazaINL/vmkUIB0J7tbAuhLwPpy86pcL+MioeXTk4Kze7ikPn8dXSdYgSzVR85CaaFwueXWFujqUDEgu6ROGH0CyLZ8aK/Z8X3NAnIxupQLl26xbRDONXwjLsEFAOHLmX+J/gpkDJBU6F+Nt/Unc7q2KBPPuQgdozr3e7ZtCRnUO7kU75UccwgqHF9o/8n4j4+/WjkepTuKy6hBP16sFRTdTWg+iP79xBPLAKyJ1urRVYRhLkzgRf15Wyk28XeGMw0Ip8hEJnDIzTc9IepsMdXDHQppHIUUfp8LIgeWZUxsb//Z7+28qwuffEr436cgKI6jf4g8eIaIFNyk1JRU2cmcDgEQEhTYaskqsEedaDfBWzjxhtqGYh2SJIDvpZkL43rZQLM68EhApFkefo/6goNRupXAEUY8xb9XP3Z6nMiIlGZMU5pl4A6NPpRSEgyipHaM8E8MqN9dyQVMgcnRBWYjk8WZIQHWrKiw0BtNTgVuVBxKkpQkqQHO7FXLdQw==
  template:
    metadata:
      creationTimestamp: null