Kubernetes: Sealed Secrets


Why use Sealed Secrets?

Sealed Secrets provides a mechanism to encrypt a Secret object so that it is safe to store in a private or public repository.

https://github.com/bitnami-labs/sealed-secrets

How it works

Sealed Secrets comprises the following components

  1. A controller deployed to cluster
  2. A CLI tool called kubeseal
  3. A custom resource definition (CRD) called SealedSecret

Upon startup, the controller looks for a cluster-wide private/public key pair, and generates a new 4096-bit RSA key pair if not found. The private key is persisted in a Secret object in the same namespace as that of the controller. The public key portion of this is made publicly available to anyone wanting to use Sealed Secrets with this cluster.

During encryption, each value in the original Secret is symmetrically encrypted using AES-256 with a randomly generated session key. The session key is then asymmetrically encrypted with the controller’s public key using SHA256 and the original Secret’s namespace/name as the input parameter. The output of the encryption process is a string that is constructed as: length (2 bytes) of encrypted session key + encrypted session key + encrypted Secret.

When a SealedSecret custom resource is deployed to the Kubernetes cluster, the controller will pick it up, unseal it using the private key, and create a Secret resource. During decryption, the SealedSecret’s namespace/name is used again as the input parameter. This ensures that the SealedSecret and Secret are strictly tied to the same namespace and name.

The companion CLI tool kubeseal is used for creating a SealedSecret custom resource definition (CRD) from a Secret resource definition using the public key. kubeseal can communicate with the controller through the Kubernetes API server and retrieve the public key needed for encrypting a Secret at runtime. The public key may also be downloaded from the controller and saved locally to be used offline.

Install kubeseal (client)

brew (mac)

brew install kubeseal

Linux x86_64 systems

https://github.com/bitnami-labs/sealed-secrets/releases # view latest release

wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.4/kubeseal-linux-amd64 -O kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

Install Sealed Secrets

https://github.com/bitnami-labs/sealed-secrets/releases # view latest release

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.4/controller.yaml

Scopes

You need to understand a little about scopes, which are basically the three different ways that you can create a Sealed Secret.

https://github.com/bitnami-labs/sealed-secrets#scopes

There are three scopes you can create your SealedSecrets with:

  • strict (default): In this case, you need to seal your Secret considering the name and the namespace. You can’t change the name and the namespaces of your SealedSecret once you’ve created it. If you try to do that, you get a decryption error.
  • namespace-wide: This scope allows you to freely rename the SealedSecret within the namespace for which you’ve sealed the Secret.
  • cluster-wide: This scope allows you to freely move the Secret to any namespace and give it any name you wish.

You can select the scope with the –scope flag while using kubeseal:

kubeseal --scope cluster-wide --format yaml < secret.yaml > sealed-secret.yaml

You can also use annotations within your Secret to apply scopes before you pass the configuration to kubeseal:

  • sealedsecrets.bitnami.com/namespace-wide: “true” for namespace-wide
  • sealedsecrets.bitnami.com/cluster-wide: “true” for cluster-wide

If you don’t specify any annotations, then kubeseal assumes a strict scope. If you set both annotations, cluster-wide takes precedence.

Private Key

The logs from the controller reveal the name of the Secret that is created in the kube-system namespace which contains the private key pair used by the controller for unsealing Sealed Secrets deployed to the cluster. It also shows the public key.

SEALED_SECRET_POD=$(kubectl get pods -n kube-system -l name=sealed-secrets-controller -o jsonpath="{.items[0].metadata.name}")

kubectl logs $SEALED_SECRET_POD -n kube-system

Example

SEALED_SECRET_POD=$(kubectl get pods -n kube-system -l name=sealed-secrets-controller -o jsonpath="{.items[0].metadata.name}")

kubectl logs $SEALED_SECRET_POD -n kube-system
controller version: v0.12.4+dirty
2020/06/24 04:34:36 Starting sealed-secrets controller version: v0.12.4+dirty
2020/06/24 04:34:36 Searching for existing private keys
2020/06/24 04:34:38 New key written to kube-system/sealed-secrets-key59nvm # private key
2020/06/24 04:34:38 Certificate is
# public key
-----BEGIN CERTIFICATE-----
xxxxx
xxxxx
xxxxx
-----END CERTIFICATE-----

2020/06/24 04:34:38 HTTP server serving on :8080

The secret in the logs should match the actual secret name.

SEALED_SECRET_SECRET=$(kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o jsonpath="{.items[0].metadata.name}")

echo $SEALED_SECRET_SECRET
sealed-secrets-key59nvm

Upon starting, the controller searches for a Secret with the label sealedsecrets.bitnami.com/sealed-secrets-key in its namespace. If it does not find one, it creates a new one in its namespace and prints the public key portion of the key pair to its output logs. The contents of this Secret, which contains the private key pair, can be seen in YAML format with the following command:

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -oyaml

Example

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -oyaml
apiVersion: v1
items:
- apiVersion: v1
data:
tls.crt: xxxxx
tls.key: xxxxx
kind: Secret
metadata:
creationTimestamp: "2020-06-24T04:34:38Z"
generateName: sealed-secrets-key
labels:
sealedsecrets.bitnami.com/sealed-secrets-key: active
name: sealed-secrets-key59nvm
namespace: kube-system
resourceVersion: "3809988"
selfLink: /api/v1/namespaces/kube-system/secrets/sealed-secrets-key59nvm
uid: 8f61d49e-0fba-4314-8a8e-bce15d41f4fc
type: kubernetes.io/tls
kind: List
metadata:
resourceVersion: ""
selfLink: ""

Public Key

You may download the public key using the following command.

kubeseal --fetch-cert > public-key-cert.pem

kubeseal encrypts the Secret using the public key that it fetches at runtime from the controller running in the Kubernetes cluster. If a user does not have direct access to the cluster, then a cluster administrator may retrieve the public key from the controller logs and make it accessible to the user.

A SealedSecret CRD is then created using kubeseal as follows using the public key file:

kubeseal --format yaml --cert=public-key-cert.pem < secret.yaml > sealed-secret.yaml

Create Sealed Secrets

strict (default)

yaml

# create yaml
kubectl create secret generic mysecret --from-literal=password=12345 -n itsmetommy --dry-run=client -oyaml > mysecret.yaml

# seal (safe to share publicly)
kubeseal < mysecret.yaml > sealed-mysecret.yaml

# apply
kubectl apply -f sealed-mysecret.yaml

json

# create json
kubectl create secret generic mysecret --from-literal=password=12345 -n itsmetommy --dry-run=client -ojson > mysecret.json

# seal (safe to share publicly)
kubeseal < mysecret.json > sealed-mysecret.json

# apply
kubectl apply -f sealed-mysecret.json

namespace-wide

yaml

# create yaml
kubectl create secret generic mysecret-nsw --from-literal=password=12345 -n itsmetommy --dry-run=client -oyaml > mysecret-nsw.yaml

# seal (safe to share publicly)
kubeseal --format yaml --scope namespace-wide < mysecret-nsw.yaml > sealed-mysecret-nsw.yaml

# apply
kubectl apply -f sealed-mysecret-nsw.yaml

json

# create json
kubectl create secret generic mysecret-nsw --from-literal=password=12345 -n itsmetommy --dry-run=client -ojson > mysecret-nsw.json

# seal (safe to share publicly)
kubeseal --scope namespace-wide < mysecret-nsw.json > sealed-mysecret-nsw.json

# apply
kubectl apply -f sealed-mysecret-nsw.json

cluster-wide

yaml

# create yaml
kubectl create secret generic mysecret-cw --from-literal=password=12345 -n itsmetommy --dry-run=client -oyaml > mysecret-cw.yaml

# seal (safe to share publicly)
kubeseal --format yaml --scope cluster-wide < mysecret-cw.yaml > sealed-mysecret-cw.yaml

# apply
kubectl apply -f sealed-mysecret-cw.yaml

json

# create json
kubectl create secret generic mysecret-cw --from-literal=password=12345 -n itsmetommy --dry-run=client -ojson > mysecret-cw.json

# seal (safe to share publicly)
kubeseal --scope cluster-wide < mysecret-cw.json > sealed-mysecret-cw.json

# apply
kubectl apply -f sealed-mysecret-cw.json

annotations

It’s also possible to request a scope via annotations in the input secret you pass to kubeseal:

  • sealedsecrets.bitnami.com/namespace-wide: “true” -> for namespace-wide
  • sealedsecrets.bitnami.com/cluster-wide: “true” -> for cluster-wide

The lack of any of such annotations means strict mode. If both are set, cluster-wide takes precedence

NOTE: next release will consolidate this into a single sealedsecrets.bitnami.com/scope annotation.

strict (default)

There is no need to add an annotation, as the default is strict.

# create yaml
cat <<EOF > mysecret-annotations-strict.yaml
apiVersion: v1
kind: Secret
metadata:
  name: mysecret-annotations-strict
  namespace: itsmetommy
type: Opaque
data:
  secret: VGhpcyBpcyBhIHNlY3JldCE=
EOF

# seal (safe to share publicly)
kubeseal < mysecret-annotations-strict.yaml > sealed-mysecret-annotations-strict.yaml

# apply
kubectl apply -f sealed-mysecret-annotations-strict.yaml

namespace-wide

# create yaml
cat <<EOF > mysecret-annotations-nsw.yaml
apiVersion: v1
kind: Secret
metadata:
  name: mysecret-annotations-nsw
  namespace: itsmetommy
  annotations:
    sealedsecrets.bitnami.com/namespace-wide: "true"
type: Opaque
data:
  secret: VGhpcyBpcyBhIHNlY3JldCE=
EOF

# seal (safe to share publicly)
kubeseal < mysecret-annotations-nsw.yaml > sealed-mysecret-annotations-nsw.yaml

# apply
kubectl apply -f sealed-mysecret-annotations-nsw.yaml

cluster-wide

# create yaml
cat <<EOF > mysecret-annotations-cw.yaml
apiVersion: v1
kind: Secret
metadata:
  name: mysecret-annotations-cw
  namespace: itsmetommy
  annotations:
    sealedsecrets.bitnami.com/cluster-wide: "true"
type: Opaque
data:
  secret: VGhpcyBpcyBhIHNlY3JldCE=
EOF

# seal (safe to share publicly)
kubeseal < mysecret-annotations-cw.yaml > sealed-mysecret-annotations-cw.yaml

# apply
kubectl apply -f sealed-mysecret-annotations-cw.yaml -n itsmetommy

Public key

Download public key (you need access) or request it from the Kubernetes cluster admin.

kubeseal --fetch-cert > public-key-cert.pem

Seal the secret using the public key.

# seal
kubeseal --format yaml --cert=public-key-cert.pem < secret.yaml > sealed-mysecret.yaml

# apply
kubectl apply -f sealed-mysecret.yaml

View Sealed Secrets

kubectl get sealedsecrets

Logs

SEALED_SECRET_POD=$(kubectl get pod -n kube-system -l name=sealed-secrets-controller -o jsonpath="{.items[0].metadata.name}")

kubectl logs --tail=2 $SEALED_SECRET_POD -n kube-system