The DevOps Odyssey, Part 11: Never Reseal Again — Backing Up Sealed Secrets Keys and Testing It with KIND

no key could decrypt secret (.dockerconfigjson)

The last time I saw this error, it cost me an afternoon.

This time, I caused it on purpose.

The Part That Did Not Come Back

In Part 9, my K3s cluster disappeared and I rebuilt it from scratch.

Terraform brought the machines back. Ansible brought K3s back. Argo CD brought the apps back.

The secrets did not come back.

Every SealedSecret in Git was still there, but the new cluster had a new Sealed Secrets controller with a new key. None of the old secrets could be decrypted. I had to dig up the original values and reseal everything by hand.

I ended Part 9 with this line: recoverability is the real goal.

This post is about fixing the part that was not recoverable, and then proving the fix works.

How Sealed Secrets Keys Actually Work

I have used Sealed Secrets since Part 4 to keep secrets in Git. Under the hood, it uses a key pair that lives inside the cluster.

The public cert is what kubeseal uses to encrypt. It is safe to share.
The private key lives in a Secret in kube-system. Only the controller can use it to decrypt.

When the cluster is gone, the private key is gone too. That is by design: an encrypted secret in Git is only useful to the cluster that holds the key.

It also means the fix is simple in theory. If the private key survives the cluster, the SealedSecrets in Git survive the cluster.

When I looked at my cluster, I found more than one key:

NAME                      TYPE                DATA   AGE
sealed-secrets-key2k6fl   kubernetes.io/tls   2      193d
sealed-secrets-keyh8rvr   kubernetes.io/tls   2      163d
sealed-secrets-keysvclv   kubernetes.io/tls   2      133d
sealed-secrets-keywddld   kubernetes.io/tls   2      103d
sealed-secrets-keygzjwx   kubernetes.io/tls   2      73d
sealed-secrets-keyk25k8   kubernetes.io/tls   2      43d
sealed-secrets-keyf6scr   kubernetes.io/tls   2      13d

Seven keys, about 30 days apart.

That is the controller’s default key rotation. Every 30 days it generates a new key, and new seals use the newest one. The part that surprised me: old keys are never deleted. They stay so that older SealedSecrets keep working.

Two things follow from that:

Back up every key, not just the newest. A secret sealed in April still needs the April key.
A backup goes stale. After the next rotation, anything sealed with the new key is not covered until I export again.

Which Key Am I Actually Sealing With?

I seal secrets with a small wrapper script around kubeseal. It can use a cached public cert file or fetch the current one from the cluster.

Out of curiosity, I compared fingerprints. The cached cert matched the oldest key, the one from rebuild day.

That is perfectly fine, since every key stays valid. But it was a good reminder that the key doing the sealing is not always the newest one, and that it is worth checking what the system is doing instead of assuming.

In practice, that meant the 30-day rotation had never touched anything I sealed. I had been running a one-stable-key setup without deciding to. I have since changed the script to fetch the current cert from the cluster by default, so new seals follow whatever key the controller is actually using.

Backing Up the Keys

The backup itself is almost anticlimactic:

kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secrets-keys.yaml

I wrapped it in a small backup.sh that strips the old cluster’s metadata (uid, resourceVersion, and friends) so the file applies cleanly to a new cluster, locks down the file permissions, and prints each key’s date and public fingerprint as a sanity check.

One important note: this file contains private keys in plain text. It never goes into Git. It is the one file that can decrypt every secret in my repos.

Where it lives long term is its own decision. I am leaning toward Vaultwarden in my homelab, plus an offline copy, so a restore does not depend on the homelab being up at the same moment the cloud cluster is down.

The Experiment

A backup you have never restored is just a hope.

So I wanted to answer one question: can a completely different cluster decrypt my real K3s secrets using only this backup?

KIND is perfect for this. It runs a throwaway Kubernetes cluster in Docker on my laptop. I gave it its own kubeconfig file, so it never touched my normal ~/.kube/config.

For test subjects, I picked two low-stakes SealedSecrets from K3s: the image pull secrets for my blog and my photo app.

Then I made two predictions.

Prediction 1: without the backup, it should fail

I created the KIND cluster and installed the same Sealed Secrets Helm chart version my K3s cluster runs. No keys restored.

Then I applied the two SealedSecrets from K3s.

PASS  janus-blog/ghcr-token stays sealed without backup (controller: "no key could decrypt secret (.dockerconfigjson)")
PASS  photo-show/ghcr-token stays sealed without backup (controller: "no key could decrypt secret (.dockerconfigjson)")

There it was. The exact Part 9 failure, reproduced on purpose.

It looks like a boring step, but it is the most important one. Without it, a success later could just mean I tested the wrong thing. With it, any success has to come from the backup.

Prediction 2: with the backup, it should work, and the data should match

I applied the backup and restarted the controller, since it only loads keys at startup:

kubectl apply -f sealed-secrets-keys.yaml
kubectl rollout restart deploy/sealed-secrets -n kube-system

Then I re-applied the same two SealedSecrets. This time the controller created both Secrets.

But “a Secret appeared” is not proof that it holds the right content. I also did not want to print secret values in my terminal, or in this post. So the script compares a sha256 hash of each Secret’s data in KIND against the same hash from the real K3s cluster:

PASS  janus-blog/ghcr-token unsealed in KIND, sha256=<matches k3s>
PASS  photo-show/ghcr-token unsealed in KIND, sha256=<matches k3s>

ALL PASS - backup restores sealing keys to a new cluster

Identical hashes means identical data, with no plaintext ever shown.

One honest caveat: both secrets happened to be sealed with the oldest key, so the newer keys only passed structure checks here. Sealing a dummy secret with each public cert would cover all seven.

Then the script deleted the KIND cluster as if it had never existed.

Bonus: no cluster at all

kubeseal also has a recovery mode that decrypts a SealedSecret using only the backup file, with no controller and no cluster:

kubeseal --recovery-unseal --recovery-private-key sealed-secrets-keys.yaml < sealed.yaml

It worked for both secrets. That is my break-glass option if everything else is down.

The Restore Order

The experiment made one thing clear: order matters.

If the Sealed Secrets controller starts before the keys are restored, it generates its own new key. That is not fatal, but it means an extra restart, failed secrets that need a re-sync, and a key that is not in any backup.

My rebuild flow from Part 9 gets one new step, right before Argo CD:

1. k3s up                        Terraform + Ansible
2. kubectl apply -f <key backup> the new step
3. Argo CD bootstrap             Ansible playbook + app-of-apps
4. Argo CD syncs sealed-secrets  controller starts and finds the restored keys
5. Argo CD syncs apps            SealedSecrets unseal, nothing to reseal

One extra kubectl apply replaces an afternoon of resealing.

So, Should Keys Rotate?

A 30-day rotation and a backup that goes stale do not get along, so I spent some time on this.

The most useful realization: rotation does not make old keys useless. The controller keeps every one of them, and my backup holds all of them anyway. Rotation only limits which future secrets a single key can open.

So the choice is simply to rotate or not, and both are valid. One gives a smaller blast radius per key. The other gives one stable key and a backup that stays good.

The blast radius argument is weaker than it sounds, though. Every key lives in the same kube-system namespace and in the same backup file. Anyone who gets one of them gets all seven. Rotation only helps if a single key leaks on its own.

The script change makes this real for me. Now that new seals follow the newest key, every 30-day rotation leaves a gap where something is sealed with a key my backup has never seen.

So I am leaning toward turning automatic rotation off. The controller’s --key-renew-period=0 flag stops new keys from being generated. I would rotate by hand about once a year instead: new key, fresh backup, and a rerun of the KIND test. One deliberate rotation with a verified backup beats twelve silent ones. Some setups go further and bring their own key, generated outside the cluster, but with a tested backup that buys me nothing extra.

The one setup I want to avoid is the middle: automatic rotation without automatic backups.

Whichever way it goes, three rules hold:

Any time a new key starts sealing, refresh the backup.
Never prune keys from the backup. A SealedSecret does not record which key sealed it, so there is no cheap way to prove a key is unused. Retire a key only after every secret it sealed has new values.
Do not auto-reseal everything after each rotation. The old ciphertext is still in Git history and the old key is still in the cluster, so resealing the same values with a newer key buys very little. Rotating the secret values is what actually matters.

Final Thoughts

In Part 9, the cluster came back, but the secrets did not.

The problem was never Sealed Secrets. It was doing exactly what it was designed to do. The problem was that the most important piece of state in the cluster, the private key, existed in only one place.

Now it exists in two. And more importantly, I have watched a completely different cluster use it to unseal my real secrets.

The best part is that none of this stays in my OCI cluster. The same questions apply to the clusters I run at work: where do the sealing keys live, how many are there, which one is actually being used, and has anyone ever restored them somewhere else? I can bring those lessons, and the KIND test, straight back to my day job.

A backup is only real once you have restored from it.