SealedSecrets on NIP

GitOps is lovely until you hit secrets. Plain Secrets can't go in Git (obvious reasons), but the GitOps model assumes Git is the source of truth. The answer: Bitnami SealedSecrets. A cluster-scoped asymmetric keypair — public key in the repo, private key in the cluster. Here is the setup, the rotation flow, and the bootstrap dance for a new cluster.

One controller per cluster

Each cluster has its own SealedSecrets controller and its own keypair. That means a SealedSecret produced for nip-cluster-prod will NOT work on nip-cluster-staging. That is intentional: secrets are bound to a cluster, which gives us a nicely isolated blast radius.

The alternative — one shared keypair across clusters — is simpler but would mean a compromised staging key also compromises prod. Not a risk profile we accept.

The sealing flow

echo -n 'super-secret-value' | kubectl create secret generic db-creds \
  --dry-run=client --from-file=password=/dev/stdin -o yaml \
  | kubeseal --controller-namespace=kube-system \
             --controller-name=sealed-secrets \
             --format=yaml > sealed-db-creds.yaml

sealed-db-creds.yaml is a SealedSecret CR containing an AES-CBC-encrypted Secret. Safe to commit.

In-cluster, the SealedSecrets controller spots it, decrypts it (with the private key), and creates a normal Secret in the namespace. Workloads mount or reference it as usual.

Rotation: re-seal in repo + commit

To rotate a secret:

1. Engineer generates the new plain Secret locally.
2. Re-seals it with kubeseal.
3. Commits the new sealed-X.yaml to Git.
4. Flux reconciles, controller decrypts, the K8s Secret updates.
5. Affected Deployments restart pods automatically (Secret hash changed — we handle this via the reloader controller).

Rotation engineer time: ~4 minutes, plus ~3 minutes for reconcile + rollout. Total: 7 minutes.

TTLs by category

Every secret carries a TTL in a nip-secret-ttl annotation. NIP runs a Sunday-evening check across all SealedSecrets, and if any will expire within 14 days, it pings the on-call on Slack.

Category	TTL
DB credentials (Postgres user/password)	90 days
API keys (OpenAI, Stripe, Sentry, etc.)	180 days
Webhook secrets (GitHub, Slack)	365 days
TLS certificates	cert-manager (60-day auto-renew)
Bootstrap keys (Flux GitRepo SSH)	not rotated (lives in 1Password)

These aren't arbitrary numbers: DB creds rotate often because in a leak scenario we want fast revocation. API keys rotate less often because every rotation also requires a trip to the 3rd-party portal.

The bootstrap dance for a new cluster

A new cluster comes up; we install the SealedSecrets controller. First-secret problem: how does the Flux SSH key get into the cluster so Flux can read the repo at all? Chicken-and-egg.

Solution: the Flux SSH key is not managed via SealedSecret. It is created manually via kubectl create secret, with the on-call pulling the key from 1Password. That is the one manual step, happening 1-2 times a year.

Once Flux is up, we generate a fresh SealedSecrets keypair, commit the public half to the GitOps repo (nip-cluster-X/sealed-secrets-pub.pem), and from there every other SealedSecret flows automatically.

What we don't use

• HashiCorp Vault — we tried it. A serious tool, but overkill at our scale of ~100 secrets. Vault's operational burden (auto-unseal, audit log, policy management) would cost more than it saves.
• Sops + age — nicer DX, but Bitnami SealedSecrets is locked-in with Flux out of the box, and we didn't want to break what works.
• Cloud KMS (GCP/AWS) — vendor lock-in, not applicable to our Hetzner footprint.

One gotcha

The kubeseal CLI version must match the in-cluster controller version. If they drift, the produced SealedSecret format may differ, and the controller cannot decrypt. The NIP UI shows the controller version up-front so engineers always know.