Kubernetes Secrets Management#

Kubernetes Secrets are the default way to store sensitive data in a cluster. The problem? They are base64-encoded, not encrypted. Anyone with API access can decode them instantly.

The Default Secret Problem#

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  password: cGFzc3dvcmQxMjM=   # echo -n 'password123' | base64

echo cGFzc3dvcmQxMjM= | base64 -d
# password123

That is not encryption. It is encoding. Every engineer with kubectl get secret access can read every secret in their namespace.

What Makes This Dangerous#

• Secrets are stored in etcd in plaintext by default
• Anyone with RBAC read on secrets sees everything
• kubectl get secret -o yaml dumps credentials to terminal
• Secrets appear in pod specs, environment variables, and logs
• GitOps workflows risk committing plaintext secrets to repos

Encryption at Rest#

The first step is enabling encryption at rest in etcd.

# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: c2VjcmV0LWtleS0xMjM0NTY3ODkwMTIzNDU2
      - identity: {}

This encrypts secrets in etcd but does not solve the access control or distribution problem.

External Secrets Operator (ESO)#

ESO syncs secrets from external providers (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault) into Kubernetes Secrets.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secret-store
    kind: ClusterSecretStore
  target:
    name: db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: prod/database/password

Why ESO Works#

• Secrets live in a dedicated secrets manager, not in Git
• Automatic refresh on a configurable interval
• Supports templating for complex secret formats
• Works across cloud providers with a unified API
• The Kubernetes Secret is a synced copy, not the source of truth

Sealed Secrets (Bitnami)#

Sealed Secrets let you encrypt secrets for Git storage. Only the cluster controller can decrypt them.

# Install the controller in-cluster
helm install sealed-secrets sealed-secrets/sealed-secrets

# Encrypt a secret client-side
kubeseal --format yaml \
  --cert pub-cert.pem \
  --secret-file my-secret.yaml \
  --sealed-secret-file my-sealed-secret.yaml

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-credentials
spec:
  encryptedData:
    password: AgBghY8a...encrypted...base64...

The SealedSecret is safe to commit. The controller decrypts it into a regular Secret inside the cluster.

Trade-offs#

• Tied to the cluster's sealing key (key rotation requires re-sealing)
• No cross-cluster secret sharing without key export
• Secret values are opaque in Git (no diff visibility)

HashiCorp Vault with CSI Driver#

The Vault CSI Provider mounts secrets as files in pods without creating Kubernetes Secret objects at all.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-db-creds
spec:
  provider: vault
  parameters:
    roleName: "app-role"
    vaultAddress: "https://vault.internal:8200"
    objects: |
      - objectName: "db-password"
        secretPath: "secret/data/prod/database"
        secretKey: "password"

# Pod spec
volumes:
  - name: secrets
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: vault-db-creds

Advantages#

• Secrets never exist as Kubernetes Secret objects
• Dynamic secrets (Vault generates credentials on demand)
• Automatic lease renewal and revocation
• Full audit trail in Vault

SOPS (Secrets OPerationS)#

Mozilla SOPS encrypts secret values in YAML/JSON files while leaving keys readable.

# Encrypt with age key
sops --encrypt --age age1... secrets.yaml > secrets.enc.yaml

# secrets.enc.yaml — keys visible, values encrypted
db:
    password: ENC[AES256_GCM,data:abc123...,iv:...,tag:...]
    host: ENC[AES256_GCM,data:def456...,iv:...,tag:...]
sops:
    age:
        - recipient: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w...

Integrate with Flux or ArgoCD for GitOps decryption at deploy time.

Secret Rotation#

Static secrets are a liability. Rotation strategies vary by tool.

Zero-Downtime Rotation Pattern#

1. Add the new secret alongside the old one
2. Update applications to accept both credentials
3. Roll out the application update
4. Remove the old secret
5. Roll out again to drop the old reference

RBAC for Secrets#

Lock down who can read secrets at the namespace level.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: secret-reader
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
    resourceNames: ["app-config"]  # restrict to specific secrets

RBAC Best Practices#

• Never grant wildcard secret access (resources: ["secrets"] with no resourceNames)
• Use separate namespaces per team or environment
• Audit secret access with Kubernetes audit logging
• Deny list and watch on secrets for most roles
• Use admission controllers to block secret mounting in unauthorized pods

Decision Matrix#

Best Practices Checklist#

1. Enable encryption at rest for etcd on every cluster
2. Never commit plaintext secrets to Git, even in private repos
3. Use an external secrets manager as the source of truth
4. Rotate secrets regularly and automate the process
5. Restrict RBAC to specific secret names, not wildcards
6. Audit secret access with Kubernetes audit logging enabled
7. Prefer mounted files over environment variables (env vars leak in logs and process listings)
8. Use network policies to restrict which pods can reach the secrets manager
9. Scan CI pipelines for accidentally committed secrets (trufflehog, gitleaks)
10. Test rotation in staging before production

Wrapping Up#

Kubernetes Secrets are a starting point, not a solution. Production workloads need encryption at rest, external secret management, strict RBAC, and automated rotation. Pick the tool that matches your GitOps workflow and operational maturity, then layer in audit logging and rotation policies.

---

Article #366 -- Codelit has mass-produced 368 articles to date. Explore them at codelit.io.