Secret Management Architecture#

Hardcoded secrets in code, environment variables in plain text, shared credentials on Slack — these are the security anti-patterns that lead to breaches. Secret management is the discipline of storing, accessing, rotating, and auditing sensitive credentials with zero trust assumptions.

Why Secret Management Matters#

A single leaked API key can cost millions:

Leaked AWS key on GitHub → bot scrapes it in minutes
→ crypto miners spin up hundreds of instances
→ $50,000+ bill before anyone notices

Secrets include: API keys, database passwords, TLS certificates, SSH keys, OAuth tokens, encryption keys, service account credentials, and webhook signing secrets.

Vault Architecture#

A secrets vault is a centralized, encrypted, access-controlled store for sensitive data.

Core Components#

┌─────────────────────────────────┐
│           Vault Server          │
│  ┌───────────┐  ┌────────────┐ │
│  │ Auth      │  │ Secret     │ │
│  │ Backend   │  │ Engines    │ │
│  ├───────────┤  ├────────────┤ │
│  │ Policies  │  │ Encryption │ │
│  │ Engine    │  │ Barrier    │ │
│  └───────────┘  └────────────┘ │
│  ┌───────────────────────────┐ │
│  │    Storage Backend        │ │
│  │  (Consul, DynamoDB, etc.) │ │
│  └───────────────────────────┘ │
└─────────────────────────────────┘

• Auth backends — authenticate clients (tokens, OIDC, AWS IAM, Kubernetes SA)
• Secret engines — generate, store, and manage secrets (KV, database, PKI, transit)
• Policies — fine-grained access control (path-based ACLs)
• Encryption barrier — all data encrypted before hitting storage
• Storage backend — durable persistence layer (Consul, DynamoDB, Raft)

Seal/Unseal Mechanism#

Vault starts in a sealed state. Unsealing requires a threshold of key shares (Shamir's Secret Sharing). Auto-unseal using cloud KMS (AWS KMS, GCP KMS, Azure Key Vault) is standard in production.

Tools Comparison#

HashiCorp Vault#

The industry standard. Supports dynamic secrets, encryption as a service, PKI, and SSH certificate management. Runs self-hosted or as HCP Vault (managed).

Strengths: flexibility, secret engines ecosystem, transit encryption, open-source core. Trade-off: operational complexity, requires dedicated infrastructure.

AWS Secrets Manager#

Fully managed, native AWS integration. Automatic rotation for RDS, Redshift, and DocumentDB credentials. Pay-per-secret, pay-per-API-call pricing.

Strengths: zero ops, Lambda-based rotation, CloudFormation support. Trade-off: AWS-only, limited to key-value secrets, no transit encryption.

Doppler#

Developer-first secret management. Syncs secrets to any platform (Vercel, AWS, GCP, Kubernetes, Docker). Universal dashboard across environments.

Strengths: excellent DX, environment syncing, change history, integrations. Trade-off: SaaS-only, less suited for advanced cryptographic use cases.

1Password (Secrets Automation)#

Extends 1Password vaults to CI/CD and infrastructure. Connect server runs in your environment, secrets never leave your network.

Strengths: teams already using 1Password, simple setup, Connect server model. Trade-off: newer product, fewer enterprise features than Vault.

Rotation Strategies#

Static secrets that never rotate are ticking time bombs.

Scheduled Rotation#

Every 30 days:
  1. Generate new credential
  2. Update vault with new value
  3. Applications fetch new secret on next read
  4. Old credential remains valid for grace period
  5. Old credential revoked after grace period

Rotation Best Practices#

• Dual-credential pattern — maintain two active credentials, rotate one at a time
• Grace periods — keep old secrets valid for 1-2 TTL cycles after rotation
• Automated rotation — never rely on humans to rotate secrets
• Rotation verification — test that new credentials work before revoking old ones
• Alert on rotation failure — failed rotation should page on-call

Dynamic Secrets#

Dynamic secrets are generated on-demand with automatic expiration. This is Vault's most powerful feature.

App requests database credentials → Vault creates new DB user
→ Credentials valid for 1 hour → Vault revokes user after TTL
→ No shared credentials, no rotation needed

Supported Dynamic Secret Types#

• Database — PostgreSQL, MySQL, MongoDB, MSSQL (unique user per lease)
• AWS — IAM users, STS assumed roles, federation tokens
• PKI — X.509 certificates with configurable TTL
• SSH — signed SSH certificates (no authorized_keys management)
• Consul — ACL tokens scoped to specific policies

Why Dynamic Secrets Win#

Encryption at Rest and In Transit#

Encryption at Rest#

Secrets must be encrypted before storage. Vault's encryption barrier handles this transparently using AES-256-GCM. Cloud providers encrypt their secret stores with service-managed or customer-managed KMS keys.

Application secret → Vault encrypts with barrier key
→ Barrier key encrypted with master key
→ Master key split via Shamir's or protected by cloud KMS
→ Encrypted data written to storage backend

Transit Encryption (Encryption as a Service)#

Vault's transit engine lets applications encrypt/decrypt data without managing keys:

POST /v1/transit/encrypt/my-key
{ "plaintext": "base64-encoded-data" }

→ Returns ciphertext
→ Application stores ciphertext in its own database
→ Key never leaves Vault

Use cases: encrypting PII in databases, signing JWTs, generating HMACs.

Secret Injection Patterns#

Environment Variables#

The simplest pattern. Secrets injected at container startup.

# Docker Compose
environment:
  DATABASE_URL: ${DATABASE_URL}  # pulled from .env or secret store

Risk: environment variables visible in process listings, crash dumps, and debug endpoints.

Sidecar/Init Container Pattern#

A sidecar container fetches secrets and writes them to a shared volume.

initContainers:
  - name: vault-agent
    image: vault:latest
    args: ["agent", "-config=/etc/vault/agent.hcl"]
    volumeMounts:
      - name: secrets
        mountPath: /vault/secrets
containers:
  - name: app
    volumeMounts:
      - name: secrets
        mountPath: /vault/secrets
        readOnly: true

SDK/API Direct Fetch#

Application fetches secrets at runtime via SDK. Most secure — secrets exist only in memory.

import hvac
client = hvac.Client(url='https://vault.internal:8200')
secret = client.secrets.kv.v2.read_secret_version(path='myapp/db')
db_password = secret['data']['data']['password']

Kubernetes Secrets vs External Secrets Operator#

Native Kubernetes Secrets#

Kubernetes secrets are base64-encoded (not encrypted) and stored in etcd.

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  password: cGFzc3dvcmQxMjM=  # base64, NOT encrypted

Problems: base64 is not encryption, secrets stored in etcd in plain text (unless etcd encryption is enabled), RBAC is the only access control, no audit trail, no rotation.

External Secrets Operator (ESO)#

ESO syncs secrets from external vaults into Kubernetes secrets automatically.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: secret/data/myapp/db
        property: password

Advantages: single source of truth in vault, automatic refresh, works with Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, and Doppler.

When to Use What#

• Native secrets — non-sensitive config, development environments
• ESO + Vault — production workloads, compliance requirements
• CSI Secret Store Driver — mount secrets as files, alternative to ESO
• SOPS + age/KMS — encrypted secrets in Git (GitOps-friendly)

Architecture Checklist#

• No secrets in source code, environment files committed to Git, or CI logs
• All secrets encrypted at rest and in transit
• Automated rotation with grace periods
• Dynamic secrets for database and cloud credentials where possible
• Least-privilege policies for secret access
• Audit logging for every secret read and write
• Break-glass procedures documented for emergency access
• Secret scanning in CI pipeline (truffleHog, gitleaks, GitHub secret scanning)

Key Takeaways#

Secret management is not optional infrastructure — it is foundational security. Start with a managed solution if operational complexity is a concern, graduate to Vault when you need dynamic secrets and transit encryption. Whatever you choose, the principles are the same: encrypt everything, rotate frequently, grant least privilege, and audit every access.

---

Design your secret management architecture on codelit.io — the system design tool for developers who ship.

This is article #156 in the Codelit engineering blog series.