Immutable Infrastructure: Build Once, Deploy Everywhere

Mutable infrastructure lets you SSH into a server, install a patch, tweak a config, and move on. Over weeks and months those ad-hoc changes accumulate. No two servers look alike, deployments become terrifying, and debugging turns into archaeology. Immutable infrastructure eliminates this class of problem by enforcing a single rule: never modify a running instance — replace it.

Mutable vs Immutable#

The distinction is straightforward:

Mutable infrastructure is not inherently wrong — it is simpler for small teams and prototypes. But once you operate at scale, drift becomes the dominant source of incidents.

Configuration Drift#

Drift happens when the actual state of a server diverges from its declared state. Common causes:

• Hot-fixes applied directly — a developer patches production at 2 AM and forgets to update the automation.
• Partial automation runs — Ansible or Chef fails midway, leaving a server half-configured.
• Package manager updates — apt upgrade pulls a newer library version on one server but not another.
• Manual troubleshooting — someone installs strace, changes an ulimit, or edits /etc/hosts and never reverts.

Drift turns identical servers into snowflakes. Immutable infrastructure eliminates drift by design: if you need a change, you build a new image and replace the running instances.

Golden Images#

A golden image is a pre-built, versioned snapshot of a server. It contains the operating system, application code, dependencies, and configuration baked in at build time.

┌──────────────────────────────────────┐
│           Golden Image v42           │
│                                      │
│  OS: Ubuntu 24.04                    │
│  Runtime: Node 22.1                  │
│  App: api-server@3.8.0              │
│  Config: baked from Vault at build   │
│  Monitoring: Datadog agent 7.x       │
└──────────────────────────────────────┘

Build pipeline: source code commit triggers CI, which produces a tested artifact, which feeds the image builder, which publishes a versioned image to a registry.

Key principles for golden images:

1. Version everything — tag images with the git SHA or a semantic version so you can trace any running instance back to a specific commit.
2. Minimize secrets at build time — bake configuration structure but inject secrets at boot via Vault, AWS Secrets Manager, or environment variables.
3. Test the image, not just the code — run integration tests against the built image before promoting it to production.

Packer#

HashiCorp Packer automates golden image creation across cloud providers. A single template can produce an AMI for AWS, a Compute Image for GCP, and a VHD for Azure.

source "amazon-ebs" "api" {
  ami_name      = "api-server-{{timestamp}}"
  instance_type = "t3.medium"
  region        = "us-east-1"
  source_ami_filter {
    filters = {
      name                = "ubuntu/images/*ubuntu-noble-24.04-amd64-server-*"
      virtualization-type = "hvm"
    }
    owners      = ["099720109477"]
    most_recent = true
  }
  ssh_username = "ubuntu"
}

build {
  sources = ["source.amazon-ebs.api"]

  provisioner "shell" {
    scripts = [
      "scripts/install-node.sh",
      "scripts/install-app.sh",
      "scripts/harden.sh"
    ]
  }
}

Packer provisions a temporary instance, runs your scripts, snapshots the disk, and tears down the instance. The output is a machine image ready for launch.

Container Images#

Containers are the most common form of immutable infrastructure today. A Dockerfile defines the image; a registry stores versioned tags; an orchestrator deploys identical replicas.

FROM node:22-slim AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci --production
COPY . .
RUN npm run build

FROM node:22-slim
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
USER node
CMD ["node", "dist/server.js"]

Best practices for container images:

• Multi-stage builds — keep the final image small by discarding build tools.
• Pin base image digests — node:22-slim@sha256:abc... prevents silent upstream changes.
• No SSH, no shell — production containers should not include debugging tools. Use ephemeral debug containers when needed.
• Read-only root filesystem — enforce immutability at the runtime level with --read-only.

Phoenix Servers#

The Phoenix server pattern (coined by Martin Fowler) takes immutability to its logical conclusion: servers are regularly destroyed and recreated from scratch, like a phoenix rising from ashes.

Instead of asking "how long has this server been running?", you ask "how recently was this server replaced?"

Benefits:

• Eliminates long-lived state — forces teams to externalize all state to databases, object stores, or managed services.
• Proves recovery — every deployment exercises the full provisioning path, so you know recovery works.
• Reduces attack surface — compromised servers are short-lived; rotation limits the window for persistent threats.

Implementation strategies:

1. Rolling replacements on deploy — each deployment launches new instances and drains old ones.
2. Scheduled rotation — even without code changes, replace instances every 24-48 hours.
3. Chaos-driven replacement — tools like Chaos Monkey randomly terminate instances, forcing the system to prove its resilience.

Blue-Green with Immutable Infrastructure#

Blue-green deployment pairs naturally with immutable infrastructure. You maintain two identical environments:

                    ┌─────────────────┐
                    │   Load Balancer  │
                    └────────┬────────┘
                             │
              ┌──────────────┴──────────────┐
              ▼                              ▼
    ┌──────────────────┐          ┌──────────────────┐
    │   Blue (v41)     │          │   Green (v42)    │
    │   ── live ──     │          │   ── staging ──  │
    │   3 instances    │          │   3 instances    │
    │   golden image   │          │   golden image   │
    └──────────────────┘          └──────────────────┘

Deployment flow:

1. Build a new golden image (v42).
2. Launch green instances from v42.
3. Run smoke tests against green.
4. Switch the load balancer to green.
5. Monitor error rates and latency.
6. If healthy, drain and terminate blue. If not, switch back.

Because images are immutable, rollback is instantaneous — you simply point the load balancer back to blue. No patches to reverse, no config to restore.

Trade-offs#

Immutable infrastructure is not free. Understand the costs:

• Longer build times — building a full image takes minutes, not seconds. Optimize with layer caching and parallel builds.
• Larger artifacts — machine images can be gigabytes. Container images are lighter but still require registry storage.
• Secret injection complexity — secrets cannot be baked in, so you need a runtime secret delivery mechanism.
• Stateful workloads — databases, message brokers, and other stateful systems resist the immutable pattern. Use managed services or separate stateful and stateless tiers.
• Local development gap — developers often run mutable local environments. Bridge the gap with dev containers or local Kubernetes clusters.

Migration Path#

Moving from mutable to immutable infrastructure is incremental:

1. Automate provisioning first — if you cannot build a server from scratch with a script, you are not ready for immutable.
2. Externalize state — move sessions, uploads, and caches to external stores (Redis, S3, managed databases).
3. Build golden images in CI — start producing images alongside your existing deployment.
4. Run both in parallel — deploy immutable instances alongside mutable ones and compare behavior.
5. Cut over — once confidence is high, stop patching and start replacing.

Key Takeaways#

• Mutable infrastructure drifts. Immutable infrastructure enforces consistency by replacing instances rather than patching them.
• Golden images and container images are the two primary forms of immutable artifacts.
• Packer builds machine images across cloud providers from a single template.
• Phoenix servers prove recovery by regularly destroying and recreating instances.
• Blue-green deployment with immutable images enables instant rollback.
• Externalize all state before adopting immutable patterns — stateful workloads need separate strategies.

---

Build and explore system design concepts hands-on at codelit.io.

295 articles on system design at codelit.io/blog.