Observability-Driven Development: Instrument Before You Ship

Most teams add observability after something breaks. Observability-driven development flips the order: instrument first, then ship. Every feature ships with the metrics, traces, and logs needed to verify it works correctly in production — before users report problems.

What Is Observability-Driven Development?#

ODD is a methodology where observability is a first-class requirement for every feature, not an afterthought bolted on during an incident. The core principle is simple: if you cannot measure a feature's behavior in production, it is not ready to ship.

Traditional development cycle:

Design → Build → Test → Ship → (break) → Add monitoring → Fix

ODD cycle:

Design → Define success metrics → Instrument → Build → Ship → Validate with data

The difference is that success criteria and instrumentation are defined before code is written, not after the first outage.

The Three Pillars in ODD Context#

ODD builds on the three pillars of observability but applies them with intent:

Metrics#

Metrics answer "what is happening?" with numbers:

• RED metrics for services: Rate, Errors, Duration.
• USE metrics for resources: Utilization, Saturation, Errors.
• Business metrics for features: conversion rate, cart abandonment, signup completion.

In ODD, every feature defines its own metrics before implementation. A checkout flow feature would define: transactions per minute, payment failure rate, P95 checkout duration, and cart-to-purchase conversion rate.

Traces#

Traces answer "where is time being spent?" across service boundaries:

• Add trace context to every new code path.
• Create spans for meaningful operations — database queries, external API calls, cache lookups.
• Attach feature-specific attributes to spans (feature flag variant, user segment, experiment ID).

Structured Logs#

Logs answer "why did this specific thing happen?":

• Emit structured logs (JSON) with consistent fields: requestId, userId, featureFlag, outcome.
• Log at decision points — not just errors, but the reasons behind code path selections.
• Correlate logs with trace IDs so you can jump from a metric anomaly to the exact log line.

The ODD Workflow#

Step 1: Define Success Criteria#

Before writing code, answer:

• What does success look like for this feature? (e.g., "P95 latency under 200ms, error rate below 0.1%")
• What SLIs (Service Level Indicators) will measure it?
• What thresholds trigger an alert or rollback?

Document these as acceptance criteria alongside functional requirements.

Step 2: Design Instrumentation#

Plan your instrumentation alongside your architecture:

• Which metrics will you emit? Name them, define their labels/dimensions, and specify the aggregation (counter, histogram, gauge).
• Where will you add trace spans? Map the critical path and mark each operation that should be independently measurable.
• What log events will you emit? Define the structured fields and severity levels.

Step 3: Implement Instrumentation First#

Write the instrumentation code before the feature logic:

// Pseudo-code: instrument first
const checkoutDuration = histogram("checkout.duration_ms", {tags: ["payment_method", "flag_variant"]})
const checkoutErrors = counter("checkout.errors_total", {tags: ["error_type", "payment_method"]})

function processCheckout(order) {
  const span = tracer.startSpan("checkout.process", {attributes: {"order.total": order.total}})
  const timer = checkoutDuration.startTimer()

  try {
    // Feature logic goes here
    span.setStatus("OK")
  } catch (err) {
    checkoutErrors.increment({error_type: err.code})
    span.recordException(err)
    throw err
  } finally {
    timer.stop()
    span.end()
  }
}

The instrumentation wraps the feature. When the feature logic is implemented, the measurements are already in place.

Step 4: Build the Dashboard Before Merge#

Create the monitoring dashboard as part of the pull request:

• Dashboard-as-code (Grafana JSON, Terraform, Pulumi) lives in the same repository.
• The PR reviewer can verify that instrumentation matches the success criteria.
• When the feature merges, the dashboard is already deployed.

Step 5: Ship and Validate#

Deploy the feature and immediately validate against the predefined success criteria using real production data — not staging, not synthetic tests.

Feature Flags + Observability#

Feature flags and observability are complementary. Together they enable controlled, measurable releases.

Instrumented rollouts:

1. Ship the feature behind a flag, disabled by default.
2. Enable for 1% of traffic. Compare metrics between flag-on and flag-off cohorts.
3. If metrics meet success criteria, increase to 10%, then 50%, then 100%.
4. If metrics degrade, disable the flag instantly — no rollback deployment needed.

Key practice: Tag all metrics, traces, and logs with the feature flag variant. This lets you slice dashboards by flag state and see the exact impact of each feature independently.

// Tag everything with the flag state
const variant = featureFlags.getVariant("new-checkout", user)
span.setAttribute("feature.new_checkout", variant)
metrics.increment("checkout.started", {flag_variant: variant})
logger.info({event: "checkout_started", flag_variant: variant, user_id: user.id})

Canary Deployments + Metrics#

Canary deployments route a small percentage of traffic to the new version while the rest stays on the old version. ODD makes canaries data-driven rather than time-based.

Automated canary analysis:

1. Deploy the canary (5% of traffic).
2. Collect metrics from both canary and baseline for a defined window (e.g., 10 minutes).
3. Run statistical comparison: error rate, latency percentiles, business metrics.
4. If the canary is statistically equivalent or better, promote. If worse, roll back automatically.

Tools like Flagger, Argo Rollouts, and Kayenta automate this analysis. The key is that promotion and rollback decisions are driven by metrics, not by an engineer watching dashboards.

SLOs as Feature Gates#

Service Level Objectives (SLOs) define the reliability targets your users expect. ODD ties SLOs directly to feature releases:

• Error budget: If your SLO is 99.9% availability, you have a 0.1% error budget. Feature releases consume error budget.
• Feature gate: If the error budget is nearly exhausted, new feature releases are paused until the budget recovers.
• Burn rate alerts: Alert when the error budget is being consumed faster than expected — a signal to investigate recent releases.

This creates a natural feedback loop: ship fast when reliability is healthy, slow down when it is not.

Building the Culture#

ODD is as much about culture as tooling:

• Code review includes instrumentation review. Reviewers check that metrics, traces, and logs are present and meaningful.
• Definition of done includes dashboards. A feature is not done until its monitoring is in place.
• Blameless postmortems reference instrumentation gaps. When an incident reveals missing observability, the remediation includes adding it.
• On-call rotation builds empathy. Engineers who operate their own code in production naturally write better instrumentation.

Anti-Patterns to Avoid#

• Logging everything — High-cardinality logs without structure create noise, not insight. Be intentional about what you log.
• Metrics without context — A counter that increments tells you something happened but not why. Add dimensions.
• Observability as a separate team's job — If the people building features are not the people instrumenting them, critical context is lost.
• Dashboard graveyards — Dashboards nobody looks at. Tie dashboards to on-call runbooks and SLO tracking.
• Alerting on symptoms only — Alert on SLO burn rate, not individual metric thresholds. Symptom-based alerts reduce noise.

Key Takeaways#

• ODD means defining success metrics and building instrumentation before writing feature logic.
• The three pillars — metrics, traces, structured logs — are planned during design, not added after incidents.
• Feature flags combined with observability enable measured, incremental rollouts with instant rollback.
• Canary deployments become data-driven when automated analysis compares canary metrics against baseline.
• SLOs and error budgets create a natural governor on release velocity — ship fast when healthy, slow down when not.
• Culture matters: code review should verify instrumentation, and the definition of done should include dashboards.

The teams that move fastest in production are not the ones that skip observability. They are the ones that invest in it upfront, so every release is a measured experiment rather than a leap of faith.

---

Build and explore system design concepts hands-on at codelit.io.

294 articles on system design at codelit.io/blog.