Observability vs Monitoring: Metrics, Logs, and Traces#

Monitoring tells you when something is broken. Observability tells you why.

Most teams start with monitoring (alerts when CPU is high) and realize too late they need observability (understanding why requests are slow for users in Europe on Tuesdays). This guide covers the three pillars and how to implement them.

Monitoring vs Observability#

You need both. Monitoring catches known issues fast. Observability helps debug novel problems.

The Three Pillars#

1. Metrics — Numbers Over Time#

Time-series data: values measured at regular intervals.

http_requests_total{service="api", status="200"} 145892
http_request_duration_seconds{service="api", quantile="0.99"} 0.42
cpu_usage_percent{host="web-1"} 72.5

Types:

• Counter — only goes up (total requests, errors)
• Gauge — goes up and down (CPU, memory, queue depth)
• Histogram — distribution of values (request duration P50/P95/P99)

Best for: Dashboards, alerting, capacity planning, SLO tracking

Tools: Prometheus, Datadog, CloudWatch, Grafana + Mimir

2. Logs — Events with Context#

Structured records of what happened:

{
  "timestamp": "2026-03-28T14:32:01Z",
  "level": "error",
  "service": "payment-service",
  "trace_id": "abc123",
  "user_id": "user_456",
  "message": "Stripe charge failed",
  "error": "card_declined",
  "amount": 99.99
}

Structured logs > unstructured. Always use JSON with consistent fields.

Best for: Debugging specific errors, audit trails, compliance

Tools: ELK (Elasticsearch + Logstash + Kibana), Grafana Loki, Datadog Logs, CloudWatch Logs

3. Traces — Request Journey#

Follow a single request across all services:

[Trace: abc123] 450ms total
├── API Gateway        12ms
├── Auth Service       8ms
├── Order Service      180ms
│   ├── PostgreSQL     45ms
│   └── Redis Cache    2ms (cache miss → DB)
├── Payment Service    230ms ← bottleneck!
│   ├── Fraud Check    15ms
│   └── Stripe API     210ms ← external dependency
└── Email Service      20ms (async, not blocking)

Best for: Finding bottlenecks, understanding latency, debugging distributed systems

Tools: Jaeger, Zipkin, Datadog APM, Grafana Tempo, OpenTelemetry

How They Work Together#

A user reports: "Checkout is slow."

1. Metrics → Dashboard shows P99 latency spiked from 200ms to 2s at 2pm
2. Traces → Filter traces where duration > 1s → Payment Service is the bottleneck
3. Logs → Filter Payment Service logs at 2pm → Stripe API timeout errors, retries causing delays

Without all three, you'd be guessing.

OpenTelemetry — The Standard#

OpenTelemetry (OTel) is the open standard for all three pillars:

Your App → OTel SDK → OTel Collector → Datadog / Grafana / Jaeger
                                      (any backend)

Why OTel matters:

• One SDK for metrics + logs + traces
• Vendor-neutral — switch backends without code changes
• Auto-instrumentation for popular frameworks (Express, Django, Spring)
• Context propagation (trace_id flows across services)

// Node.js with OpenTelemetry
import { trace } from "@opentelemetry/api";

const tracer = trace.getTracer("order-service");
const span = tracer.startSpan("process-order");
span.setAttribute("order.id", orderId);
span.setAttribute("order.amount", amount);
// ... do work ...
span.end();

Observability Stack Comparison#

Decision Guide#

• Startup (< 20 engineers): Grafana Cloud free tier or Datadog
• Scale-up: Datadog or New Relic (full platform)
• Cost-conscious: Self-hosted Prometheus + Loki + Jaeger
• AWS-native: CloudWatch + X-Ray
• Enterprise: Datadog or Elastic

Key Metrics to Track#

The Four Golden Signals (Google SRE)#

RED Method (Microservices)#

• Rate — requests per second
• Errors — errors per second
• Duration — latency histogram

USE Method (Infrastructure)#

• Utilization — % of resource used
• Saturation — work queued
• Errors — error count

SLOs, SLIs, and SLAs#

SLI ← you measure it. SLO ← you set it. SLA ← you promise it.

Error budget: if SLO is 99.9%, you have 0.1% budget for failures (43 minutes/month). Spend it on deployments and experiments.

Architecture Example#

Microservices Observability Stack#

Services (with OTel SDK)
    → OTel Collector
        → Prometheus (metrics) → Grafana Dashboards
        → Loki (logs) → Grafana Explore
        → Jaeger (traces) → Grafana Traces
    → Alert Manager → PagerDuty / Slack

Generate your observability architecture →

Codelit can also generate Datadog, Sentry, and Grafana configs automatically from any architecture — just use the Monitoring Setup export.

Summary#

1. Metrics for dashboards and alerts — "is it broken?"
2. Logs for debugging — "what went wrong?"
3. Traces for latency — "where is it slow?"
4. OpenTelemetry as the standard — one SDK, any backend
5. Four Golden Signals for every service — latency, traffic, errors, saturation
6. Set SLOs — measure what matters to users, not what's easy to measure

---

Generate observability configs at codelit.io — Datadog, Sentry, and Grafana dashboard exports from any architecture diagram.