Service Level Objectives (SLOs)#

Uptime is not reliability. Users don't care whether your server is "up" — they care whether their requests succeed fast enough. Service Level Objectives give you a precise, measurable definition of "reliable enough."

SLI vs SLO vs SLA#

These three terms form a hierarchy:

The relationship flows upward: SLIs feed SLOs, and SLOs back SLAs.

SLI (measurement) --> SLO (internal target) --> SLA (external contract)

Key insight: You can have SLOs without SLAs. Every team should have SLOs. Not every team needs SLAs.

Choosing the Right SLIs#

Bad SLIs measure infrastructure. Good SLIs measure what users experience.

Avoid These SLIs#

• CPU utilization
• Memory usage
• Pod restart count
• Disk IOPS

Use These Instead#

Availability SLI:

successful requests / total requests

Latency SLI:

requests faster than threshold / total requests

Correctness SLI:

requests returning correct data / total requests

Freshness SLI:

data updates arriving within threshold / total updates

SLI Selection by Service Type#

Error Budgets#

An error budget is the inverse of your SLO. If your SLO is 99.9% availability over 30 days, your error budget is 0.1% — roughly 43 minutes of allowed downtime.

Error Budget = 1 - SLO target

99.9% SLO --> 0.1% budget --> 43.2 min/month
99.5% SLO --> 0.5% budget --> 3.6 hours/month
99.0% SLO --> 1.0% budget --> 7.2 hours/month

Error Budget Policy#

Define what happens as budget depletes:

This creates a self-balancing system: ship fast when you have budget, slow down when reliability suffers.

Burn Rate Alerts#

Traditional threshold alerts fire too late. Burn rate alerts tell you how fast you're consuming your error budget.

Burn Rate = (error rate observed) / (error rate allowed by SLO)

A burn rate of 1.0 means you'll exhaust your budget exactly at the end of the window. A burn rate of 10.0 means you'll burn through your entire monthly budget in 3 days.

Multi-Window Burn Rate Strategy#

Google SRE recommends alerting on two windows simultaneously to reduce false positives:

Both windows must be breaching to trigger the alert. This eliminates short spikes from waking anyone up.

Burn Rate Alert in Prometheus#

# Fast burn: 14.4x over 1 hour (pages immediately)
- alert: HighBurnRate
  expr: |
    (
      sum(rate(http_requests_total{code=~"5.."}[1h]))
      /
      sum(rate(http_requests_total[1h]))
    ) > (14.4 * 0.001)
  for: 2m
  labels:
    severity: critical

Building an SLO Dashboard#

Every SLO dashboard should show four things:

1. Current SLI value — are we meeting the objective right now?
2. Error budget remaining — how much room do we have?
3. Burn rate — how fast are we consuming budget?
4. Time series — SLI trend over the SLO window

+--------------------------------------------------+
|  API Availability SLO: 99.9%                     |
|                                                  |
|  Current SLI:     99.94%     [=====|----]        |
|  Budget Remaining: 62%       [========|--]       |
|  Burn Rate:        0.8x      (healthy)           |
|  Window:           30 days   (18 days remaining) |
+--------------------------------------------------+

Dashboard Anti-Patterns#

• Showing raw error counts instead of ratios
• Missing the SLO target line on graphs
• No burn rate visualization
• Mixing infrastructure metrics with SLIs

The Google SRE Approach#

Google's SRE book codified SLOs as the core reliability practice. Key principles:

1. 100% is the wrong target. Pursuing perfect reliability is infinitely expensive and slows innovation.
2. Users set the SLO. If users can't tell the difference between 99.99% and 99.999%, the tighter target wastes engineering effort.
3. Error budgets create alignment. Product and SRE teams agree on the budget. No more "move fast" vs "don't break things" arguments.
4. SLOs drive decisions. Launch reviews, architecture choices, and staffing all reference SLOs.

Practical SLO Targets#

Consequences of SLO Breach#

When your SLO is breached (error budget exhausted), escalation kicks in:

Engineering consequences:

• Feature freeze until budget recovers
• Mandatory postmortem for every incident
• Architecture review for the failing service
• Increased testing and canary requirements

Organizational consequences:

• SLO breach appears in team health dashboards
• Leadership review if breaches are recurring
• Staffing adjustments (more SRE support)
• Possible re-architecture or service decomposition

What NOT to do:

• Blame individuals
• Loosen the SLO to avoid breaches
• Ignore the breach and keep shipping

Tools for SLO Management#

Nobl9#

A dedicated SLO platform that integrates with your existing observability stack:

• Connects to Datadog, Prometheus, New Relic, Splunk, and more
• Calculates error budgets automatically
• Provides burn rate alerting out of the box
• Supports composite SLOs across multiple services

Sloth (Open Source)#

Generates Prometheus alerting rules from SLO definitions:

# sloth.yml
version: "prometheus/v1"
service: "api-gateway"
slos:
  - name: "requests-availability"
    objective: 99.9
    sli:
      events:
        error_query: sum(rate(http_requests_total{code=~"5.."}[{{.window}}]))
        total_query: sum(rate(http_requests_total[{{.window}}]))
    alerting:
      page_alert:
        labels:
          severity: critical
      ticket_alert:
        labels:
          severity: warning

Run sloth generate and it produces multi-window, multi-burn-rate alerts automatically.

Other Tools#

• Datadog SLO Tracking — built into Datadog with SLO widgets
• Google Cloud SLO Monitoring — native GCP service
• Prometheus + Grafana — DIY with recording rules
• Dynatrace — SLO tiles with automatic baselining

Getting Started Checklist#

1. Pick your most critical service
2. Identify 1-2 SLIs that reflect user experience
3. Set an SLO target (start conservative, tighten later)
4. Calculate the error budget
5. Set up burn rate alerts
6. Build a dashboard
7. Write an error budget policy
8. Review monthly

---

This is article #354 in the Codelit engineering series. Explore more at codelit.io.