A DevOps and SRE AI Agent Workflow That Does Not Make Incidents Worse#

The worst SRE agent is the one that sounds confident during an incident.

Incidents do not need more confidence. They need context, restraint, and a clean trail.

A good DevOps agent should make the first 15 minutes less chaotic. It should not pretend to be the incident commander.

The job#

Give the agent a narrow first job:

Collect evidence, summarize likely causes, identify owners, suggest next actions, and require approval before any production change.

That is enough.

If it does that well, the team will use it.

Trigger sources#

The workflow can start from:

• PagerDuty alert.
• Datadog monitor.
• Sentry issue.
• Slack incident channel.
• Failed deploy.
• Error budget burn.
• Synthetic check failure.
• Customer support spike.

Each trigger should create the same basic run object: service, severity, timestamp, symptoms, links, and current owner.

What the agent should read#

Start read-only:

• Recent deploys.
• Error traces.
• Logs.
• Metrics.
• Runbooks.
• Ownership map.
• Feature flags.
• Incident history.
• Pull requests.
• Change calendar.

Most incident value comes from connecting existing evidence.

What the agent should not do first#

Do not start with autonomous rollback.

Do not start with autonomous scaling.

Do not start with autonomous config changes.

Those can come later, behind approval and scoped permissions. The first version should reduce diagnosis time without increasing blast radius.

The output format#

The agent should post something like this:

Incident packet
Service: billing-webhooks
Severity: sev2 candidate
First seen: 14:03 UTC

Signals:
- Error rate rose from 0.3% to 8.7%
- Latest deploy landed 11 minutes before first spike
- Failures cluster around missing customer_id

Likely owner:
- Payments platform

Suggested next action:
- Review parser change in PR 4821
- Confirm legacy payload handling

Requires approval:
- Rollback
- Customer-facing status update

That format is intentionally plain. Incidents are not the place for prose theater.

Guardrails#

The policy should be explicit:

• Reads are allowed inside service scope.
• Posting an evidence summary is allowed.
• Creating an incident ticket is allowed with source links.
• Rollbacks require human approval.
• Scaling changes require human approval.
• Customer status updates require human approval.
• Any action touching customer data gets logged.

The agent should know when it is just an analyst.

Evals#

Replay cases should include:

• No deploy near the incident.
• Multiple deploys near the incident.
• Misleading log spike.
• Downstream dependency failure.
• Missing runbook.
• Conflicting dashboard signals.
• Customer data exposure risk.
• Rollback requested but unsafe.

If your evals only test happy paths, you are measuring demos.

Build it in Codelit#

Try this:

Design a DevOps and SRE AI agent workflow for incident evidence collection. Include PagerDuty, Datadog, Sentry, GitHub, runbooks, ownership routing, Slack updates, approval gates, evals, audit logs, and production architecture.

Build the SRE agent workflow

The agent should not be a hero. It should be the teammate who has the packet ready when humans arrive.