Agent Reliability Engineering#

Reliable agents are not created by asking the model to be careful.

Reliability comes from architecture.

The same old engineering ideas still matter: timeouts, retries, idempotency, isolation, observability, rollbacks, SLOs, and human override. Agents just make the failure modes stranger.

Agent failure modes#

Plan for:

• Wrong tool.
• Right tool, wrong arguments.
• Missing context.
• Stale memory.
• Conflicting sources.
• Prompt injection.
• Tool timeout.
• Model degradation.
• Cost runaway.
• Approval bypass attempt.
• Human correction ignored.

If you cannot name the failure modes, you cannot design the guardrails.

Retries need policy#

Do not retry everything.

Retry:

• Transient network errors.
• Tool rate limits within budget.
• Recoverable timeouts.

Do not blindly retry:

• Write actions.
• Payment operations.
• Customer messages.
• Deploys.
• Data deletion.

Retries without idempotency are a bug generator.

Idempotency matters#

Every action should answer:

• If this runs twice, what happens?
• How do we detect duplicate work?
• Can we preview before execution?
• Can we roll back?
• Is the result auditable?

Agents love loops. Production systems need brakes.

SLOs for agents#

Define SLOs:

• Successful task completion.
• Unsafe action blocked.
• Correct tool selection.
• Source citation coverage.
• Human correction rate.
• Latency to first useful response.
• Cost per useful run.

Do not use one generic "accuracy" metric and call it done.

Build it in Codelit#

Try this:

Design agent reliability engineering for a production AI workflow. Include failure modes, retries, idempotency, tool timeouts, evals, observability, SLOs, human override, rollback, and incident runbooks.

Map the agent reliability workflow

Agent reliability is just reliability engineering with a more interesting failure surface.