Reliability Engineering Best Practices

Ten practices for building resilient, boundable agent systems - plus operating checks so production stays honest under load and outages.

Use them as a design rubric and a launch checklist before agents gain real tools and real credentials.

How to Use This List

Walk A when designing bounds; B when wiring dependencies; C when operating SLOs and incidents.
Tick items only when true in code, config, and runbooks - not as aspirations.
Pair with Why Agents Fail Differently Than Traditional Services for the failure catalog.
Re-check after adding tools, model routes, MCP servers, or multi-agent handoffs.

A - Bound Every Run

1. Cap turns and wall-clock time on every production agent. Turns alone miss hung tools; clocks alone miss tight thrash loops.
2. Enforce nested timeouts (run > turn > tool > HTTP). Inner deadlines never exceed remaining outer budget.
3. Detect no-progress (identical tool signature + error) and stop or escalate. Do not donate unlimited turns to hope.
4. Return structured tool observations on failure (ok, error_type, retryable). Empty catches and swallowed errors cause confident lies.
5. Share retry budgets across the run, not only per call. Collapse SDK, wrapper, and model retries into one policy.

B - Protect Dependencies and Degrade Safely

6. Put circuit breakers on model providers and critical tools. Fail fast with circuit_open; share state across workers.
7. Define explicit fallback chains for user-critical capabilities. Ordered, testable, honest about degrade; never widen blast radius.
8. Prefer idempotent side effects and record partial mutations. Timeouts and retries must not double-send or double-charge.
9. Ship kill switches and keep them reachable from on-call. Per-run and global; cancel in-flight work where possible.
10. Gate irreversible actions with HITL privilege policy. Reliability includes not doing the wrong durable thing quickly.

C - Measure, Alert, and Learn

11. Define SLOs for success rate, latency, and cost per run. Uptime-only dashboards are insufficient for agents.
12. Alert on burn rates and cost spikes, not single failures. Page on storms; ticket on slow burns.
13. Trace runs with model id, tools, stop_reason, and cost. You cannot tune what you cannot attribute.
14. Canary agent changes (prompt, schema, model, graph) under SLOs. Rollback when success, latency, or cost burns.
15. Run agent-shaped postmortems after SEVs. Use bounds and gates as action items, not "remind the model."

Applying These Practices in Order

Bounds (1-5): cheapest protection against thrash and hang; ship before broad tool access.
Dependency safety (6-10): launch blockers whenever paid APIs or side effects exist.
Operate (11-15): keeps reliability true after traffic, model swaps, and fatigue set in.

FAQs

Why "ten practices" if there are fifteen checkboxes?

Items 1-10 are the core design practices for resilient agents. Items 11-15 extend them into measurement and learning so the list stays useful after launch.

What is the first practice if we can only adopt one?

Hard run bounds: max turns + wall clock + structured stop reasons (practices 1-3). Everything else builds on finite runs.

How do these practices relate to frameworks?

They are framework-agnostic. LangGraph timeouts, SDK retries, and router fallbacks are mechanisms; budgets, breakers, and SLOs are the policy.

Do internal admin agents need the same rigor?

Yes when they can touch prod data, money, customer messages, or shared credentials. Internal does not mean low blast radius.

How do we reduce cost without killing success SLOs?

Tier tasks, cache safe reads, prefer cheaper models with eval proof, and stop no-progress early so spend concentrates on useful work.

When is a circuit breaker required vs a retry budget alone?

When many runs share a dependency. Budgets protect one run; breakers protect the dependency and the fleet.

How should timeouts behave under human approval waits?

Pause execution deadlines while waiting on humans, or use a separate approval SLA so overnight queues do not false-timeout.

Where should kill switches live?

Run detail UI, authenticated chatops, and on-call CLI. Drill time-to-trip; seconds matter for spend incidents.

Can we count degraded fallbacks as full success?

Count capability success if the user outcome is acceptable, but track primary serve rate and quality separately.

What evidence shows practice 4 was violated?

Traces where tool errors are missing, empty, or untyped while the final answer claims completion.

How often should we review SLOs?

Weekly for burn and outliers; after every model or tool surface change; after each SEV.

Where should teams link this list?

In the section sidebar as the close-out checklist, and from ADRs or runbooks that introduce new agent dependencies or autonomy.

Why Agents Fail Differently Than Traditional Services - failure modes
Reliability Engineering Basics - first retry wrapper
Circuit Breakers for Agent Tool Calls - fail fast
Fallback Chains: Degrading Gracefully When a Model or Tool Fails - ordered degrade
Setting SLOs for Agent Latency, Success Rate, and Cost - measurable targets
Timeout Strategies for Long-Running Agent Steps - nested clocks
Postmortems for Agent Incidents: A Template - incident review
Kill Switches: Stopping a Runaway Agent Mid-Execution - emergency stop

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Reliability Engineering Best Practices

Ten practices for building resilient, boundable agent systems - plus operating checks so production stays honest under load and outages.

Use them as a design rubric and a launch checklist before agents gain real tools and real credentials.

How to Use This List

Walk A when designing bounds; B when wiring dependencies; C when operating SLOs and incidents.
Tick items only when true in code, config, and runbooks - not as aspirations.
Pair with Why Agents Fail Differently Than Traditional Services for the failure catalog.
Re-check after adding tools, model routes, MCP servers, or multi-agent handoffs.

A - Bound Every Run

1. Cap turns and wall-clock time on every production agent. Turns alone miss hung tools; clocks alone miss tight thrash loops.
2. Enforce nested timeouts (run > turn > tool > HTTP). Inner deadlines never exceed remaining outer budget.
3. Detect no-progress (identical tool signature + error) and stop or escalate. Do not donate unlimited turns to hope.
4. Return structured tool observations on failure (ok, error_type, retryable). Empty catches and swallowed errors cause confident lies.
5. Share retry budgets across the run, not only per call. Collapse SDK, wrapper, and model retries into one policy.

B - Protect Dependencies and Degrade Safely

6. Put circuit breakers on model providers and critical tools. Fail fast with circuit_open; share state across workers.
7. Define explicit fallback chains for user-critical capabilities. Ordered, testable, honest about degrade; never widen blast radius.
8. Prefer idempotent side effects and record partial mutations. Timeouts and retries must not double-send or double-charge.
9. Ship kill switches and keep them reachable from on-call. Per-run and global; cancel in-flight work where possible.
10. Gate irreversible actions with HITL privilege policy. Reliability includes not doing the wrong durable thing quickly.

C - Measure, Alert, and Learn

11. Define SLOs for success rate, latency, and cost per run. Uptime-only dashboards are insufficient for agents.
12. Alert on burn rates and cost spikes, not single failures. Page on storms; ticket on slow burns.
13. Trace runs with model id, tools, stop_reason, and cost. You cannot tune what you cannot attribute.
14. Canary agent changes (prompt, schema, model, graph) under SLOs. Rollback when success, latency, or cost burns.
15. Run agent-shaped postmortems after SEVs. Use bounds and gates as action items, not "remind the model."

Applying These Practices in Order

Bounds (1-5): cheapest protection against thrash and hang; ship before broad tool access.
Dependency safety (6-10): launch blockers whenever paid APIs or side effects exist.
Operate (11-15): keeps reliability true after traffic, model swaps, and fatigue set in.

FAQs

Why "ten practices" if there are fifteen checkboxes?

Items 1-10 are the core design practices for resilient agents. Items 11-15 extend them into measurement and learning so the list stays useful after launch.

What is the first practice if we can only adopt one?

Hard run bounds: max turns + wall clock + structured stop reasons (practices 1-3). Everything else builds on finite runs.

How do these practices relate to frameworks?

They are framework-agnostic. LangGraph timeouts, SDK retries, and router fallbacks are mechanisms; budgets, breakers, and SLOs are the policy.

Do internal admin agents need the same rigor?

Yes when they can touch prod data, money, customer messages, or shared credentials. Internal does not mean low blast radius.

How do we reduce cost without killing success SLOs?

Tier tasks, cache safe reads, prefer cheaper models with eval proof, and stop no-progress early so spend concentrates on useful work.

When is a circuit breaker required vs a retry budget alone?

When many runs share a dependency. Budgets protect one run; breakers protect the dependency and the fleet.

How should timeouts behave under human approval waits?

Pause execution deadlines while waiting on humans, or use a separate approval SLA so overnight queues do not false-timeout.

Where should kill switches live?

Run detail UI, authenticated chatops, and on-call CLI. Drill time-to-trip; seconds matter for spend incidents.

Can we count degraded fallbacks as full success?

Count capability success if the user outcome is acceptable, but track primary serve rate and quality separately.

What evidence shows practice 4 was violated?

Traces where tool errors are missing, empty, or untyped while the final answer claims completion.

How often should we review SLOs?

Weekly for burn and outliers; after every model or tool surface change; after each SEV.

Where should teams link this list?

In the section sidebar as the close-out checklist, and from ADRs or runbooks that introduce new agent dependencies or autonomy.

Why Agents Fail Differently Than Traditional Services - failure modes
Reliability Engineering Basics - first retry wrapper
Circuit Breakers for Agent Tool Calls - fail fast
Fallback Chains: Degrading Gracefully When a Model or Tool Fails - ordered degrade
Setting SLOs for Agent Latency, Success Rate, and Cost - measurable targets
Timeout Strategies for Long-Running Agent Steps - nested clocks
Postmortems for Agent Incidents: A Template - incident review
Kill Switches: Stopping a Runaway Agent Mid-Execution - emergency stop

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Reliability Engineering Best Practices

Search Documentation

Reliability Engineering Best Practices