Agent Observability Best Practices

Ten practices for tracing, alerting, and reviewing agent behavior in production. Use them as a launch rubric and a weekly ops checklist for agent systems.

How to Use This List

Walk A when instrumenting; B when wiring alerts and cost; C when operating reviews and quality loops.
Tick items only when true in code, config, or runbooks - not as future hopes.
Pair with What Agent Observability Actually Needs to Capture for the field model.
Re-check after new tools, multi-agent roles, or traffic jumps.

A - Capture the Run Correctly

1. One run_id per user task, propagated everywhere. Workers, tools, queues, and logs must reuse the same id so reconstruction is a filter, not archaeology.
2. Nest model, tool, and handoff spans under that run. Flat "final answer" logs hide the path where failures live.
3. Always emit a host-side stop_reason. Include goal_met, max_turns, timeout, budget_exceeded, error, escalated, killed (extend as needed).
4. Redact secrets and cap payloads at emission time. Traces are sensitive data stores; UI-only masking is not enough.
5. Record model identity, tokens, and tool status on every step. You need cost and error forensics without re-running production.

B - Alert, Cost, and Stack Pairing

6. Page on SLOs, not on vibes. Track success rate, p95 latency, tool error rate, and cost per successful run (or per minute) in your metrics system.
7. Pair agent traces with infrastructure APM. Shared ids into Datadog/Honeycomb/Grafana so host and dependency faults are visible beside tool trees.
8. Sample successes if needed; keep failures and high-risk runs. Irreversible tool paths and error classes must not share a casual 1% sample with debug noise.

C - Review, Evals, and Change Control

9. Review real traces on a schedule. Weekly samples, release diffs, and SEV reconstructions beat dashboards nobody opens.
10. Join evals and human labels to run_id. Quality scores without path context do not produce fixes.
11. Require instrumentation for new tools/nodes in PR review. No span, no merge for side-effecting tools.
12. Tag agent_version / git sha on every run. Canaries and rollbacks need version filters in both agent UI and APM.
13. Separate debug retention from audit retention. Prompt-heavy debug stores and compliance audit events follow different rules.
14. Practice a 15-minute reconstruction drill. Pick a random prod run_id and explain side effects from traces alone.
15. Own a runbook that starts with ids. Alert → metrics → copy run_id → agent UI → classify logic vs infra → mitigate.

Applying These Practices in Order

Capture (1-5): without these, every later practice lies.
Alert and pairing (6-8): how you learn about pain before Twitter/Slack.
Review and evals (9-15): how the system stays honest after launch.

FAQs

Why "ten practices" if there are fifteen checkboxes?

The section promise is ten core production practices (items 1-10). Items 11-15 extend them into engineering process and drills so the list stays usable after launch.

What is the first practice if we can only adopt one?

Stable run_id propagation plus tool/model nesting. Everything else depends on being able to fetch one task's path.

Do demos need the full checklist?

Demos can be lighter, but anything with real tools, credentials, or spend should implement redaction, stop reasons, and basic success/latency metrics before wide access.

Which platform is required?

None specifically. LangSmith, Langfuse, Phoenix, or a disciplined custom logger can all satisfy the practices if the data model is right.

How is this different from generic microservices observability?

Agents need trajectory fields (tools, turns, stop reasons, eval joins) that request logs alone do not provide - but they still need classic APM underneath.

Where should alerts page?

Into the same on-call path as other production services. Agent-specific Slack channels without ownership become wallpaper.

How often should humans review traces?

At least weekly for active agents, plus every release that changes prompts, tools, or models, plus every SEV.

What belongs in metrics vs traces?

Metrics: rates, latencies, cost aggregates for alerting. Traces: full path for diagnosis. Evals: quality scores joined by id.

How do multi-agent systems change the checklist?

Add specialist labels and handoff packet ids; keep a parent run_id for the user task so ownership stays clear.

Can we delay APM pairing until later?

Only if the agent has no meaningful dependencies. As soon as tools hit networks, queues, or DBs, pairing prevents false "model is slow" root causes.

What evidence shows practice 4 was violated?

API keys, session cookies, or raw credentials appearing in exported traces or support screenshots.

Where should teams link this list?

In the section sidebar as the close-out checklist, and from deploy runbooks for agent services.

What Agent Observability Actually Needs to Capture - field model
Agent Observability Basics - first instrumentation sketches
LangSmith for LangChain/LangGraph-Native Tracing - LangGraph-native path
Langfuse: OpenTelemetry-Based Tracing for Any Framework - multi-framework path
Arize Phoenix for Eval-Rigorous Agent Monitoring - eval-heavy path
Pairing Agent Observability with Infrastructure Monitoring - APM join
Building Custom Trace Logging Without a Third-Party Platform - DIY schema

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Agent Observability Best Practices

Ten practices for tracing, alerting, and reviewing agent behavior in production. Use them as a launch rubric and a weekly ops checklist for agent systems.

How to Use This List

Walk A when instrumenting; B when wiring alerts and cost; C when operating reviews and quality loops.
Tick items only when true in code, config, or runbooks - not as future hopes.
Pair with What Agent Observability Actually Needs to Capture for the field model.
Re-check after new tools, multi-agent roles, or traffic jumps.

A - Capture the Run Correctly

1. One run_id per user task, propagated everywhere. Workers, tools, queues, and logs must reuse the same id so reconstruction is a filter, not archaeology.
2. Nest model, tool, and handoff spans under that run. Flat "final answer" logs hide the path where failures live.
3. Always emit a host-side stop_reason. Include goal_met, max_turns, timeout, budget_exceeded, error, escalated, killed (extend as needed).
4. Redact secrets and cap payloads at emission time. Traces are sensitive data stores; UI-only masking is not enough.
5. Record model identity, tokens, and tool status on every step. You need cost and error forensics without re-running production.

B - Alert, Cost, and Stack Pairing

6. Page on SLOs, not on vibes. Track success rate, p95 latency, tool error rate, and cost per successful run (or per minute) in your metrics system.
7. Pair agent traces with infrastructure APM. Shared ids into Datadog/Honeycomb/Grafana so host and dependency faults are visible beside tool trees.
8. Sample successes if needed; keep failures and high-risk runs. Irreversible tool paths and error classes must not share a casual 1% sample with debug noise.

C - Review, Evals, and Change Control

9. Review real traces on a schedule. Weekly samples, release diffs, and SEV reconstructions beat dashboards nobody opens.
10. Join evals and human labels to run_id. Quality scores without path context do not produce fixes.
11. Require instrumentation for new tools/nodes in PR review. No span, no merge for side-effecting tools.
12. Tag agent_version / git sha on every run. Canaries and rollbacks need version filters in both agent UI and APM.
13. Separate debug retention from audit retention. Prompt-heavy debug stores and compliance audit events follow different rules.
14. Practice a 15-minute reconstruction drill. Pick a random prod run_id and explain side effects from traces alone.
15. Own a runbook that starts with ids. Alert → metrics → copy run_id → agent UI → classify logic vs infra → mitigate.

Applying These Practices in Order

Capture (1-5): without these, every later practice lies.
Alert and pairing (6-8): how you learn about pain before Twitter/Slack.
Review and evals (9-15): how the system stays honest after launch.

FAQs

Why "ten practices" if there are fifteen checkboxes?

The section promise is ten core production practices (items 1-10). Items 11-15 extend them into engineering process and drills so the list stays usable after launch.

What is the first practice if we can only adopt one?

Stable run_id propagation plus tool/model nesting. Everything else depends on being able to fetch one task's path.

Do demos need the full checklist?

Demos can be lighter, but anything with real tools, credentials, or spend should implement redaction, stop reasons, and basic success/latency metrics before wide access.

Which platform is required?

None specifically. LangSmith, Langfuse, Phoenix, or a disciplined custom logger can all satisfy the practices if the data model is right.

How is this different from generic microservices observability?

Agents need trajectory fields (tools, turns, stop reasons, eval joins) that request logs alone do not provide - but they still need classic APM underneath.

Where should alerts page?

Into the same on-call path as other production services. Agent-specific Slack channels without ownership become wallpaper.

How often should humans review traces?

At least weekly for active agents, plus every release that changes prompts, tools, or models, plus every SEV.

What belongs in metrics vs traces?

Metrics: rates, latencies, cost aggregates for alerting. Traces: full path for diagnosis. Evals: quality scores joined by id.

How do multi-agent systems change the checklist?

Add specialist labels and handoff packet ids; keep a parent run_id for the user task so ownership stays clear.

Can we delay APM pairing until later?

Only if the agent has no meaningful dependencies. As soon as tools hit networks, queues, or DBs, pairing prevents false "model is slow" root causes.

What evidence shows practice 4 was violated?

API keys, session cookies, or raw credentials appearing in exported traces or support screenshots.

Where should teams link this list?

In the section sidebar as the close-out checklist, and from deploy runbooks for agent services.

What Agent Observability Actually Needs to Capture - field model
Agent Observability Basics - first instrumentation sketches
LangSmith for LangChain/LangGraph-Native Tracing - LangGraph-native path
Langfuse: OpenTelemetry-Based Tracing for Any Framework - multi-framework path
Arize Phoenix for Eval-Rigorous Agent Monitoring - eval-heavy path
Pairing Agent Observability with Infrastructure Monitoring - APM join
Building Custom Trace Logging Without a Third-Party Platform - DIY schema

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Agent Observability Best Practices

Buscar en la documentación

Agent Observability Best Practices