What Agent Observability Actually Needs to Capture
Request-level logs that work for REST APIs fail agents. An agent run is a tree of model steps, tool calls, observations, retries, and stop reasons, not one HTTP hit.
If you only capture "status 200 in 2.1s," you cannot explain a wrong refund, a stuck loop, or a $40 thrash on a single ticket.
Summary
- Agent observability must reconstruct a full run - every loop turn, tool attempt, model call, state snapshot, and terminal reason - joined by a stable
run_id. - Insight: Agent failures are path-dependent. The bug is often "which branch, which tool observation, which missing stop condition," not a single exception stack.
- Key Concepts: run / trace, span / step, tool call + observation, state / checkpoint, token and cost attributes, stop reason, correlation with infra.
- When to Use: Designing what to instrument before you pick LangSmith, Langfuse, Phoenix, or a custom logger.
- Limitations/Trade-offs: Full fidelity costs storage and creates privacy risk. You still need sampling, redaction, and separate audit retention rules.
- Related Topics: agent loops, evals, SLOs, HITL audit trails, infrastructure APM.
Foundations
A traditional service trace answers: which dependency was slow or erroring on this request?
An agent trace must also answer: what did the model decide, what tools ran with which args, what did those tools return, how did state change, and why did the loop stop?
Minimum capture surface for a useful agent run:
| Field / span | Why you need it |
|---|---|
run_id / trace_id | Join every step of one task |
| User / tenant / session ids | Find "this customer's bad run" |
| Goal or task summary | Context without dumping full history |
| Model call spans | Model id, latency, tokens, finish reason |
| Tool call spans | Name, args (or fingerprint), result status |
| Loop / graph node labels | ReAct turn, plan step, handoff, replan |
| State or checkpoint refs | Reconstruct decisions without infinite logs |
| Errors and retries | Distinguish flaky tools from bad reasoning |
| Stop reason | goal_met, max_turns, timeout, kill, escalated |
| Cost attributes | Tokens and estimated $ per run |
If any of these are missing, on-call reconstructs with guesses.
run_id
├── turn 1
│ ├── model.chat
│ └── tool.search (args, observation)
├── turn 2
│ ├── model.chat
│ └── tool.refund (args, observation)
└── terminal: stop_reason=goal_met | max_turns | errorMechanics & Interactions
Loops, not requests
Agents iterate. One user message can produce ten model calls and twenty tools. Observability must nest those under one parent run so "the refund agent" is one object, not twenty uncorrelated lines.
Tool calls are first-class
Most production damage lives in tools: wrong args, partial writes, silent success with empty bodies.
Log intent (tool_requested) and outcome (tool_executed / tool_failed) as separate moments when side effects matter.
Args need enough detail to debug, but secrets must be redacted before write. When policy forbids raw PII, store fingerprints and retrieve full args from a secure store only for incident review.
State and checkpoints
Message lists and graph state explain why the next decision happened.
You rarely want every intermediate blob in the hot log forever.
Prefer: small state summaries in spans, large artifacts by reference, and checkpoints you can load by thread_id when resuming or replaying.
Cost and latency are product metrics
Token counts and wall time per node are not vanity. They power SLOs, budget caps, and "this prompt change doubled cost" reviews. Attach model name, provider, and estimated cost when your stack can compute it.
Quality signals sit beside traces
Observability captures what happened. Evals and human labels capture whether it was good. You need both: trace for path, eval for score, join key for both.
Privacy and retention
Agents see CRM fields, messages, and credentials in tool observations. Defaulting to "log everything forever" is a security incident waiting for a ticket. Define redaction at emission time, separate debug retention from audit retention, and restrict who can open production traces.
Advanced Considerations & Applications
Multi-agent and handoffs need span boundaries per agent role and a handoff packet id so you can see which specialist owned a failure.
Human-in-the-loop needs approval events with actor, decision id, and args (or fingerprint), not only approved: true.
That is adjacent to audit trails; reuse the same ids in both systems.
Sampling: full traces for errors, canaries, and high-value tenants; sample happy paths if volume forces it. Never sample away the audit path for irreversible actions.
Open standards vs vendor UIs: OpenTelemetry and OpenInference give portable spans; agent platforms add message views, datasets, and evals. Capture the data model first so you can switch UIs without reinventing fields.
Infra correlation: Agent platforms do not replace Datadog or Honeycomb for pod OOM, queue lag, or DB pool exhaustion.
Propagate trace_id / run_id into infrastructure spans so "slow tool" and "saturated worker" meet in one investigation.
| Approach | What you see well | What you miss |
|---|---|---|
| HTTP access logs only | Status and latency | Tool path, decisions, stop reason |
| LLM provider dashboard only | Token bills | Graph path and tool args |
| Structured agent traces | Full loop reconstruction | Host/OS failures without APM join |
| Custom print debugging | Instant local insight | Prod multi-worker correlation |
Common Misconceptions
- "If the API returned 200, observability is fine." Agents return 200 with wrong side effects constantly.
- "Logging the final answer is enough." Side effects hide in intermediate tools.
- "Prompt bodies must always be stored forever." Often hashes, samples, or short-lived debug stores are enough; legal may require less, not more.
- "Metrics replace traces." Metrics tell you rates; traces tell you one bad path.
- "Vendor UI = compliance audit log." Trace UIs are for engineering; audit retention and integrity are separate requirements.
- "More agents only need more logs." They need clearer ownership labels and handoff contracts in the trace tree.
FAQs
What is the smallest useful agent trace?
One run_id, ordered model and tool spans with latency and status, and a terminal stop_reason. That already beats request logs for most SEVs.
Do I need to store full tool observations?
Store enough to debug typical failures. Cap size, truncate large blobs, and put artifacts in object storage referenced by id.
How is this different from application logging?
Logs are event streams. Agent observability is a hierarchical run model with explicit parent/child spans and decision metadata.
Should the model write its own "thought log" as the source of truth?
No. Host-side instrumentation is authoritative. Model rationales are optional attributes, not the only record.
What about streaming tokens?
Prefer step-boundary spans for ops. Per-token streams help debug UX latency; they are rarely needed for every production run.
How do I capture multi-turn sessions?
Use a stable thread_id / session id plus per-message run_id values so you can zoom out to the conversation and in to one turn.
Where do evals fit?
Offline and online scores attach to the same run_id. Capture first, score second, then filter traces by score.
What stop reasons should I standardize?
At least: goal_met, max_turns, timeout, budget_exceeded, tool_circuit_open, escalated, killed, error.
How do secrets end up in traces?
Through tool args, headers, and pasted user content. Redact at the instrumentation layer before export.
Is OpenTelemetry enough by itself?
It is an excellent transport and schema base. You still need agent-specific attributes and a UI or warehouse that understands message trees.
When is a custom logger better than a platform?
Tiny internal tools, air-gapped constraints, or when you only need a few structured events - see the custom logging recipe in this section.
What should PRs require before merging agent features?
Instrumentation for new tools and nodes, redaction tests, and a sample trace attached to the PR for non-trivial paths.
Related
- Agent Observability Basics - first hands-on loop trace
- LangSmith for LangChain/LangGraph-Native Tracing - graph-native platform
- Langfuse: OpenTelemetry-Based Tracing for Any Framework - OTEL-friendly path
- Building Custom Trace Logging Without a Third-Party Platform - minimal host logger
- Agent Observability Best Practices - production checklist
- Audit Trails: Logging Every Agent Action for Later Review - compliance-grade events
Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana
@solana/web3.js+@solana/spl-token.