Search across all documentation pages
When an agent "thinks," it is not running a separate brain process outside the model. It is producing tokens that represent intermediate reasoning, then (ideally) a structured decision such as a tool call or final answer.
Those intermediate tokens are the reasoning trace. Builders care about them because they affect quality, latency, cost, debuggability, and sometimes safety review.
This page explains what traces are, how they relate to tool calling, and when to expose, hide, or require them.
The reason phase of the agent loop is a model completion. That completion can be structured in layers:
Not every product exposes layer 1. Many only log layer 2 when you prompt for it. Layer 3 is what the host runtime must parse to act.
Chain-of-thought originally meant prompting the model to write step-by-step reasoning before the answer. In agent systems, the same idea appears as:
A practical analogy: a pilot's checklist and radio chatter before moving the controls. The checklist helps, but what matters for the aircraft is the control input. For agents, the control input is the tool call or final answer.
context (perceive)
│
v
[reasoning trace] ← optional / provider-specific
│
v
tool call or final answer ← commitment the host must handleYou can ask the model to reason in plain text before tools:
Think step by step about which tool to use.
Then emit a single tool call.In classic ReAct formatting, the model alternates Thought / Action / Observation.
Modern APIs often replace free-form Action lines with native function calling while still allowing a thought field or preceding text.
Many production agents only receive structured tool_calls with no user-visible reasoning.
The model still "thinks" in its forward pass, but you do not get a separate trace object.
This is simpler for UX and safer against leaking internal strategy. It is harder to debug "why did it pick that tool?"
Some frontier models offer modes that spend more tokens on internal deliberation before answering. Those modes often improve hard multi-hop tasks and careful tool sequencing.
Trade-offs are direct: higher latency and cost per reason step. For trivial lookups, extended reasoning is usually waste.
| Placement | Effect |
|---|---|
| Inside one completion before tool JSON | Helps that single decision; may or may not be stored |
| As a separate "planner" call | Explicit plan message becomes context for later steps |
| Across ReAct iterations | Each thought can reference prior observations |
| Summarized into memory | Compresses long traces so context does not explode |
If you keep full traces in the message list forever, context fills with self-talk. Many systems log full traces externally but only keep short summaries in the live window.
A visible trace is not always a true report of the model's internal computation. It is still useful as a debug narrative: if the thought says "I will look up the user by email" and the tool call uses a phone number, you found a concrete inconsistency.
Do not treat traces as audit-proof explanations of model cognition. Do treat them as high-signal logs for builder review.
Showing raw traces to end users can:
Common pattern: full traces for operators, short status for users ("Checking order status…"), final answer for everyone.
Longer deliberation tends to help when:
It tends to hurt when:
Budget two pots: tokens spent thinking and tokens spent on tool I/O. A beautiful 800-token plan that calls the wrong API still fails. A 20-token decision that calls the right API often wins.
See Planning vs Reacting: Two Philosophies for Agent Decision-Making.
| Approach | Strength | Weakness | Best fit |
|---|---|---|---|
| No visible trace, native tools only | Fast, simple UX, less leakage | Harder human debugging | High-volume simple agents |
| Short prompted thoughts | Cheap interpretability | Can be shallow or theatrical | Mid complexity ReAct loops |
| Extended reasoning mode | Better hard-task quality | Cost and latency | Complex research / coding steps |
| Separate planner agent/call | Clear plan artifact | Extra hop; plans can go stale | Long workflows with replan |
Evaluation harnesses should store:
Graders can check whether the plan mentioned the right entity and whether the tool args matched.
def log_reason_step(run_id: str, thought: str | None, decision: dict) -> None:
# Send to your observability backend; do not always show `thought` to end users
record = {"run_id": run_id, "thought": thought, "decision": decision}
print(record) # replace with structured loggerKeep this under 15 lines of illustrative code; wire real SDKs in framework sections.
Model-produced intermediate content that appears before a committed action or answer, used to structure multi-step decisions.
Related but not identical. Classic CoT is a prompting pattern. Extended modes are product features that allocate more internal deliberation, sometimes with separate token accounting.
No. Native tool calling works without any user-visible thought text. Visible thoughts are optional for debugging and some quality patterns.
Usually no, or only a sanitized progress summary. Full traces are better for operators, support, and evals.
Yes. They may restate system prompts, tool schemas, prior user PII, or injection payloads. Treat traces as sensitive logs.
Traces are not guaranteed faithful. Log both, fix prompts/tools, and score consistency in evals when it matters.
If you keep full thoughts in the live message list, they consume tokens every later turn. Prefer external logs plus short in-context summaries.
No. ReAct popularized interleaved thought and action, but modern stacks often use native tools with optional free-text rationales.
When task failure is expensive and ambiguity is high. Keep a cheaper default for simple turns.
No. Observe is environment feedback after acting. Traces belong to the reason phase before or while deciding.
You can prompt for structured plans (goals, risks, next tool). Enforcement is soft unless you validate with a schema and reject bad shapes.
Map tool names to human status strings: "Looking up order," "Checking inventory," "Drafting reply."
They help diagnosis (repetitive thoughts are a signal) but do not replace max-turn stops and tool-result checks.
It can reduce accidental user exposure, but server-side logs and provider policies still matter. Security is a system property, not only a UX choice.
Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana
@solana/web3.js+@solana/spl-token.
Reviewed by Chris St. John·Last updated Jul 16, 2026