Reasoning Traces: How Agents "Think" Before Acting

When an agent "thinks," it is not running a separate brain process outside the model. It is producing tokens that represent intermediate reasoning, then (ideally) a structured decision such as a tool call or final answer.

Those intermediate tokens are the reasoning trace. Builders care about them because they affect quality, latency, cost, debuggability, and sometimes safety review.

This page explains what traces are, how they relate to tool calling, and when to expose, hide, or require them.

Summary

• A reasoning trace is model-generated intermediate text (or provider-specific "thinking" content) that sits between the current context and the next action, helping the model plan before it commits to a tool or answer.
• Insight: Traces can improve multi-step tool choice and make failures inspectable, but they also cost tokens, may leak sensitive planning, and are not a guarantee of correctness.
• Key Concepts: chain-of-thought (CoT), extended / deliberate reasoning, scratchpad, hidden vs visible traces, tool-call commitment, faithfulness.
• When to Use This Model: Choosing models with "thinking" modes, designing prompts that ask for plans, logging agent runs, or deciding whether end users should see intermediate reasoning.
• Limitations/Trade-offs: Longer traces are slower and more expensive. Visible CoT can be wrong while the final action is right (or the reverse). Some providers do not return raw thinking tokens to clients.
• Related Topics: agent loop reason phase, planning vs reacting, context budget, observability.

Foundations

The reason phase of the agent loop is a model completion. That completion can be structured in layers:

1. Optional private or semi-private thinking tokens
2. Optional natural-language plan or scratchpad
3. Commitment: tool call JSON, multi-tool batch, or final user-facing text

Not every product exposes layer 1. Many only log layer 2 when you prompt for it. Layer 3 is what the host runtime must parse to act.

Chain-of-thought originally meant prompting the model to write step-by-step reasoning before the answer. In agent systems, the same idea appears as:

• Explicit "Thought:" lines in ReAct-style prompts
• Separate plan messages before tool use
• Provider "extended thinking" or high-reasoning modes that allocate more internal tokens

A practical analogy: a pilot's checklist and radio chatter before moving the controls. The checklist helps, but what matters for the aircraft is the control input. For agents, the control input is the tool call or final answer.

context (perceive)
        │
        v
  [reasoning trace]  ← optional / provider-specific
        │
        v
  tool call or final answer  ← commitment the host must handle

Mechanics & Interactions

Prompted scratchpads

You can ask the model to reason in plain text before tools:

Think step by step about which tool to use.
Then emit a single tool call.

In classic ReAct formatting, the model alternates Thought / Action / Observation. Modern APIs often replace free-form Action lines with native function calling while still allowing a thought field or preceding text.

Native tool calling without visible thought

Many production agents only receive structured tool_calls with no user-visible reasoning. The model still "thinks" in its forward pass, but you do not get a separate trace object.

This is simpler for UX and safer against leaking internal strategy. It is harder to debug "why did it pick that tool?"

Extended reasoning modes

Some frontier models offer modes that spend more tokens on internal deliberation before answering. Those modes often improve hard multi-hop tasks and careful tool sequencing.

Trade-offs are direct: higher latency and cost per reason step. For trivial lookups, extended reasoning is usually waste.

Where the trace lives in the loop

If you keep full traces in the message list forever, context fills with self-talk. Many systems log full traces externally but only keep short summaries in the live window.

Faithfulness and debugging

A visible trace is not always a true report of the model's internal computation. It is still useful as a debug narrative: if the thought says "I will look up the user by email" and the tool call uses a phone number, you found a concrete inconsistency.

Do not treat traces as audit-proof explanations of model cognition. Do treat them as high-signal logs for builder review.

Safety and product surfaces

Showing raw traces to end users can:

• Reveal tool names, internal IDs, or policy wording
• Expose failed attempts and uncertainty that undermines trust
• Leak prompt-injection content that was only meant for the model

Common pattern: full traces for operators, short status for users ("Checking order status…"), final answer for everyone.

Advanced Considerations & Applications

When longer reasoning helps

Longer deliberation tends to help when:

• Multiple tools could apply and the wrong one is costly
• The goal is ambiguous and needs decomposition
• Observations contradict each other and need reconciliation
• The task is math-heavy, code-heavy, or policy-heavy

It tends to hurt when:

• The task is a single known lookup
• Latency budgets are tight (chat UX)
• Cost per session must stay tiny

Planning tokens vs acting tokens

Budget two pots: tokens spent thinking and tokens spent on tool I/O. A beautiful 800-token plan that calls the wrong API still fails. A 20-token decision that calls the right API often wins.

See Planning vs Reacting: Two Philosophies for Agent Decision-Making.

Storing traces for evals

Evaluation harnesses should store:

• Thought/plan text if available
• Tool calls and args
• Tool results
• Final answer
• Stop reason

Graders can check whether the plan mentioned the right entity and whether the tool args matched.

Python-shaped logging sketch

def log_reason_step(run_id: str, thought: str | None, decision: dict) -> None:
    # Send to your observability backend; do not always show `thought` to end users
    record = {"run_id": run_id, "thought": thought, "decision": decision}
    print(record)  # replace with structured logger

Keep this under 15 lines of illustrative code; wire real SDKs in framework sections.

Common Misconceptions

• "If I can see the chain-of-thought, the model is trustworthy." Visible reasoning can be wrong, incomplete, or post-hoc. Verify with tools and tests.
• "More thinking tokens always improve agents." Past a point you pay for verbosity and hesitation without better tool choices.
• "Reasoning traces replace tools." Thinking cannot fetch live order state. Tools still do the real work.
• "Hiding traces means the model is not reasoning." Hidden forward-pass computation still happens; you just do not receive a text dump.
• "Users should see every thought for transparency." Transparency for operators ≠ dumping scratchpads on consumers. Product trust needs curated status.
• "A plan in the trace is a contract." Without replan logic, a stale plan after a failed tool call becomes a liability.

FAQs

Related

• Inside the Agent Loop: Perceive, Reason, Act, Observe - where the reason phase sits
• Agent Loop Basics - tool call vs final answer branching
• Planning vs Reacting: Two Philosophies for Agent Decision-Making - how much to think up front
• How Context Shapes Every Decision an Agent Makes - what the reasoner can see
• Agent Loop Best Practices - logging and budget habits
• ReAct: Reasoning and Acting in an Interleaved Loop - architecture that popularized thought/action interleaving

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.