How Function Calling Actually Works Under the Hood

Function calling (also called tool use) is a structured contract between your host process and a model.

The model never runs your code. It proposes a named operation with JSON arguments. Your host validates those arguments, executes the real function, and feeds the result back as the next observation.

Summary

Tool use is a multi-message protocol: you declare tools, the model may emit structured tool_calls, you run them, append tool results, and call the model again until it answers in plain text or you stop the loop.
Insight: Almost every agent product is this loop plus policy. Misunderstanding who executes code, how messages accumulate, or when the model may call zero/one/many tools causes silent failures and unsafe side effects.
Key Concepts: tool schema, tool_calls, tool message, host validation, multi-turn trajectory, finish reason, parallel calls, tool choice.
When to Use This Model: Designing an agent runtime, debugging wrong tool selection, or mapping a framework's "agents" API onto the underlying chat API.
Limitations/Trade-offs: Schemas and wire formats differ slightly by provider. Native function calling still needs host-side authz, timeouts, and error shaping. Structured calls reduce parsing errors but do not guarantee correct business logic.
Related Topics: agent loops, tool descriptions, JSON Schema parameters, parallel tool calls, tool errors, MCP tool servers.

Foundations

Without tools, a chat model only returns text from its parameters and the current context.

With tools, you also send a list of tool definitions (name, description, JSON Schema for parameters). On each reason step the model chooses one of three outcomes:

Final text - no tool needed (or work is done).
One or more tool calls - structured proposals, not executions.
Refusal / empty / stop - provider-specific edge cases your host must handle.

The important ownership split:

Party	Owns
Model	Selecting tools and filling argument fields from context
Host	Validating args, authz, execution, timeouts, redaction, logging
Tool implementation	Side effects and returning a compact, truthful observation

If you let the model emit free-form "run this shell" text and you execute it, that is not safe function calling. Native tool use keeps the action space closed: only tools you registered can be requested by name.

request  = messages + tools + tool_choice + model params
response = assistant message (content and/or tool_calls)
if tool_calls:
    for each call:
        result = host.execute(call)
        messages += tool result bound to call id
    request again with updated messages
else:
    return assistant content (or stop for other reasons)

That cycle is the same whether you use raw OpenAI-compatible APIs, OpenRouter, LangGraph tool nodes, Pydantic AI, CrewAI tools, or MCP-backed tool lists. Frameworks hide the plumbing; they do not remove the host's duty to execute safely.

Mechanics & Interactions

Message roles in the trajectory

A minimal successful path looks like this:

system - policy and role.
user - goal.
assistant with tool_calls - proposed action(s).
tool (one per call id) - observation string or JSON.
assistant - final natural-language answer (or more tool_calls).

Providers differ on field names (tool_calls vs function_call legacy, role: "tool" vs vendor variants). The logical roles stay stable: proposal, then observation, then next decision.

What the model actually emits

A tool call typically includes:

name - must match a registered tool
arguments - JSON object (sometimes delivered as a JSON string to parse)
id - correlates the later tool result to this call

Your host should never trust the name blindly. Map name → implementation through an allowlist. Reject unknown names with a tool-result error the model can read.

Host validation before act

Schema validity is the first gate, not the last.

import json
from typing import Any, Callable
 
REGISTRY: dict[str, Callable[..., Any]] = {}
 
def handle_tool_call(name: str, arguments: str | dict) -> dict:
    if name not in REGISTRY:
        return {"ok": False, "error": f"unknown_tool:{name}"}
    args = json.loads(arguments) if isinstance(arguments, str) else arguments
    # schema validate args here (jsonschema / pydantic)
    try:
        value = REGISTRY[name](**args)
        return {"ok": True, "data": value}
    except Exception as exc:  # narrow in production
        return {"ok": False, "error": type(exc).__name__, "message": str(exc)[:300]}

Business rules (positive ids, allowed tenants, idempotency keys) belong in this gate. Crashing the process on bad args aborts the agent loop; returning a structured error usually lets the model recover.

Tool results re-enter context

The model only "sees" what you put back.

Prefer short, structured payloads:

{"ok": True, "order_id": 1042, "status": "shipped", "eta": "2026-07-18"}

Avoid dumping full HTTP transcripts, HTML pages, or secrets. Those burn tokens, hide the goal, and can inject untrusted instructions into the next reason step.

Finish reasons and tool_choice

APIs expose controls such as:

auto - model decides whether to call tools
none - forbid tools this turn (useful for finalize)
required / forced function - must call a tool (or a specific one)

Forced tool use is helpful for extraction steps. It is harmful if you force a write tool when the user only asked a question.

Always read the provider finish reason (tool_calls, stop, length, and so on). A length finish mid-arguments is a broken call; do not execute partial JSON.

Parallel calls and multi-step chains

Modern models may return several tool_calls in one assistant message. That is batching independent reads, not true multi-threaded reasoning inside the model.

Dependent steps still need sequential turns: call A, observe A, then call B with A's id. See Parallel Tool Calls.

Where MCP and frameworks sit

MCP standardizes how hosts discover and invoke remote tools. Frameworks (LangGraph, CrewAI, OpenAI Agents SDK, Pydantic AI, Vercel AI SDK) own graph/state helpers and often auto-loop tool calls.

Under every abstraction you still have: schema in, structured call out, host execute, observation back.

Advanced Considerations & Applications

Token cost of tools

Tool schemas sit in context on every reason call (unless the provider caches a stable prefix). Large libraries raise cost and worsen selection. Patterns like deferred tool loading / tool search address that; see Tool Search.

Streaming tool calls

Streaming APIs may deliver argument fragments token-by-token. Buffer until the call is complete and JSON-parseable before executing. Never run side effects on partial arguments.

Determinism and replay

Log call id, name, args (redacted), result hash, latency, and model identity. Replay with stubbed tools is the foundation of agent tests. Without call ids, parallel results are hard to re-associate.

Security boundary

Treat tool outputs as untrusted text in the next prompt. They can contain prompt-injection payloads from web pages, tickets, or emails. Keep hard deny rules in host code, not only in the system prompt.

Design	Strength	Weakness	Best fit
Native function calling + host loop	Clear contract, portable mental model	You implement validation and stops	Most production agents
Free-form "Action:" text parsing	Works on older models	Brittle parsing, easier injection	Legacy demos only
Forced single tool	High reliability for extraction	No multi-tool flexibility	Form-fill / structured ETL
Huge always-on tool list	Maximum capability surface	Cost, confusion, risky writes	Avoid; prefer phased tools

Common Misconceptions

"The model runs my function." It only proposes calls. Your process runs code.
"If the schema is valid, the call is safe." Schema does not encode authz, tenancy, or "should we do this."
"Tool calling is the same as an agent." Tool calling is a capability. An agent also needs a loop, stops, and state policy.
"More tools always make a smarter agent." Overlapping tools increase mis-selection and token use.
"Empty tool results mean success." Empty observations often cause invent-from-memory answers; return explicit ok / error fields.
"Framework = correct protocol." Frameworks still need your max turns, error shaping, and credential scoping.

FAQs

What is the difference between "function calling" and "tool use"?

Mostly branding. Both mean the model emits structured calls against declared tools instead of only free text.

Does the model see the Python source of my tool?

No. It sees the name, description, and parameter schema you send (plus any docs you put in prompts).

Why did the model answer without calling my tool?

It may believe parametric knowledge is enough, the description was unclear, tools were not passed, or tool_choice forbade tools. Improve descriptions, require tools when ground truth is mandatory, or instruct "use tools for live data."

Should I execute tools concurrently?

Only when the host can safely parallelize independent side-effect-free calls. Preserve result-to-call-id mapping. See the parallel-calls page in this section.

How do I bind a tool result to the right call?

Use the provider's tool call id on the tool message. Never rely on order alone when multiple calls exist.

What if arguments are invalid JSON?

Do not execute. Return a tool error describing the parse failure, or repair once under a strict policy, then re-ask the model.

Can tool results include binary data?

Prefer references (urls, file ids) or short base64 with hard size caps. Huge blobs destroy the context budget.

How is this different from JSON mode / structured outputs?

Structured outputs constrain the final answer shape. Function calling selects and parameterizes actions mid-loop. Many products use both.

Do all models support tools equally?

No. Capability, parallel-call quality, and schema strictness vary. Verify at build for each model slug you route to.

Where do system prompts fit relative to tools?

System policy still governs when tools may be used, what to refuse, and how to present answers. Tools expand actions; they do not replace policy.

Should errors crash the loop or return as tool messages?

Default to structured tool errors so the model can adapt. Crash only on host-fatal conditions (auth misconfig, out-of-memory). See Handling Tool Errors.

Is MCP required for function calling?

No. MCP is a transport/discovery standard for tools. Local in-process tools work with plain function calling.

How many round trips are normal?

Task-dependent. Simple lookups may need one tool turn. Multi-step workflows need several. Always enforce max turns and timeouts in the host.

Why log tool args if I already log the final answer?

Most production bugs are wrong args, wrong tool choice, or bad observations. Final prose alone is not enough to debug.

Tool Use & Function Calling Basics - hands-on first cycle
Writing Tool Descriptions the Model Will Actually Use Correctly - selection quality
JSON Schema for Tool Parameters: A Reference Guide - parameter shapes
Parallel Tool Calls: When and How Models Call Multiple Tools at Once - batched calls
Handling Tool Errors Gracefully Inside an Agent Loop - failure as observation
Tool Use & Function Calling Best Practices - operational habits
Inside the Agent Loop: Perceive, Reason, Act, Observe - loop framing
Agent Loop Basics - minimal loop code

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

How Function Calling Actually Works Under the Hood

Function calling (also called tool use) is a structured contract between your host process and a model.

The model never runs your code. It proposes a named operation with JSON arguments. Your host validates those arguments, executes the real function, and feeds the result back as the next observation.

Summary

Tool use is a multi-message protocol: you declare tools, the model may emit structured tool_calls, you run them, append tool results, and call the model again until it answers in plain text or you stop the loop.
Insight: Almost every agent product is this loop plus policy. Misunderstanding who executes code, how messages accumulate, or when the model may call zero/one/many tools causes silent failures and unsafe side effects.
Key Concepts: tool schema, tool_calls, tool message, host validation, multi-turn trajectory, finish reason, parallel calls, tool choice.
When to Use This Model: Designing an agent runtime, debugging wrong tool selection, or mapping a framework's "agents" API onto the underlying chat API.
Limitations/Trade-offs: Schemas and wire formats differ slightly by provider. Native function calling still needs host-side authz, timeouts, and error shaping. Structured calls reduce parsing errors but do not guarantee correct business logic.
Related Topics: agent loops, tool descriptions, JSON Schema parameters, parallel tool calls, tool errors, MCP tool servers.

Foundations

Without tools, a chat model only returns text from its parameters and the current context.

With tools, you also send a list of tool definitions (name, description, JSON Schema for parameters). On each reason step the model chooses one of three outcomes:

Final text - no tool needed (or work is done).
One or more tool calls - structured proposals, not executions.
Refusal / empty / stop - provider-specific edge cases your host must handle.

The important ownership split:

Party	Owns
Model	Selecting tools and filling argument fields from context
Host	Validating args, authz, execution, timeouts, redaction, logging
Tool implementation	Side effects and returning a compact, truthful observation

request  = messages + tools + tool_choice + model params
response = assistant message (content and/or tool_calls)
if tool_calls:
    for each call:
        result = host.execute(call)
        messages += tool result bound to call id
    request again with updated messages
else:
    return assistant content (or stop for other reasons)

Mechanics & Interactions

Message roles in the trajectory

A minimal successful path looks like this:

system - policy and role.
user - goal.
assistant with tool_calls - proposed action(s).
tool (one per call id) - observation string or JSON.
assistant - final natural-language answer (or more tool_calls).

Providers differ on field names (tool_calls vs function_call legacy, role: "tool" vs vendor variants). The logical roles stay stable: proposal, then observation, then next decision.

What the model actually emits

A tool call typically includes:

name - must match a registered tool
arguments - JSON object (sometimes delivered as a JSON string to parse)
id - correlates the later tool result to this call

Your host should never trust the name blindly. Map name → implementation through an allowlist. Reject unknown names with a tool-result error the model can read.

Host validation before act

Schema validity is the first gate, not the last.

import json
from typing import Any, Callable
 
REGISTRY: dict[str, Callable[..., Any]] = {}
 
def handle_tool_call(name: str, arguments: str | dict) -> dict:
    if name not in REGISTRY:
        return {"ok": False, "error": f"unknown_tool:{name}"}
    args = json.loads(arguments) if isinstance(arguments, str) else arguments
    # schema validate args here (jsonschema / pydantic)
    try:
        value = REGISTRY[name](**args)
        return {"ok": True, "data": value}
    except Exception as exc:  # narrow in production
        return {"ok": False, "error": type(exc).__name__, "message": str(exc)[:300]}

Tool results re-enter context

The model only "sees" what you put back.

Prefer short, structured payloads:

{"ok": True, "order_id": 1042, "status": "shipped", "eta": "2026-07-18"}

Avoid dumping full HTTP transcripts, HTML pages, or secrets. Those burn tokens, hide the goal, and can inject untrusted instructions into the next reason step.

Finish reasons and tool_choice

APIs expose controls such as:

auto - model decides whether to call tools
none - forbid tools this turn (useful for finalize)
required / forced function - must call a tool (or a specific one)

Forced tool use is helpful for extraction steps. It is harmful if you force a write tool when the user only asked a question.

Always read the provider finish reason (tool_calls, stop, length, and so on). A length finish mid-arguments is a broken call; do not execute partial JSON.

Parallel calls and multi-step chains

Modern models may return several tool_calls in one assistant message. That is batching independent reads, not true multi-threaded reasoning inside the model.

Dependent steps still need sequential turns: call A, observe A, then call B with A's id. See Parallel Tool Calls.

Where MCP and frameworks sit

Under every abstraction you still have: schema in, structured call out, host execute, observation back.

Advanced Considerations & Applications

Token cost of tools

Streaming tool calls

Streaming APIs may deliver argument fragments token-by-token. Buffer until the call is complete and JSON-parseable before executing. Never run side effects on partial arguments.

Determinism and replay

Log call id, name, args (redacted), result hash, latency, and model identity. Replay with stubbed tools is the foundation of agent tests. Without call ids, parallel results are hard to re-associate.

Security boundary

Design	Strength	Weakness	Best fit
Native function calling + host loop	Clear contract, portable mental model	You implement validation and stops	Most production agents
Free-form "Action:" text parsing	Works on older models	Brittle parsing, easier injection	Legacy demos only
Forced single tool	High reliability for extraction	No multi-tool flexibility	Form-fill / structured ETL
Huge always-on tool list	Maximum capability surface	Cost, confusion, risky writes	Avoid; prefer phased tools

Common Misconceptions

"The model runs my function." It only proposes calls. Your process runs code.
"If the schema is valid, the call is safe." Schema does not encode authz, tenancy, or "should we do this."
"Tool calling is the same as an agent." Tool calling is a capability. An agent also needs a loop, stops, and state policy.
"More tools always make a smarter agent." Overlapping tools increase mis-selection and token use.
"Empty tool results mean success." Empty observations often cause invent-from-memory answers; return explicit ok / error fields.
"Framework = correct protocol." Frameworks still need your max turns, error shaping, and credential scoping.

FAQs

What is the difference between "function calling" and "tool use"?

Mostly branding. Both mean the model emits structured calls against declared tools instead of only free text.

Does the model see the Python source of my tool?

No. It sees the name, description, and parameter schema you send (plus any docs you put in prompts).

Why did the model answer without calling my tool?

Should I execute tools concurrently?

Only when the host can safely parallelize independent side-effect-free calls. Preserve result-to-call-id mapping. See the parallel-calls page in this section.

How do I bind a tool result to the right call?

Use the provider's tool call id on the tool message. Never rely on order alone when multiple calls exist.

What if arguments are invalid JSON?

Do not execute. Return a tool error describing the parse failure, or repair once under a strict policy, then re-ask the model.

Can tool results include binary data?

Prefer references (urls, file ids) or short base64 with hard size caps. Huge blobs destroy the context budget.

How is this different from JSON mode / structured outputs?

Structured outputs constrain the final answer shape. Function calling selects and parameterizes actions mid-loop. Many products use both.

Do all models support tools equally?

No. Capability, parallel-call quality, and schema strictness vary. Verify at build for each model slug you route to.

Where do system prompts fit relative to tools?

System policy still governs when tools may be used, what to refuse, and how to present answers. Tools expand actions; they do not replace policy.

Should errors crash the loop or return as tool messages?

Default to structured tool errors so the model can adapt. Crash only on host-fatal conditions (auth misconfig, out-of-memory). See Handling Tool Errors.

Is MCP required for function calling?

No. MCP is a transport/discovery standard for tools. Local in-process tools work with plain function calling.

How many round trips are normal?

Task-dependent. Simple lookups may need one tool turn. Multi-step workflows need several. Always enforce max turns and timeouts in the host.

Why log tool args if I already log the final answer?

Most production bugs are wrong args, wrong tool choice, or bad observations. Final prose alone is not enough to debug.

Tool Use & Function Calling Basics - hands-on first cycle
Writing Tool Descriptions the Model Will Actually Use Correctly - selection quality
JSON Schema for Tool Parameters: A Reference Guide - parameter shapes
Parallel Tool Calls: When and How Models Call Multiple Tools at Once - batched calls
Handling Tool Errors Gracefully Inside an Agent Loop - failure as observation
Tool Use & Function Calling Best Practices - operational habits
Inside the Agent Loop: Perceive, Reason, Act, Observe - loop framing
Agent Loop Basics - minimal loop code

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

How Function Calling Actually Works Under the Hood

Search Documentation

How Function Calling Actually Works Under the Hood