Unit Testing Individual Agent Tools

Q: What about async tools?

Use your async test support (pytest-asyncio or equivalent) and still inject fakes. Do not require a running event loop against real services in unit tests.

Q: How fine-grained should return errors be?

Fine enough for the model to correct args or stop. Include error_code, human message, and retryable when possible.

Tool unit tests prove that each model-callable function behaves correctly for given arguments, without paying for a planner or a live LLM.

They are the cheapest layer in an agent test pyramid and the one most teams underbuild.

Summary

Extract tools as plain functions (or thin adapters), inject dependencies, cover happy path plus validation and failure modes, and keep the agent loop out of these tests entirely.

Recipe

List every tool the agent may call (name, args schema, side effects, auth scope).
Move implementation into importable modules with no global model client required to exercise them.
Inject HTTP/DB/clock/randomness so tests can pass fakes.
Write pure unit tests for: valid args, invalid args, auth failures, empty results, timeouts, and idempotent retries.
Assert on return payloads the model will see (shape and error codes), not only on internal side effects.
Keep tests offline and fast; mark rare live integration tests separately.
Run tool unit tests on every PR before any model eval job.
When a production bug is tool-shaped, add a regression test at this layer first.

Working Example

# tools/tickets.py
from typing import Protocol
 
 
class TicketApi(Protocol):
    def create(self, title: str, priority: str, idempotency_key: str) -> dict: ...
 
 
ALLOWED = {"P1", "P2", "P3"}
 
 
def create_ticket(
    api: TicketApi,
    *,
    title: str,
    priority: str,
    idempotency_key: str,
) -> dict:
    title = title.strip()
    if not title:
        return {"ok": False, "error_code": "VALIDATION", "message": "empty title", "retryable": False}
    if priority not in ALLOWED:
        return {
            "ok": False,
            "error_code": "VALIDATION",
            "message": f"priority must be one of {sorted(ALLOWED)}",
            "retryable": False,
        }
    if not idempotency_key:
        return {"ok": False, "error_code": "VALIDATION", "message": "idempotency_key required", "retryable": False}
    try:
        ticket = api.create(title, priority, idempotency_key)
    except TimeoutError:
        return {"ok": False, "error_code": "TIMEOUT", "message": "upstream timeout", "retryable": True}
    return {"ok": True, "ticket": ticket}
 
 
# tests/test_tickets_tool.py
class FakeApi:
    def __init__(self):
        self.calls = []
        self.by_key = {}
 
    def create(self, title, priority, idempotency_key):
        self.calls.append((title, priority, idempotency_key))
        if idempotency_key in self.by_key:
            return self.by_key[idempotency_key]
        row = {"id": "t1", "title": title, "priority": priority}
        self.by_key[idempotency_key] = row
        return row
 
 
def test_create_ticket_happy_path():
    api = FakeApi()
    out = create_ticket(
        api, title="  Outage ", priority="P1", idempotency_key="k1"
    )
    assert out["ok"] is True
    assert out["ticket"]["title"] == "Outage"
    assert api.calls == [("Outage", "P1", "k1")]
 
 
def test_create_ticket_validation_and_idempotency():
    api = FakeApi()
    bad = create_ticket(api, title="", priority="P1", idempotency_key="k")
    assert bad["ok"] is False and bad["error_code"] == "VALIDATION"
 
    a = create_ticket(api, title="A", priority="P2", idempotency_key="same")
    b = create_ticket(api, title="A", priority="P2", idempotency_key="same")
    assert a == b and len(api.by_key) == 1

Timeout path:

class SlowApi:
    def create(self, title, priority, idempotency_key):
        raise TimeoutError()
 
 
def test_create_ticket_timeout_is_retryable():
    out = create_ticket(
        SlowApi(), title="A", priority="P1", idempotency_key="k"
    )
    assert out == {
        "ok": False,
        "error_code": "TIMEOUT",
        "message": "upstream timeout",
        "retryable": True,
    }

Deep Dive

Why isolate tools from the loop

The agent loop multiplies variance: model choice, prompt text, and multi-step history. Tool unit tests freeze the contract so you know failures above this layer are planner or prompt problems.

What good tool tests cover

Category	Examples
Schema / validation	Missing fields, enums, bounds, stripping whitespace
Authorization	Scoped token cannot hit admin routes
Happy path	Correct upstream call and model-facing JSON
Empty / not found	Clear `NOT_FOUND`, non-retryable when appropriate
Transient failure	Timeouts and 503 map to `retryable: true`
Idempotency	Same key → same business effect
Safety caps	Amount limits, allowlists, path sandboxing
Observability hooks	Request ids present when your wrapper adds them

Design habits that make testing easy

Pure-ish core. Parse and validate before I/O.
Explicit dependencies. Pass clients in; avoid hidden singletons.
Stable error envelopes. Models recover better from structured failures than raw stack traces.
Side-effect ranking. Read tools tested differently from write tools (writes need idempotency and audit).
No network in default pytest path. Use markers like @pytest.mark.integration for live calls.

Mapping tools to test ownership

Tool type	Unit focus	Often needs integration too
Calculator / formatter	Exact outputs	Rarely
REST wrapper	Mapping, errors, retries	Contract vs staging API
DB query	SQL builder, row limits, read-only user	Real DB migration tests
Sandboxed shell	Path allowlist, timeouts	Container smoke
Browser / search	Query building, result truncation	Recorded fixtures

Gotchas

Testing only the happy path. Models invent weird args; validation tests are mandatory.
Asserting internal mocks but not the returned JSON. The model only sees the return value.
Hidden real network in "unit" tests. Flakes train people to ignore red CI.
Shared mutable fakes across tests. Isolate state per test.
Swallowing exceptions into empty strings. You lose retry signals and unit clarity.
Coupling tests to full framework Agent classes. Import the tool function directly.
Skipping idempotency for writes. Retries will double-apply in production even if demos look fine.

Alternatives

Approach	When to use	Trade-off
Direct function unit tests	Default for every tool	Requires extractable tools
Framework-level tool tests	When registration wiring is fragile	Heavier fixtures
Contract tests vs real staging	External APIs you do not own	Needs env and credentials
Property-based tests (Hypothesis)	Parsers and numeric tools	More upfront design
Snapshot of OpenAPI-derived fixtures	Large REST surfaces	Snapshots can rot

FAQs

Should every tool have unit tests before the first demo?

At least every write tool and every tool that enforces policy. Read-only pure helpers can start thinner, but anything that spends money or mutates state needs coverage early.

How do I test tools that call LLMs themselves?

Split "build prompt / parse response" from "HTTP to model." Unit-test parse and policy; mock or record the model boundary.

Do I need to test tool descriptions and schemas?

Yes at a lightweight level: required fields, enums, and that the registered name matches docs. Description quality is more of a prompt/eval concern.

What about async tools?

Use your async test support (pytest-asyncio or equivalent) and still inject fakes. Do not require a running event loop against real services in unit tests.

How fine-grained should return errors be?

Fine enough for the model to correct args or stop. Include error_code, human message, and retryable when possible.

Can I unit-test MCP or third-party tool servers?

Test your adapter and policy layer unit-style. Treat the remote server with contract or sandbox integration tests.

Where do permission checks live?

In the tool host before side effects. Unit-test that missing scopes never call the underlying API.

How do tool tests relate to gold-set evals?

Tool tests guarantee the screwdriver works. Gold sets check that the agent picks up the right screwdriver for the job.

Should fakes mirror production latency?

Not in unit tests. Optionally simulate timeouts. Load and latency belong in separate jobs.

What is a good CI time budget for tool unit tests?

Seconds to low minutes for the whole tool pack. If it is slow, you accidentally included network or heavy containers.

Related: Testing AI Agents Basics

Related: Why Testing an Agent Differs from Testing Deterministic Code

Related: Simulating Full Agent Loops in a Test Harness

Related: Building a Gold-Set Eval Suite for Agent Quality

Related: Testing AI Agents Best Practices

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

They are the cheapest layer in an agent test pyramid and the one most teams underbuild.

Summary

Extract tools as plain functions (or thin adapters), inject dependencies, cover happy path plus validation and failure modes, and keep the agent loop out of these tests entirely.

Recipe

List every tool the agent may call (name, args schema, side effects, auth scope).
Move implementation into importable modules with no global model client required to exercise them.
Inject HTTP/DB/clock/randomness so tests can pass fakes.
Write pure unit tests for: valid args, invalid args, auth failures, empty results, timeouts, and idempotent retries.
Assert on return payloads the model will see (shape and error codes), not only on internal side effects.
Keep tests offline and fast; mark rare live integration tests separately.
Run tool unit tests on every PR before any model eval job.
When a production bug is tool-shaped, add a regression test at this layer first.

Working Example

# tools/tickets.py
from typing import Protocol
 
 
class TicketApi(Protocol):
    def create(self, title: str, priority: str, idempotency_key: str) -> dict: ...
 
 
ALLOWED = {"P1", "P2", "P3"}
 
 
def create_ticket(
    api: TicketApi,
    *,
    title: str,
    priority: str,
    idempotency_key: str,
) -> dict:
    title = title.strip()
    if not title:
        return {"ok": False, "error_code": "VALIDATION", "message": "empty title", "retryable": False}
    if priority not in ALLOWED:
        return {
            "ok": False,
            "error_code": "VALIDATION",
            "message": f"priority must be one of {sorted(ALLOWED)}",
            "retryable": False,
        }
    if not idempotency_key:
        return {"ok": False, "error_code": "VALIDATION", "message": "idempotency_key required", "retryable": False}
    try:
        ticket = api.create(title, priority, idempotency_key)
    except TimeoutError:
        return {"ok": False, "error_code": "TIMEOUT", "message": "upstream timeout", "retryable": True}
    return {"ok": True, "ticket": ticket}
 
 
# tests/test_tickets_tool.py
class FakeApi:
    def __init__(self):
        self.calls = []
        self.by_key = {}
 
    def create(self, title, priority, idempotency_key):
        self.calls.append((title, priority, idempotency_key))
        if idempotency_key in self.by_key:
            return self.by_key[idempotency_key]
        row = {"id": "t1", "title": title, "priority": priority}
        self.by_key[idempotency_key] = row
        return row
 
 
def test_create_ticket_happy_path():
    api = FakeApi()
    out = create_ticket(
        api, title="  Outage ", priority="P1", idempotency_key="k1"
    )
    assert out["ok"] is True
    assert out["ticket"]["title"] == "Outage"
    assert api.calls == [("Outage", "P1", "k1")]
 
 
def test_create_ticket_validation_and_idempotency():
    api = FakeApi()
    bad = create_ticket(api, title="", priority="P1", idempotency_key="k")
    assert bad["ok"] is False and bad["error_code"] == "VALIDATION"
 
    a = create_ticket(api, title="A", priority="P2", idempotency_key="same")
    b = create_ticket(api, title="A", priority="P2", idempotency_key="same")
    assert a == b and len(api.by_key) == 1

Timeout path:

class SlowApi:
    def create(self, title, priority, idempotency_key):
        raise TimeoutError()
 
 
def test_create_ticket_timeout_is_retryable():
    out = create_ticket(
        SlowApi(), title="A", priority="P1", idempotency_key="k"
    )
    assert out == {
        "ok": False,
        "error_code": "TIMEOUT",
        "message": "upstream timeout",
        "retryable": True,
    }

Deep Dive

Why isolate tools from the loop

The agent loop multiplies variance: model choice, prompt text, and multi-step history. Tool unit tests freeze the contract so you know failures above this layer are planner or prompt problems.

What good tool tests cover

Category	Examples
Schema / validation	Missing fields, enums, bounds, stripping whitespace
Authorization	Scoped token cannot hit admin routes
Happy path	Correct upstream call and model-facing JSON
Empty / not found	Clear `NOT_FOUND`, non-retryable when appropriate
Transient failure	Timeouts and 503 map to `retryable: true`
Idempotency	Same key → same business effect
Safety caps	Amount limits, allowlists, path sandboxing
Observability hooks	Request ids present when your wrapper adds them

Design habits that make testing easy

Pure-ish core. Parse and validate before I/O.
Explicit dependencies. Pass clients in; avoid hidden singletons.
Stable error envelopes. Models recover better from structured failures than raw stack traces.
Side-effect ranking. Read tools tested differently from write tools (writes need idempotency and audit).
No network in default pytest path. Use markers like @pytest.mark.integration for live calls.

Mapping tools to test ownership

Tool type	Unit focus	Often needs integration too
Calculator / formatter	Exact outputs	Rarely
REST wrapper	Mapping, errors, retries	Contract vs staging API
DB query	SQL builder, row limits, read-only user	Real DB migration tests
Sandboxed shell	Path allowlist, timeouts	Container smoke
Browser / search	Query building, result truncation	Recorded fixtures

Gotchas

Testing only the happy path. Models invent weird args; validation tests are mandatory.
Asserting internal mocks but not the returned JSON. The model only sees the return value.
Hidden real network in "unit" tests. Flakes train people to ignore red CI.
Shared mutable fakes across tests. Isolate state per test.
Swallowing exceptions into empty strings. You lose retry signals and unit clarity.
Coupling tests to full framework Agent classes. Import the tool function directly.
Skipping idempotency for writes. Retries will double-apply in production even if demos look fine.

Alternatives

Approach	When to use	Trade-off
Direct function unit tests	Default for every tool	Requires extractable tools
Framework-level tool tests	When registration wiring is fragile	Heavier fixtures
Contract tests vs real staging	External APIs you do not own	Needs env and credentials
Property-based tests (Hypothesis)	Parsers and numeric tools	More upfront design
Snapshot of OpenAPI-derived fixtures	Large REST surfaces	Snapshots can rot

FAQs

Should every tool have unit tests before the first demo?

At least every write tool and every tool that enforces policy. Read-only pure helpers can start thinner, but anything that spends money or mutates state needs coverage early.

How do I test tools that call LLMs themselves?

Split "build prompt / parse response" from "HTTP to model." Unit-test parse and policy; mock or record the model boundary.

Do I need to test tool descriptions and schemas?

Yes at a lightweight level: required fields, enums, and that the registered name matches docs. Description quality is more of a prompt/eval concern.

What about async tools?

Use your async test support (pytest-asyncio or equivalent) and still inject fakes. Do not require a running event loop against real services in unit tests.

How fine-grained should return errors be?

Fine enough for the model to correct args or stop. Include error_code, human message, and retryable when possible.

Can I unit-test MCP or third-party tool servers?

Test your adapter and policy layer unit-style. Treat the remote server with contract or sandbox integration tests.

Where do permission checks live?

In the tool host before side effects. Unit-test that missing scopes never call the underlying API.

How do tool tests relate to gold-set evals?

Tool tests guarantee the screwdriver works. Gold sets check that the agent picks up the right screwdriver for the job.

Should fakes mirror production latency?

Not in unit tests. Optionally simulate timeouts. Load and latency belong in separate jobs.

What is a good CI time budget for tool unit tests?

Seconds to low minutes for the whole tool pack. If it is slow, you accidentally included network or heavy containers.

Related: Testing AI Agents Basics

Related: Why Testing an Agent Differs from Testing Deterministic Code

Related: Simulating Full Agent Loops in a Test Harness

Related: Building a Gold-Set Eval Suite for Agent Quality

Related: Testing AI Agents Best Practices

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Unit Testing Individual Agent Tools

Search Documentation

Unit Testing Individual Agent Tools