Why Testing an Agent Differs from Testing Deterministic Code
Deterministic unit tests assert that the same input always yields the same output. Agent systems break that contract because models sample, tool results drift, and multi-step plans take different paths to the same goal.
You still need tests. You need a layered strategy: hard assertions where code is deterministic, and evals where quality is graded rather than string-matched.
Summary
- Agent quality is scored with distributions and rubrics, not only byte-identical snapshots of free-form text.
- Insight: Exact-match tests on model text flake, hide real regressions, and train teams to freeze brittle prompts instead of measuring task success.
- Key Concepts: non-determinism, tool isolation, trajectory checks, gold sets, LLM-as-judge, pass rates, regression gates.
- When to Use: Any multi-step LLM agent with tools, planning, or open-ended answers; also any single-turn product surface where wording can vary.
- Limitations/Trade-offs: Evals cost money and time, judges can be biased, and flaky suites erode trust if thresholds and seeds are sloppy.
- Related Topics: unit-testing tools, simulated loops, gold-set suites, LLM judges, prompt/model regression checklists.
Foundations
What classic tests assume
Ordinary code is a pure or nearly pure function of inputs and state you control.
assert add(2, 2) == 4
assert parse_invoice(blob)["total"] == Decimal("19.99")Those assertions stay green forever if the implementation does not change.
What agents actually do
An agent typically:
- Interprets a user goal in natural language.
- Chooses tools and arguments under uncertainty.
- Observes tool results and updates a plan.
- Stops when a stopping condition fires or a budget is exhausted.
- Emits a final answer whose wording is not unique.
Two healthy runs can call tools in different orders, phrase answers differently, and still complete the task. Two broken runs can look fluent while skipping a required tool or inventing facts.
Layers of "correct"
| Layer | Example pass condition | Test style |
|---|---|---|
| Tool / pure functions | Schema, math, DB row shape | Deterministic unit tests |
| Policy and host code | Caps, allowlists, timeouts | Deterministic unit/integration tests |
| Single step behavior | Valid tool call shape | Contract tests + light mocks |
| Task success | Ticket created with right fields | Scenario sims + gold checks |
| Answer quality | Helpful, faithful, on-policy | Rubrics / LLM-as-judge |
| Safety | No PII leak, no forbidden tools | Red-team suites + hard guards |
If you only unit-test tools, you never catch planner failure. If you only snapshot full transcripts, every model bump breaks CI for the wrong reason.
Mechanics & Interactions
Why exact string equality fails
Models draw from a distribution. Temperature, system prompts, provider routing, and even "identical" model versions can change tokens.
Exact equality on free-form text:
- Flakes under tiny sampling noise.
- Misses semantic regressions that still match a lucky template.
- Blocks useful rewrites that improve clarity without changing meaning.
Prefer invariants over full string equality:
- Structured fields that must be present.
- Required tool names in the trajectory.
- Numeric tolerances or set equality on extracted entities.
- Rubric scores above a threshold averaged over N runs.
Where determinism still lives
Much of an agent stack is ordinary software.
Test it like ordinary software:
- Tool implementations and argument validation.
- Idempotency stores and approval gates.
- Context assembly and truncation rules.
- Stop conditions, retries, and circuit breakers.
- Parsers for tool JSON and final structured output.
These tests should be fast, free of live model calls, and mandatory in every PR.
The agent loop as a test surface
A full loop couples three moving parts: model, tools, and orchestrator.
| Strategy | What you freeze | What you measure |
|---|---|---|
| Unit-test tools | Model out of the picture | Correct side effects for given args |
| Scripted model / recorded responses | Tool world real or mocked | Orchestrator chooses right next step |
| Simulated scenarios | Partial mocks for external systems | End-to-end task success |
| Live model evals | Tools often real or sandboxed | Quality distribution on a gold set |
Production confidence needs more than one row of that table.
Metrics that replace assert equal
- Pass@k / pass rate: fraction of runs that satisfy hard success criteria.
- Trajectory score: did required tools run (and forbidden ones not)?
- Faithfulness: final answer consistent with tool observations.
- Latency and cost per successful task: regressions that quality scores miss.
- Judge scores: rubric dimensions (correctness, completeness, tone, safety).
Report scores with sample size and version pins (prompt hash, model slug, tool schema version).
# Sketch: hard success is deterministic; wording is not compared
def task_passed(run) -> bool:
return (
run.final_status == "completed"
and "create_ticket" in run.tools_called
and run.ticket["priority"] == "P1"
and run.cost_usd < 0.50
)Advanced Considerations & Applications
Flakiness budgets
Non-determinism does not excuse chaos.
Control variance with:
- Fixed seeds when the provider supports them (still not a full guarantee across vendors).
- Multiple trials per case and a minimum pass rate (for example 4/5) instead of one-shot pass/fail.
- Separate smoke (cheap, PR) and nightly (full gold set, multi-model) lanes.
- Cached or recorded model responses for pure orchestrator tests.
Cost of truth
Live evals spend money. Budget them like load tests: small on every PR, larger on main and release candidates.
Interaction with CI/CD
A model or prompt change is a behavior change, not a docs-only tweak. Treat eval gates the way you treat integration tests for a payment service: block deploys when pass rate or safety scores drop below agreed thresholds (see CI lifecycle docs in this category).
Trade-off table
| Approach | Strength | Weakness |
|---|---|---|
| Exact text snapshot | Simple | Flaky / brittle |
| Structured field asserts | Stable, cheap | Misses open-ended quality |
| Gold set + hard checks | Grounded task success | Needs curation |
| LLM-as-judge | Scales soft criteria | Judge bias and cost |
| Human review sample | Highest trust | Slow, not PR-speed |
Healthy pipelines combine hard checks for must-never-fail cases with judges for style and partial credit.
Common Misconceptions
- "Agents are untestable." Tools and host policy are fully testable; the model layer is evaluable with rates and rubrics.
- "Temperature zero makes it deterministic." It reduces variance but does not freeze providers, tools, or multi-step branching.
- "One golden transcript is enough." A single path overfits yesterday's plan and misses alternate valid trajectories.
- "If the demo works, ship it." Demos are n=1 marketing; production needs distributional evidence.
- "Unit tests replace evals." They complement evals. They do not score planner quality.
- "Judges remove the need for humans." Judges need calibration against human labels on a seed set.
FAQs
Can I still use pytest for agents?
Yes. Use pytest (or any unit framework) for tools, parsers, and host policy. Use the same runner to orchestrate eval jobs that assert on pass rates and thresholds, not only on raw strings.
What should fail a PR immediately?
Deterministic regressions (tool bugs, schema breaks, policy bypasses) and hard safety failures. Soft quality dips may warn on PR and block only on larger suites.
How many trials per case do I need?
Start with 1 for pure unit tests, 3-5 for flaky model paths on critical cases, and more for release gates. Raise N until the pass-rate confidence interval is useful for your risk level.
Should I mock the model always?
Mock for orchestrator and tool-wiring tests. Use live or recorded-but-refreshed model calls for quality evals. Mocking the model forever hides real prompt regressions.
How do I test open-ended answers without a judge?
Extract structure (JSON schema, required citations, entity sets), check tool trajectories, and reserve free-form prose for rubric scoring when structure is not enough.
Are snapshot tests ever useful?
Yes for stable structured artifacts (tool schemas, prompt templates, fixture JSON). Avoid snapshotting free-form model prose as a hard gate.
What is the difference between an eval and a test?
A test usually expects a fixed result. An eval measures quality against criteria, often aggregating many cases and runs into scores and thresholds.
How do tool flakiness and model flakiness interact?
External tools add their own nondeterminism. Sandbox or script external systems in CI so failures point at the agent, not at a flaky third-party API.
Do multi-agent systems change the picture?
They add handoff and privilege surfaces. You still unit-test tools, but scenarios must cover wrong handoffs and shared-state races.
When is exact match acceptable on model output?
When the model is constrained to a tiny closed set (enums, yes/no, fixed JSON schema with strict decoding) and you still allow for rare provider failures with retries.
How should product managers read eval reports?
As task success rates, cost per success, and safety incident counts - not as "the prompt looks better."
What is the first step if our suite is only manual demos?
Extract pure tools into unit tests, write ten gold tasks with hard success checks, and run them on every model/prompt change before adding fancy judges.
Related
Related: Testing AI Agents Basics
Related: Unit Testing Individual Agent Tools
Related: Simulating Full Agent Loops in a Test Harness
Related: Building a Gold-Set Eval Suite for Agent Quality
Related: LLM-as-Judge: Scoring Agent Output Automatically
Related: Regression Testing an Agent After a Model or Prompt Change
Related: Testing AI Agents Best Practices
Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana
@solana/web3.js+@solana/spl-token.