Testing AI Agents Best Practices
Ten practices for building a testing and eval pipeline that catches agent regressions early without drowning the team in flaky CI.
Use this list when you stand up agent quality gates or when demos are shipping without evidence.
How to Use This Checklist
- Work A → D: foundations, suite design, automation, then operations.
- Tick items only when they exist in code, CI, or runbooks - not as slideware.
- Re-check after new tools, model swaps, or major prompt pack edits.
- Pair unchecked items with an owner and a date.
A - Foundations
- 1. Separate deterministic tests from probabilistic evals. Unit-test tools, parsers, and host policy with exact asserts; score model behavior with pass rates and rubrics.
- 2. Make every tool independently testable. Inject dependencies, return structured errors, and cover validation, timeouts, and idempotent writes without a live model.
- 3. Version prompts, models, tools, and gold packs together. Store hashes/slugs in eval reports so a score always maps to a known bundle.
B - Suite Design
- 4. Curate a gold set with hard, machine-checkable outcomes. Prefer trajectories, structured fields, and world state over free-form string equality.
- 5. Slice metrics and protect P0 paths. Track overall and per-intent scores; never let averages hide must-never-fail collapses.
- 6. Budget turns, wall time, and cost per successful task. Quality that triples spend is still a regression.
- 7. Add adversarial and policy cases. Injection, forbidden tools, refusal paths, and empty-tool results belong in the pack.
C - Automation and Judges
- 8. Layer CI: fast PR smoke, fuller nightly, strict release gates. Scripted simulations on every PR; live multi-trial gold on main/release.
- 9. Use LLM-as-judge only after calibration. Structured rubrics, pinned judge models, human agreement checks; hard safety stays rule-based.
D - Operations
- 10. Close the loop from incidents to gold. Every serious production miss becomes a permanent case, and model/prompt changes follow a written regression checklist with rollback.
Applying the Habits in Order
| Stage | Habits | Exit criterion |
|---|---|---|
| Bootstrap | 1-3 | Tools unit-tested; versions pinned in reports |
| Measure | 4-7 | Gold pack + slices + budgets + adversarial cases |
| Automate | 8-9 | CI lanes live; judges calibrated where used |
| Operate | 10 | Incident→gold habit and regression checklist in use |
Quick Anti-Patterns
- Exact-match snapshots on free-form model prose as the only gate.
- Demo notebooks instead of versioned gold data.
- Live production APIs inside default unit tests.
- One shared unlimited API key for CI evals.
- Shipping model swaps because "it felt better" without score tables.
- Judge-only safety without host allowlists and caps.
- Ignoring cost/latency while chasing a 1% quality bump.
- Editing gold labels in the same PR that hides a regression.
Minimal Pipeline Sketch
PR:
tool unit tests
scripted loop scenarios (smoke)
optional tiny gold smoke
Nightly / release candidate:
full gold set x N trials
cost + latency report
calibrated judge dimensions (optional)
safety / injection pack
On fail:
diff by case id -> fix agent OR fix fixtures with review
never raise thresholds silently to go greenFAQs
Which three habits are non-negotiable?
Deterministic tool tests (1-2), a gold set with hard checks (4), and version-pinned regression runs before model/prompt ship (3, 10).
How large should the first gold set be?
Start with tens of high-value cases including P0 and policy edges. Expand from incidents rather than inventing hundreds of low-signal chats.
How do we reduce flake without lying to ourselves?
Raise trials on live lanes, use pass-rate thresholds, stub flaky externals, and keep a pure deterministic lane that must always be green.
When should we add LLM-as-judge?
After hard checks exist and you have a small human-labeled calibration set. Judges are for soft quality, not for inventing ground truth.
Do multi-agent systems need a different pipeline?
Same layers, plus handoff and privilege scenarios. Each agent's tools still need unit tests.
How often should packs be reviewed?
After every incident, on major product policy changes, and on a calendar review (for example monthly) for duplicates and dead cases.
Where should thresholds live?
In versioned config next to the suite, not in a chat thread. Changes to thresholds need the same review as code.
Can open-source eval frameworks replace this list?
They can accelerate harnesses and dashboards. You still own cases, thresholds, and the determinism split.
How do we keep CI affordable?
Smoke on PR, cache judgments, stub tools, sample live models, and reserve large N for nightly/release.
What proves the pipeline works?
Regressions caught pre-prod, short time-to-detect on the few that escape, and a growing gold set tied to real failures rather than vanity metrics.
Should product managers see eval reports?
Yes in business terms: task success, P0 health, cost per success, and open safety issues - not raw token dumps.
How does this relate to observability?
Evals guard known intents before ship. Observability finds novel failures that should become the next gold cases.
Related
Related: Why Testing an Agent Differs from Testing Deterministic Code
Related: Testing AI Agents Basics
Related: Unit Testing Individual Agent Tools
Related: Simulating Full Agent Loops in a Test Harness
Related: Building a Gold-Set Eval Suite for Agent Quality
Related: LLM-as-Judge: Scoring Agent Output Automatically
Related: Regression Testing an Agent After a Model or Prompt Change
Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana
@solana/web3.js+@solana/spl-token.