Regression Testing an Agent After a Model or Prompt Change
A model bump or prompt edit is a behavior change with production blast radius. Use this cheatsheet to catch quality, cost, safety, and wiring regressions before users do.
How to Use This List
- Run top to bottom for every prompt pack or model slug change.
- Tick items only with evidence (CI logs, eval reports, dashboards).
- Pair with your gold-set suite and tool unit tests; this page is the release gate workflow, not a substitute for those assets.
- Re-run after hotfixes that "only changed wording."
Quick Map
| Change type | Highest risk | Minimum gate |
|---|---|---|
| System prompt tweak | Tone, tool over/under-use | Smoke gold + P0 pass rate |
| Tool description edit | Wrong args, missed tools | Tool unit + trajectory scenarios |
| Model slug swap | Quality and cost cliffs | Full gold + cost slice |
| Temperature / decoding | Variance, schema breaks | Multi-trial pass rates |
| Provider routing change | Subtle quality drift | Shadow or canary + eval sample |
| New tool added | Permission and loop bugs | Unit + forbid/allow scenarios |
A - Freeze What You Are Comparing
- 1. Pin versions. Record previous and candidate
modelslug, prompt hash/commit, tool schema version, and gold pack id. - 2. Name the hypothesis. Write one line: what should improve and what must not regress.
- 3. Snapshot baseline scores. Export last green overall, P0, per-intent, cost/success, and latency numbers.
- 4. Isolate the change. Do not combine model, prompt, and tool refactors in one experiment unless you accept messy bisects.
B - Deterministic Layers First
- 5. Run tool unit tests. All write tools, validators, and idempotency cases green.
- 6. Run scripted loop scenarios. Required/forbidden tools, max-turn stops, approval deny paths.
- 7. Verify structured output parsers. JSON schema / enum paths still parse at temperature used in prod.
- 8. Confirm policy guards in host code. Caps, allowlists, and kill switches are not prompt-only.
C - Gold-Set and Quality Gates
- 9. Execute smoke gold on the candidate. Fail fast if P0 cases drop.
- 10. Execute full gold with N trials. Use the same N as baseline (for example 3) for fair pass rates.
- 11. Diff failures by case id. Classify: real regression, flake, or outdated gold label.
- 12. Run soft metrics if calibrated. LLM-as-judge dimensions only where agreement is trusted.
- 13. Check cost and latency slices. Cost per successful task and p95 turns/latency vs baseline budget.
D - Safety and Abuse
- 14. Replay injection and policy cases. Prompt injection, secrets in context, disallowed tool coercion.
- 15. Confirm refuse/escalate paths. Refund blocks, auth failures, and human handoff still fire.
- 16. Scan outputs for secret leakage patterns. Tokens, raw credentials, private URLs.
E - Ship Path
- 17. Record a go/no-go table. Overall, P0, safety, cost, open flakes.
- 18. Choose rollout mode. Full, canary percent, or shadow compare against current prod.
- 19. Prepare rollback. Prior prompt pack and model slug one click or one config flip away.
- 20. Schedule post-deploy watch. Error rate, override rate, cost burn, and top new traces for 24-72h.
Threshold Cheatsheet (example defaults)
| Metric | Warn | Block deploy |
|---|---|---|
| P0 pass rate delta | -2% | -5% or absolute < 0.95 |
| Overall pass rate delta | -3% | -7% |
| Forbidden tool hits on policy cases | any | any |
| Cost per success | +15% | +30% without approval |
| Judge safety dimension | -0.2 mean | any drop below rubric floor |
| New parse failures | > 0.5% | > 2% |
Tune numbers to your risk; write them down before the experiment so you do not move goalposts.
Failure Triage Cheatsheet
| Symptom | Likely cause | Next step |
|---|---|---|
| Many trajectory misses | Tool descriptions / model tool skill | Diff tool schemas; add examples |
| Hard checks pass, users hate tone | Prompt style regression | Judge tone + human sample |
| Flaky pass rates | N too low / hot sampling | Raise trials; lower temperature on extract steps |
| Cost spike | Extra turns or bigger model | Cap turns; route easy intents cheaper |
| Only one intent breaks | Overfitting or policy mismatch | Inspect that slice's gold labels |
| All green in CI, bad in prod | Fixture unreality | Add redacted prod traces to gold |
Minimal Report Template
Change: prompt X -> Y / model A -> B
Baseline pack: gold@DATE Candidate commit: SHA
P0 pass: 0.98 -> 0.96 (N=3)
Overall: 0.91 -> 0.90
Cost/success: $0.04 -> $0.05
Safety forbidden tools: 0 -> 0
Decision: ship canary 10% / block
Rollback: config flag agent.prompt=X model=AFAQs
Do tiny prompt typo fixes need the full suite?
Run deterministic tests plus smoke P0 at minimum. Full gold is still wise if the typo sits in tool instructions or safety text.
What if the new model is better overall but worse on one P0?
Block or fix before full ship. P0 is a product promise, not an average.
How do I compare two candidates fairly?
Same gold pack, same N, same tools/fixtures, same decoding settings unless decoding is the experiment.
Should regression tests call production APIs?
Prefer sandboxes and fakes. If you must use staging, isolate credentials and rate limits.
How long should a regression run take?
PR smoke: minutes. Release full gold: whatever your pack needs, often under an hour with concurrency. Overnight is fine for huge matrices.
Can I auto-merge when evals pass?
Only if thresholds are stable and safety cases are strong. Many teams still require a human ack on model swaps.
What belongs in the PR description?
Hypothesis, version pins, score table, known flakes, and rollback pointer.
How do prompt templates in code differ from hosted prompt stores?
Same process: version, diff, eval, roll forward/back. Hosted stores still need the same gates.
What if gold labels are wrong after a product policy change?
Update gold in a dedicated PR with product sign-off, then rebaseline. Do not hide a model regression by silently editing labels in the same change.
Is shadow traffic a substitute for gold sets?
No. Shadow helps validate production mix; gold sets give labeled truth and fast iteration.
Related
Related: Why Testing an Agent Differs from Testing Deterministic Code
Related: Building a Gold-Set Eval Suite for Agent Quality
Related: LLM-as-Judge: Scoring Agent Output Automatically
Related: Simulating Full Agent Loops in a Test Harness
Related: Unit Testing Individual Agent Tools
Related: Testing AI Agents Best Practices
Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana
@solana/web3.js+@solana/spl-token.