Regression Testing an Agent After a Model or Prompt Change

A model bump or prompt edit is a behavior change with production blast radius. Use this cheatsheet to catch quality, cost, safety, and wiring regressions before users do.

How to Use This List

Run top to bottom for every prompt pack or model slug change.
Tick items only with evidence (CI logs, eval reports, dashboards).
Pair with your gold-set suite and tool unit tests; this page is the release gate workflow, not a substitute for those assets.
Re-run after hotfixes that "only changed wording."

Quick Map

Change type	Highest risk	Minimum gate
System prompt tweak	Tone, tool over/under-use	Smoke gold + P0 pass rate
Tool description edit	Wrong args, missed tools	Tool unit + trajectory scenarios
Model slug swap	Quality and cost cliffs	Full gold + cost slice
Temperature / decoding	Variance, schema breaks	Multi-trial pass rates
Provider routing change	Subtle quality drift	Shadow or canary + eval sample
New tool added	Permission and loop bugs	Unit + forbid/allow scenarios

A - Freeze What You Are Comparing

1. Pin versions. Record previous and candidate model slug, prompt hash/commit, tool schema version, and gold pack id.
2. Name the hypothesis. Write one line: what should improve and what must not regress.
3. Snapshot baseline scores. Export last green overall, P0, per-intent, cost/success, and latency numbers.
4. Isolate the change. Do not combine model, prompt, and tool refactors in one experiment unless you accept messy bisects.

B - Deterministic Layers First

5. Run tool unit tests. All write tools, validators, and idempotency cases green.
6. Run scripted loop scenarios. Required/forbidden tools, max-turn stops, approval deny paths.
7. Verify structured output parsers. JSON schema / enum paths still parse at temperature used in prod.
8. Confirm policy guards in host code. Caps, allowlists, and kill switches are not prompt-only.

C - Gold-Set and Quality Gates

9. Execute smoke gold on the candidate. Fail fast if P0 cases drop.
10. Execute full gold with N trials. Use the same N as baseline (for example 3) for fair pass rates.
11. Diff failures by case id. Classify: real regression, flake, or outdated gold label.
12. Run soft metrics if calibrated. LLM-as-judge dimensions only where agreement is trusted.
13. Check cost and latency slices. Cost per successful task and p95 turns/latency vs baseline budget.

D - Safety and Abuse

14. Replay injection and policy cases. Prompt injection, secrets in context, disallowed tool coercion.
15. Confirm refuse/escalate paths. Refund blocks, auth failures, and human handoff still fire.
16. Scan outputs for secret leakage patterns. Tokens, raw credentials, private URLs.

E - Ship Path

17. Record a go/no-go table. Overall, P0, safety, cost, open flakes.
18. Choose rollout mode. Full, canary percent, or shadow compare against current prod.
19. Prepare rollback. Prior prompt pack and model slug one click or one config flip away.
20. Schedule post-deploy watch. Error rate, override rate, cost burn, and top new traces for 24-72h.

Threshold Cheatsheet (example defaults)

Metric	Warn	Block deploy
P0 pass rate delta	-2%	-5% or absolute < 0.95
Overall pass rate delta	-3%	-7%
Forbidden tool hits on policy cases	any	any
Cost per success	+15%	+30% without approval
Judge safety dimension	-0.2 mean	any drop below rubric floor
New parse failures	> 0.5%	> 2%

Tune numbers to your risk; write them down before the experiment so you do not move goalposts.

Failure Triage Cheatsheet

Symptom	Likely cause	Next step
Many trajectory misses	Tool descriptions / model tool skill	Diff tool schemas; add examples
Hard checks pass, users hate tone	Prompt style regression	Judge tone + human sample
Flaky pass rates	N too low / hot sampling	Raise trials; lower temperature on extract steps
Cost spike	Extra turns or bigger model	Cap turns; route easy intents cheaper
Only one intent breaks	Overfitting or policy mismatch	Inspect that slice's gold labels
All green in CI, bad in prod	Fixture unreality	Add redacted prod traces to gold

Minimal Report Template

Change: prompt X -> Y / model A -> B
Baseline pack: gold@DATE  Candidate commit: SHA
P0 pass: 0.98 -> 0.96 (N=3)
Overall: 0.91 -> 0.90
Cost/success: $0.04 -> $0.05
Safety forbidden tools: 0 -> 0
Decision: ship canary 10% / block
Rollback: config flag agent.prompt=X model=A

FAQs

Do tiny prompt typo fixes need the full suite?

Run deterministic tests plus smoke P0 at minimum. Full gold is still wise if the typo sits in tool instructions or safety text.

What if the new model is better overall but worse on one P0?

Block or fix before full ship. P0 is a product promise, not an average.

How do I compare two candidates fairly?

Same gold pack, same N, same tools/fixtures, same decoding settings unless decoding is the experiment.

Should regression tests call production APIs?

Prefer sandboxes and fakes. If you must use staging, isolate credentials and rate limits.

How long should a regression run take?

PR smoke: minutes. Release full gold: whatever your pack needs, often under an hour with concurrency. Overnight is fine for huge matrices.

Can I auto-merge when evals pass?

Only if thresholds are stable and safety cases are strong. Many teams still require a human ack on model swaps.

What belongs in the PR description?

Hypothesis, version pins, score table, known flakes, and rollback pointer.

How do prompt templates in code differ from hosted prompt stores?

Same process: version, diff, eval, roll forward/back. Hosted stores still need the same gates.

What if gold labels are wrong after a product policy change?

Update gold in a dedicated PR with product sign-off, then rebaseline. Do not hide a model regression by silently editing labels in the same change.

Is shadow traffic a substitute for gold sets?

No. Shadow helps validate production mix; gold sets give labeled truth and fast iteration.

Related: Why Testing an Agent Differs from Testing Deterministic Code

Related: Building a Gold-Set Eval Suite for Agent Quality

Related: LLM-as-Judge: Scoring Agent Output Automatically

Related: Simulating Full Agent Loops in a Test Harness

Related: Unit Testing Individual Agent Tools

Related: Testing AI Agents Best Practices

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Regression Testing an Agent After a Model or Prompt Change

A model bump or prompt edit is a behavior change with production blast radius. Use this cheatsheet to catch quality, cost, safety, and wiring regressions before users do.

How to Use This List

Run top to bottom for every prompt pack or model slug change.
Tick items only with evidence (CI logs, eval reports, dashboards).
Pair with your gold-set suite and tool unit tests; this page is the release gate workflow, not a substitute for those assets.
Re-run after hotfixes that "only changed wording."

Quick Map

Change type	Highest risk	Minimum gate
System prompt tweak	Tone, tool over/under-use	Smoke gold + P0 pass rate
Tool description edit	Wrong args, missed tools	Tool unit + trajectory scenarios
Model slug swap	Quality and cost cliffs	Full gold + cost slice
Temperature / decoding	Variance, schema breaks	Multi-trial pass rates
Provider routing change	Subtle quality drift	Shadow or canary + eval sample
New tool added	Permission and loop bugs	Unit + forbid/allow scenarios

A - Freeze What You Are Comparing

1. Pin versions. Record previous and candidate model slug, prompt hash/commit, tool schema version, and gold pack id.
2. Name the hypothesis. Write one line: what should improve and what must not regress.
3. Snapshot baseline scores. Export last green overall, P0, per-intent, cost/success, and latency numbers.
4. Isolate the change. Do not combine model, prompt, and tool refactors in one experiment unless you accept messy bisects.

B - Deterministic Layers First

5. Run tool unit tests. All write tools, validators, and idempotency cases green.
6. Run scripted loop scenarios. Required/forbidden tools, max-turn stops, approval deny paths.
7. Verify structured output parsers. JSON schema / enum paths still parse at temperature used in prod.
8. Confirm policy guards in host code. Caps, allowlists, and kill switches are not prompt-only.

C - Gold-Set and Quality Gates

9. Execute smoke gold on the candidate. Fail fast if P0 cases drop.
10. Execute full gold with N trials. Use the same N as baseline (for example 3) for fair pass rates.
11. Diff failures by case id. Classify: real regression, flake, or outdated gold label.
12. Run soft metrics if calibrated. LLM-as-judge dimensions only where agreement is trusted.
13. Check cost and latency slices. Cost per successful task and p95 turns/latency vs baseline budget.

D - Safety and Abuse

14. Replay injection and policy cases. Prompt injection, secrets in context, disallowed tool coercion.
15. Confirm refuse/escalate paths. Refund blocks, auth failures, and human handoff still fire.
16. Scan outputs for secret leakage patterns. Tokens, raw credentials, private URLs.

E - Ship Path

17. Record a go/no-go table. Overall, P0, safety, cost, open flakes.
18. Choose rollout mode. Full, canary percent, or shadow compare against current prod.
19. Prepare rollback. Prior prompt pack and model slug one click or one config flip away.
20. Schedule post-deploy watch. Error rate, override rate, cost burn, and top new traces for 24-72h.

Threshold Cheatsheet (example defaults)

Metric	Warn	Block deploy
P0 pass rate delta	-2%	-5% or absolute < 0.95
Overall pass rate delta	-3%	-7%
Forbidden tool hits on policy cases	any	any
Cost per success	+15%	+30% without approval
Judge safety dimension	-0.2 mean	any drop below rubric floor
New parse failures	> 0.5%	> 2%

Tune numbers to your risk; write them down before the experiment so you do not move goalposts.

Failure Triage Cheatsheet

Symptom	Likely cause	Next step
Many trajectory misses	Tool descriptions / model tool skill	Diff tool schemas; add examples
Hard checks pass, users hate tone	Prompt style regression	Judge tone + human sample
Flaky pass rates	N too low / hot sampling	Raise trials; lower temperature on extract steps
Cost spike	Extra turns or bigger model	Cap turns; route easy intents cheaper
Only one intent breaks	Overfitting or policy mismatch	Inspect that slice's gold labels
All green in CI, bad in prod	Fixture unreality	Add redacted prod traces to gold

Minimal Report Template

Change: prompt X -> Y / model A -> B
Baseline pack: gold@DATE  Candidate commit: SHA
P0 pass: 0.98 -> 0.96 (N=3)
Overall: 0.91 -> 0.90
Cost/success: $0.04 -> $0.05
Safety forbidden tools: 0 -> 0
Decision: ship canary 10% / block
Rollback: config flag agent.prompt=X model=A

FAQs

Do tiny prompt typo fixes need the full suite?

Run deterministic tests plus smoke P0 at minimum. Full gold is still wise if the typo sits in tool instructions or safety text.

What if the new model is better overall but worse on one P0?

Block or fix before full ship. P0 is a product promise, not an average.

How do I compare two candidates fairly?

Same gold pack, same N, same tools/fixtures, same decoding settings unless decoding is the experiment.

Should regression tests call production APIs?

Prefer sandboxes and fakes. If you must use staging, isolate credentials and rate limits.

How long should a regression run take?

PR smoke: minutes. Release full gold: whatever your pack needs, often under an hour with concurrency. Overnight is fine for huge matrices.

Can I auto-merge when evals pass?

Only if thresholds are stable and safety cases are strong. Many teams still require a human ack on model swaps.

What belongs in the PR description?

Hypothesis, version pins, score table, known flakes, and rollback pointer.

How do prompt templates in code differ from hosted prompt stores?

Same process: version, diff, eval, roll forward/back. Hosted stores still need the same gates.

What if gold labels are wrong after a product policy change?

Update gold in a dedicated PR with product sign-off, then rebaseline. Do not hide a model regression by silently editing labels in the same change.

Is shadow traffic a substitute for gold sets?

No. Shadow helps validate production mix; gold sets give labeled truth and fast iteration.

Related: Why Testing an Agent Differs from Testing Deterministic Code

Related: Building a Gold-Set Eval Suite for Agent Quality

Related: LLM-as-Judge: Scoring Agent Output Automatically

Related: Simulating Full Agent Loops in a Test Harness

Related: Unit Testing Individual Agent Tools

Related: Testing AI Agents Best Practices

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Regression Testing an Agent After a Model or Prompt Change

Search Documentation

Regression Testing an Agent After a Model or Prompt Change