Building a Gold-Set Eval Suite for Agent Quality

Q: How do I handle multiple valid tool orders?

Encode partial orders or sets (must_call) instead of a single linear script when both orders are correct.

A gold set is a curated collection of tasks with known-good outcomes, labels, or graders used to score agent quality over time.

It is the backbone of regression testing when free-form behavior cannot be asserted with a single string equality check.

Summary

Collect real and synthetic tasks, attach hard success criteria (and optional rubrics), freeze them under version control, run the agent N times per case, and gate releases on pass rate and slice-level scores.

Recipe

Define product-critical intents (for example refund, lookup, escalate, research summary).
Source cases from production traces, support tickets, and synthetic edge cases (with privacy redaction).
For each case write: id, input, context fixtures, hard checks, optional rubric, tags (severity, locale, tool).
Prefer outcomes you can grade without prose equality: tool trajectories, structured fields, DB state, citation ids.
Split smoke (fast, always-on) vs full (nightly) packs.
Run the agent with pinned model/prompt/tool versions; store run artifacts.
Aggregate pass rates by slice; fail the suite when global or must-never-fail slices drop below thresholds.
Review misses weekly; add cases for every production incident.

Working Example

# evals/gold_cases.py
GOLD = [
    {
        "id": "order-status-42",
        "tags": ["lookup", "P0"],
        "input": "What is the status of order 42?",
        "fixtures": {"orders": {"42": {"status": "shipped", "eta": "2026-07-20"}}},
        "must_call": ["get_order"],
        "forbid_call": ["issue_refund"],
        "expect_contains_any": ["shipped", "2026-07-20"],
    },
    {
        "id": "refund-eligible",
        "tags": ["refund", "P0"],
        "input": "Refund order 99 please",
        "fixtures": {"orders": {"99": {"status": "delivered", "refundable": True}}},
        "must_call": ["get_order", "issue_refund"],
        "expect_refund": {"order_id": "99", "status": "ok"},
    },
    {
        "id": "refund-blocked",
        "tags": ["refund", "policy", "P0"],
        "input": "Refund order 12",
        "fixtures": {"orders": {"12": {"status": "delivered", "refundable": False}}},
        "must_call": ["get_order"],
        "forbid_call": ["issue_refund"],
        "expect_contains_any": ["not eligible", "cannot refund", "policy"],
    },
]
 
 
def hard_pass(case: dict, run: dict) -> bool:
    called = set(run.get("tools_called", []))
    if not set(case.get("must_call", [])).issubset(called):
        return False
    if called & set(case.get("forbid_call", [])):
        return False
    if "expect_refund" in case:
        if run.get("refund") != case["expect_refund"]:
            return False
    final = (run.get("final") or "").lower()
    needles = [s.lower() for s in case.get("expect_contains_any", [])]
    if needles and not any(n in final for n in needles):
        return False
    return True
 
 
def evaluate(cases, runner, trials: int = 1) -> dict:
    """runner(case) -> run dict; trials > 1 for flaky live models."""
    rows = []
    for case in cases:
        ok = 0
        for _ in range(trials):
            ok += int(hard_pass(case, runner(case)))
        rows.append(
            {
                "id": case["id"],
                "pass_rate": ok / trials,
                "tags": case.get("tags", []),
            }
        )
    overall = sum(r["pass_rate"] for r in rows) / max(len(rows), 1)
    p0 = [r for r in rows if "P0" in r["tags"]]
    p0_rate = sum(r["pass_rate"] for r in p0) / max(len(p0), 1)
    return {"overall": overall, "p0": p0_rate, "rows": rows}
 
 
def test_gold_suite_thresholds():
    def fake_runner(case):
        # Stand-in: production would call your agent entrypoint with fixtures
        if case["id"] == "order-status-42":
            return {
                "tools_called": ["get_order"],
                "final": "Order 42 is shipped; eta 2026-07-20.",
            }
        if case["id"] == "refund-eligible":
            return {
                "tools_called": ["get_order", "issue_refund"],
                "refund": {"order_id": "99", "status": "ok"},
                "final": "Refund completed.",
            }
        return {
            "tools_called": ["get_order"],
            "final": "This order is not eligible for refund under policy.",
        }
 
    report = evaluate(GOLD, fake_runner, trials=1)
    assert report["overall"] >= 1.0
    assert report["p0"] >= 1.0

Deep Dive

What belongs in a gold case

Field	Purpose
Stable `id`	Diff reports across commits
Natural language `input`	Realistic user wording
Fixtures / seed world	Hermetic tools and RAG docs
Hard checks	Machine-gradable success
Optional rubric	Soft quality dimensions
Tags	Slice metrics (locale, intent, severity)
Notes / owner	Why the case exists

Avoid cases that only a human can grade unless you also budget human review or a calibrated judge.

Sourcing strategy

Production traces (redacted): highest realism.
Incident reviews: every Sev-1 becomes a permanent case.
Policy edges: refunds, authz, self-harm, prompt injection samples.
Synthetic paraphrases: same intent, different wording, to reduce overfitting.
Adversarial: tool abuse, contradictory instructions, huge context.

Hard checks beat vibes

Prefer:

Required tool sets and argument constraints.
Exact structured outputs (JSON schema).
Database or ticket field equality.
Citation document ids that must appear.
Explicit refusal on disallowed actions.

Use substring lists sparingly; prefer structured fields when you can change the agent to emit them.

Thresholds and slices

Global accuracy can hide a collapse on refunds while lookups look fine.

Track at least:

Overall pass rate
P0 / must-never-fail pass rate
Per-intent pass rate
Cost per successful task
Average turns

Example release policy:

P0 pass rate ≥ 0.95 over N=3 trials
Overall ≥ 0.85
No new forbidden-tool hits on policy cases

Lifecycle

draft cases -> peer review labels -> freeze v1 pack
  -> run on main baseline -> store scores
  -> every prompt/model change: re-run and diff
  -> monthly prune duplicates / add incidents

Version the pack (gold_set@2026-07-15) in artifacts so score drops are interpretable.

Gotchas

Gold sets that only contain demos. They never catch long-tail failures.
Unredacted PII in fixtures. Treat eval data as production data.
Labels that disagree with current policy. Update gold when product policy changes; do not silently weaken checks.
One trial on a hot model. Variance will thrash your gates.
Teaching to the test. If engineers hardcode scenario ids, add paraphrases and holdout cases.
No owner. Orphan packs rot; assign a maintainer.
Mixing smoke and full without tags. PR CI becomes either too slow or too weak.

Alternatives

Approach	Role relative to gold sets
Unit tests for tools	Necessary floor; not task quality
Scripted simulations	Wiring and budgets; weaker on open language
LLM-as-judge only	Scales soft metrics; needs gold seed for calibration
Human eval only	Highest trust; too slow as sole gate
Production A/B metrics	Ultimate truth; too late as first signal

FAQs

How many cases do I need to start?

Twenty well-labeled, policy-aware cases beat two hundred unlabeled chats. Grow toward the hundreds as the product surface expands.

Should gold answers be full transcripts?

Usually no. Store expected outcomes and constraints. Full transcripts overfit path choices.

How do I handle multiple valid tool orders?

Encode partial orders or sets (must_call) instead of a single linear script when both orders are correct.

Can product managers edit gold cases?

Yes via reviewed data files. Require engineering review for hard checks that touch safety or money.

What is a holdout set?

A slice you do not stare at while prompt tuning, used to detect overfitting to the main pack.

How often should we rebaseline scores?

When you intentionally accept a behavior change (new policy, new UX). Otherwise compare to the last release baseline, not to an ever-moving average that hides drift.

Do gold sets replace observability?

No. Evals catch known intents. Observability finds novel failures in the wild that should become new gold cases.

How do RAG agents differ?

Fixtures must include the documents that should be retrieved, and checks should require correct citation ids, not only fluent summaries.

What about multilingual packs?

Tag locale and enforce slice thresholds so English success cannot mask other languages.

How do I store large fixtures?

Keep compact seed data in repo; large binaries in object storage with content hashes referenced by case id.

Related: Why Testing an Agent Differs from Testing Deterministic Code

Related: Testing AI Agents Basics

Related: Simulating Full Agent Loops in a Test Harness

Related: LLM-as-Judge: Scoring Agent Output Automatically

Related: Regression Testing an Agent After a Model or Prompt Change

Related: Testing AI Agents Best Practices

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

It is the backbone of regression testing when free-form behavior cannot be asserted with a single string equality check.

Summary

Recipe

Define product-critical intents (for example refund, lookup, escalate, research summary).
Source cases from production traces, support tickets, and synthetic edge cases (with privacy redaction).
For each case write: id, input, context fixtures, hard checks, optional rubric, tags (severity, locale, tool).
Prefer outcomes you can grade without prose equality: tool trajectories, structured fields, DB state, citation ids.
Split smoke (fast, always-on) vs full (nightly) packs.
Run the agent with pinned model/prompt/tool versions; store run artifacts.
Aggregate pass rates by slice; fail the suite when global or must-never-fail slices drop below thresholds.
Review misses weekly; add cases for every production incident.

Working Example

# evals/gold_cases.py
GOLD = [
    {
        "id": "order-status-42",
        "tags": ["lookup", "P0"],
        "input": "What is the status of order 42?",
        "fixtures": {"orders": {"42": {"status": "shipped", "eta": "2026-07-20"}}},
        "must_call": ["get_order"],
        "forbid_call": ["issue_refund"],
        "expect_contains_any": ["shipped", "2026-07-20"],
    },
    {
        "id": "refund-eligible",
        "tags": ["refund", "P0"],
        "input": "Refund order 99 please",
        "fixtures": {"orders": {"99": {"status": "delivered", "refundable": True}}},
        "must_call": ["get_order", "issue_refund"],
        "expect_refund": {"order_id": "99", "status": "ok"},
    },
    {
        "id": "refund-blocked",
        "tags": ["refund", "policy", "P0"],
        "input": "Refund order 12",
        "fixtures": {"orders": {"12": {"status": "delivered", "refundable": False}}},
        "must_call": ["get_order"],
        "forbid_call": ["issue_refund"],
        "expect_contains_any": ["not eligible", "cannot refund", "policy"],
    },
]
 
 
def hard_pass(case: dict, run: dict) -> bool:
    called = set(run.get("tools_called", []))
    if not set(case.get("must_call", [])).issubset(called):
        return False
    if called & set(case.get("forbid_call", [])):
        return False
    if "expect_refund" in case:
        if run.get("refund") != case["expect_refund"]:
            return False
    final = (run.get("final") or "").lower()
    needles = [s.lower() for s in case.get("expect_contains_any", [])]
    if needles and not any(n in final for n in needles):
        return False
    return True
 
 
def evaluate(cases, runner, trials: int = 1) -> dict:
    """runner(case) -> run dict; trials > 1 for flaky live models."""
    rows = []
    for case in cases:
        ok = 0
        for _ in range(trials):
            ok += int(hard_pass(case, runner(case)))
        rows.append(
            {
                "id": case["id"],
                "pass_rate": ok / trials,
                "tags": case.get("tags", []),
            }
        )
    overall = sum(r["pass_rate"] for r in rows) / max(len(rows), 1)
    p0 = [r for r in rows if "P0" in r["tags"]]
    p0_rate = sum(r["pass_rate"] for r in p0) / max(len(p0), 1)
    return {"overall": overall, "p0": p0_rate, "rows": rows}
 
 
def test_gold_suite_thresholds():
    def fake_runner(case):
        # Stand-in: production would call your agent entrypoint with fixtures
        if case["id"] == "order-status-42":
            return {
                "tools_called": ["get_order"],
                "final": "Order 42 is shipped; eta 2026-07-20.",
            }
        if case["id"] == "refund-eligible":
            return {
                "tools_called": ["get_order", "issue_refund"],
                "refund": {"order_id": "99", "status": "ok"},
                "final": "Refund completed.",
            }
        return {
            "tools_called": ["get_order"],
            "final": "This order is not eligible for refund under policy.",
        }
 
    report = evaluate(GOLD, fake_runner, trials=1)
    assert report["overall"] >= 1.0
    assert report["p0"] >= 1.0

Deep Dive

What belongs in a gold case

Field	Purpose
Stable `id`	Diff reports across commits
Natural language `input`	Realistic user wording
Fixtures / seed world	Hermetic tools and RAG docs
Hard checks	Machine-gradable success
Optional rubric	Soft quality dimensions
Tags	Slice metrics (locale, intent, severity)
Notes / owner	Why the case exists

Avoid cases that only a human can grade unless you also budget human review or a calibrated judge.

Sourcing strategy

Production traces (redacted): highest realism.
Incident reviews: every Sev-1 becomes a permanent case.
Policy edges: refunds, authz, self-harm, prompt injection samples.
Synthetic paraphrases: same intent, different wording, to reduce overfitting.
Adversarial: tool abuse, contradictory instructions, huge context.

Hard checks beat vibes

Prefer:

Required tool sets and argument constraints.
Exact structured outputs (JSON schema).
Database or ticket field equality.
Citation document ids that must appear.
Explicit refusal on disallowed actions.

Use substring lists sparingly; prefer structured fields when you can change the agent to emit them.

Thresholds and slices

Global accuracy can hide a collapse on refunds while lookups look fine.

Track at least:

Overall pass rate
P0 / must-never-fail pass rate
Per-intent pass rate
Cost per successful task
Average turns

Example release policy:

P0 pass rate ≥ 0.95 over N=3 trials
Overall ≥ 0.85
No new forbidden-tool hits on policy cases

Lifecycle

draft cases -> peer review labels -> freeze v1 pack
  -> run on main baseline -> store scores
  -> every prompt/model change: re-run and diff
  -> monthly prune duplicates / add incidents

Version the pack (gold_set@2026-07-15) in artifacts so score drops are interpretable.

Gotchas

Gold sets that only contain demos. They never catch long-tail failures.
Unredacted PII in fixtures. Treat eval data as production data.
Labels that disagree with current policy. Update gold when product policy changes; do not silently weaken checks.
One trial on a hot model. Variance will thrash your gates.
Teaching to the test. If engineers hardcode scenario ids, add paraphrases and holdout cases.
No owner. Orphan packs rot; assign a maintainer.
Mixing smoke and full without tags. PR CI becomes either too slow or too weak.

Alternatives

Approach	Role relative to gold sets
Unit tests for tools	Necessary floor; not task quality
Scripted simulations	Wiring and budgets; weaker on open language
LLM-as-judge only	Scales soft metrics; needs gold seed for calibration
Human eval only	Highest trust; too slow as sole gate
Production A/B metrics	Ultimate truth; too late as first signal

FAQs

How many cases do I need to start?

Twenty well-labeled, policy-aware cases beat two hundred unlabeled chats. Grow toward the hundreds as the product surface expands.

Should gold answers be full transcripts?

Usually no. Store expected outcomes and constraints. Full transcripts overfit path choices.

How do I handle multiple valid tool orders?

Encode partial orders or sets (must_call) instead of a single linear script when both orders are correct.

Can product managers edit gold cases?

Yes via reviewed data files. Require engineering review for hard checks that touch safety or money.

What is a holdout set?

A slice you do not stare at while prompt tuning, used to detect overfitting to the main pack.

How often should we rebaseline scores?

When you intentionally accept a behavior change (new policy, new UX). Otherwise compare to the last release baseline, not to an ever-moving average that hides drift.

Do gold sets replace observability?

No. Evals catch known intents. Observability finds novel failures in the wild that should become new gold cases.

How do RAG agents differ?

Fixtures must include the documents that should be retrieved, and checks should require correct citation ids, not only fluent summaries.

What about multilingual packs?

Tag locale and enforce slice thresholds so English success cannot mask other languages.

How do I store large fixtures?

Keep compact seed data in repo; large binaries in object storage with content hashes referenced by case id.

Related: Why Testing an Agent Differs from Testing Deterministic Code

Related: Testing AI Agents Basics

Related: Simulating Full Agent Loops in a Test Harness

Related: LLM-as-Judge: Scoring Agent Output Automatically

Related: Regression Testing an Agent After a Model or Prompt Change

Related: Testing AI Agents Best Practices

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Building a Gold-Set Eval Suite for Agent Quality

Search Documentation

Building a Gold-Set Eval Suite for Agent Quality