Case Studies Best Practices

Q: Where should case studies live?

In-repo docs/case-studies/ or an internal docs space with PR review. Chat threads are not an archive.

Ten practices for documenting and learning from your own agent case studies so the next build starts from evidence, not folklore.

Use this list when a pilot ends, after a costly incident, or when someone asks "can we do what Team X did?"

How to Use This Checklist

Work A → D in order; early habits make later ones cheap.
Check items that are true for the case study you are writing or reviewing.
If A fails, pause publication - you do not yet have a case study, only a demo memory.
Attach the filled list to the case study doc or PR.
Revisit when metrics windows roll or architecture changes.

A - Capture the Frame Early

1. Write problem, done_when, and non-goals before celebrating the demo. Retrofitting a frame invites revisionist success.
2. Name success metrics that include quality, cost, and safety/noise - not only vibes. Three honest metrics beat twelve vanity ones.
3. Snapshot architecture as responsibilities (host, tools, memory, gates, sinks), not logo stacks alone. Logos rot; roles stay teachable.

B - Keep Evidence Beside Narrative

4. Link golden sets, redacted traces, and metric tables from the case study. Unlinked claims expire into marketing.
5. Record rejected alternatives and constraints in a short trade-off log. Future teams need the why not, not only the winner.
6. Date the metric window and traffic mix. A friendly pilot is not a launch certificate.

C - Document Failure and Bounds

7. Include bounds (max turns, budgets, timeouts) and escalation paths that actually fired. Hidden safety is unrepeatable safety.
8. File incidents and near-misses as first-class sections, not footnotes. Negative results prevent expensive clones.

D - Make Learning Operational

9. Schedule a revisit when models, prices, tools, or goals change - append, do not silent-edit history. Living systems need living docs.
10. Require a case study (or explicit waiver) before another team copies the pattern into production. Pattern reuse without preconditions is how outages spread.

Applying the Habits in Order

Stage	Habits	Exit criterion
Kickoff	1-3	Frame + metrics + architecture sketch exist
Pilot end	4-6	Evidence links + trade-offs + dated numbers
Hardening	7-8	Bounds and incidents written
Org reuse	9-10	Revisit plan + copy gate

Quick "Do Not Publish Yet" List

Only happy-path screenshots.
No cost or success denominator.
Framework tutorial with no problem frame.
Metrics without time range.
"It felt faster" with no latency table.
Secrets or customer PII in pasted traces.

Case Study Skeleton (Copy)

# <Name> - case study (YYYY-MM-DD)
## Problem & done_when
## Architecture (roles)
## Tools & permissions
## Bounds & human gates
## Eval & metrics (window)
## Outcomes (before/after if any)
## Trade-offs rejected
## Incidents / near-misses
## Open questions & revisit date
## Links (ADR, eval, dashboards)

FAQs

How long should a case study take to write?

Often 60-90 minutes if you captured metrics during the pilot. If it takes days, your instrumentation was missing - fix that next time.

Who owns the document?

The tech lead of the pilot owns accuracy; a staff+ engineer may own the org template. Ownership beats wiki orphans.

Should marketing rewrite engineering case studies?

They may produce external versions with legal review. Keep an internal engineering source of truth with full trade-offs and incidents.

What if the pilot failed?

Still write it. Failed pilots with clear causes are high-leverage teaching artifacts.

Do we need a case study for every experiment?

No. Require them for multi-week pilots, production exposure, or patterns others will copy. Spike notes can be ten lines.

How do case studies relate to ADRs?

ADRs freeze single decisions. Case studies narrate the journey and outcomes. Link them; do not merge them into one muddy doc.

Can we use vendor case studies instead?

As inspiration only. Their traffic, risk, and ops maturity are not yours. Write the internal version before copying architecture.

What is the minimum evidence set?

Golden-set definition and scores, cost per successful task (or honest proxy), and one redacted trace showing a normal run and a failure run.

How often should we revisit?

At least when model routing, major tools, or success criteria change - and on a calendar cadence (for example quarterly) for production agents.

Where should case studies live?

In-repo docs/case-studies/ or an internal docs space with PR review. Chat threads are not an archive.

Should multi-agent systems use a different template?

Add handoff contracts, per-agent allowlists, and orchestrator budgets to the same skeleton. Do not invent a parallel culture.

What is the most common documentation failure?

Shipping a polished architecture diagram with no metrics, no rejects, and no bounds - then watching another team clone the diagram into an outage.

What a Case Study Should Capture for Future Reference - capture checklist depth
Case Studies Basics - first end-to-end mini write-up
Case Study: Building a Customer-Support Agent with RAG and Escalation - worked example
Case Study: A Code-Review Agent Wired into GitHub CI - CI-shaped example
Before-and-After Case Study: Cutting an Agent's Cost by Switching to OpenRouter Routing - metrics-heavy example
Why Agent Systems Need Their Own ADR Discipline - decision records companion

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Case Studies Best Practices

Ten practices for documenting and learning from your own agent case studies so the next build starts from evidence, not folklore.

Use this list when a pilot ends, after a costly incident, or when someone asks "can we do what Team X did?"

How to Use This Checklist

Work A → D in order; early habits make later ones cheap.
Check items that are true for the case study you are writing or reviewing.
If A fails, pause publication - you do not yet have a case study, only a demo memory.
Attach the filled list to the case study doc or PR.
Revisit when metrics windows roll or architecture changes.

A - Capture the Frame Early

1. Write problem, done_when, and non-goals before celebrating the demo. Retrofitting a frame invites revisionist success.
2. Name success metrics that include quality, cost, and safety/noise - not only vibes. Three honest metrics beat twelve vanity ones.
3. Snapshot architecture as responsibilities (host, tools, memory, gates, sinks), not logo stacks alone. Logos rot; roles stay teachable.

B - Keep Evidence Beside Narrative

4. Link golden sets, redacted traces, and metric tables from the case study. Unlinked claims expire into marketing.
5. Record rejected alternatives and constraints in a short trade-off log. Future teams need the why not, not only the winner.
6. Date the metric window and traffic mix. A friendly pilot is not a launch certificate.

C - Document Failure and Bounds

7. Include bounds (max turns, budgets, timeouts) and escalation paths that actually fired. Hidden safety is unrepeatable safety.
8. File incidents and near-misses as first-class sections, not footnotes. Negative results prevent expensive clones.

D - Make Learning Operational

9. Schedule a revisit when models, prices, tools, or goals change - append, do not silent-edit history. Living systems need living docs.
10. Require a case study (or explicit waiver) before another team copies the pattern into production. Pattern reuse without preconditions is how outages spread.

Applying the Habits in Order

Stage	Habits	Exit criterion
Kickoff	1-3	Frame + metrics + architecture sketch exist
Pilot end	4-6	Evidence links + trade-offs + dated numbers
Hardening	7-8	Bounds and incidents written
Org reuse	9-10	Revisit plan + copy gate

Quick "Do Not Publish Yet" List

Only happy-path screenshots.
No cost or success denominator.
Framework tutorial with no problem frame.
Metrics without time range.
"It felt faster" with no latency table.
Secrets or customer PII in pasted traces.

Case Study Skeleton (Copy)

# <Name> - case study (YYYY-MM-DD)
## Problem & done_when
## Architecture (roles)
## Tools & permissions
## Bounds & human gates
## Eval & metrics (window)
## Outcomes (before/after if any)
## Trade-offs rejected
## Incidents / near-misses
## Open questions & revisit date
## Links (ADR, eval, dashboards)

FAQs

How long should a case study take to write?

Often 60-90 minutes if you captured metrics during the pilot. If it takes days, your instrumentation was missing - fix that next time.

Who owns the document?

The tech lead of the pilot owns accuracy; a staff+ engineer may own the org template. Ownership beats wiki orphans.

Should marketing rewrite engineering case studies?

They may produce external versions with legal review. Keep an internal engineering source of truth with full trade-offs and incidents.

What if the pilot failed?

Still write it. Failed pilots with clear causes are high-leverage teaching artifacts.

Do we need a case study for every experiment?

No. Require them for multi-week pilots, production exposure, or patterns others will copy. Spike notes can be ten lines.

How do case studies relate to ADRs?

ADRs freeze single decisions. Case studies narrate the journey and outcomes. Link them; do not merge them into one muddy doc.

Can we use vendor case studies instead?

As inspiration only. Their traffic, risk, and ops maturity are not yours. Write the internal version before copying architecture.

What is the minimum evidence set?

Golden-set definition and scores, cost per successful task (or honest proxy), and one redacted trace showing a normal run and a failure run.

How often should we revisit?

At least when model routing, major tools, or success criteria change - and on a calendar cadence (for example quarterly) for production agents.

Where should case studies live?

In-repo docs/case-studies/ or an internal docs space with PR review. Chat threads are not an archive.

Should multi-agent systems use a different template?

Add handoff contracts, per-agent allowlists, and orchestrator budgets to the same skeleton. Do not invent a parallel culture.

What is the most common documentation failure?

Shipping a polished architecture diagram with no metrics, no rejects, and no bounds - then watching another team clone the diagram into an outage.

What a Case Study Should Capture for Future Reference - capture checklist depth
Case Studies Basics - first end-to-end mini write-up
Case Study: Building a Customer-Support Agent with RAG and Escalation - worked example
Case Study: A Code-Review Agent Wired into GitHub CI - CI-shaped example
Before-and-After Case Study: Cutting an Agent's Cost by Switching to OpenRouter Routing - metrics-heavy example
Why Agent Systems Need Their Own ADR Discipline - decision records companion

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Case Studies Best Practices

Search Documentation

Case Studies Best Practices