What a Case Study Should Capture for Future Reference

Q: Should I include full prompts?

Include versioned prompt pointers and key policy excerpts. Full prompts may be long or sensitive; store them in repo with the case study linking to the commit.

A good agent case study is not a marketing write-up.

It is a durable record of what you built, why you chose that design, what it cost, how it failed, and what you would repeat or avoid next time.

Summary

Capture architecture, tools, bounds, metrics, incidents, and rejected alternatives so future you (or another team) can reuse judgment instead of only cloning code.
Insight: Agent systems rot faster than typical apps - models, pricing, frameworks, and tool APIs shift - so undocumented wins become folklore and undocumented failures get rediscovered.
Key Concepts: problem frame, success metric, architecture sketch, tool allowlist, bounds and escalations, eval evidence, cost and latency, trade-off log, outcomes and open questions.
When to Use: After any pilot that took real effort, after a production incident, and before you recommend the same pattern elsewhere.
Limitations/Trade-offs: Case studies take time; over-documenting demos creates noise. Prefer short, evidence-backed notes over novel-length retrospectives.
Related Topics: ADRs, eval suites, observability traces, cost routing, human-in-the-loop gates.

Foundations

Why agents need case studies more than CRUD services

A fixed service has a relatively stable path from request to response.

An agent has a loop, model variance, tool side effects, and cost that scales with thrash.

Six months later you rarely remember:

Which model tier handled which step.
Why you rejected multi-agent for v1.
What false-positive rate made the bot unusable.
Which gate stopped a bad write in production.

Without that memory, the next project starts from vibes.

Case study vs ADR vs runbook

Artifact	Primary job	Time horizon
Case study	Narrative of problem → design → results	Reuse of judgment across projects
ADR	Single decision and context	Lifecycle of this system
Runbook	How to operate and recover now	Incident and day-2 ops

Use all three when the system is serious.

A case study can link to ADRs and runbooks; it should not replace them.

Audience

Write for a skeptical peer who was not in the room.

They should answer:

Would I start from this design for a similar problem?
What must be true for these results to hold?
What broke, and how was it contained?

Mechanics & Interactions

The capture checklist (minimum viable case study)

Aim for one short doc (or wiki page) with these sections.

Section	What to write	Bad substitute
Problem frame	User, goal, `done_when`, constraints	"We wanted an AI assistant"
Success metrics	Quality, latency, cost, human override rate	"Stakeholders were happy"
Architecture sketch	Loop shape, agents, stores, deploy target	Framework logo only
Tools and permissions	Allowlist, read vs write, secrets scope	"Integrations"
Bounds	Max turns, budgets, timeouts, stop rules	"The model stops when done"
Human gates	What pauses for approval	"Support can take over" with no mechanism
Eval evidence	Golden tasks, pass rates, sample size	One demo transcript
Observability	Trace fields you actually used	"We have logging"
Outcomes	Before/after numbers and qualitative notes	Screenshots of chat only
Trade-offs and rejects	What you did not choose and why	Silence
Incidents	Failure modes and fixes	Blaming the model alone
Open questions	What you still do not know	Claiming completeness

Architecture sketch that ages well

Prefer boxes that name responsibilities, not only product names:

Host / orchestrator process.
Model gateway (direct provider or OpenRouter-style router).
Tool layer (HTTP, MCP, repo APIs).
Memory (session vs long-term).
Gates (HITL, policy, spend caps).
Sinks (tickets, chat, CI comments, CRM).

Note framework versions and "verify at build" for anything that moves quarterly.

Metrics worth keeping

Metric class	Examples	Why it survives hype
Task success	Rubric pass rate on golden set	Quality without vibes
Safety	Bad-write count, escalation rate	Trust
Cost	$/successful task, p95 $/run	Budget realism
Latency	p50/p95 end-to-end	UX and CI budgets
Noise	Comments per PR, false S0 rate	Adoption
Ops	Loop-cap hits, tool error rate	Reliability

Record period and traffic mix. A pilot on 50 friendly tickets is not a launch claim.

Trade-off log (the part people skip)

For each major fork, write one line:

Option A vs B → Chose A because constraint → Cost/risk accepted.

Examples:

Single agent vs multi-agent → single for v1 → simpler traces; may split later if context bloats.
Advisory vs blocking review bot → advisory first → lower merge friction until precision is proven.
Always frontier model vs tiered routing → tiered after cost blowup → quality risk on hard steps needs eval coverage.

Future you needs the constraint, not the brand preference.

Linking evidence

A case study without evidence is a blog post.

Attach or link:

Sample traces (redacted).
Eval set definition and scores.
Dashboard screenshots or tables for cost/latency.
Incident tickets and postmortems.
ADR IDs for framework and provider choices.

Advanced Considerations

Depth vs speed

Situation	Case study depth
Weekend spike, no users	10-line note + scrap link
Internal pilot (weeks)	Full checklist, light numbers
Customer-facing or money/risk path	Full checklist + incidents + eval tables
Pattern you will teach the org	Publish as reusable case study (this section's style)

Updating living systems

Case studies go stale when models and tools change.

Practice:

Date the document and the metric window.
Add a short "revisited" section when routing, tools, or success criteria change.
Do not silently rewrite history; append outcomes so sequences stay honest.

Multi-team reuse

If another team will copy your pattern, add:

Preconditions (data quality, APIs, review culture).
Non-goals for v1.
What broke when we scaled 10×.

Without preconditions, others clone the diagram and miss the constraints that made it work.

Common Misconceptions

Misconception	Reality
"The PR description is enough"	PRs lose metrics, rejects, and multi-month outcomes
"We only need ADRs"	ADRs freeze decisions; case studies teach journeys
"Show the happy path only"	Failures teach more than demos
"Vendor case studies replace ours"	Their traffic, risk, and ops are not yours
"Case studies are for marketing"	Internal engineering memory is the primary value
"No numbers until production"	Pilot golden-set scores still beat anecdotes

FAQs

How long should an internal case study be?

Often 1-3 pages plus links. If nobody will read it, cut narrative and keep tables, metrics, and trade-offs.

When should I write it - during or after the project?

Capture decisions and metrics as you go; write the coherent narrative at pilot end or after a major milestone. Waiting six months guarantees fiction.

What if results were bad?

Document them carefully. Negative case studies prevent expensive repeats and are often more valuable than victory posts.

Should I include full prompts?

Include versioned prompt pointers and key policy excerpts. Full prompts may be long or sensitive; store them in repo with the case study linking to the commit.

How do I handle confidential customer data?

Redact entities, use synthetic examples, and keep real traces in a private store. The architecture and metrics still publish cleanly.

Is a slide deck enough?

Slides fade. Prefer a text document with tables that greps and links. Decks can summarize for leadership if the source of truth is text.

What is the difference between a case study and an eval report?

An eval report is a measurement artifact for a model or prompt version. A case study explains the whole system and the human decisions around those measurements.

How many metrics are enough?

Three strong ones beat twelve vanity ones. Prefer task success, cost per success, and one safety or noise metric tied to trust.

Should framework choice dominate the write-up?

No. Lead with problem, tools, bounds, and outcomes. Frameworks are implementation detail unless the decision was the story.

Can I reuse this template for multi-agent systems?

Yes. Add handoff contracts, per-agent allowlists, and orchestrator stop rules to the architecture and bounds sections.

How do case studies relate to postmortems?

Postmortems deep-dive one incident. Case studies summarize the system story and may link several postmortems as evidence.

What is the first section to write if I am short on time?

Problem frame, success metrics, tool allowlist, bounds, and the trade-off log. Everything else can be thin.

Case Studies Basics - first end-to-end mini case study
Case Studies Best Practices - habits for writing and revisiting case studies
Why Agent Systems Need Their Own ADR Discipline - decision records companion
Architecture Decisions Basics - capturing forks early
Postmortems for Agent Incidents: A Template - incident evidence

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

It is a durable record of what you built, why you chose that design, what it cost, how it failed, and what you would repeat or avoid next time.

Summary

Capture architecture, tools, bounds, metrics, incidents, and rejected alternatives so future you (or another team) can reuse judgment instead of only cloning code.
Insight: Agent systems rot faster than typical apps - models, pricing, frameworks, and tool APIs shift - so undocumented wins become folklore and undocumented failures get rediscovered.
Key Concepts: problem frame, success metric, architecture sketch, tool allowlist, bounds and escalations, eval evidence, cost and latency, trade-off log, outcomes and open questions.
When to Use: After any pilot that took real effort, after a production incident, and before you recommend the same pattern elsewhere.
Limitations/Trade-offs: Case studies take time; over-documenting demos creates noise. Prefer short, evidence-backed notes over novel-length retrospectives.
Related Topics: ADRs, eval suites, observability traces, cost routing, human-in-the-loop gates.

Foundations

Why agents need case studies more than CRUD services

A fixed service has a relatively stable path from request to response.

An agent has a loop, model variance, tool side effects, and cost that scales with thrash.

Six months later you rarely remember:

Which model tier handled which step.
Why you rejected multi-agent for v1.
What false-positive rate made the bot unusable.
Which gate stopped a bad write in production.

Without that memory, the next project starts from vibes.

Case study vs ADR vs runbook

Artifact	Primary job	Time horizon
Case study	Narrative of problem → design → results	Reuse of judgment across projects
ADR	Single decision and context	Lifecycle of this system
Runbook	How to operate and recover now	Incident and day-2 ops

Use all three when the system is serious.

A case study can link to ADRs and runbooks; it should not replace them.

Audience

Write for a skeptical peer who was not in the room.

They should answer:

Would I start from this design for a similar problem?
What must be true for these results to hold?
What broke, and how was it contained?

Mechanics & Interactions

The capture checklist (minimum viable case study)

Aim for one short doc (or wiki page) with these sections.

Section	What to write	Bad substitute
Problem frame	User, goal, `done_when`, constraints	"We wanted an AI assistant"
Success metrics	Quality, latency, cost, human override rate	"Stakeholders were happy"
Architecture sketch	Loop shape, agents, stores, deploy target	Framework logo only
Tools and permissions	Allowlist, read vs write, secrets scope	"Integrations"
Bounds	Max turns, budgets, timeouts, stop rules	"The model stops when done"
Human gates	What pauses for approval	"Support can take over" with no mechanism
Eval evidence	Golden tasks, pass rates, sample size	One demo transcript
Observability	Trace fields you actually used	"We have logging"
Outcomes	Before/after numbers and qualitative notes	Screenshots of chat only
Trade-offs and rejects	What you did not choose and why	Silence
Incidents	Failure modes and fixes	Blaming the model alone
Open questions	What you still do not know	Claiming completeness

Architecture sketch that ages well

Prefer boxes that name responsibilities, not only product names:

Host / orchestrator process.
Model gateway (direct provider or OpenRouter-style router).
Tool layer (HTTP, MCP, repo APIs).
Memory (session vs long-term).
Gates (HITL, policy, spend caps).
Sinks (tickets, chat, CI comments, CRM).

Note framework versions and "verify at build" for anything that moves quarterly.

Metrics worth keeping

Metric class	Examples	Why it survives hype
Task success	Rubric pass rate on golden set	Quality without vibes
Safety	Bad-write count, escalation rate	Trust
Cost	$/successful task, p95 $/run	Budget realism
Latency	p50/p95 end-to-end	UX and CI budgets
Noise	Comments per PR, false S0 rate	Adoption
Ops	Loop-cap hits, tool error rate	Reliability

Record period and traffic mix. A pilot on 50 friendly tickets is not a launch claim.

Trade-off log (the part people skip)

For each major fork, write one line:

Option A vs B → Chose A because constraint → Cost/risk accepted.

Examples:

Single agent vs multi-agent → single for v1 → simpler traces; may split later if context bloats.
Advisory vs blocking review bot → advisory first → lower merge friction until precision is proven.
Always frontier model vs tiered routing → tiered after cost blowup → quality risk on hard steps needs eval coverage.

Future you needs the constraint, not the brand preference.

Linking evidence

A case study without evidence is a blog post.

Attach or link:

Sample traces (redacted).
Eval set definition and scores.
Dashboard screenshots or tables for cost/latency.
Incident tickets and postmortems.
ADR IDs for framework and provider choices.

Advanced Considerations

Depth vs speed

Situation	Case study depth
Weekend spike, no users	10-line note + scrap link
Internal pilot (weeks)	Full checklist, light numbers
Customer-facing or money/risk path	Full checklist + incidents + eval tables
Pattern you will teach the org	Publish as reusable case study (this section's style)

Updating living systems

Case studies go stale when models and tools change.

Practice:

Date the document and the metric window.
Add a short "revisited" section when routing, tools, or success criteria change.
Do not silently rewrite history; append outcomes so sequences stay honest.

Multi-team reuse

If another team will copy your pattern, add:

Preconditions (data quality, APIs, review culture).
Non-goals for v1.
What broke when we scaled 10×.

Without preconditions, others clone the diagram and miss the constraints that made it work.

Common Misconceptions

Misconception	Reality
"The PR description is enough"	PRs lose metrics, rejects, and multi-month outcomes
"We only need ADRs"	ADRs freeze decisions; case studies teach journeys
"Show the happy path only"	Failures teach more than demos
"Vendor case studies replace ours"	Their traffic, risk, and ops are not yours
"Case studies are for marketing"	Internal engineering memory is the primary value
"No numbers until production"	Pilot golden-set scores still beat anecdotes

FAQs

How long should an internal case study be?

Often 1-3 pages plus links. If nobody will read it, cut narrative and keep tables, metrics, and trade-offs.

When should I write it - during or after the project?

Capture decisions and metrics as you go; write the coherent narrative at pilot end or after a major milestone. Waiting six months guarantees fiction.

What if results were bad?

Document them carefully. Negative case studies prevent expensive repeats and are often more valuable than victory posts.

Should I include full prompts?

Include versioned prompt pointers and key policy excerpts. Full prompts may be long or sensitive; store them in repo with the case study linking to the commit.

How do I handle confidential customer data?

Redact entities, use synthetic examples, and keep real traces in a private store. The architecture and metrics still publish cleanly.

Is a slide deck enough?

Slides fade. Prefer a text document with tables that greps and links. Decks can summarize for leadership if the source of truth is text.

What is the difference between a case study and an eval report?

An eval report is a measurement artifact for a model or prompt version. A case study explains the whole system and the human decisions around those measurements.

How many metrics are enough?

Three strong ones beat twelve vanity ones. Prefer task success, cost per success, and one safety or noise metric tied to trust.

Should framework choice dominate the write-up?

No. Lead with problem, tools, bounds, and outcomes. Frameworks are implementation detail unless the decision was the story.

Can I reuse this template for multi-agent systems?

Yes. Add handoff contracts, per-agent allowlists, and orchestrator stop rules to the architecture and bounds sections.

How do case studies relate to postmortems?

Postmortems deep-dive one incident. Case studies summarize the system story and may link several postmortems as evidence.

What is the first section to write if I am short on time?

Problem frame, success metrics, tool allowlist, bounds, and the trade-off log. Everything else can be thin.

Case Studies Basics - first end-to-end mini case study
Case Studies Best Practices - habits for writing and revisiting case studies
Why Agent Systems Need Their Own ADR Discipline - decision records companion
Architecture Decisions Basics - capturing forks early
Postmortems for Agent Incidents: A Template - incident evidence

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

What a Case Study Should Capture for Future Reference

Buscar na documentação

What a Case Study Should Capture for Future Reference