What a Case Study Should Capture for Future Reference
A good agent case study is not a marketing write-up.
It is a durable record of what you built, why you chose that design, what it cost, how it failed, and what you would repeat or avoid next time.
Summary
- Capture architecture, tools, bounds, metrics, incidents, and rejected alternatives so future you (or another team) can reuse judgment instead of only cloning code.
- Insight: Agent systems rot faster than typical apps - models, pricing, frameworks, and tool APIs shift - so undocumented wins become folklore and undocumented failures get rediscovered.
- Key Concepts: problem frame, success metric, architecture sketch, tool allowlist, bounds and escalations, eval evidence, cost and latency, trade-off log, outcomes and open questions.
- When to Use: After any pilot that took real effort, after a production incident, and before you recommend the same pattern elsewhere.
- Limitations/Trade-offs: Case studies take time; over-documenting demos creates noise. Prefer short, evidence-backed notes over novel-length retrospectives.
- Related Topics: ADRs, eval suites, observability traces, cost routing, human-in-the-loop gates.
Foundations
Why agents need case studies more than CRUD services
A fixed service has a relatively stable path from request to response.
An agent has a loop, model variance, tool side effects, and cost that scales with thrash.
Six months later you rarely remember:
- Which model tier handled which step.
- Why you rejected multi-agent for v1.
- What false-positive rate made the bot unusable.
- Which gate stopped a bad write in production.
Without that memory, the next project starts from vibes.
Case study vs ADR vs runbook
| Artifact | Primary job | Time horizon |
|---|---|---|
| Case study | Narrative of problem → design → results | Reuse of judgment across projects |
| ADR | Single decision and context | Lifecycle of this system |
| Runbook | How to operate and recover now | Incident and day-2 ops |
Use all three when the system is serious.
A case study can link to ADRs and runbooks; it should not replace them.
Audience
Write for a skeptical peer who was not in the room.
They should answer:
- Would I start from this design for a similar problem?
- What must be true for these results to hold?
- What broke, and how was it contained?
Mechanics & Interactions
The capture checklist (minimum viable case study)
Aim for one short doc (or wiki page) with these sections.
| Section | What to write | Bad substitute |
|---|---|---|
| Problem frame | User, goal, done_when, constraints | "We wanted an AI assistant" |
| Success metrics | Quality, latency, cost, human override rate | "Stakeholders were happy" |
| Architecture sketch | Loop shape, agents, stores, deploy target | Framework logo only |
| Tools and permissions | Allowlist, read vs write, secrets scope | "Integrations" |
| Bounds | Max turns, budgets, timeouts, stop rules | "The model stops when done" |
| Human gates | What pauses for approval | "Support can take over" with no mechanism |
| Eval evidence | Golden tasks, pass rates, sample size | One demo transcript |
| Observability | Trace fields you actually used | "We have logging" |
| Outcomes | Before/after numbers and qualitative notes | Screenshots of chat only |
| Trade-offs and rejects | What you did not choose and why | Silence |
| Incidents | Failure modes and fixes | Blaming the model alone |
| Open questions | What you still do not know | Claiming completeness |
Architecture sketch that ages well
Prefer boxes that name responsibilities, not only product names:
- Host / orchestrator process.
- Model gateway (direct provider or OpenRouter-style router).
- Tool layer (HTTP, MCP, repo APIs).
- Memory (session vs long-term).
- Gates (HITL, policy, spend caps).
- Sinks (tickets, chat, CI comments, CRM).
Note framework versions and "verify at build" for anything that moves quarterly.
Metrics worth keeping
| Metric class | Examples | Why it survives hype |
|---|---|---|
| Task success | Rubric pass rate on golden set | Quality without vibes |
| Safety | Bad-write count, escalation rate | Trust |
| Cost | $/successful task, p95 $/run | Budget realism |
| Latency | p50/p95 end-to-end | UX and CI budgets |
| Noise | Comments per PR, false S0 rate | Adoption |
| Ops | Loop-cap hits, tool error rate | Reliability |
Record period and traffic mix. A pilot on 50 friendly tickets is not a launch claim.
Trade-off log (the part people skip)
For each major fork, write one line:
- Option A vs B → Chose A because constraint → Cost/risk accepted.
Examples:
- Single agent vs multi-agent → single for v1 → simpler traces; may split later if context bloats.
- Advisory vs blocking review bot → advisory first → lower merge friction until precision is proven.
- Always frontier model vs tiered routing → tiered after cost blowup → quality risk on hard steps needs eval coverage.
Future you needs the constraint, not the brand preference.
Linking evidence
A case study without evidence is a blog post.
Attach or link:
- Sample traces (redacted).
- Eval set definition and scores.
- Dashboard screenshots or tables for cost/latency.
- Incident tickets and postmortems.
- ADR IDs for framework and provider choices.
Advanced Considerations
Depth vs speed
| Situation | Case study depth |
|---|---|
| Weekend spike, no users | 10-line note + scrap link |
| Internal pilot (weeks) | Full checklist, light numbers |
| Customer-facing or money/risk path | Full checklist + incidents + eval tables |
| Pattern you will teach the org | Publish as reusable case study (this section's style) |
Updating living systems
Case studies go stale when models and tools change.
Practice:
- Date the document and the metric window.
- Add a short "revisited" section when routing, tools, or success criteria change.
- Do not silently rewrite history; append outcomes so sequences stay honest.
Multi-team reuse
If another team will copy your pattern, add:
- Preconditions (data quality, APIs, review culture).
- Non-goals for v1.
- What broke when we scaled 10×.
Without preconditions, others clone the diagram and miss the constraints that made it work.
Common Misconceptions
| Misconception | Reality |
|---|---|
| "The PR description is enough" | PRs lose metrics, rejects, and multi-month outcomes |
| "We only need ADRs" | ADRs freeze decisions; case studies teach journeys |
| "Show the happy path only" | Failures teach more than demos |
| "Vendor case studies replace ours" | Their traffic, risk, and ops are not yours |
| "Case studies are for marketing" | Internal engineering memory is the primary value |
| "No numbers until production" | Pilot golden-set scores still beat anecdotes |
FAQs
How long should an internal case study be?
Often 1-3 pages plus links. If nobody will read it, cut narrative and keep tables, metrics, and trade-offs.
When should I write it - during or after the project?
Capture decisions and metrics as you go; write the coherent narrative at pilot end or after a major milestone. Waiting six months guarantees fiction.
What if results were bad?
Document them carefully. Negative case studies prevent expensive repeats and are often more valuable than victory posts.
Should I include full prompts?
Include versioned prompt pointers and key policy excerpts. Full prompts may be long or sensitive; store them in repo with the case study linking to the commit.
How do I handle confidential customer data?
Redact entities, use synthetic examples, and keep real traces in a private store. The architecture and metrics still publish cleanly.
Is a slide deck enough?
Slides fade. Prefer a text document with tables that greps and links. Decks can summarize for leadership if the source of truth is text.
What is the difference between a case study and an eval report?
An eval report is a measurement artifact for a model or prompt version. A case study explains the whole system and the human decisions around those measurements.
How many metrics are enough?
Three strong ones beat twelve vanity ones. Prefer task success, cost per success, and one safety or noise metric tied to trust.
Should framework choice dominate the write-up?
No. Lead with problem, tools, bounds, and outcomes. Frameworks are implementation detail unless the decision was the story.
Can I reuse this template for multi-agent systems?
Yes. Add handoff contracts, per-agent allowlists, and orchestrator stop rules to the architecture and bounds sections.
How do case studies relate to postmortems?
Postmortems deep-dive one incident. Case studies summarize the system story and may link several postmortems as evidence.
What is the first section to write if I am short on time?
Problem frame, success metrics, tool allowlist, bounds, and the trade-off log. Everything else can be thin.
Related
- Case Studies Basics - first end-to-end mini case study
- Case Studies Best Practices - habits for writing and revisiting case studies
- Why Agent Systems Need Their Own ADR Discipline - decision records companion
- Architecture Decisions Basics - capturing forks early
- Postmortems for Agent Incidents: A Template - incident evidence
Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana
@solana/web3.js+@solana/spl-token.