Choosing a Framework Best Practices

Ten practices for evaluating and committing to an agent framework without over-committing early. Use them as a pre-ADR checklist and a quarterly review rubric.

Honest trade-offs beat vendor hype: optimize for operable control flow, team skills, and portable assets - not for the longest feature checklist.

How to Use This List

Walk A before a spike, B during evaluation, C when you commit and operate.
Tick items only when evidence exists (repo, scorecard, ADR), not as aspirations.
Pair with the comparison matrix and team skill checklist.
Re-check when ownership, compliance needs, or topology complexity changes.

A - Frame the Decision

1. Write the job story before the shortlist. One paragraph: user goal, tools, risk, latency/cost envelope, and whether this is even an agent. Frameworks cannot rescue a script-shaped problem.
2. Name the orchestration shape you need. Graph, crew/roles, thin typed loop, provider-native handoffs, UI streaming, no-code automation, or custom. Feature shopping without a shape produces random winners.
3. Inventory team skills and on-call ownership. Language fluency and who debugs production failures outrank blog benchmarks. See Matching Team Skill Set to Framework Choice.
4. Define non-negotiable ops constraints up front. HITL/resume, SSO, data residency, audit logs, multi-tenant isolation, air-gap, or budget caps. Drop candidates that cannot meet them without heroics.
5. Cap the shortlist at two finalists (three max). More than that becomes theater. Use the matrix for elimination, not for endless debate.

B - Evaluate with Evidence

6. Spike both finalists on the same real task. Identical tools, success criteria, and stop budgets. Capture install friction, lines of glue, trace readability, and failure behavior - see Choosing a Framework Basics.
7. Scorecard more than happy-path demos. Include tool errors, timeouts, injection-ish inputs, approval pauses (if required), and cost per success. A pretty multi-agent demo is not a production signal.
8. Keep tools, schemas, and prompts portable during the spike. If the spike only works as framework-native spaghetti, you learned the wrong lesson. Apply lock-in cheatsheet patterns immediately.
9. Separate UI SDK choices from durable runtime choices. Vercel AI SDK (or peers) can win the presentation layer while LangGraph, Pydantic AI, MS Agent Framework, or custom code owns the worker. Do not force one library to do every job.
10. Prefer the simplest stack that passes the scorecard. Thin native SDK or Pydantic AI can beat a full graph platform for short tool agents. CrewAI can beat graphs for role-shaped work. No-code can beat all of them for SaaS glue.

C - Commit, Contain, Revisit

11. Write an ADR with conscious bets and revisit triggers. Record why finalists lost, what lock-in you accept, and metrics that force a rematch. Link spike artifacts.
12. Establish host interfaces and eval cases as code. AgentHost.run, shared schemas, and CI evals make future migration a project, not a crisis.
13. Enforce stop conditions and least-privilege tools in the host. Framework defaults are not a safety program. Max turns, timeouts, and allowlists ship with v1.
14. Limit production frameworks per product surface. One primary orchestrator per agent product; optional UI SDK; optional no-code for peripheral automation. Sprawl is an ownership bug.
15. Schedule a post-launch review using production metrics. Turns, cost, stop reasons, incident themes, and team debug time decide whether to harden, simplify, or migrate (migration paths).

Applying These Practices in Order

Frame (1-5): cheap on a whiteboard; expensive after three half-integrated POCs.
Evidence (6-10): the antidote to vendor keynotes and internal politics.
Commit (11-15): how you avoid both endless churn and permanent accidental lock-in.

Scorecard Template (minimum columns)

Criterion	Weight	Notes
Correct on golden cases	Must
Stop/timeout behavior	Must
Team can debug	High
Portable tools/schemas	High
HITL/resume fit	As needed
Observability quality	High
Time to first secure deploy	Medium
Dependency / upgrade risk	Medium
Cost per successful task	Medium
Conscious lock-in accepted?	High	Y/N + description

Framework-Specific Honesty Checks

If you are leaning…	Ask hard
LangGraph	Are we ready to design state and edges, or are we cargo-culting graphs for a chatbot?
CrewAI	Do we truly need roles, or is this one agent in a trench coat?
Microsoft Agent Framework	Is MS ecosystem gravity a discount or a tax for us?
LlamaIndex	Are we solving retrieval, orchestration, or both - and who owns each?
Pydantic AI	Will typed single-agent services cover the roadmap for 6+ months?
Native Agents SDK	Is provider alignment a strategy or an accident?
Vercel AI SDK	What component owns durable tools, secrets, and long jobs?
No-code	Who reviews changes, secrets, and failure modes?
Custom	Who maintains the runtime when the author changes teams?

Anti-Patterns

Anti-pattern	Why it fails	Better practice
Framework bake-off without shared tasks	Incomparable demos	Practice 6
Choosing for resume lines	Unowned complexity	Practices 3 and 10
Standardizing company-wide on day one	Ignores product diversity	Practice 14 with interfaces
Ignoring no-code out of pride	Wasted eng time on glue	Practice 10
Migration without eval gates	Silent regressions	Practices 12 and 15
"Temporary" super-agent with all tools	Security and lock-in	Practice 13

FAQs

Why "ten practices" if there are fifteen checkboxes?

Items 1-10 are the core evaluation loop. Items 11-15 extend into commitment and operations so the choice stays honest after launch.

What is the single highest-leverage practice?

Spike two finalists on one real task with shared stop budgets and a written scorecard (practices 5-7). Everything else gets sharper after that.

How long should a spike take?

Usually 1-2 engineering days per finalist for a thin agent, longer if enterprise auth and connectors are in scope. Time-box before perfectionism sets in.

Should we always pick open source to avoid lock-in?

No. Open source can still trap you in DSLs and state formats. Portability is about seams (tools, schemas, evals), not license text alone.

When is custom runtime the best-practice outcome?

When finalists fail non-negotiables or add more escape hatches than features, and a named platform owner will maintain the loop. See Framework-Free vs Framework-Based.

How do we stop framework churn every quarter?

ADR revisit triggers tied to metrics, not vibes. New blogs are not triggers; repeated max-turn incidents or compliance gaps might be.

Can product and platform teams use different frameworks?

Yes if interfaces and observability fields are shared. Document the split so support paths stay clear.

Where do architecture patterns fit relative to framework choice?

Patterns first (ReAct, plan-execute, multi-agent), framework second. A framework that fights your pattern is a bad fit regardless of popularity.

How formal should the ADR be for a startup?

One page is enough: context, finalists, scorecard link, decision, conscious lock-in, revisit triggers. Formality scales with blast radius.

What evidence shows practice 10 was violated?

A multi-agent graph platform deployed for a single tool call that a 50-line loop handled with equal quality and lower cost.

Should procurement-driven framework choices skip spikes?

Never for technical fit. Procurement can constrain the shortlist; spikes still validate operability.

How do these practices interact with model gateways like OpenRouter?

Gateways help model portability and cost routing. They do not replace orchestration evaluation, tool design, or stop policies.

What Actually Differs Between Agent Frameworks - comparison axes
Framework Comparison Matrix: LangGraph, CrewAI, Microsoft Agent Framework & More - ranked reference
Choosing a Framework Basics - hands-on dual spike
Matching Team Skill Set to Framework Choice - people fit
Framework Lock-In Risks and How to Avoid Them - portability bar
An ADR Template for Agent Framework Selection - write the decision down

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Choosing a Framework Best Practices

Ten practices for evaluating and committing to an agent framework without over-committing early. Use them as a pre-ADR checklist and a quarterly review rubric.

Honest trade-offs beat vendor hype: optimize for operable control flow, team skills, and portable assets - not for the longest feature checklist.

How to Use This List

Walk A before a spike, B during evaluation, C when you commit and operate.
Tick items only when evidence exists (repo, scorecard, ADR), not as aspirations.
Pair with the comparison matrix and team skill checklist.
Re-check when ownership, compliance needs, or topology complexity changes.

A - Frame the Decision

1. Write the job story before the shortlist. One paragraph: user goal, tools, risk, latency/cost envelope, and whether this is even an agent. Frameworks cannot rescue a script-shaped problem.
2. Name the orchestration shape you need. Graph, crew/roles, thin typed loop, provider-native handoffs, UI streaming, no-code automation, or custom. Feature shopping without a shape produces random winners.
3. Inventory team skills and on-call ownership. Language fluency and who debugs production failures outrank blog benchmarks. See Matching Team Skill Set to Framework Choice.
4. Define non-negotiable ops constraints up front. HITL/resume, SSO, data residency, audit logs, multi-tenant isolation, air-gap, or budget caps. Drop candidates that cannot meet them without heroics.
5. Cap the shortlist at two finalists (three max). More than that becomes theater. Use the matrix for elimination, not for endless debate.

B - Evaluate with Evidence

6. Spike both finalists on the same real task. Identical tools, success criteria, and stop budgets. Capture install friction, lines of glue, trace readability, and failure behavior - see Choosing a Framework Basics.
7. Scorecard more than happy-path demos. Include tool errors, timeouts, injection-ish inputs, approval pauses (if required), and cost per success. A pretty multi-agent demo is not a production signal.
8. Keep tools, schemas, and prompts portable during the spike. If the spike only works as framework-native spaghetti, you learned the wrong lesson. Apply lock-in cheatsheet patterns immediately.
9. Separate UI SDK choices from durable runtime choices. Vercel AI SDK (or peers) can win the presentation layer while LangGraph, Pydantic AI, MS Agent Framework, or custom code owns the worker. Do not force one library to do every job.
10. Prefer the simplest stack that passes the scorecard. Thin native SDK or Pydantic AI can beat a full graph platform for short tool agents. CrewAI can beat graphs for role-shaped work. No-code can beat all of them for SaaS glue.

C - Commit, Contain, Revisit

11. Write an ADR with conscious bets and revisit triggers. Record why finalists lost, what lock-in you accept, and metrics that force a rematch. Link spike artifacts.
12. Establish host interfaces and eval cases as code. AgentHost.run, shared schemas, and CI evals make future migration a project, not a crisis.
13. Enforce stop conditions and least-privilege tools in the host. Framework defaults are not a safety program. Max turns, timeouts, and allowlists ship with v1.
14. Limit production frameworks per product surface. One primary orchestrator per agent product; optional UI SDK; optional no-code for peripheral automation. Sprawl is an ownership bug.
15. Schedule a post-launch review using production metrics. Turns, cost, stop reasons, incident themes, and team debug time decide whether to harden, simplify, or migrate (migration paths).

Applying These Practices in Order

Frame (1-5): cheap on a whiteboard; expensive after three half-integrated POCs.
Evidence (6-10): the antidote to vendor keynotes and internal politics.
Commit (11-15): how you avoid both endless churn and permanent accidental lock-in.

Scorecard Template (minimum columns)

Criterion	Weight	Notes
Correct on golden cases	Must
Stop/timeout behavior	Must
Team can debug	High
Portable tools/schemas	High
HITL/resume fit	As needed
Observability quality	High
Time to first secure deploy	Medium
Dependency / upgrade risk	Medium
Cost per successful task	Medium
Conscious lock-in accepted?	High	Y/N + description

Framework-Specific Honesty Checks

If you are leaning…	Ask hard
LangGraph	Are we ready to design state and edges, or are we cargo-culting graphs for a chatbot?
CrewAI	Do we truly need roles, or is this one agent in a trench coat?
Microsoft Agent Framework	Is MS ecosystem gravity a discount or a tax for us?
LlamaIndex	Are we solving retrieval, orchestration, or both - and who owns each?
Pydantic AI	Will typed single-agent services cover the roadmap for 6+ months?
Native Agents SDK	Is provider alignment a strategy or an accident?
Vercel AI SDK	What component owns durable tools, secrets, and long jobs?
No-code	Who reviews changes, secrets, and failure modes?
Custom	Who maintains the runtime when the author changes teams?

Anti-Patterns

Anti-pattern	Why it fails	Better practice
Framework bake-off without shared tasks	Incomparable demos	Practice 6
Choosing for resume lines	Unowned complexity	Practices 3 and 10
Standardizing company-wide on day one	Ignores product diversity	Practice 14 with interfaces
Ignoring no-code out of pride	Wasted eng time on glue	Practice 10
Migration without eval gates	Silent regressions	Practices 12 and 15
"Temporary" super-agent with all tools	Security and lock-in	Practice 13

FAQs

Why "ten practices" if there are fifteen checkboxes?

Items 1-10 are the core evaluation loop. Items 11-15 extend into commitment and operations so the choice stays honest after launch.

What is the single highest-leverage practice?

Spike two finalists on one real task with shared stop budgets and a written scorecard (practices 5-7). Everything else gets sharper after that.

How long should a spike take?

Usually 1-2 engineering days per finalist for a thin agent, longer if enterprise auth and connectors are in scope. Time-box before perfectionism sets in.

Should we always pick open source to avoid lock-in?

No. Open source can still trap you in DSLs and state formats. Portability is about seams (tools, schemas, evals), not license text alone.

When is custom runtime the best-practice outcome?

When finalists fail non-negotiables or add more escape hatches than features, and a named platform owner will maintain the loop. See Framework-Free vs Framework-Based.

How do we stop framework churn every quarter?

ADR revisit triggers tied to metrics, not vibes. New blogs are not triggers; repeated max-turn incidents or compliance gaps might be.

Can product and platform teams use different frameworks?

Yes if interfaces and observability fields are shared. Document the split so support paths stay clear.

Where do architecture patterns fit relative to framework choice?

Patterns first (ReAct, plan-execute, multi-agent), framework second. A framework that fights your pattern is a bad fit regardless of popularity.

How formal should the ADR be for a startup?

One page is enough: context, finalists, scorecard link, decision, conscious lock-in, revisit triggers. Formality scales with blast radius.

What evidence shows practice 10 was violated?

A multi-agent graph platform deployed for a single tool call that a 50-line loop handled with equal quality and lower cost.

Should procurement-driven framework choices skip spikes?

Never for technical fit. Procurement can constrain the shortlist; spikes still validate operability.

How do these practices interact with model gateways like OpenRouter?

Gateways help model portability and cost routing. They do not replace orchestration evaluation, tool design, or stop policies.

What Actually Differs Between Agent Frameworks - comparison axes
Framework Comparison Matrix: LangGraph, CrewAI, Microsoft Agent Framework & More - ranked reference
Choosing a Framework Basics - hands-on dual spike
Matching Team Skill Set to Framework Choice - people fit
Framework Lock-In Risks and How to Avoid Them - portability bar
An ADR Template for Agent Framework Selection - write the decision down

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Choosing a Framework Best Practices

Buscar en la documentación

Choosing a Framework Best Practices