Why Some Teams Build a Custom Agent Runtime Instead of Adopting a Framework
A custom agent runtime is your own control plane: you own the model call, tool dispatch, stop rules, and run metadata without LangGraph, CrewAI, Microsoft Agent Framework, or a provider Agents SDK sitting in the middle. Some teams pick that path deliberately. This page explains when that decision is rational and when it is not-invented-here dressed up as architecture.
Summary
- Hand-rolled runtimes win when orchestration needs are small, dependency and compliance constraints are hard, or framework abstractions fight your product shape.
- Insight: Frameworks speed demos and multi-pattern orchestration; they also add upgrade risk, opaque magic, and lock-in around tools and state.
- Key Concepts: agent runtime, thin loop, control plane, dependency budget, undifferentiated heavy lifting, escape hatch.
- When to Use: Writing an ADR, reviewing a POC that "just used LangGraph," or deciding whether a provider Agents SDK is already enough without a third-party graph library.
- Limitations/Trade-offs: Custom still uses HTTP clients, queues, and OpenTelemetry. Framework-free means you own the loop, not that you invent TLS or JSON Schema.
- Related Topics: framework-free trade-offs, ReAct loops as APIs, eval harnesses, when to abandon the custom runtime.
Foundations
An agent runtime is the host that repeatedly: builds context, calls a model, executes tools, appends observations, enforces budgets, and emits a final result or stop reason.
A framework-based runtime borrows that host from a library (LangGraph graphs, CrewAI crews, OpenAI Agents Runner, Pydantic AI agents, and peers).
A custom runtime keeps those semantics in your service code, usually as a bounded loop behind FastAPI, Express, a worker, or a CLI.
Framework path:
App -> Framework control plane -> Model + tools -> Framework state helpers
Custom path:
App owns loop -> Model client -> Tool plugins -> Your store + tracesNeither path is "no dependencies." The question is who owns orchestration truth: stop enums, tool allowlists, retries, and what a "run" means in your product.
Teams that build custom usually share at least one pressure:
- Dependency freezes - regulated or air-gapped environments reject fast-moving agent frameworks.
- Simple loop shape - tools + budgets, not durable multi-day graphs with joins.
- Strict observability contracts - every field must land in their warehouse schema, not a vendor UI alone.
- Product is the runtime - they sell agent hosting and cannot ship a third-party control plane as the core.
- Framework mismatch - they tried a framework and spent more time fighting its graph model than shipping tools.
Analogy: buying a workflow engine versus writing a state machine for one process. One process with clear stop rules can be a 150-line loop. Twenty processes with pause/resume and visual ops usually want a platform.
Mechanics & Interactions
What custom actually buys
| Benefit | What it looks like in practice |
|---|---|
| Full stop control | max_turns, wall clock, cost caps, and cancel flags in your code path |
| Transparent tool path | Dispatcher you can unit test without framework mocks |
| Stable domain contracts | Tools stay pure functions; the loop is a thin adapter |
| Predictable upgrades | Model SDK upgrades only; no graph DSL major every quarter |
| Compliance narrative | You can point auditors at the exact file that opens circuits and denies tools |
What you still pay for
| Cost | Symptom if ignored |
|---|---|
| You own retries and backoff | Flaky tools thrash the model into spend |
| You own parallel tool semantics | Partial failures become inconsistent transcripts |
| You own multi-agent later | Ad-hoc handoffs become an undocumented second framework |
| You own HITL resume | "Just add a queue" becomes a half-built workflow engine |
| You own docs and hiring | New engineers learn your loop, not a public skill market |
Custom vs native provider SDKs vs third-party frameworks
| Approach | Owns the loop | Best when |
|---|---|---|
| Custom thin runtime | You | Hard constraints + simple tool loop |
| Native Agents SDK (OpenAI, Google ADK, Anthropic Messages loop) | Vendor/you mix | Single-provider product, handoffs/MCP wanted fast |
| Third-party framework | Library | Graphs, crews, enterprise connectors, multi-surface ops |
Custom is not automatically more portable than a framework. Portability comes from pure tools, versioned schemas, and prompts as data. A custom loop that embeds business rules only in prompt strings is still a trap.
Signals that custom is rational
- The product needs a single bounded ReAct-style loop for months, not branching workflows.
- Security review will not approve large orchestration packages without months of legal work.
- The team already operates solid HTTP services and prefers boring code over framework lore.
- You can staff ownership: one named engineer/team owns stop rules, evals, and on-call for the loop.
- A one-week spike shows the framework adding escape hatches faster than features.
Signals that custom is cosplay
- You are already implementing generic checkpoint packages, join semantics, and a plugin graph DSL.
- Product needs multi-day human approval with resume across deploys.
- Five agents and fan-out/fan-in appear on the roadmap this quarter.
- Nobody owns runtime reliability; "the loop" lives in a notebook and a Slack thread.
- The real motivation is dislike of docs, not a measured dependency or control problem.
Interaction with the rest of the stack
Custom runtimes still plug into:
- Model gateways (direct provider or OpenRouter-style routing)
- Tool plugins (local functions, MCP clients, internal APIs)
- Eval harnesses (gold sets, LLM-as-judge, CI gates)
- Observability (structured logs, OpenTelemetry spans, cost metrics)
- Product APIs (FastAPI/Express endpoints that start runs and stream events)
The runtime is the seam between product HTTP and model I/O. Keep that seam thin so domain services do not import "agent" types.
Advanced Considerations & Applications
Layering that keeps an exit hatch
domain/tools/* # pure, testable side effects
domain/schemas/* # JSON Schema / Pydantic contracts
runtime/loop.py # model call + dispatch + budgets
runtime/plugins/* # tool registration adapters
adapters/http.py # FastAPI / Express surfaceIf you later adopt LangGraph or a native Agents SDK, you rewrite runtime/* and keep domain/*.
Trade-off table
| Approach | Best fit | Main cost |
|---|---|---|
| Custom thin loop | Simple agents, compliance niches, platform builders | You own reliability forever |
| Native provider SDK | Single-provider products needing handoffs/MCP fast | Provider coupling |
| Full third-party framework | Complex orchestration and shared multi-product ops | Dependency weight and dual learning curves |
| Custom + borrowed pieces | Thin loop + OTel + your DB checkpoints | Discipline not to reinvent graphs |
When custom becomes a framework anyway
Watch for these code smells:
- A private
WorkflowEnginewith more docs than your product - YAML DSLs for agent graphs "just for us"
- Reimplemented memory, RAG, and multi-agent routers without product need
- Incompatible loop copies in three services
At that point you did not avoid a framework. You forked one without community support. See When a Custom Runtime Stops Being Worth Maintaining.
Organizational fit
| Team shape | Custom tends to… |
|---|---|
| Small backend-strong team, one agent product | Work well |
| Platform team shipping agent host as product | Work well if staffed |
| Large org with many agent apps | Prefer shared framework or internal platform once |
| Frontend-heavy team needing graphs | Struggle without framework DX |
Skill match matters as much as architecture. See Matching Team Skill Set to Framework Choice.
Measuring whether custom paid off
Track:
- Days from spike to first production tool path
- Share of agent code in
domain/vsruntime/ - Incident count from missing stop conditions
- Cost of adopting one new model provider (should be small)
- Hours spent reimplementing features frameworks already ship
If provider swaps are cheap and stop-related incidents stay near zero, custom is earning rent. If you spend every sprint on generic orchestration, migrate.
Common Misconceptions
- "Custom means zero dependencies." You still depend on model SDKs, HTTP stacks, and observability libraries.
- "Frameworks are always slower." For complex graphs, a mature framework is often faster to correctness than a buggy custom state machine.
- "Native Agents SDKs are not frameworks." They are provider-aligned frameworks; custom is a third option beside them and multi-vendor libs.
- "We can add stop conditions later." Unbounded loops are a production incident generator; budgets belong in v1.
- "Custom is more secure by default." Security comes from tool least privilege, sandboxes, and authZ - not from avoiding pip packages.
- "If we write it, we understand it." Only if it is small, tested, and documented. A 3k-line private orchestrator is not automatic clarity.
FAQs
Is a custom runtime the same as calling Chat Completions in a while-loop?
Often yes at the core. Production custom runtimes add budgets, structured stop reasons, tool plugins, auth context, tracing, and an HTTP or worker API around that loop.
How is this different from OpenAI Agents SDK or Anthropic tool loops?
Provider kits give official agent objects, handoffs, guardrails, or Messages tool primitives. Custom means you own those pieces (or deliberately omit them) without adopting that kit as your control plane.
When is LangGraph still the better default?
When durable resume, complex branching, joins, or multi-path HITL are first-class product needs. Reimplementing those well is expensive.
Can we use OpenRouter inside a custom runtime?
Yes. Treat OpenRouter (or any gateway) as the model client behind your loop. Routing is orthogonal to who owns the agent loop.
What should never live only in the runtime module?
Business invariants, authorization, durable customer records, and tool implementations other services must call. Keep those in domain modules with independent tests.
How big should a "thin" custom loop stay?
If the loop file grows past a few hundred lines of generic orchestration (graphs, joins, DSL parsers), you are inventing a framework. Extract only product-specific policy or switch stacks.
Do custom runtimes support streaming UIs?
Yes. Stream token or event chunks from the host; keep privileged tools on the server. UI SDKs (for example Vercel AI SDK) can present streams without owning the runtime.
Is MCP compatible with custom runtimes?
Yes. MCP is a tool-connection protocol. Your dispatcher can list MCP tools, translate schemas, and call MCP clients like any other plugin source.
What belongs in the ADR when we choose custom?
Drivers (compliance, simplicity, platform product), alternatives considered, stop/budget requirements, ownership, eval plan, and explicit exit triggers that force migration.
How do we prevent each team from forking a private loop?
Publish an internal thin runtime package with plugin interfaces, or standardize on one framework. Custom without a shared platform multiplies operational cost.
Are evals optional if the loop is small?
No. Small loops still change prompts, models, and tools. Gold-set evals and CI gates are how you know the custom path still works after edits.
When should we abandon custom for a maintained framework?
When roadmap needs outgrow the loop, staffing cannot own reliability, or you are reimplementing generic orchestration weekly. Use the decision cheatsheet in this section.
Related
- Custom Agent Runtimes Basics - first hand-rolled loop and dispatcher
- Designing a ReAct Loop from Scratch in FastAPI or Express - API-shaped recipe
- When a Custom Runtime Stops Being Worth Maintaining - exit checklist
- Custom Agent Runtimes Best Practices - maintainability practices
- Framework-Free vs Framework-Based: When to Roll Your Own - complementary trade-off frame
- What Native Provider Agent Primitives Give You Over a Framework - native SDK alternative
Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana
@solana/web3.js+@solana/spl-token.