Trade-off Guide: Managed Gateway vs Direct Provider Integration
Agents rarely call "a model" in the abstract.
They call an access path: a managed multi-model gateway (for example OpenRouter) or a direct provider/cloud API.
This trade-off guide helps you pick a path (or hybrid), encode it in config, and write an ADR that future migrations can trust.
Summary
Choose a managed gateway when multi-model routing, failover, and one client surface matter more than minimal hops and single-vendor contracts; choose direct when compliance, IAM, exclusive features, or committed discounts dominate - and document a hybrid when both are true.
Recipe
- List models and providers you need this quarter (real list, not aspirational).
- List hard constraints: subprocessors, regions, ZDR, air-gap, procurement.
- Estimate engineering cost of N direct SDKs vs one OpenAI-compatible client.
- Model cost: inference + platform fees + eng time (verify fees at build).
- Decide primary path: gateway, direct, or hybrid (primary + backup).
- Define fallback behavior on 429/5xx and on policy blocks.
- Pin base URL, auth, and model roles in config - not only in chat lore.
- Write the ADR with revisit triggers (volume, legal, incident, fee change).
Working Example
A small config module that makes the access-path decision explicit.
from dataclasses import dataclass
from enum import Enum
class AccessPath(str, Enum):
GATEWAY = "managed_gateway" # e.g. OpenRouter
DIRECT = "direct_provider"
HYBRID = "hybrid"
@dataclass
class InferenceAccess:
path: AccessPath
base_url: str
default_model_role: str
notes: str
def choose_access(
providers_needed: int,
legal_allows_gateway: bool,
needs_cross_provider_failover: bool,
) -> InferenceAccess:
if not legal_allows_gateway:
return InferenceAccess(
path=AccessPath.DIRECT,
base_url="https://api.openai.com/v1", # example; pin per env
default_model_role="mid_default",
notes="Gateway subprocessor blocked; direct only.",
)
if providers_needed >= 2 or needs_cross_provider_failover:
return InferenceAccess(
path=AccessPath.GATEWAY,
base_url="https://openrouter.ai/api/v1",
default_model_role="mid_default",
notes="One client, multi-model routing and failover.",
)
return InferenceAccess(
path=AccessPath.DIRECT,
base_url="https://api.anthropic.com", # illustrative
default_model_role="mid_default",
notes="Single provider this quarter; revisit if eval matrix grows.",
)
print(choose_access(3, True, True))Exact endpoints, headers, and model slugs change - verify at build against vendor and OpenRouter docs.
Deep Dive
What a managed gateway buys agents
| Capability | Why agents care |
|---|---|
| One OpenAI-compatible surface | Frameworks and thin loops swap models with config |
| Broad model catalog | Eval harness and tiered routing without N SDKs |
| Cross-provider failover | Host can continue when one upstream is down |
| Central activity views | Faster cost attribution while you build full tracing |
| Optional BYOK | Use your provider accounts through the gateway API |
OpenRouter is the common example in this guide (~315+ models in the category pin set - verify current catalog at build).
What direct integration buys agents
| Capability | Why agents care |
|---|---|
| Fewest hops / earliest betas | Latency and brand-new tool features |
| Enterprise MSA and IAM | Cloud org policies, VPC endpoints, committed use |
| Clear subprocessor story | Security questionnaires and DPAs |
| Provider-native agent APIs | Handoffs, batch, or proprietary tool protocols |
| No gateway platform fee | When volume and discounts dominate |
Dimension comparison
| Dimension | Managed gateway | Direct provider |
|---|---|---|
| Integration | One client | One client per vendor (or glue) |
| Model swap | Change slug/role | Often new options and auth |
| Failover | Often built-in multi-host | You build multi-home |
| Latency | Extra hop (often small vs decode) | Minimal hop |
| Cost shape | Pass-through inference + platform/credit fees | List or commit pricing |
| Compliance | Gateway + upstream chain | Shorter chain |
| Feature lag | Possible lag on betas | Earliest on that vendor |
| Lock-in | Lower API lock-in; still model-behavior lock-in | Higher API lock-in risk |
Hybrid patterns worth documenting
| Pattern | When |
|---|---|
| Direct primary, gateway backup | Existing vendor prod + failover need |
| Gateway primary, BYOK for top model | Unified API + volume on one account |
| Gateway for dev/eval, direct for prod | Research speed vs regulated launch |
| Per-tenant path | Some tenants require named provider only |
Hybrids need explicit retry and data-policy rules so traffic does not "accidentally" leave the approved path.
Cost modeling (honest)
Do not compare sticker token prices alone.
Include:
- Credit purchase or BYOK fees on gateways (verify at build).
- Engineering time for multi-SDK auth, retries, and normalization.
- Incident cost of single-provider outages.
- Eval cost of testing N models (gateway usually cheaper here).
Agent-specific failure modes
| Failure | Gateway angle | Direct angle |
|---|---|---|
| Provider outage | Route to alternate host/model | Hard fail unless multi-homed |
| Policy block (data) | Filters may refuse a route | Contractual controls per vendor |
| Rate limits | Platform + upstream | Upstream (+ your quota) |
| Schema drift across models | Normalize carefully | Still your problem per API |
| Secret sprawl | One gateway key (+ BYOK) | Many vendor keys |
ADR fields unique to this decision
- Approved path(s) and environments (dev/stage/prod).
- Data retention / training / ZDR assumptions as of date.
- Fallback chain and whether fallback may change providers.
- Who pays which invoice (credits vs cloud commit).
- Revisit triggers: monthly spend, legal review, SEV from provider outage.
Gotchas
- "Gateway removes lock-in completely" - prompts and tool behavior still model-lock.
- "Direct is always cheaper" - eng time and outages have cost too.
- Hardcoding model marketing names in business logic - use roles + config.
- Silent hybrid without policy - traffic violates the DPA you thought you had.
- Ignoring free-tier data policies - free endpoints often log more; agents may send customer text.
- Framework default base URL left in examples - prod still points at the wrong host.
- No eval when swapping providers - "compatible API" is not identical behavior.
- Stacking retries at SDK, gateway, and agent loop - define one budget.
Alternatives
| Approach | Best for | Trade-off |
|---|---|---|
| Managed gateway only | Multi-model agents, small teams | Subprocessor + fees |
| Direct single provider | Simple prod, strong MSA | Weak multi-model story |
| Direct multi-provider DIY | Full control, large platform team | You own normalization |
| Cloud model garden (Bedrock/Vertex/Azure) | Enterprise IAM and regions | Catalog and UX differ |
| Self-hosted / local models | Privacy, offline, marginal cost | Quality and ops variance |
| Hybrid (documented) | Mixed legal and product needs | Complexity in routing policy |
FAQs
Is OpenRouter required to use multiple models?
No. It is one managed way. You can also multi-home direct APIs or use cloud gardens.
Does a gateway replace agent observability?
No. Keep run traces (tools, stop reasons, cost) in your system; use gateway activity as a supplement.
When is BYOK the right compromise?
When you want one API shape but billing, limits, or compliance must stay on your provider accounts. Confirm fee thresholds at build.
Can legal reject gateways but still allow direct OpenAI?
Yes. That is a common enterprise outcome. Encode it as a hard force in the ADR.
Should eval harnesses use the same path as production?
Prefer the same path for parity, or document intentional differences (for example gateway in CI, direct in prod) and test both when they diverge.
How do agent frameworks affect this choice?
Most accept any OpenAI-compatible base URL. Framework neutrality makes gateways easy; it does not forbid direct APIs.
What belongs in config vs ADR?
ADR: path choice and constraints. Config: base URL, keys via secret store, model role map, fallback list.
Is latency always worse on a gateway?
Often slightly higher TTFT, but routing to a healthier region can win. Measure on your workloads.
How often should we revisit this ADR?
After major fee or policy changes, after provider SEVs, when a second production model becomes required, and on a quarterly calendar.
Where can I read a dimension-by-dimension cheatsheet?
See OpenRouter vs Calling Provider APIs Directly for a focused comparison list.
Related
- Why Agent Systems Need Their Own ADR Discipline - ADR habit
- Documenting Model and Provider Decisions for Future Migrations - migration notes
- OpenRouter vs Calling Provider APIs Directly - comparison cheatsheet
- What OpenRouter Actually Is: One API for 300+ Models - gateway overview
- Bring Your Own Key (BYOK): Using Your Own Provider Accounts via OpenRouter - BYOK
- OpenRouter's Data Policies and Why They Matter for Enterprises - data policy
- Why Model Choice Is a Runtime Decision, Not a One-Time One - runtime routing
- Architecture Decisions Best Practices - checklist
Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana
@solana/web3.js+@solana/spl-token.