Model and Provider Rules: Routing, Fallback, and Cost Discipline
These rules keep model identity, provider routing, fallback, and spend as runtime policy instead of scattered constants and tribal knowledge.
Multi-turn tool use multiplies tokens quickly.
Without provider-agnostic design, predeclared fallbacks, and cost budgets, the first real traffic week becomes either an outage or an invoice incident.
How to Use This List
- Apply the policy-interface rules before optimizing any single model choice.
- Require numeric budgets (per run, per tenant, per key) before enabling production traffic.
- Re-run this list when you add a route, change a gateway, or raise autonomy.
- Pair with OpenRouter cost and fallback guides when you route through a multi-model gateway.
Policy Interface Rules
- Resolve models through a logical policy or tier, not raw ids in business logic. Call sites ask for
fast,balanced,strong, or a named route (triage,planner,coder). - Keep one config map as the source of truth for id mappings. Changing a model must not require hunting string literals across the repo.
- Support runtime override only as a time-boxed experiment path. Permanent overrides become shadow hardcodes; promote winners into config.
- Inject the resolver so tests can pin a fake model without patching globals. Testability is part of the rule, not an afterthought.
- Separate difficulty policy from latency policy when needed. A slow batch job may use a strong tier; interactive chat may require a fast tier even at lower quality.
Cost Discipline Rules
- Default to the cheapest tier that meets the quality bar. Escalate with sample evidence, not "safer feels better."
- Enforce a per-run token or dollar budget in host code. Prompts that say "be frugal" do not stop a loop.
- Enforce per-key and per-tenant spend caps at the gateway or billing layer. Key-level caps catch runaway agents when app logic fails.
- Account for turns times tools times model price in capacity planning. Single-call demos understate multi-turn cost by an order of magnitude.
- Log tokens in, tokens out, estimated cost, policy name, and resolved model id per run. Cost retros without those fields are guesswork.
- Route cheap subtasks (classification, extraction, routing) to cheap tiers. Reserve frontier models for steps that evals prove need them.
- Re-eval tool-calling quality whenever the model map changes. Tiers are not interchangeable without proof on your tool schemas.
Routing and Portability Rules
- Prefer provider-portable prompts and tool schemas in core domain code. Thin adapters absorb vendor-specific request shapes.
- Do not bake one vendor's proprietary tool IR into business logic. Portability dies when the domain imports a single SDK's objects everywhere.
- Use a multi-model gateway when you need many providers; still keep app-level policies. Gateway routing does not replace logical tier names in your code.
- Record data-retention and training policies per provider when handling sensitive content. Cost-optimal routes that retain prompts may be out of policy.
- Prefer explicit provider order when latency or compliance requires it. Silent "auto" routing is fine for demos; production needs documented preferences.
Fallback and Resilience Rules
- Predeclare an acyclic fallback chain per policy. Improvised swaps during an outage are not a strategy.
- Fail over on transport and availability errors; do not blindly fail over on every quality miss. Quality issues need evals and canaries, not infinite provider hopping.
- Budget fallback attempts. Fallback storms (strong → balanced → strong) amplify cost and latency.
- Surface degraded mode to operators and, when appropriate, users. Silent quality cliffs after fallback destroy trust.
- Canary or shadow traffic when changing primary model or provider order. Compare success and cost before full cutover.
- Keep kill switches for spend and provider routes. You must be able to disable a bad route without a full deploy theater.
Decision Cheatsheet
| Situation | Default action |
|---|---|
| New agent call site | Request a policy name, never embed a model id |
| Quality not meeting bar | Escalate tier with eval evidence on a fixed suite |
| Cost too high | De-escalate cheap steps first; re-eval; cut turns and tools next |
| Provider outage | Follow predeclared fallback; alert; do not invent chains live |
| Sensitive data | Restrict providers by data policy before optimizing price |
| Experiment | Time-boxed override + promote via config if it wins |
Minimal Policy Shape
from dataclasses import dataclass
@dataclass(frozen=True)
class ModelRef:
provider: str
model_id: str
POLICY = {
"fast": ModelRef("openrouter", "meta-llama/llama-3.3-70b-instruct"),
"balanced": ModelRef("openrouter", "anthropic/claude-sonnet-4.5"),
"strong": ModelRef("openrouter", "openai/gpt-5"),
}
FALLBACK = {"strong": "balanced", "balanced": "fast"} # acyclic
def resolve(policy: str) -> ModelRef:
return POLICY[policy]Model ids are illustrative; verify current provider and OpenRouter names at build time.
FAQs
Is OpenRouter required to follow these rules?
No.
It is one convenient multi-model gateway.
The rules are logical policies, budgets, and fallbacks in your system; transport can be direct APIs or a gateway.
How many tiers do most teams need?
Two or three (fast / balanced / strong) cover most designs.
Add named routes when a step has a special quality or compliance bar.
Should planners and executors use different models?
Often yes.
Validate with evals; do not assume a stronger planner always wins.
What belongs in config versus code?
Model ids, fallbacks, budgets, and default policies belong in config.
Agent goal logic and tool handlers belong in code that requests policies by name.
How do I know a cheaper tier is safe?
Run the same golden tasks and production samples; compare task success and safety metrics, not vibe.
What if fallback lowers quality unacceptably?
Define a degraded mode: reduce autonomy, queue work, or fail closed with a clear operator alert rather than silently serving bad answers.
Do spend limits live in the app or at the provider?
Both when possible.
App per-run budgets catch loop waste; provider or key caps catch total runaway spend.
How often should we revisit the model map?
On any material quality or cost incident, when providers change pricing or retention policy, and on a fixed quarterly (or similar) review.
Are these rules only for large fleets?
No.
Even one multi-turn agent benefits from a policy name, a budget, and a fallback.
How do these relate to fundamentals model-swapping rules?
Fundamentals introduce policy-based selection for early agents.
This page adds production cost discipline, provider routing, and fallback operations for tech leads.
What about latency-aware routing?
Treat latency as a first-class axis: some routes optimize for p95 response time, others for cost or quality.
Document which axis each policy optimizes.
Can we hardcode a model for a regulated path?
Yes as an explicit, reviewed exception with owner and rationale - not as an unlogged special case in one handler.
Related
- How Opinionated Rules Prevent Agent Production Incidents - why policy beats ad hoc model choice.
- The Core Agent Rules: A Quick-Reference List - condensed tables including this category.
- Observability and Eval Rules for Production Agent Systems - logging and eval gates that make cost discipline real.
- AI Agent Rules Best Practices - checklist form including cost and routing items.
- Model-Swapping Rules for Cost and Capability Flexibility - fundamentals-level model policy.
- Cost-Based Routing: Sending Cheap Tasks to Cheap Models - gateway-level cost routing.
- How OpenRouter's Provider Fallback Actually Protects Uptime - provider fallback mechanics.
- Setting Spend Limits and Usage Caps per API Key - key-level spend controls.
Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana
@solana/web3.js+@solana/spl-token.