Model and Provider Rules: Routing, Fallback, and Cost Discipline

These rules keep model identity, provider routing, fallback, and spend as runtime policy instead of scattered constants and tribal knowledge.

Multi-turn tool use multiplies tokens quickly.

Without provider-agnostic design, predeclared fallbacks, and cost budgets, the first real traffic week becomes either an outage or an invoice incident.

How to Use This List

• Apply the policy-interface rules before optimizing any single model choice.
• Require numeric budgets (per run, per tenant, per key) before enabling production traffic.
• Re-run this list when you add a route, change a gateway, or raise autonomy.
• Pair with OpenRouter cost and fallback guides when you route through a multi-model gateway.

Policy Interface Rules

1. Resolve models through a logical policy or tier, not raw ids in business logic. Call sites ask for fast, balanced, strong, or a named route (triage, planner, coder).
2. Keep one config map as the source of truth for id mappings. Changing a model must not require hunting string literals across the repo.
3. Support runtime override only as a time-boxed experiment path. Permanent overrides become shadow hardcodes; promote winners into config.
4. Inject the resolver so tests can pin a fake model without patching globals. Testability is part of the rule, not an afterthought.
5. Separate difficulty policy from latency policy when needed. A slow batch job may use a strong tier; interactive chat may require a fast tier even at lower quality.

Cost Discipline Rules

6. Default to the cheapest tier that meets the quality bar. Escalate with sample evidence, not "safer feels better."
7. Enforce a per-run token or dollar budget in host code. Prompts that say "be frugal" do not stop a loop.
8. Enforce per-key and per-tenant spend caps at the gateway or billing layer. Key-level caps catch runaway agents when app logic fails.
9. Account for turns times tools times model price in capacity planning. Single-call demos understate multi-turn cost by an order of magnitude.
10. Log tokens in, tokens out, estimated cost, policy name, and resolved model id per run. Cost retros without those fields are guesswork.
11. Route cheap subtasks (classification, extraction, routing) to cheap tiers. Reserve frontier models for steps that evals prove need them.
12. Re-eval tool-calling quality whenever the model map changes. Tiers are not interchangeable without proof on your tool schemas.

Routing and Portability Rules

13. Prefer provider-portable prompts and tool schemas in core domain code. Thin adapters absorb vendor-specific request shapes.
14. Do not bake one vendor's proprietary tool IR into business logic. Portability dies when the domain imports a single SDK's objects everywhere.
15. Use a multi-model gateway when you need many providers; still keep app-level policies. Gateway routing does not replace logical tier names in your code.
16. Record data-retention and training policies per provider when handling sensitive content. Cost-optimal routes that retain prompts may be out of policy.
17. Prefer explicit provider order when latency or compliance requires it. Silent "auto" routing is fine for demos; production needs documented preferences.

Fallback and Resilience Rules

18. Predeclare an acyclic fallback chain per policy. Improvised swaps during an outage are not a strategy.
19. Fail over on transport and availability errors; do not blindly fail over on every quality miss. Quality issues need evals and canaries, not infinite provider hopping.
20. Budget fallback attempts. Fallback storms (strong → balanced → strong) amplify cost and latency.
21. Surface degraded mode to operators and, when appropriate, users. Silent quality cliffs after fallback destroy trust.
22. Canary or shadow traffic when changing primary model or provider order. Compare success and cost before full cutover.
23. Keep kill switches for spend and provider routes. You must be able to disable a bad route without a full deploy theater.

Decision Cheatsheet

Minimal Policy Shape

from dataclasses import dataclass
 
 
@dataclass(frozen=True)
class ModelRef:
    provider: str
    model_id: str
 
 
POLICY = {
    "fast": ModelRef("openrouter", "meta-llama/llama-3.3-70b-instruct"),
    "balanced": ModelRef("openrouter", "anthropic/claude-sonnet-4.5"),
    "strong": ModelRef("openrouter", "openai/gpt-5"),
}
FALLBACK = {"strong": "balanced", "balanced": "fast"}  # acyclic
 
 
def resolve(policy: str) -> ModelRef:
    return POLICY[policy]

Model ids are illustrative; verify current provider and OpenRouter names at build time.

FAQs

Related

• How Opinionated Rules Prevent Agent Production Incidents - why policy beats ad hoc model choice.
• The Core Agent Rules: A Quick-Reference List - condensed tables including this category.
• Observability and Eval Rules for Production Agent Systems - logging and eval gates that make cost discipline real.
• AI Agent Rules Best Practices - checklist form including cost and routing items.
• Model-Swapping Rules for Cost and Capability Flexibility - fundamentals-level model policy.
• Cost-Based Routing: Sending Cheap Tasks to Cheap Models - gateway-level cost routing.
• How OpenRouter's Provider Fallback Actually Protects Uptime - provider fallback mechanics.
• Setting Spend Limits and Usage Caps per API Key - key-level spend controls.

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.