What Actually Bottlenecks an Agent System at Scale
Web apps usually hit CPU, memory, or database connections first.
//
Search across all documentation pages
Web apps usually hit CPU, memory, or database connections first.
Agent systems hit provider rate limits, tool concurrency, and per-request cost long before the worker host looks busy.
Scaling agents is therefore less about adding pods and more about shaping demand, sharing budgets, and making each run cheaper.
A classic API service often scales like this:
An agent run usually:
Each step has a different bottleneck profile. CPU on the worker is rarely the first wall.
| Resource | What saturates first | Symptom at scale |
|---|---|---|
| Model provider | RPM, TPM, concurrent streams | 429/503, rising p95 of model calls |
| Tool backends | Connection pools, vendor rate limits, write locks | Timeouts, partial mutations, circuit trips |
| Money | Budget per day/tenant/run | Kill switches, degraded quality, finance alerts |
| Context / memory | Token window growth across turns | Truncation, quality drop, higher cost per turn |
| Queue / workers | Job backlog, worker concurrency | Long wait before first token |
| Host compute | CPU/GPU for local models or sandboxes | High load average (often last) |
If you only autoscale workers on CPU, you will often scale into rate limits and cost ceilings.
Each new worker can:
Without a shared concurrency budget and a queue that absorbs spikes, horizontal scale is a load amplifier, not a cure.
Providers publish or enforce:
Agent loops burn these quickly because one user goal is many model calls.
Example shape:
That is not one HTTP hit against the provider. It is a burst of RPM and TPM for a single product request.
At fleet scale, correlated bursts (product launches, morning batch jobs, retries after an outage) create retry storms. Cascading retries are covered in Why Agents Fail Differently Than Traditional Services.
Tools are often the real wall:
Even if the model provider has headroom, five parallel agent runs each opening three tools can exhaust a bulkhead.
Bulkheads (separate concurrency pools per dependency) keep one hot tool from starving the whole worker. They pair with Circuit Breakers for Agent Tool Calls.
Cost is a first-class bottleneck for agents.
A 2x traffic increase can be a 5-20x bill increase if:
Measure cost per successful run and cost per accepted user outcome, not only cost per 1k tokens.
When budget pressure rises, cost-aware routing and caching matter more than another replica. See Cost-Aware Scaling: Routing Load to Cheaper Models Under Pressure and Caching Agent Responses and Tool Results to Cut Redundant Calls.
A queue decouples intake from processing.
Without admission rules, the queue only delays the same overload:
Healthy systems combine:
| Pressure | What fails next if you ignore it |
|---|---|
| Hit RPM hard | Retries stack; TPM still rises; cost spikes |
| Cap model concurrency only | Tools thrash; partial side effects |
| Cap tools only | Model wait time grows; user latency SLOs burn |
| No cache | Identical FAQ traffic burns TPM needlessly |
| No cost SLOs | Finance discovers the outage first |
Scaling is a control problem: which gauge do you protect, and what do you shed?
One tenant with a runaway loop can consume the shared RPM budget.
Mitigations:
Specialists multiply concurrent calls.
A research graph with planner + 3 workers + critic can open several model streams and many tools for one user goal. Cap concurrency at the graph level, not only per leaf tool.
Self-hosted models shift the bottleneck to GPU memory and batching.
They remove some vendor RPM ceilings but introduce:
Hybrid fleets (cheap remote for easy tasks, local for private/hot paths) need routing policy, not only capacity.
| Approach | Strength | Weakness |
|---|---|---|
| Scale workers only | Familiar k8s pattern | Hits rate limits and cost first |
| Queue + concurrency caps | Protects providers and spend | Higher queue latency under load |
| Always-premium models | Max quality headroom | Linear (or worse) cost scale |
| Aggressive cache + cheaper models | Throughput without bill panic | Staleness and quality risk if mis-keyed |
| Reject overload early | Honest UX and stable SLOs | Product must handle 429/queue-full |
Prefer shared budgets + queues + cost controls over replica count as the primary dial.
A scaled agent system:
Uptime of workers is necessary. It is not the definition of scale.
Provider rate limits (RPM/TPM) and unexpected spend, often in the same week traffic grows.
Trace each step. If model calls return 429 while tools are fine, fix model concurrency. If tools time out with green model metrics, fix tool bulkheads.
Usually no at scale. Prefer enqueue + worker + status channel so intake never blocks on multi-minute loops.
It shifts ops pain (cold starts, max duration) but still hits the same provider, tool, and cost ceilings. See deploying-agents content on serverless mismatch.
Cache hits skip model and tool work, which frees RPM, TPM, tool slots, and budget for novel traffic.
Sustained 429 rate, queue depth SLO burn, and cost burn rate - not only worker CPU.
They can improve quality through specialization, but they usually increase concurrent demand unless you share a hard budget across the graph.
Uncoordinated retries convert a blip into a storm. Shared budgets and breakers are scale features, not only reliability features.
Yes for long-running agents. Larger contexts cost more and slow generation; context discipline is capacity planning.
At worker process, shared Redis/semaphore, queue prefetch, and provider-side if available. Details in Concurrency Limits and Backpressure for Agent Workers.
Define success, latency, and cost SLOs so scale decisions protect user outcomes, not only replica count. See Setting SLOs for Agent Latency, Success Rate, and Cost.
HTTP intake that enqueues jobs, a worker pool with low concurrency, structured results store, and metrics for queue depth, 429s, and cost per run. Expand in Architecture Guide: A Horizontally Scalable Agent Worker Pool.
Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana
@solana/web3.js+@solana/spl-token.
Reviewed by Chris St. John·Last updated Jul 16, 2026