How an LLM's Context Window Bounds What an Agent Can Do
An agent is only as capable as what fits in the model call at each step.
//
Search across all documentation pages
An agent is only as capable as what fits in the model call at each step.
The context window is that hard limit: system instructions, user goals, tool schemas, prior messages, tool results, and any retrieved memory all compete for the same token budget.
When the window fills, the agent does not "remember more cleverly."
It either truncates, summarizes, fails, or starts making decisions from an incomplete picture.
A context window is the maximum number of tokens a model can accept as input (and sometimes count toward combined input plus output, depending on the provider).
Tokens are subword pieces, not words, so English text often lands around a few hundred tokens per page of prose, while JSON tool dumps and source code can be denser.
Everything the model uses to decide the next action must fit inside that window on that call.
For a chat completion this is mostly conversation history.
For an agent it is a growing stack: system prompt, tool definitions, current goal, earlier reasoning or messages, tool results, retrieved documents, and sometimes explicit scratchpads or plans.
That stack is the agent's working memory for the turn.
Nothing outside it is visible unless you re-inject it on a later call.
A useful analogy is a desk of fixed size.
The agent can only rearrange papers that fit on the desk.
Filing cabinets (databases, vector stores, logs) exist, but the agent only works with whatever you pull onto the desk right now.
Frontier models commonly advertise windows from roughly 128K to 1M+ tokens (verify exact product limits at build time).
Those numbers sound huge until an agent runs ten tool calls that each return multi-kilobyte JSON, or until a research agent pastes full web pages into the thread.
Agents fail on context more often than teams expect because the failure is gradual: quality drifts, tools get re-called, and goals get restated incorrectly long before a hard "context length exceeded" error appears.
Every agent loop iteration typically re-sends some form of the full working state to the model.
That means cost and context pressure both grow with turns, not just with the final answer length.
Tool results are often the heaviest items.
A search snippet, a database row set, a file read, or a browser page can dwarf the original user prompt.
If your loop appends every tool result forever, the window becomes a landfill of past observations the model must re-read on every subsequent step.
Tool schemas also consume budget.
Each function name, description, and JSON Schema parameter block is part of the prompt.
Agents with twenty tools can spend a non-trivial share of the window on capability ads before the user goal even appears.
Conversation history is the third major consumer.
User messages, assistant thoughts or replies, and tool-call bookkeeping all accumulate.
Without compaction, a long-running agent will either hit the limit or force the runtime to drop older turns.
When the budget is exceeded, runtimes choose one of a few strategies:
| Strategy | What happens | Typical effect |
|---|---|---|
| Hard fail | API returns a context-length error | Loud failure; operator must shrink inputs |
| FIFO drop | Oldest messages removed first | Agent "forgets" early goals or constraints |
| Keep system + recent | Drop middle turns, keep prompt and tail | Better short-term coherence; loses mid-run facts |
| Summarize | Compress older turns into a digest | Saves space; can lose detail or introduce errors |
| Retrieve on demand | Store history externally; pull only relevant slices | Scales better; needs retrieval design |
None of these is free.
Dropping context loses facts.
Summarizing can invent or omit critical constraints.
Retrieval can miss the one detail that mattered.
The mechanical truth is simple: the agent cannot act on what is not present in the window at decision time.
That is why context design is part of agent architecture, not an afterthought for "memory features."
Treat the context window as a product of three budgets: fixed overhead, variable working set, and headroom for the model's next reply.
Fixed overhead is system instructions, tool schemas, safety rules, and stable policy text.
Variable working set is the current goal, recent turns, active tool results, and retrieved passages.
Headroom is tokens reserved so the model can still emit a plan, tool call, or final answer without overflowing.
If fixed overhead alone is large, the agent has little room to work even on short tasks.
That is a common failure mode of "kitchen-sink" system prompts and oversized tool catalogs.
For long-running agents, move durable state out of the window and into structured stores.
Keep plans, open todos, entity facts, and citations in external memory, then re-inject only the slice needed for the next decision.
This is the practical boundary between short-term conversational context and longer-term agent memory.
For tool-heavy agents, shrink tool results before they re-enter context.
Prefer filtered fields, row limits, summaries with links to full artifacts, and structured extracts over raw dumps.
A 50-line tool result the model can scan beats a 5,000-line log that crowds out the goal.
For research or RAG agents, retrieval is not a free lunch.
Every retrieved chunk is a conscious spend of context budget.
Rank, cap, and cite; do not dump the top-20 passages because "more context is safer."
| Approach | Strength | Weakness | Best fit |
|---|---|---|---|
| Giant window, naive append-all | Simple to build | Cost explodes; quality drifts | Short demos only |
| Sliding window of recent turns | Predictable size | Forgets early constraints | Short interactive agents |
| Periodic summarization | Extends horizon | Summary errors compound | Medium multi-turn workflows |
| External state + selective rehydrate | Scales and stays controllable | More engineering | Production multi-step agents |
Also budget for reasoning or "thinking" tokens when a provider meters them separately or counts them against the window.
Extended reasoning can improve hard decisions while silently eating the same scarce space the tools need.
Finally, couple context discipline with stopping conditions.
An agent that cannot finish within a safe context envelope should stop, checkpoint, or escalate rather than thrash until the window collapses.
It is the model's maximum token capacity for the text (and multimodal inputs, if supported) it can process in one request.
For agents, that capacity is shared by instructions, tools, history, retrieval, and the space needed for the next response.
Tokens are model-specific pieces of text, often subwords or punctuation units.
A rough English rule of thumb is that one token is often around three-quarters of a word, but code and JSON can tokenize very differently, so measure with the provider's tokenizer when precision matters.
Agents re-send growing histories and often inject large tool outputs on every turn.
A chatbot reply might be one short exchange; an agent run can be dozens of tool-augmented steps.
It raises the ceiling, but it does not design the memory system.
Without truncation, summarization, or external state, a larger window just delays the same packing problem and raises cost.
Put stable policies, role, tool-use rules, and non-negotiable constraints in the system prompt.
Put run-specific facts, intermediate artifacts, and long documents outside the prompt and rehydrate only when needed.
Each tool's name, description, and parameter schema is part of the prompt tokens.
Large tool catalogs shrink the budget available for the actual task and can confuse tool selection.
No.
Keep what the next decision needs: the goal, active constraints, recent tool outcomes, and any open plan steps.
Archive or summarize the rest.
Context is the ephemeral working set sent on a given model call.
Long-term memory is durable storage (files, databases, vector indexes) that only becomes useful when selected content is loaded back into context.
Models still produce an answer when the window is crowded with low-value history, but they may ignore early constraints, repeat work, or miss a buried fact.
The hard error is only the final failure mode.
Cap, filter, and summarize at the tool boundary.
Return identifiers or artifact URLs for full data, and keep only the fields the model needs to choose the next action.
Often yes, in the sense that images, audio, or files are converted into token-equivalent cost that competes with text.
Exact accounting varies by provider, so verify the current metering rules for your model.
Larger prompts mean more input tokens billed on every turn.
A loop that re-sends a bloated history multiplies cost even if each new user message is short.
Measure a real run: log tokens for system prompt, tools, history, and tool results separately.
Then cut the largest bucket first instead of guessing.
Retrieval reduces what must stay resident, but it does not remove the need for a coherent working set.
You still need enough window to hold the query, selected evidence, and the model's next action cleanly.
Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana
@solana/web3.js+@solana/spl-token.
Reviewed by Chris St. John·Last updated Jul 16, 2026