Managing Conversation History Within a Context Budget
Keep multi-turn agents inside the model context window without silently deleting goals, constraints, or the last useful tool results.
Summary
Use a measured token (or char) budget, protect the system prompt and recent turns, then drop or summarize older middle turns so each model call stays under the limit with room for the reply.
Recipe
- Pick a hard input budget: model context minus max output minus safety margin (tools, RAG, scratchpad).
- Always keep: system / policy messages, the latest user turn, and the last K turns (or last tool-call chain the provider requires).
- Measure history size before each model call (tokenizer when available; char heuristic for demos).
- While over budget: drop oldest non-protected messages, or replace a contiguous older span with one summary message.
- Log what you dropped or summarized (counts, not full PII) so debugging remains possible.
- Re-test long chats: goal retention, tool-call validity, and cost per turn.
Working Example
from __future__ import annotations
# Demo heuristic: ~4 chars per token. Replace with a real tokenizer in production.
def estimate_tokens(messages: list[dict]) -> int:
total = 0
for m in messages:
total += len(m.get("content") or "") // 4 + 4 # role overhead fudge
return total
def trim_to_budget(
messages: list[dict],
max_input_tokens: int = 3000,
keep_last: int = 6,
) -> list[dict]:
system = [m for m in messages if m.get("role") == "system"]
rest = [m for m in messages if m.get("role") != "system"]
if estimate_tokens(system + rest) <= max_input_tokens:
return system + rest
# Protect recent dialogue; drop from the oldest side of the middle
protected = rest[-keep_last:] if keep_last else []
head = rest[:-keep_last] if keep_last else rest[:]
while head and estimate_tokens(system + head + protected) > max_input_tokens:
head.pop(0)
if estimate_tokens(system + head + protected) > max_input_tokens:
# Last resort: shrink protected from the oldest end
while len(protected) > 1 and estimate_tokens(system + protected) > max_input_tokens:
protected.pop(0)
return system + head + protected
history = [
{"role": "system", "content": "You are a brief support agent."},
{"role": "user", "content": "I need help with order 99."},
{"role": "assistant", "content": "I can help. What is wrong?"},
# ... many turns later ...
{"role": "user", "content": "Summarize my issue in one line."},
]
bounded = trim_to_budget(history, max_input_tokens=500, keep_last=4)For lossy but safer retention, replace head with a single synthetic summary instead of deleting it (see summarization article).
Deep Dive
Budget composition
Think in layers that compete for the same window:
| Layer | Typical contents | Budget stance |
|---|---|---|
| Policy | System rules, tool schemas (if counted as input) | Fixed, rarely trimmed |
| Session memory | Scratchpad JSON, retrieved long-term facts | Small, controlled |
| History | User / assistant / tool messages | Elastic - primary trim target |
| Generation reserve | Room for the model reply | Reserved, never fill 100% of context |
If you only measure "messages before tools," tool results will blow the limit on the next turn.
Count the full payload you will send.
Truncation strategies
- Sliding window (message count): simple; may drop critical early constraints.
- Token budget with protected tail: production default for chat.
- Role-aware drop: drop old assistant small-talk before user constraints; never drop an unpaired tool-call sequence required by the API.
- Middle-out: keep head (goal) + tail (recent), compress the middle - usually via summary.
When to summarize vs delete
Delete chit-chat and redundant acknowledgments.
Summarize when the span holds decisions, ids, or user constraints.
Summaries are also model calls: budget their cost and run them asynchronously when possible.
Tool-call integrity
Many providers require that every tool message remains adjacent to its assistant tool_calls message.
Trimming one without the other causes API errors.
Trim in atomic groups (assistant tool-call turn + tool results) rather than single messages.
Measurement
Prefer the provider tokenizer or a matching open tokenizer.
Char/4 is a teaching approximation only.
Unit-test with long synthetic histories so budget bugs appear in CI, not in the longest customer chat.
Gotchas
- Forgetting output reserve. A full input window leaves no room for the reply and causes hard failures or silent truncation depending on the API.
- Breaking tool-call pairs. Trimming a tool result but leaving the call (or the reverse) yields 4xx errors.
- Dropping the goal. Aggressive windows forget the user's original constraint; pin goals on a scratchpad.
- Double-counting RAG. Retrieved chunks added after trim can re-overflow the budget.
- Token estimator drift. Heuristics diverge from real billing tokens; re-check with production logs.
- Summarizing without a marker. Models cannot tell summary from primary source; prefix clearly (
[summary of earlier turns]). - Per-user fairness. One runaway session can dominate cost if you only cap per request, not per session.
Alternatives
| Approach | Pros | Cons |
|---|---|---|
| Sliding message window | Easy to implement | Drops early facts |
| Token budget + protected tail | Predictable size | Needs measurement code |
| Periodic summarization | Keeps long goals | Summary errors; extra cost |
| Full history every turn | Perfect recall until limit | Cost and latency explode |
| Framework checkpoint pruning | Integrated with graph state | Learn framework semantics |
FAQs
What is a good default for keep_last?
Start with the last 6-12 conversational turns or the last complete tool chain, then tune with traces. Product chats often need more human turns; tool-heavy agents need intact call groups.
Should I trim before or after adding the new user message?
Add the new user message first, then trim so the latest request is never dropped in favor of older filler.
Can I store full history and only trim for the model?
Yes - and you should. Keep an audit/log copy (with retention policy) separate from the prompt projection.
How do I handle images or large tool payloads?
Store blobs outside the transcript; leave short references or hashes in history. Re-fetch into cache when a later step needs detail.
Does a larger context model remove the need for this?
It delays pressure but does not remove cost, latency, or distraction from irrelevant early turns.
Where does scratchpad fit in the budget?
Count it as session memory. Prefer small structured fields over pasting the entire log into the scratchpad.
How often should I summarize?
When remaining budget crosses a threshold (for example 70% full) or every N turns - not on every message unless chats are huge.
What if trim removes a safety refusal earlier in the chat?
Safety policy belongs in the system layer you never trim, not only in past assistant messages.
Related
Related: Short-Term Memory Basics
Related: Summarization Strategies for Long-Running Conversations
Related: Short-Term Memory: What an Agent Actually Needs Within One Run
Related: Short-Term Memory Best Practices
Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana
@solana/web3.js+@solana/spl-token.