Managing Conversation History Within a Context Budget

Keep multi-turn agents inside the model context window without silently deleting goals, constraints, or the last useful tool results.

Summary

Use a measured token (or char) budget, protect the system prompt and recent turns, then drop or summarize older middle turns so each model call stays under the limit with room for the reply.

Recipe

Pick a hard input budget: model context minus max output minus safety margin (tools, RAG, scratchpad).
Always keep: system / policy messages, the latest user turn, and the last K turns (or last tool-call chain the provider requires).
Measure history size before each model call (tokenizer when available; char heuristic for demos).
While over budget: drop oldest non-protected messages, or replace a contiguous older span with one summary message.
Log what you dropped or summarized (counts, not full PII) so debugging remains possible.
Re-test long chats: goal retention, tool-call validity, and cost per turn.

Working Example

from __future__ import annotations
 
# Demo heuristic: ~4 chars per token. Replace with a real tokenizer in production.
def estimate_tokens(messages: list[dict]) -> int:
    total = 0
    for m in messages:
        total += len(m.get("content") or "") // 4 + 4  # role overhead fudge
    return total
 
def trim_to_budget(
    messages: list[dict],
    max_input_tokens: int = 3000,
    keep_last: int = 6,
) -> list[dict]:
    system = [m for m in messages if m.get("role") == "system"]
    rest = [m for m in messages if m.get("role") != "system"]
    if estimate_tokens(system + rest) <= max_input_tokens:
        return system + rest
 
    # Protect recent dialogue; drop from the oldest side of the middle
    protected = rest[-keep_last:] if keep_last else []
    head = rest[:-keep_last] if keep_last else rest[:]
    while head and estimate_tokens(system + head + protected) > max_input_tokens:
        head.pop(0)
    if estimate_tokens(system + head + protected) > max_input_tokens:
        # Last resort: shrink protected from the oldest end
        while len(protected) > 1 and estimate_tokens(system + protected) > max_input_tokens:
            protected.pop(0)
    return system + head + protected
 
history = [
    {"role": "system", "content": "You are a brief support agent."},
    {"role": "user", "content": "I need help with order 99."},
    {"role": "assistant", "content": "I can help. What is wrong?"},
    # ... many turns later ...
    {"role": "user", "content": "Summarize my issue in one line."},
]
bounded = trim_to_budget(history, max_input_tokens=500, keep_last=4)

For lossy but safer retention, replace head with a single synthetic summary instead of deleting it (see summarization article).

Deep Dive

Budget composition

Think in layers that compete for the same window:

Layer	Typical contents	Budget stance
Policy	System rules, tool schemas (if counted as input)	Fixed, rarely trimmed
Session memory	Scratchpad JSON, retrieved long-term facts	Small, controlled
History	User / assistant / tool messages	Elastic - primary trim target
Generation reserve	Room for the model reply	Reserved, never fill 100% of context

If you only measure "messages before tools," tool results will blow the limit on the next turn.

Count the full payload you will send.

Truncation strategies

Sliding window (message count): simple; may drop critical early constraints.
Token budget with protected tail: production default for chat.
Role-aware drop: drop old assistant small-talk before user constraints; never drop an unpaired tool-call sequence required by the API.
Middle-out: keep head (goal) + tail (recent), compress the middle - usually via summary.

When to summarize vs delete

Delete chit-chat and redundant acknowledgments.

Summarize when the span holds decisions, ids, or user constraints.

Summaries are also model calls: budget their cost and run them asynchronously when possible.

Tool-call integrity

Many providers require that every tool message remains adjacent to its assistant tool_calls message.

Trimming one without the other causes API errors.

Trim in atomic groups (assistant tool-call turn + tool results) rather than single messages.

Measurement

Prefer the provider tokenizer or a matching open tokenizer.

Char/4 is a teaching approximation only.

Unit-test with long synthetic histories so budget bugs appear in CI, not in the longest customer chat.

Gotchas

Forgetting output reserve. A full input window leaves no room for the reply and causes hard failures or silent truncation depending on the API.
Breaking tool-call pairs. Trimming a tool result but leaving the call (or the reverse) yields 4xx errors.
Dropping the goal. Aggressive windows forget the user's original constraint; pin goals on a scratchpad.
Double-counting RAG. Retrieved chunks added after trim can re-overflow the budget.
Token estimator drift. Heuristics diverge from real billing tokens; re-check with production logs.
Summarizing without a marker. Models cannot tell summary from primary source; prefix clearly ([summary of earlier turns]).
Per-user fairness. One runaway session can dominate cost if you only cap per request, not per session.

Alternatives

Approach	Pros	Cons
Sliding message window	Easy to implement	Drops early facts
Token budget + protected tail	Predictable size	Needs measurement code
Periodic summarization	Keeps long goals	Summary errors; extra cost
Full history every turn	Perfect recall until limit	Cost and latency explode
Framework checkpoint pruning	Integrated with graph state	Learn framework semantics

FAQs

What is a good default for keep_last?

Start with the last 6-12 conversational turns or the last complete tool chain, then tune with traces. Product chats often need more human turns; tool-heavy agents need intact call groups.

Should I trim before or after adding the new user message?

Add the new user message first, then trim so the latest request is never dropped in favor of older filler.

Can I store full history and only trim for the model?

Yes - and you should. Keep an audit/log copy (with retention policy) separate from the prompt projection.

How do I handle images or large tool payloads?

Store blobs outside the transcript; leave short references or hashes in history. Re-fetch into cache when a later step needs detail.

Does a larger context model remove the need for this?

It delays pressure but does not remove cost, latency, or distraction from irrelevant early turns.

Where does scratchpad fit in the budget?

Count it as session memory. Prefer small structured fields over pasting the entire log into the scratchpad.

How often should I summarize?

When remaining budget crosses a threshold (for example 70% full) or every N turns - not on every message unless chats are huge.

What if trim removes a safety refusal earlier in the chat?

Safety policy belongs in the system layer you never trim, not only in past assistant messages.

Related: Short-Term Memory Basics

Related: Summarization Strategies for Long-Running Conversations

Related: Short-Term Memory: What an Agent Actually Needs Within One Run

Related: Short-Term Memory Best Practices

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Summary

Use a measured token (or char) budget, protect the system prompt and recent turns, then drop or summarize older middle turns so each model call stays under the limit with room for the reply.

Recipe

Pick a hard input budget: model context minus max output minus safety margin (tools, RAG, scratchpad).
Always keep: system / policy messages, the latest user turn, and the last K turns (or last tool-call chain the provider requires).
Measure history size before each model call (tokenizer when available; char heuristic for demos).
While over budget: drop oldest non-protected messages, or replace a contiguous older span with one summary message.
Log what you dropped or summarized (counts, not full PII) so debugging remains possible.
Re-test long chats: goal retention, tool-call validity, and cost per turn.

Working Example

from __future__ import annotations
 
# Demo heuristic: ~4 chars per token. Replace with a real tokenizer in production.
def estimate_tokens(messages: list[dict]) -> int:
    total = 0
    for m in messages:
        total += len(m.get("content") or "") // 4 + 4  # role overhead fudge
    return total
 
def trim_to_budget(
    messages: list[dict],
    max_input_tokens: int = 3000,
    keep_last: int = 6,
) -> list[dict]:
    system = [m for m in messages if m.get("role") == "system"]
    rest = [m for m in messages if m.get("role") != "system"]
    if estimate_tokens(system + rest) <= max_input_tokens:
        return system + rest
 
    # Protect recent dialogue; drop from the oldest side of the middle
    protected = rest[-keep_last:] if keep_last else []
    head = rest[:-keep_last] if keep_last else rest[:]
    while head and estimate_tokens(system + head + protected) > max_input_tokens:
        head.pop(0)
    if estimate_tokens(system + head + protected) > max_input_tokens:
        # Last resort: shrink protected from the oldest end
        while len(protected) > 1 and estimate_tokens(system + protected) > max_input_tokens:
            protected.pop(0)
    return system + head + protected
 
history = [
    {"role": "system", "content": "You are a brief support agent."},
    {"role": "user", "content": "I need help with order 99."},
    {"role": "assistant", "content": "I can help. What is wrong?"},
    # ... many turns later ...
    {"role": "user", "content": "Summarize my issue in one line."},
]
bounded = trim_to_budget(history, max_input_tokens=500, keep_last=4)

For lossy but safer retention, replace head with a single synthetic summary instead of deleting it (see summarization article).

Deep Dive

Budget composition

Think in layers that compete for the same window:

Layer	Typical contents	Budget stance
Policy	System rules, tool schemas (if counted as input)	Fixed, rarely trimmed
Session memory	Scratchpad JSON, retrieved long-term facts	Small, controlled
History	User / assistant / tool messages	Elastic - primary trim target
Generation reserve	Room for the model reply	Reserved, never fill 100% of context

If you only measure "messages before tools," tool results will blow the limit on the next turn.

Count the full payload you will send.

Truncation strategies

Sliding window (message count): simple; may drop critical early constraints.
Token budget with protected tail: production default for chat.
Role-aware drop: drop old assistant small-talk before user constraints; never drop an unpaired tool-call sequence required by the API.
Middle-out: keep head (goal) + tail (recent), compress the middle - usually via summary.

When to summarize vs delete

Delete chit-chat and redundant acknowledgments.

Summarize when the span holds decisions, ids, or user constraints.

Summaries are also model calls: budget their cost and run them asynchronously when possible.

Tool-call integrity

Many providers require that every tool message remains adjacent to its assistant tool_calls message.

Trimming one without the other causes API errors.

Trim in atomic groups (assistant tool-call turn + tool results) rather than single messages.

Measurement

Prefer the provider tokenizer or a matching open tokenizer.

Char/4 is a teaching approximation only.

Unit-test with long synthetic histories so budget bugs appear in CI, not in the longest customer chat.

Gotchas

Forgetting output reserve. A full input window leaves no room for the reply and causes hard failures or silent truncation depending on the API.
Breaking tool-call pairs. Trimming a tool result but leaving the call (or the reverse) yields 4xx errors.
Dropping the goal. Aggressive windows forget the user's original constraint; pin goals on a scratchpad.
Double-counting RAG. Retrieved chunks added after trim can re-overflow the budget.
Token estimator drift. Heuristics diverge from real billing tokens; re-check with production logs.
Summarizing without a marker. Models cannot tell summary from primary source; prefix clearly ([summary of earlier turns]).
Per-user fairness. One runaway session can dominate cost if you only cap per request, not per session.

Alternatives

Approach	Pros	Cons
Sliding message window	Easy to implement	Drops early facts
Token budget + protected tail	Predictable size	Needs measurement code
Periodic summarization	Keeps long goals	Summary errors; extra cost
Full history every turn	Perfect recall until limit	Cost and latency explode
Framework checkpoint pruning	Integrated with graph state	Learn framework semantics

FAQs

What is a good default for keep_last?

Start with the last 6-12 conversational turns or the last complete tool chain, then tune with traces. Product chats often need more human turns; tool-heavy agents need intact call groups.

Should I trim before or after adding the new user message?

Add the new user message first, then trim so the latest request is never dropped in favor of older filler.

Can I store full history and only trim for the model?

Yes - and you should. Keep an audit/log copy (with retention policy) separate from the prompt projection.

How do I handle images or large tool payloads?

Store blobs outside the transcript; leave short references or hashes in history. Re-fetch into cache when a later step needs detail.

Does a larger context model remove the need for this?

It delays pressure but does not remove cost, latency, or distraction from irrelevant early turns.

Where does scratchpad fit in the budget?

Count it as session memory. Prefer small structured fields over pasting the entire log into the scratchpad.

How often should I summarize?

When remaining budget crosses a threshold (for example 70% full) or every N turns - not on every message unless chats are huge.

What if trim removes a safety refusal earlier in the chat?

Safety policy belongs in the system layer you never trim, not only in past assistant messages.

Related: Short-Term Memory Basics

Related: Summarization Strategies for Long-Running Conversations

Related: Short-Term Memory: What an Agent Actually Needs Within One Run

Related: Short-Term Memory Best Practices

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Managing Conversation History Within a Context Budget

Search Documentation

Managing Conversation History Within a Context Budget