Caching Agent Responses and Tool Results to Cut Redundant Calls

Many agent fleets pay repeatedly for the same FAQ answer and the same search snippet.

Caching repeatable tool results and safe model responses frees RPM, TPM, tool slots, and budget for novel work.

Done poorly, cache keys leak data across tenants or serve stale facts as truth.

Summary

Key stable reads by normalized inputs and tenant scope, cache tool observations more aggressively than final answers, set TTLs by freshness needs, and never cache personalized or side-effecting operations without strict isolation.

Recipe

1. Inventory calls: which tools are pure reads vs writes; which prompts are templated FAQs.
2. Prefer caching tool results (search, KB fetch, catalog) before caching full agent transcripts.
3. Build keys from tenant, tool name, normalized args, and schema version.
4. Choose store (memory for dev; Redis/Memcached for multi-worker; HTTP cache where applicable).
5. Set TTLs from data volatility (seconds for prices, days for static docs).
6. On hit, return a structured observation labeled source=cache so traces stay honest.
7. Bypass cache for authorized personal data paths unless the key includes the right subject id.
8. Metric hit rate, latency saved, and $ saved; alert on sudden hit-rate collapse.

Working Example

import hashlib
import json
import time
from dataclasses import dataclass
from typing import Any, Callable
 
 
@dataclass
class CacheEntry:
    value: Any
    expires_at: float
 
 
class TtlCache:
    def __init__(self) -> None:
        self._store: dict[str, CacheEntry] = {}
 
    def get(self, key: str) -> Any | None:
        ent = self._store.get(key)
        if not ent:
            return None
        if time.time() >= ent.expires_at:
            del self._store[key]
            return None
        return ent.value
 
    def set(self, key: str, value: Any, ttl_s: float) -> None:
        self._store[key] = CacheEntry(value=value, expires_at=time.time() + ttl_s)
 
 
cache = TtlCache()
 
 
def make_key(tenant_id: str, tool: str, args: dict) -> str:
    normalized = json.dumps(args, sort_keys=True, separators=(",", ":"))
    raw = f"{tenant_id}|{tool}|{normalized}"
    return hashlib.sha256(raw.encode()).hexdigest()
 
 
def cached_tool(
    tenant_id: str,
    tool: str,
    args: dict,
    fn: Callable[[], Any],
    ttl_s: float,
) -> dict:
    key = make_key(tenant_id, tool, args)
    hit = cache.get(key)
    if hit is not None:
        return {"ok": True, "data": hit, "source": "cache"}
    data = fn()
    cache.set(key, data, ttl_s=ttl_s)
    return {"ok": True, "data": data, "source": "live"}
 
 
def kb_fetch(doc_id: str) -> dict:
    # stand-in for network fetch
    return {"doc_id": doc_id, "body": "password reset steps..."}
 
 
print(cached_tool("tenant-a", "kb.fetch", {"doc_id": "faq-1"}, lambda: kb_fetch("faq-1"), ttl_s=60))
print(cached_tool("tenant-a", "kb.fetch", {"doc_id": "faq-1"}, lambda: kb_fetch("faq-1"), ttl_s=60))

Deep Dive

What to cache first

Tool-result caches often beat end-to-end answer caches on both safety and hit rate.

Key design rules

1. Normalize: trim whitespace, stable JSON key order, canonical enums.
2. Scope: include tenant_id always; include user_id when data is personal.
3. Version: bump when tool schema or prompt product version changes.
4. Exclude secrets: never put raw API keys or full auth headers in keys/values logs.
5. Separate product surfaces: support bot vs internal admin may share tools but not cache namespaces.

Final-answer caching

Cache final answers only when:

• Input is a closed FAQ or template.
• Output is non-personal.
• Freshness is acceptable.
• You still enforce safety filters on serve.

Semantic cache (embed query → nearest prior answer) can raise hit rate but adds wrong-answer risk. Start with exact keys; add semantic only with similarity thresholds and eval.

Provider prompt caching vs your cache

Some model providers offer prompt/prefix caching for repeated system prompts.

That reduces token pricing for large stable prefixes. It does not replace:

• Tool-result caches
• Cross-user FAQ caches
• Skipping a loop entirely on exact hits

Use both layers when available (verify provider behavior and pricing at build).

Invalidation

Strategies:

• TTL-only: simple; fine for low-risk reads
• Event-driven: purge on CMS publish, price update, catalog change
• Version stamp: store content_version in key
• Manual bust: ops endpoint for incidents

Stale cache is a correctness bug, not only a scale footgun.

Multi-worker and multi-region

In-process dicts do not share hits across workers.

Use Redis (or similar) for fleet hit rate. For multi-region, decide whether cross-region cache sharing is worth latency and consistency complexity.

Observability

Log:

• cache_hit / cache_miss
• key space (not raw PII values)
• TTL and tool name
• latency delta vs live

A hit-rate cliff often means a deploy changed normalization or traffic shifted to novel goals.

Gotchas

1. Caching without tenant in the key. Cross-tenant data leaks.
2. Caching tool writes. Duplicate side effects or masking failures.
3. Giant values. Blowing Redis memory with full page HTML dumps.
4. Non-normalized args. "Q:" vs "q:" destroys hit rate.
5. Serving cache without source in traces. You cannot debug freshness.
6. Semantic cache with low thresholds. Confident wrong FAQ answers.
7. Infinite TTL on "static" content. Legal and pricing pages still change.

Alternatives

Combine caching with Cost-Aware Scaling: Routing Load to Cheaper Models Under Pressure for spike survival.

FAQs

Related

• What Actually Bottlenecks an Agent System at Scale - why redundant calls hurt
• Scaling Agent Systems Basics - first cache sketch
• Cost-Aware Scaling: Routing Load to Cheaper Models Under Pressure - next degrade layer
• Concurrency Limits and Backpressure for Agent Workers - capacity pairing
• Architecture Guide: A Horizontally Scalable Agent Worker Pool - cache in the pool
• Scaling Agent Systems Best Practices - checklist
• Cost-Based Routing: Sending Cheap Tasks to Cheap Models - complementary savings

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.