Versioning Prompts and Tool Schemas Alongside Code

Prompts and tool JSON schemas drive agent control flow as much as Python modules do.

Version them in the same repo, review them in the same PRs, pin them at deploy time, and stamp every run with the versions that actually ran.

Summary

Keep prompts and tool schemas as immutable packs under git, load them through a resolver that takes an explicit version, emit a deploy manifest that pairs pack versions with code_sha, and refuse "edit prod prompt in place" as a release process.

Recipe

1. Create an agent_pack/ (or similar) tree: prompts/, tools/, policies/, MANIFEST.json.
2. Name artifacts with explicit versions (system_v3.txt, tools_2026_04.json) or content hashes.
3. Load only through a resolver API - no ad-hoc open("prompt.txt") scattered in nodes.
4. On release, write a pin: {code_sha, prompt_version, tool_schema_version, model_route, policy_version}.
5. Require PR review for pack changes; run evals when pack paths change.
6. Log the resolved pin on every agent run (and in traces).
7. Keep prior pins immutable for fast rollback.
8. Deprecate tool fields with dual-support windows; never silently rename required args in place.

Working Example

from __future__ import annotations
 
import hashlib
import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any
 
ROOT = Path("agent_pack")
 
 
@dataclass(frozen=True)
class AgentPin:
    code_sha: str
    prompt_version: str
    tool_schema_version: str
    model_id: str
    policy_version: str = "default"
 
    def as_log_fields(self) -> dict[str, str]:
        return {
            "code_sha": self.code_sha,
            "prompt_version": self.prompt_version,
            "tool_schema_version": self.tool_schema_version,
            "model_id": self.model_id,
            "policy_version": self.policy_version,
        }
 
 
class PackStore:
    """Load versioned prompts and tool schemas from the repo tree."""
 
    def __init__(self, root: Path = ROOT) -> None:
        self.root = root
 
    def load_prompt(self, version: str) -> str:
        path = self.root / "prompts" / f"system_{version}.txt"
        text = path.read_text(encoding="utf-8")
        return text.strip()
 
    def load_tools(self, version: str) -> list[dict[str, Any]]:
        path = self.root / "tools" / f"{version}.json"
        data = json.loads(path.read_text(encoding="utf-8"))
        assert isinstance(data, list), "tools file must be a JSON array"
        return data
 
    def content_hash(self, versioned_path: Path) -> str:
        digest = hashlib.sha256(versioned_path.read_bytes()).hexdigest()
        return digest[:12]
 
 
def build_release_manifest(pin: AgentPin, store: PackStore) -> dict[str, Any]:
    prompt_path = store.root / "prompts" / f"system_{pin.prompt_version}.txt"
    tools_path = store.root / "tools" / f"{pin.tool_schema_version}.json"
    return {
        **pin.as_log_fields(),
        "prompt_sha256_12": store.content_hash(prompt_path),
        "tools_sha256_12": store.content_hash(tools_path),
    }
 
 
# Demo layout would include:
# agent_pack/prompts/system_v1.txt
# agent_pack/tools/tools_v1.json
store = PackStore()
pin = AgentPin(
    code_sha="deadbeef",
    prompt_version="v1",
    tool_schema_version="tools_v1",
    model_id="openrouter/auto",  # verify at build
)
# print(json.dumps(build_release_manifest(pin, store), indent=2))
print(pin.as_log_fields())

Deep Dive

Why "alongside code" beats a loose CMS

A CMS or flag service can serve the active pin. The source of truth should still be reviewable artifacts.

Suggested tree

agent_pack/
  prompts/
    system_v1.txt
    system_v2.txt
    fewshot_v1.json
  tools/
    tools_v1.json
    tools_v2.json
  policies/
    spend_v1.yaml
  MANIFEST.json          # optional catalog of known versions
deploy/
  pins/
    active.json
    2026-04-01T12-00Z-deadbee.json

Keep generated dumps (pretty-printed API exports) out of the pack unless they are the real source.

Version naming strategies

Common pattern: human label + content hash in the pin (prompt_version=v12, prompt_sha256_12=...).

Tool schemas need compatibility rules

Agents plan against the schema the model sees.

Breaking changes:

• Rename or remove a required property
• Change enum values the prompt teaches
• Split one tool into two without dual registration

Safer sequence:

1. Add new fields as optional; support both in host code.
2. Update prompt few-shots and evals.
3. Ship code that accepts old and new.
4. Make new fields required only after traffic and evals clear.
5. Remove old fields in a later pin.

See also Versioning and Deprecating Agent Tools Without Breaking Agents.

Pairing with code_sha

A pack that references a tool implemented only in commit B must not pin on workers still at A.

Options:

• Release train: one deploy updates workers then pack (or atomic together).
• Feature flag: new tool name registered only when code flag is on.
• Schema registry: workers advertise supported schema max; router picks compatible pin.

Runtime resolution

request → read active pin (or flag override) → PackStore.load_* → agent run → log pin

Do not read "latest file mtime" in production. That is non-reproducible.

CODEOWNERS and review

Route agent_pack/** to agent owners and safety reviewers.

Treat prompt PRs like security-sensitive config when tools can spend money or send messages.

Secrets

Never put API keys, customer PII, or raw prod transcripts into pack files.

Few-shots must be redacted and licensed for retention.

Gotchas

1. Editing the live prompt file in place on the server. History and rollback die.
2. Version in filename but loader always opens system.txt. Labels become theater.
3. Tool schema in code strings and JSON files diverge. Single source only.
4. Hash not recorded on the pin. You cannot prove what text ran after a force-push.
5. Per-tenant overrides without version IDs. One VIP experiment becomes undebuggable.
6. Breaking schema on the same version number. Versions must be immutable once shipped.
7. Forgetting evals on pack-only PRs. Path filters that omit agent_pack/** are a footgun.

Alternatives

Many mature teams combine git source + flag service for percentage rollout of an already-built pin.

FAQs

Related

• What "Continuous Deployment" Means for a Prompt-Driven System - why packs are deployables
• CI/CD & Agent Lifecycle Basics - first loader and pin
• Running Eval Suites as a CI Gate Before Deploy - gate pack changes
• Feature-Flagging Agent Behavior Changes - rollout of versions
• Rollback Strategies When a Deployed Agent Regresses - restore prior pin
• Versioning and Deprecating Agent Tools Without Breaking Agents - tool lifecycle
• CI/CD & Agent Lifecycle Best Practices - section habits

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.