Versioning Prompts and Tool Schemas Alongside Code

Q: Do model IDs belong in the pack?

Yes in the pin. Model choice is behavior. Keep provider secrets out of the pack.

Q: How do pack versions show up in observability?

As high-cardinality-careful attributes on the root span: prompt_version, tool_schema_version, code_sha. See What Agent Observability Actually Needs to Capture.

Prompts and tool JSON schemas drive agent control flow as much as Python modules do.

Version them in the same repo, review them in the same PRs, pin them at deploy time, and stamp every run with the versions that actually ran.

Summary

Keep prompts and tool schemas as immutable packs under git, load them through a resolver that takes an explicit version, emit a deploy manifest that pairs pack versions with code_sha, and refuse "edit prod prompt in place" as a release process.

Recipe

Create an agent_pack/ (or similar) tree: prompts/, tools/, policies/, MANIFEST.json.
Name artifacts with explicit versions (system_v3.txt, tools_2026_04.json) or content hashes.
Load only through a resolver API - no ad-hoc open("prompt.txt") scattered in nodes.
On release, write a pin: {code_sha, prompt_version, tool_schema_version, model_route, policy_version}.
Require PR review for pack changes; run evals when pack paths change.
Log the resolved pin on every agent run (and in traces).
Keep prior pins immutable for fast rollback.
Deprecate tool fields with dual-support windows; never silently rename required args in place.

Working Example

from __future__ import annotations
 
import hashlib
import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any
 
ROOT = Path("agent_pack")
 
 
@dataclass(frozen=True)
class AgentPin:
    code_sha: str
    prompt_version: str
    tool_schema_version: str
    model_id: str
    policy_version: str = "default"
 
    def as_log_fields(self) -> dict[str, str]:
        return {
            "code_sha": self.code_sha,
            "prompt_version": self.prompt_version,
            "tool_schema_version": self.tool_schema_version,
            "model_id": self.model_id,
            "policy_version": self.policy_version,
        }
 
 
class PackStore:
    """Load versioned prompts and tool schemas from the repo tree."""
 
    def __init__(self, root: Path = ROOT) -> None:
        self.root = root
 
    def load_prompt(self, version: str) -> str:
        path = self.root / "prompts" / f"system_{version}.txt"
        text = path.read_text(encoding="utf-8")
        return text.strip()
 
    def load_tools(self, version: str) -> list[dict[str, Any]]:
        path = self.root / "tools" / f"{version}.json"
        data = json.loads(path.read_text(encoding="utf-8"))
        assert isinstance(data, list), "tools file must be a JSON array"
        return data
 
    def content_hash(self, versioned_path: Path) -> str:
        digest = hashlib.sha256(versioned_path.read_bytes()).hexdigest()
        return digest[:12]
 
 
def build_release_manifest(pin: AgentPin, store: PackStore) -> dict[str, Any]:
    prompt_path = store.root / "prompts" / f"system_{pin.prompt_version}.txt"
    tools_path = store.root / "tools" / f"{pin.tool_schema_version}.json"
    return {
        **pin.as_log_fields(),
        "prompt_sha256_12": store.content_hash(prompt_path),
        "tools_sha256_12": store.content_hash(tools_path),
    }
 
 
# Demo layout would include:
# agent_pack/prompts/system_v1.txt
# agent_pack/tools/tools_v1.json
store = PackStore()
pin = AgentPin(
    code_sha="deadbeef",
    prompt_version="v1",
    tool_schema_version="tools_v1",
    model_id="openrouter/auto",  # verify at build
)
# print(json.dumps(build_release_manifest(pin, store), indent=2))
print(pin.as_log_fields())

Deep Dive

Why "alongside code" beats a loose CMS

Concern	Git-backed packs	Only in a prod admin UI
Review	PR diffs, CODEOWNERS	Often none
CI	Path filters + evals	Easy to skip
Rollback	Prior pin / git tag	"Who changed it?"
Audit	Commit + release ID	Chat archaeology
Env parity	Same files in stage/prod	Drift by hand

A CMS or flag service can serve the active pin. The source of truth should still be reviewable artifacts.

Suggested tree

agent_pack/
  prompts/
    system_v1.txt
    system_v2.txt
    fewshot_v1.json
  tools/
    tools_v1.json
    tools_v2.json
  policies/
    spend_v1.yaml
  MANIFEST.json          # optional catalog of known versions
deploy/
  pins/
    active.json
    2026-04-01T12-00Z-deadbee.json

Keep generated dumps (pretty-printed API exports) out of the pack unless they are the real source.

Version naming strategies

Strategy	Example	Pros	Cons
Monotonic integer	`v12`	Simple	Collisions across branches
Date stamp	`2026_04_15`	Chronology	Multiple same-day
Semver for tools	`2.1.0`	Compat rules	Overkill for prose prompts
Content hash	`a3f1c9`	Exact identity	Harder human chat

Common pattern: human label + content hash in the pin (prompt_version=v12, prompt_sha256_12=...).

Tool schemas need compatibility rules

Agents plan against the schema the model sees.

Breaking changes:

Rename or remove a required property
Change enum values the prompt teaches
Split one tool into two without dual registration

Safer sequence:

Add new fields as optional; support both in host code.
Update prompt few-shots and evals.
Ship code that accepts old and new.
Make new fields required only after traffic and evals clear.
Remove old fields in a later pin.

Pairing with `code_sha`

A pack that references a tool implemented only in commit B must not pin on workers still at A.

Options:

Release train: one deploy updates workers then pack (or atomic together).
Feature flag: new tool name registered only when code flag is on.
Schema registry: workers advertise supported schema max; router picks compatible pin.

Runtime resolution

request → read active pin (or flag override) → PackStore.load_* → agent run → log pin

Do not read "latest file mtime" in production. That is non-reproducible.

CODEOWNERS and review

Route agent_pack/** to agent owners and safety reviewers.

Treat prompt PRs like security-sensitive config when tools can spend money or send messages.

Secrets

Never put API keys, customer PII, or raw prod transcripts into pack files.

Few-shots must be redacted and licensed for retention.

Gotchas

Editing the live prompt file in place on the server. History and rollback die.
Version in filename but loader always opens system.txt. Labels become theater.
Tool schema in code strings and JSON files diverge. Single source only.
Hash not recorded on the pin. You cannot prove what text ran after a force-push.
Per-tenant overrides without version IDs. One VIP experiment becomes undebuggable.
Breaking schema on the same version number. Versions must be immutable once shipped.
Forgetting evals on pack-only PRs. Path filters that omit agent_pack/** are a footgun.

Alternatives

Approach	Strength	Weakness
Git packs + pin (this recipe)	Reviewable, CI-friendly	Needs loader discipline
Feature-flag payload only	Fast toggles	Weak review unless flags are PR-generated
DB rows edited by ops	Hotfix	Audit and env drift
Fully dynamic prompt from user prefs	Personalization	Hard evals; injection surface
Monolith config in app YAML	Simple	No fine-grained rollback of prompt alone

Many mature teams combine git source + flag service for percentage rollout of an already-built pin.

FAQs

Should prompts be Python string constants?

Short constants are fine for tiny demos. Production packs should be files (or generated from files) so non-engineers can review diffs without hunting string literals.

How do we version multi-message chat templates?

Store a JSON or YAML list of roles/contents under a versioned name. The loader validates role enums before run.

What about LangSmith / Langfuse prompt hubs?

Hubs are fine as distribution if each pulled revision is pinned by ID and mirrored or recorded in your release manifest (verify vendor APIs at build).

Do model IDs belong in the pack?

Yes in the pin. Model choice is behavior. Keep provider secrets out of the pack.

How long do we keep old pins?

At least long enough for incident review and compliance (often 90 days or more). Immutable object storage works well.

Can two services share one pack repo?

Yes via a submodule, package, or internal artifact. Pin the pack version independently per service deploy.

How do we test schema changes without production tools?

Contract tests on JSON Schema, plus loop simulations with mocked tool handlers for old and new shapes.

What goes in MANIFEST.json?

A catalog of known versions, owners, deprecation dates, and min code SHA - optional but helpful for humans and validators.

Should we semver prompts?

Optional. Prompts rarely have clean compat. Immutability + pin + eval matters more than 1.2.3.

How do pack versions show up in observability?

As high-cardinality-careful attributes on the root span: prompt_version, tool_schema_version, code_sha. See What Agent Observability Actually Needs to Capture.

What "Continuous Deployment" Means for a Prompt-Driven System - why packs are deployables
CI/CD & Agent Lifecycle Basics - first loader and pin
Running Eval Suites as a CI Gate Before Deploy - gate pack changes
Feature-Flagging Agent Behavior Changes - rollout of versions
Rollback Strategies When a Deployed Agent Regresses - restore prior pin
Versioning and Deprecating Agent Tools Without Breaking Agents - tool lifecycle
CI/CD & Agent Lifecycle Best Practices - section habits

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Version them in the same repo, review them in the same PRs, pin them at deploy time, and stamp every run with the versions that actually ran.

Summary

Recipe

Create an agent_pack/ (or similar) tree: prompts/, tools/, policies/, MANIFEST.json.
Name artifacts with explicit versions (system_v3.txt, tools_2026_04.json) or content hashes.
Load only through a resolver API - no ad-hoc open("prompt.txt") scattered in nodes.
On release, write a pin: {code_sha, prompt_version, tool_schema_version, model_route, policy_version}.
Require PR review for pack changes; run evals when pack paths change.
Log the resolved pin on every agent run (and in traces).
Keep prior pins immutable for fast rollback.
Deprecate tool fields with dual-support windows; never silently rename required args in place.

Working Example

from __future__ import annotations
 
import hashlib
import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any
 
ROOT = Path("agent_pack")
 
 
@dataclass(frozen=True)
class AgentPin:
    code_sha: str
    prompt_version: str
    tool_schema_version: str
    model_id: str
    policy_version: str = "default"
 
    def as_log_fields(self) -> dict[str, str]:
        return {
            "code_sha": self.code_sha,
            "prompt_version": self.prompt_version,
            "tool_schema_version": self.tool_schema_version,
            "model_id": self.model_id,
            "policy_version": self.policy_version,
        }
 
 
class PackStore:
    """Load versioned prompts and tool schemas from the repo tree."""
 
    def __init__(self, root: Path = ROOT) -> None:
        self.root = root
 
    def load_prompt(self, version: str) -> str:
        path = self.root / "prompts" / f"system_{version}.txt"
        text = path.read_text(encoding="utf-8")
        return text.strip()
 
    def load_tools(self, version: str) -> list[dict[str, Any]]:
        path = self.root / "tools" / f"{version}.json"
        data = json.loads(path.read_text(encoding="utf-8"))
        assert isinstance(data, list), "tools file must be a JSON array"
        return data
 
    def content_hash(self, versioned_path: Path) -> str:
        digest = hashlib.sha256(versioned_path.read_bytes()).hexdigest()
        return digest[:12]
 
 
def build_release_manifest(pin: AgentPin, store: PackStore) -> dict[str, Any]:
    prompt_path = store.root / "prompts" / f"system_{pin.prompt_version}.txt"
    tools_path = store.root / "tools" / f"{pin.tool_schema_version}.json"
    return {
        **pin.as_log_fields(),
        "prompt_sha256_12": store.content_hash(prompt_path),
        "tools_sha256_12": store.content_hash(tools_path),
    }
 
 
# Demo layout would include:
# agent_pack/prompts/system_v1.txt
# agent_pack/tools/tools_v1.json
store = PackStore()
pin = AgentPin(
    code_sha="deadbeef",
    prompt_version="v1",
    tool_schema_version="tools_v1",
    model_id="openrouter/auto",  # verify at build
)
# print(json.dumps(build_release_manifest(pin, store), indent=2))
print(pin.as_log_fields())

Deep Dive

Why "alongside code" beats a loose CMS

Concern	Git-backed packs	Only in a prod admin UI
Review	PR diffs, CODEOWNERS	Often none
CI	Path filters + evals	Easy to skip
Rollback	Prior pin / git tag	"Who changed it?"
Audit	Commit + release ID	Chat archaeology
Env parity	Same files in stage/prod	Drift by hand

A CMS or flag service can serve the active pin. The source of truth should still be reviewable artifacts.

Suggested tree

agent_pack/
  prompts/
    system_v1.txt
    system_v2.txt
    fewshot_v1.json
  tools/
    tools_v1.json
    tools_v2.json
  policies/
    spend_v1.yaml
  MANIFEST.json          # optional catalog of known versions
deploy/
  pins/
    active.json
    2026-04-01T12-00Z-deadbee.json

Keep generated dumps (pretty-printed API exports) out of the pack unless they are the real source.

Version naming strategies

Strategy	Example	Pros	Cons
Monotonic integer	`v12`	Simple	Collisions across branches
Date stamp	`2026_04_15`	Chronology	Multiple same-day
Semver for tools	`2.1.0`	Compat rules	Overkill for prose prompts
Content hash	`a3f1c9`	Exact identity	Harder human chat

Common pattern: human label + content hash in the pin (prompt_version=v12, prompt_sha256_12=...).

Tool schemas need compatibility rules

Agents plan against the schema the model sees.

Breaking changes:

Rename or remove a required property
Change enum values the prompt teaches
Split one tool into two without dual registration

Safer sequence:

Add new fields as optional; support both in host code.
Update prompt few-shots and evals.
Ship code that accepts old and new.
Make new fields required only after traffic and evals clear.
Remove old fields in a later pin.

Pairing with `code_sha`

A pack that references a tool implemented only in commit B must not pin on workers still at A.

Options:

Release train: one deploy updates workers then pack (or atomic together).
Feature flag: new tool name registered only when code flag is on.
Schema registry: workers advertise supported schema max; router picks compatible pin.

Runtime resolution

request → read active pin (or flag override) → PackStore.load_* → agent run → log pin

Do not read "latest file mtime" in production. That is non-reproducible.

CODEOWNERS and review

Route agent_pack/** to agent owners and safety reviewers.

Treat prompt PRs like security-sensitive config when tools can spend money or send messages.

Secrets

Never put API keys, customer PII, or raw prod transcripts into pack files.

Few-shots must be redacted and licensed for retention.

Gotchas

Editing the live prompt file in place on the server. History and rollback die.
Version in filename but loader always opens system.txt. Labels become theater.
Tool schema in code strings and JSON files diverge. Single source only.
Hash not recorded on the pin. You cannot prove what text ran after a force-push.
Per-tenant overrides without version IDs. One VIP experiment becomes undebuggable.
Breaking schema on the same version number. Versions must be immutable once shipped.
Forgetting evals on pack-only PRs. Path filters that omit agent_pack/** are a footgun.

Alternatives

Approach	Strength	Weakness
Git packs + pin (this recipe)	Reviewable, CI-friendly	Needs loader discipline
Feature-flag payload only	Fast toggles	Weak review unless flags are PR-generated
DB rows edited by ops	Hotfix	Audit and env drift
Fully dynamic prompt from user prefs	Personalization	Hard evals; injection surface
Monolith config in app YAML	Simple	No fine-grained rollback of prompt alone

Many mature teams combine git source + flag service for percentage rollout of an already-built pin.

FAQs

Should prompts be Python string constants?

Short constants are fine for tiny demos. Production packs should be files (or generated from files) so non-engineers can review diffs without hunting string literals.

How do we version multi-message chat templates?

Store a JSON or YAML list of roles/contents under a versioned name. The loader validates role enums before run.

What about LangSmith / Langfuse prompt hubs?

Hubs are fine as distribution if each pulled revision is pinned by ID and mirrored or recorded in your release manifest (verify vendor APIs at build).

Do model IDs belong in the pack?

Yes in the pin. Model choice is behavior. Keep provider secrets out of the pack.

How long do we keep old pins?

At least long enough for incident review and compliance (often 90 days or more). Immutable object storage works well.

Can two services share one pack repo?

Yes via a submodule, package, or internal artifact. Pin the pack version independently per service deploy.

How do we test schema changes without production tools?

Contract tests on JSON Schema, plus loop simulations with mocked tool handlers for old and new shapes.

What goes in MANIFEST.json?

A catalog of known versions, owners, deprecation dates, and min code SHA - optional but helpful for humans and validators.

Should we semver prompts?

Optional. Prompts rarely have clean compat. Immutability + pin + eval matters more than 1.2.3.

How do pack versions show up in observability?

As high-cardinality-careful attributes on the root span: prompt_version, tool_schema_version, code_sha. See What Agent Observability Actually Needs to Capture.

What "Continuous Deployment" Means for a Prompt-Driven System - why packs are deployables
CI/CD & Agent Lifecycle Basics - first loader and pin
Running Eval Suites as a CI Gate Before Deploy - gate pack changes
Feature-Flagging Agent Behavior Changes - rollout of versions
Rollback Strategies When a Deployed Agent Regresses - restore prior pin
Versioning and Deprecating Agent Tools Without Breaking Agents - tool lifecycle
CI/CD & Agent Lifecycle Best Practices - section habits

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Versioning Prompts and Tool Schemas Alongside Code

Search Documentation

Versioning Prompts and Tool Schemas Alongside Code