CI/CD & Agent Lifecycle Best Practices

Ten practices for shipping agent changes safely and reversibly - plus operating checks so prompt, model, and tool updates do not become unbisectable incidents.

Use them as a design rubric for the release train and as a go-live checklist before continuous deployment of behavior packs.

How to Use This List

Walk A when defining artifacts and versioning; B when wiring CI/CD gates; C when operating production promote and recovery.
Tick items only when true in repo layout, pipeline config, and runbooks - not as aspirations.
Pair with What "Continuous Deployment" Means for a Prompt-Driven System for the mental model.
Re-check after adding agents, model routers, MCP tool packs, or multi-tenant prompt overrides.

A - Version Behavior Like Code

1. Store prompts, tool schemas, and policy packs in git (or a reviewable source fed by git). No prod-only paste as the system of record.
2. Load packs only through a versioned resolver; pin code_sha with pack versions on every release. Immutability after ship is mandatory.
3. Stamp every run trace with the resolved pin (prompt, schema, model, policy, code). You cannot roll back what you cannot name.
4. Review pack PRs with CODEOWNERS and treat tool-schema breaks as compatibility projects, not drive-by renames.
5. Keep secrets and raw prod transcripts out of packs. Redact few-shots; inject credentials at runtime from a secret store.

B - Gate and Deliver Progressively

6. Run eval suites as required CI checks on pack and agent code changes. Absolute floors plus max drop vs baseline; safety slices harder than average.
7. Separate fast PR suites from full pre-promote suites and fail closed when credentials or scorers are misconfigured.
8. Ship candidate pins dark; expose them via feature flags with sticky cohorts. Flags select versions - they do not replace git review of prompt bodies.
9. Use shadow and canary with pin-dimensioned SLOs (success, latency, cost, tool errors, safety) before 100% traffic.
10. Automate abort rules for canaries and pause CD on incident; never auto-promote without a path to stop.

C - Recover, Learn, and Keep the Train Honest

11. Practice pin/flag rollback to last known good in minutes with a written runbook and game days.
12. Pair rollbacks with kill switches for in-flight thrash and irreversible tools.
13. Quarantine bad pins and add gold-set cases after every SEV so the same regression cannot pass silently.
14. Dashboard and alert on pin-level burn, not only host CPU and HTTP 5xx.
15. Expire feature flags after full rollout and rebaseline suites only with explicit product intent - not to greenwash a bad pack.

Applying These Practices in Order

Versioning (1-5): foundation; without pins, gates and rollbacks are theater.
Gates and progressive delivery (6-10): stop known failures before users, then prove live traffic.
Operate (11-15): keep recovery fast and the suite honest after real incidents.

FAQs

Why "ten practices" if there are fifteen checkboxes?

Items 1-10 are the core shipping practices. Items 11-15 extend them into incident recovery and long-term hygiene so the list stays useful after the first launch.

What is the first practice if we can only adopt one?

Immutable versioned pins logged on every run (practices 1-3). Everything else depends on knowing what ran.

Do these practices require a specific framework?

No. LangGraph, CrewAI, OpenAI Agents SDK, Pydantic AI, and custom runtimes all need pins, evals, and rollback. Frameworks do not replace the release discipline (verify versions at build).

How strict should CI gates be for experimental agents?

Still gate safety and smoke quality. Loosen product-tone thresholds only in non-prod projects with no prod credentials.

Can we continuous-deploy prompts without canaries?

Only for the lowest blast radius (draft-only, internal, no tools). Tools, money, or external send require progressive exposure.

Where should platform own vs agent teams own?

Platform: pipeline, flag service, pin store, rollback plumbing. Agent team: gold sets, thresholds, go/no-go on quality, pack content.

How do multi-tenant custom prompts fit?

Each override needs a version ID, owner, and eval subset. Never let one tenant's hot edit become the global default path.

What evidence shows practice 6 is real?

Branch protection requires the eval job, and a deliberately broken pack PR fails the check in CI logs.

How often should we game-day rollback?

At least quarterly, and after any change to pin storage or flag plumbing.

Do model provider outages count as "agent deploy regressions"?

They are dependency incidents. Still use model-route flags and fallbacks; postmortems should separate provider fail from your pin fail.

How do we avoid eval flake destroying trust?

Pin scorers, retry only transport errors, quarantine flaky cases with owners, and keep a deterministic core gate.

Where should teams link this list?

In the section sidebar as the close-out checklist, and from ADRs or runbooks that introduce new agent products or CD automation.

What "Continuous Deployment" Means for a Prompt-Driven System - mental model
CI/CD & Agent Lifecycle Basics - first pipeline
Versioning Prompts and Tool Schemas Alongside Code - packs and pins
Running Eval Suites as a CI Gate Before Deploy - hard gates
Canary and Shadow Deployments for Agent Updates - progressive traffic
Feature-Flagging Agent Behavior Changes - controlled rollout
Rollback Strategies When a Deployed Agent Regresses - recovery cheatsheet
Building a Gold-Set Eval Suite for Agent Quality - suite design
What Agent Observability Actually Needs to Capture - pin on traces
Kill Switches: Stopping a Runaway Agent Mid-Execution - emergency stop

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

CI/CD & Agent Lifecycle Best Practices

Ten practices for shipping agent changes safely and reversibly - plus operating checks so prompt, model, and tool updates do not become unbisectable incidents.

Use them as a design rubric for the release train and as a go-live checklist before continuous deployment of behavior packs.

How to Use This List

Walk A when defining artifacts and versioning; B when wiring CI/CD gates; C when operating production promote and recovery.
Tick items only when true in repo layout, pipeline config, and runbooks - not as aspirations.
Pair with What "Continuous Deployment" Means for a Prompt-Driven System for the mental model.
Re-check after adding agents, model routers, MCP tool packs, or multi-tenant prompt overrides.

A - Version Behavior Like Code

1. Store prompts, tool schemas, and policy packs in git (or a reviewable source fed by git). No prod-only paste as the system of record.
2. Load packs only through a versioned resolver; pin code_sha with pack versions on every release. Immutability after ship is mandatory.
3. Stamp every run trace with the resolved pin (prompt, schema, model, policy, code). You cannot roll back what you cannot name.
4. Review pack PRs with CODEOWNERS and treat tool-schema breaks as compatibility projects, not drive-by renames.
5. Keep secrets and raw prod transcripts out of packs. Redact few-shots; inject credentials at runtime from a secret store.

B - Gate and Deliver Progressively

6. Run eval suites as required CI checks on pack and agent code changes. Absolute floors plus max drop vs baseline; safety slices harder than average.
7. Separate fast PR suites from full pre-promote suites and fail closed when credentials or scorers are misconfigured.
8. Ship candidate pins dark; expose them via feature flags with sticky cohorts. Flags select versions - they do not replace git review of prompt bodies.
9. Use shadow and canary with pin-dimensioned SLOs (success, latency, cost, tool errors, safety) before 100% traffic.
10. Automate abort rules for canaries and pause CD on incident; never auto-promote without a path to stop.

C - Recover, Learn, and Keep the Train Honest

11. Practice pin/flag rollback to last known good in minutes with a written runbook and game days.
12. Pair rollbacks with kill switches for in-flight thrash and irreversible tools.
13. Quarantine bad pins and add gold-set cases after every SEV so the same regression cannot pass silently.
14. Dashboard and alert on pin-level burn, not only host CPU and HTTP 5xx.
15. Expire feature flags after full rollout and rebaseline suites only with explicit product intent - not to greenwash a bad pack.

Applying These Practices in Order

Versioning (1-5): foundation; without pins, gates and rollbacks are theater.
Gates and progressive delivery (6-10): stop known failures before users, then prove live traffic.
Operate (11-15): keep recovery fast and the suite honest after real incidents.

FAQs

Why "ten practices" if there are fifteen checkboxes?

Items 1-10 are the core shipping practices. Items 11-15 extend them into incident recovery and long-term hygiene so the list stays useful after the first launch.

What is the first practice if we can only adopt one?

Immutable versioned pins logged on every run (practices 1-3). Everything else depends on knowing what ran.

Do these practices require a specific framework?

No. LangGraph, CrewAI, OpenAI Agents SDK, Pydantic AI, and custom runtimes all need pins, evals, and rollback. Frameworks do not replace the release discipline (verify versions at build).

How strict should CI gates be for experimental agents?

Still gate safety and smoke quality. Loosen product-tone thresholds only in non-prod projects with no prod credentials.

Can we continuous-deploy prompts without canaries?

Only for the lowest blast radius (draft-only, internal, no tools). Tools, money, or external send require progressive exposure.

Where should platform own vs agent teams own?

Platform: pipeline, flag service, pin store, rollback plumbing. Agent team: gold sets, thresholds, go/no-go on quality, pack content.

How do multi-tenant custom prompts fit?

Each override needs a version ID, owner, and eval subset. Never let one tenant's hot edit become the global default path.

What evidence shows practice 6 is real?

Branch protection requires the eval job, and a deliberately broken pack PR fails the check in CI logs.

How often should we game-day rollback?

At least quarterly, and after any change to pin storage or flag plumbing.

Do model provider outages count as "agent deploy regressions"?

They are dependency incidents. Still use model-route flags and fallbacks; postmortems should separate provider fail from your pin fail.

How do we avoid eval flake destroying trust?

Pin scorers, retry only transport errors, quarantine flaky cases with owners, and keep a deterministic core gate.

Where should teams link this list?

In the section sidebar as the close-out checklist, and from ADRs or runbooks that introduce new agent products or CD automation.

What "Continuous Deployment" Means for a Prompt-Driven System - mental model
CI/CD & Agent Lifecycle Basics - first pipeline
Versioning Prompts and Tool Schemas Alongside Code - packs and pins
Running Eval Suites as a CI Gate Before Deploy - hard gates
Canary and Shadow Deployments for Agent Updates - progressive traffic
Feature-Flagging Agent Behavior Changes - controlled rollout
Rollback Strategies When a Deployed Agent Regresses - recovery cheatsheet
Building a Gold-Set Eval Suite for Agent Quality - suite design
What Agent Observability Actually Needs to Capture - pin on traces
Kill Switches: Stopping a Runaway Agent Mid-Execution - emergency stop

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

CI/CD & Agent Lifecycle Best Practices

Search Documentation

CI/CD & Agent Lifecycle Best Practices