Rollback Strategies When a Deployed Agent Regresses

Use this when production quality, safety, latency, or cost burns after an agent change.

The goal is minutes to safe, not a perfect root cause in the first five minutes.

How to Use This Cheatsheet

Start at A if users are actively harmed or spend is spiking; otherwise walk A → E in order.
Prefer the smallest reversible unit (flag/pin) before rebuilding the world.
Tick only actions you actually completed; note times in the incident channel.
Pair with Kill Switches: Stopping a Runaway Agent Mid-Execution when runs are still thrashing.
After stabilize, finish E so the same pin cannot silently return.

A - Stabilize Now (First 5-15 Minutes)

1. Declare the incident - severity, commander, comms channel, customer impact one-liner.
2. Identify the active pin - prompt_version, tool_schema_version, model_id, code_sha, flag variations from a failing run trace.
3. Trip kill switch / disable risky tools if side effects are wrong (refunds, send, delete, shell).
4. Flip traffic to last known good pin - flag percent → 0, active → previous manifest, or route table → baseline.
5. Confirm workers see the flip - sample new runs show baseline versions (watch config TTL).
6. Pause CD / auto-promote so a pipeline cannot re-push the bad pin.
7. Snapshot evidence - failing run IDs, dashboards, the bad and good manifest JSON, recent deploy times.
8. Customer comms if user-visible errors or wrong actions occurred (status page / support macro).

B - Choose the Rollback Unit

Pick the narrowest lever that removes the regression:

Symptom cluster	Try first	Then
Quality drop after prompt-only change	Previous `prompt_version`	Full previous pin
Tool call storms / schema errors	Previous `tool_schema_version` + matching code	Disable tool via flag
Latency/cost after model swap	Previous `model_id` / route	Provider failover
Only canary cohort hurt	Canary percent → 0	Keep baseline
Host crash / OOM / bad binary	Redeploy previous `code_sha`	Scale / restart
Partial migrations	Expand-contract down path	Feature-flag off new writes
Unsafe content / injection surge	Tighten policy pack + tool deny	Full agent off

9. Name the unit you rolled - one line in the incident doc ("rolled prompt v17 → v16").
10. Avoid hot-editing prod prompt text as the primary fix - that creates a third unknown version.

C - Execute Pin / Flag / Deploy Rollback

Flag or config pin (preferred)

11. Set production flag/config to baseline pin IDs (not "latest").
12. Clear or respect caches - document max staleness; force refresh if supported.
13. Verify with synthetic probe - scripted case that failed under candidate, passes under baseline.
14. Watch live metrics 15-30 minutes - success, latency, cost, tool errors, escalations.

Application / container rollback

15. Redeploy last good artifact only if code_sha is implicated or pin lives only in the image.
16. Ensure pack files inside the image match the intended versions (no external override still pointing at bad pack).
17. Run smoke eval subset against the restored revision in prod-like env if time allows.

Data and side effects

18. List irreversible actions already taken - emails sent, tickets closed, charges, deletes.
19. Start compensating actions - reverse charges, notify users, reopen tickets - with humans in the loop.
20. Preserve audit trails - do not delete bad runs; mark them for review.

D - Confirm Recovery

21. Error budget / SLO burn returning to normal for success, latency, cost.
22. No increase in safety blocks or policy violations beyond baseline noise.
23. Canary cohort (if any) empty or healthy - no residual traffic on bad pin.
24. Support queue trend - similar ticket rate to pre-incident window.
25. On-call agrees "stable" - explicit statement in channel before leaving D.

E - Prevent Instant Repeat

26. Quarantine the bad pin - mark undeployable; block that version ID in CD.
27. Add or fix gold-set cases that would have failed the gate (link PR).
28. Lower auto-promote confidence if the gate was green but prod burned - tighten thresholds or add canary abort rules.
29. Patch runbooks with the actual flip commands/UI paths you used.
30. Schedule postmortem within policy window - use agent-shaped template (bounds, pins, gates).
31. Owner for re-release - who will ship a fixed pin only after eval + canary.

Decision Card (Print / Pin in Channel)

IMPACT?  safety/money  → kill tools + pin flip + human review of side effects
         quality only  → pin/flag flip first
         cost/latency  → model route + max_turns/spend caps, then pin
STILL BAD after pin flip?
         → code redeploy + check external pack overrides + dependency outages
NEED TIME?
         → leave agent in degrade mode (retrieve-only / draft-only) rather than thrash

Common Failure Modes During Rollback

Mistake	Result	Fix
Roll image but not pack	Behavior unchanged	Flip pack pin / flags
Roll pack but not schema-compatible code	Mass tool errors	Roll both as a train
Non-sticky residual canary	Random bad sessions	Percent 0 + verify sticky maps
"Fixed" by editing live prompt	Unknown version	Ship proper pin later
No probe after flip	False recovery	Synthetic + metric confirm
Re-enable canary same day without new eval	Repeat SEV	Quarantine + new gate evidence

FAQs

How fast should pin rollback be?

Target under five minutes from decision to new runs on baseline, including config propagation. If slower, invest in flags and pin stores.

Is redeploying always safer than a flag flip?

No. Redeploys are slower and coarser. Use them when the binary or migrations are wrong.

What if we do not have a previous pin?

That is the incident behind the incident. Freeze changes, reconstruct from git, and build pin history before the next launch.

Should we roll back during an upstream model outage?

If the outage is provider-wide, failover routes or degrade. If only your new model ID fails, roll the model pin.

Can we partial-roll one tenant?

Yes - force baseline for the affected org flag override while others stay candidate, then global baseline if pattern spreads.

Where do kill switches stop and rollback starts?

Kill switch stops ongoing damage now. Rollback restores known good behavior for new runs. Often you do both.

How long do we stay on baseline after a SEV?

Until a fixed pin passes eval, canary abort rules are satisfied, and the postmortem action items that block re-release are done.

What metrics prove rollback worked?

Pin dimension shows baseline, proxy success recovered, cost/latency back in band, and no new safety SEVs - not "feels better."

Do we roll RAG indexes the same way?

If the index is versioned, point retrieval at the prior index ID. If not versioned, restore from snapshot or reindex from last good corpus pin.

Who can authorize break-glass stay on bad pin?

Rare and written: executive/product + eng lead with explicit residual risk. Default is roll back.

How do we practice this?

Game-day: deploy a deliberate bad pin to staging, time the flip, verify probes, and fix doc gaps.

What belongs in the postmortem every time?

Pin IDs, gate results (why green), canary gaps, time-to-detect, time-to-rollback, and new suite cases.

What "Continuous Deployment" Means for a Prompt-Driven System - rollback as a product feature
Versioning Prompts and Tool Schemas Alongside Code - pins to restore
Feature-Flagging Agent Behavior Changes - flip mechanics
Canary and Shadow Deployments for Agent Updates - abort rules
Running Eval Suites as a CI Gate Before Deploy - stop repeats
CI/CD & Agent Lifecycle Best Practices - preventive habits
Kill Switches: Stopping a Runaway Agent Mid-Execution - stop in-flight harm
Postmortems for Agent Incidents: A Template - after action
Setting SLOs for Agent Latency, Success Rate, and Cost - recovery signals

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Rollback Strategies When a Deployed Agent Regresses

Use this when production quality, safety, latency, or cost burns after an agent change.

The goal is minutes to safe, not a perfect root cause in the first five minutes.

How to Use This Cheatsheet

Start at A if users are actively harmed or spend is spiking; otherwise walk A → E in order.
Prefer the smallest reversible unit (flag/pin) before rebuilding the world.
Tick only actions you actually completed; note times in the incident channel.
Pair with Kill Switches: Stopping a Runaway Agent Mid-Execution when runs are still thrashing.
After stabilize, finish E so the same pin cannot silently return.

A - Stabilize Now (First 5-15 Minutes)

1. Declare the incident - severity, commander, comms channel, customer impact one-liner.
2. Identify the active pin - prompt_version, tool_schema_version, model_id, code_sha, flag variations from a failing run trace.
3. Trip kill switch / disable risky tools if side effects are wrong (refunds, send, delete, shell).
4. Flip traffic to last known good pin - flag percent → 0, active → previous manifest, or route table → baseline.
5. Confirm workers see the flip - sample new runs show baseline versions (watch config TTL).
6. Pause CD / auto-promote so a pipeline cannot re-push the bad pin.
7. Snapshot evidence - failing run IDs, dashboards, the bad and good manifest JSON, recent deploy times.
8. Customer comms if user-visible errors or wrong actions occurred (status page / support macro).

B - Choose the Rollback Unit

Pick the narrowest lever that removes the regression:

Symptom cluster	Try first	Then
Quality drop after prompt-only change	Previous `prompt_version`	Full previous pin
Tool call storms / schema errors	Previous `tool_schema_version` + matching code	Disable tool via flag
Latency/cost after model swap	Previous `model_id` / route	Provider failover
Only canary cohort hurt	Canary percent → 0	Keep baseline
Host crash / OOM / bad binary	Redeploy previous `code_sha`	Scale / restart
Partial migrations	Expand-contract down path	Feature-flag off new writes
Unsafe content / injection surge	Tighten policy pack + tool deny	Full agent off

9. Name the unit you rolled - one line in the incident doc ("rolled prompt v17 → v16").
10. Avoid hot-editing prod prompt text as the primary fix - that creates a third unknown version.

C - Execute Pin / Flag / Deploy Rollback

Flag or config pin (preferred)

11. Set production flag/config to baseline pin IDs (not "latest").
12. Clear or respect caches - document max staleness; force refresh if supported.
13. Verify with synthetic probe - scripted case that failed under candidate, passes under baseline.
14. Watch live metrics 15-30 minutes - success, latency, cost, tool errors, escalations.

Application / container rollback

15. Redeploy last good artifact only if code_sha is implicated or pin lives only in the image.
16. Ensure pack files inside the image match the intended versions (no external override still pointing at bad pack).
17. Run smoke eval subset against the restored revision in prod-like env if time allows.

Data and side effects

18. List irreversible actions already taken - emails sent, tickets closed, charges, deletes.
19. Start compensating actions - reverse charges, notify users, reopen tickets - with humans in the loop.
20. Preserve audit trails - do not delete bad runs; mark them for review.

D - Confirm Recovery

21. Error budget / SLO burn returning to normal for success, latency, cost.
22. No increase in safety blocks or policy violations beyond baseline noise.
23. Canary cohort (if any) empty or healthy - no residual traffic on bad pin.
24. Support queue trend - similar ticket rate to pre-incident window.
25. On-call agrees "stable" - explicit statement in channel before leaving D.

E - Prevent Instant Repeat

26. Quarantine the bad pin - mark undeployable; block that version ID in CD.
27. Add or fix gold-set cases that would have failed the gate (link PR).
28. Lower auto-promote confidence if the gate was green but prod burned - tighten thresholds or add canary abort rules.
29. Patch runbooks with the actual flip commands/UI paths you used.
30. Schedule postmortem within policy window - use agent-shaped template (bounds, pins, gates).
31. Owner for re-release - who will ship a fixed pin only after eval + canary.

Decision Card (Print / Pin in Channel)

IMPACT?  safety/money  → kill tools + pin flip + human review of side effects
         quality only  → pin/flag flip first
         cost/latency  → model route + max_turns/spend caps, then pin
STILL BAD after pin flip?
         → code redeploy + check external pack overrides + dependency outages
NEED TIME?
         → leave agent in degrade mode (retrieve-only / draft-only) rather than thrash

Common Failure Modes During Rollback

Mistake	Result	Fix
Roll image but not pack	Behavior unchanged	Flip pack pin / flags
Roll pack but not schema-compatible code	Mass tool errors	Roll both as a train
Non-sticky residual canary	Random bad sessions	Percent 0 + verify sticky maps
"Fixed" by editing live prompt	Unknown version	Ship proper pin later
No probe after flip	False recovery	Synthetic + metric confirm
Re-enable canary same day without new eval	Repeat SEV	Quarantine + new gate evidence

FAQs

How fast should pin rollback be?

Target under five minutes from decision to new runs on baseline, including config propagation. If slower, invest in flags and pin stores.

Is redeploying always safer than a flag flip?

No. Redeploys are slower and coarser. Use them when the binary or migrations are wrong.

What if we do not have a previous pin?

That is the incident behind the incident. Freeze changes, reconstruct from git, and build pin history before the next launch.

Should we roll back during an upstream model outage?

If the outage is provider-wide, failover routes or degrade. If only your new model ID fails, roll the model pin.

Can we partial-roll one tenant?

Yes - force baseline for the affected org flag override while others stay candidate, then global baseline if pattern spreads.

Where do kill switches stop and rollback starts?

Kill switch stops ongoing damage now. Rollback restores known good behavior for new runs. Often you do both.

How long do we stay on baseline after a SEV?

Until a fixed pin passes eval, canary abort rules are satisfied, and the postmortem action items that block re-release are done.

What metrics prove rollback worked?

Pin dimension shows baseline, proxy success recovered, cost/latency back in band, and no new safety SEVs - not "feels better."

Do we roll RAG indexes the same way?

If the index is versioned, point retrieval at the prior index ID. If not versioned, restore from snapshot or reindex from last good corpus pin.

Who can authorize break-glass stay on bad pin?

Rare and written: executive/product + eng lead with explicit residual risk. Default is roll back.

How do we practice this?

Game-day: deploy a deliberate bad pin to staging, time the flip, verify probes, and fix doc gaps.

What belongs in the postmortem every time?

Pin IDs, gate results (why green), canary gaps, time-to-detect, time-to-rollback, and new suite cases.

What "Continuous Deployment" Means for a Prompt-Driven System - rollback as a product feature
Versioning Prompts and Tool Schemas Alongside Code - pins to restore
Feature-Flagging Agent Behavior Changes - flip mechanics
Canary and Shadow Deployments for Agent Updates - abort rules
Running Eval Suites as a CI Gate Before Deploy - stop repeats
CI/CD & Agent Lifecycle Best Practices - preventive habits
Kill Switches: Stopping a Runaway Agent Mid-Execution - stop in-flight harm
Postmortems for Agent Incidents: A Template - after action
Setting SLOs for Agent Latency, Success Rate, and Cost - recovery signals

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Rollback Strategies When a Deployed Agent Regresses

Search Documentation

Rollback Strategies When a Deployed Agent Regresses