Rollback Strategies When a Deployed Agent Regresses
Use this when production quality, safety, latency, or cost burns after an agent change.
The goal is minutes to safe, not a perfect root cause in the first five minutes.
How to Use This Cheatsheet
- Start at A if users are actively harmed or spend is spiking; otherwise walk A → E in order.
- Prefer the smallest reversible unit (flag/pin) before rebuilding the world.
- Tick only actions you actually completed; note times in the incident channel.
- Pair with Kill Switches: Stopping a Runaway Agent Mid-Execution when runs are still thrashing.
- After stabilize, finish E so the same pin cannot silently return.
A - Stabilize Now (First 5-15 Minutes)
- 1. Declare the incident - severity, commander, comms channel, customer impact one-liner.
- 2. Identify the active pin -
prompt_version,tool_schema_version,model_id,code_sha, flag variations from a failing run trace. - 3. Trip kill switch / disable risky tools if side effects are wrong (refunds, send, delete, shell).
- 4. Flip traffic to last known good pin - flag percent → 0,
active→ previous manifest, or route table → baseline. - 5. Confirm workers see the flip - sample new runs show baseline versions (watch config TTL).
- 6. Pause CD / auto-promote so a pipeline cannot re-push the bad pin.
- 7. Snapshot evidence - failing run IDs, dashboards, the bad and good manifest JSON, recent deploy times.
- 8. Customer comms if user-visible errors or wrong actions occurred (status page / support macro).
B - Choose the Rollback Unit
Pick the narrowest lever that removes the regression:
| Symptom cluster | Try first | Then |
|---|---|---|
| Quality drop after prompt-only change | Previous prompt_version | Full previous pin |
| Tool call storms / schema errors | Previous tool_schema_version + matching code | Disable tool via flag |
| Latency/cost after model swap | Previous model_id / route | Provider failover |
| Only canary cohort hurt | Canary percent → 0 | Keep baseline |
| Host crash / OOM / bad binary | Redeploy previous code_sha | Scale / restart |
| Partial migrations | Expand-contract down path | Feature-flag off new writes |
| Unsafe content / injection surge | Tighten policy pack + tool deny | Full agent off |
- 9. Name the unit you rolled - one line in the incident doc ("rolled prompt v17 → v16").
- 10. Avoid hot-editing prod prompt text as the primary fix - that creates a third unknown version.
C - Execute Pin / Flag / Deploy Rollback
Flag or config pin (preferred)
- 11. Set production flag/config to baseline pin IDs (not "latest").
- 12. Clear or respect caches - document max staleness; force refresh if supported.
- 13. Verify with synthetic probe - scripted case that failed under candidate, passes under baseline.
- 14. Watch live metrics 15-30 minutes - success, latency, cost, tool errors, escalations.
Application / container rollback
- 15. Redeploy last good artifact only if code_sha is implicated or pin lives only in the image.
- 16. Ensure pack files inside the image match the intended versions (no external override still pointing at bad pack).
- 17. Run smoke eval subset against the restored revision in prod-like env if time allows.
Data and side effects
- 18. List irreversible actions already taken - emails sent, tickets closed, charges, deletes.
- 19. Start compensating actions - reverse charges, notify users, reopen tickets - with humans in the loop.
- 20. Preserve audit trails - do not delete bad runs; mark them for review.
D - Confirm Recovery
- 21. Error budget / SLO burn returning to normal for success, latency, cost.
- 22. No increase in safety blocks or policy violations beyond baseline noise.
- 23. Canary cohort (if any) empty or healthy - no residual traffic on bad pin.
- 24. Support queue trend - similar ticket rate to pre-incident window.
- 25. On-call agrees "stable" - explicit statement in channel before leaving D.
E - Prevent Instant Repeat
- 26. Quarantine the bad pin - mark undeployable; block that version ID in CD.
- 27. Add or fix gold-set cases that would have failed the gate (link PR).
- 28. Lower auto-promote confidence if the gate was green but prod burned - tighten thresholds or add canary abort rules.
- 29. Patch runbooks with the actual flip commands/UI paths you used.
- 30. Schedule postmortem within policy window - use agent-shaped template (bounds, pins, gates).
- 31. Owner for re-release - who will ship a fixed pin only after eval + canary.
Decision Card (Print / Pin in Channel)
IMPACT? safety/money → kill tools + pin flip + human review of side effects
quality only → pin/flag flip first
cost/latency → model route + max_turns/spend caps, then pin
STILL BAD after pin flip?
→ code redeploy + check external pack overrides + dependency outages
NEED TIME?
→ leave agent in degrade mode (retrieve-only / draft-only) rather than thrashCommon Failure Modes During Rollback
| Mistake | Result | Fix |
|---|---|---|
| Roll image but not pack | Behavior unchanged | Flip pack pin / flags |
| Roll pack but not schema-compatible code | Mass tool errors | Roll both as a train |
| Non-sticky residual canary | Random bad sessions | Percent 0 + verify sticky maps |
| "Fixed" by editing live prompt | Unknown version | Ship proper pin later |
| No probe after flip | False recovery | Synthetic + metric confirm |
| Re-enable canary same day without new eval | Repeat SEV | Quarantine + new gate evidence |
FAQs
How fast should pin rollback be?
Target under five minutes from decision to new runs on baseline, including config propagation. If slower, invest in flags and pin stores.
Is redeploying always safer than a flag flip?
No. Redeploys are slower and coarser. Use them when the binary or migrations are wrong.
What if we do not have a previous pin?
That is the incident behind the incident. Freeze changes, reconstruct from git, and build pin history before the next launch.
Should we roll back during an upstream model outage?
If the outage is provider-wide, failover routes or degrade. If only your new model ID fails, roll the model pin.
Can we partial-roll one tenant?
Yes - force baseline for the affected org flag override while others stay candidate, then global baseline if pattern spreads.
Where do kill switches stop and rollback starts?
Kill switch stops ongoing damage now. Rollback restores known good behavior for new runs. Often you do both.
How long do we stay on baseline after a SEV?
Until a fixed pin passes eval, canary abort rules are satisfied, and the postmortem action items that block re-release are done.
What metrics prove rollback worked?
Pin dimension shows baseline, proxy success recovered, cost/latency back in band, and no new safety SEVs - not "feels better."
Do we roll RAG indexes the same way?
If the index is versioned, point retrieval at the prior index ID. If not versioned, restore from snapshot or reindex from last good corpus pin.
Who can authorize break-glass stay on bad pin?
Rare and written: executive/product + eng lead with explicit residual risk. Default is roll back.
How do we practice this?
Game-day: deploy a deliberate bad pin to staging, time the flip, verify probes, and fix doc gaps.
What belongs in the postmortem every time?
Pin IDs, gate results (why green), canary gaps, time-to-detect, time-to-rollback, and new suite cases.
Related
- What "Continuous Deployment" Means for a Prompt-Driven System - rollback as a product feature
- Versioning Prompts and Tool Schemas Alongside Code - pins to restore
- Feature-Flagging Agent Behavior Changes - flip mechanics
- Canary and Shadow Deployments for Agent Updates - abort rules
- Running Eval Suites as a CI Gate Before Deploy - stop repeats
- CI/CD & Agent Lifecycle Best Practices - preventive habits
- Kill Switches: Stopping a Runaway Agent Mid-Execution - stop in-flight harm
- Postmortems for Agent Incidents: A Template - after action
- Setting SLOs for Agent Latency, Success Rate, and Cost - recovery signals
Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana
@solana/web3.js+@solana/spl-token.