Feature-Flagging Agent Behavior Changes

Feature flags let you ship agent code and packs dark, then turn behavior on for cohorts without a scramble deploy.

For agents, flags should select pins and policies (prompt version, tools, model route, autonomy), not sprinkle if flag through every prompt string.

Summary

Define flag keys for pin selection and behavior gates, resolve them once per run into a concrete AgentPin, log the flag variation with the pin, roll out by percent or allowlist, and keep kill/rollback as a flag flip to the last safe variation.

Recipe

1. Inventory behavior dimensions worth flagging: prompt pack, tool pack, model route, max turns, tool allowlist, autonomy level.
2. Store flags in a central service or config file read by all workers (short TTL).
3. Resolve flags → AgentPin + runtime policy before the first model call.
4. Use sticky keys (user_id, org_id, session_id) for percent rollouts.
5. Default fail-safe: missing flag service → last known good pin, not "newest experimental."
6. Gate irreversible tools with separate flags or privilege rings.
7. Emit flag keys/values on traces and analytics.
8. Clean up flags after full rollout; do not leave permanent boolean soup.

Working Example

from __future__ import annotations
 
import hashlib
from dataclasses import dataclass
from typing import Any
 
 
@dataclass(frozen=True)
class AgentPin:
    prompt_version: str
    tool_schema_version: str
    model_id: str
    max_turns: int = 8
 
 
# Simulated flag payload from LaunchDarkly/Unleash/Homebrew config (verify vendors at build)
DEFAULT_FLAGS: dict[str, Any] = {
    "agent.support.prompt": "v1",
    "agent.support.tools": "tools_v1",
    "agent.support.model": "model-a",
    "agent.support.canary_prompt": "v2",
    "agent.support.canary_percent": 0,
    "agent.support.allow_refund_tool": False,
}
 
 
def bucket(key: str, flag_name: str) -> int:
    raw = hashlib.sha256(f"{flag_name}:{key}".encode()).hexdigest()
    return int(raw[:8], 16) % 100
 
 
def resolve_pin(subject_key: str, flags: dict[str, Any]) -> AgentPin:
    percent = int(flags.get("agent.support.canary_percent", 0))
    use_canary = percent > 0 and bucket(subject_key, "agent.support.canary_percent") < percent
    prompt = (
        flags["agent.support.canary_prompt"]
        if use_canary
        else flags["agent.support.prompt"]
    )
    return AgentPin(
        prompt_version=str(prompt),
        tool_schema_version=str(flags["agent.support.tools"]),
        model_id=str(flags["agent.support.model"]),
        max_turns=8,
    )
 
 
def tool_allowed(name: str, flags: dict[str, Any]) -> bool:
    if name == "refund_order":
        return bool(flags.get("agent.support.allow_refund_tool", False))
    return True
 
 
def start_run(user_id: str, flags: dict[str, Any] | None = None) -> dict[str, Any]:
    f = flags or DEFAULT_FLAGS
    pin = resolve_pin(user_id, f)
    return {
        "user_id": user_id,
        "pin": pin,
        "refund_enabled": tool_allowed("refund_order", f),
        "flag_snapshot": {
            "prompt": pin.prompt_version,
            "canary_percent": f.get("agent.support.canary_percent"),
        },
    }
 
 
print(start_run("org_9:user_1"))
print(start_run("org_9:user_1", {**DEFAULT_FLAGS, "agent.support.canary_percent": 100}))

Deep Dive

What to flag (and what not to)

Flags select versioned artifacts. They should not be the only store of the prompt body.

Resolution order

request context (user, org, plan)
  → flag evaluation
  → pin + policy
  → load pack from store
  → agent loop
  → log pin + flag variation

Resolve once per run. Re-evaluating mid-loop can switch prompts mid-thought.

Cohort strategies

Sticky org-level canaries prevent one company from seeing two policies across seats inconsistently - choose deliberately.

Fail-open vs fail-closed

Agent safety-related flags should fail closed.

Pairing with CI and canary metrics

1. PR merges pack v2 while flag still points at v1.
2. Eval gate already passed for v2.
3. Flag percent → 5%; watch SLOs.
4. Percent → 100%; then change default in config; remove dead flag later.

See Canary and Shadow Deployments for Agent Updates.

Local and test environments

• Inject flag fixtures in unit tests.
• Snapshot evaluation for critical paths.
• Never require production flag credentials to run the test suite.

Cleanup and debt

Track flags with owners and expiry.

Long-lived agent_v2_enabled flags become untestable matrix hell. After full rollout, make v2 the code default and delete the branch.

Multi-agent systems

Flag at the orchestrator policy level when possible (which specialist pin to call).

Per-node flag sprawl without a single resolved plan is hard to reason about in incidents.

Gotchas

1. Prompt text only in the flag payload. No PR review, no eval on change.
2. Non-sticky percent flags on chat agents. Jittery behavior mid-session.
3. Different flag values per worker without TTL discipline. Split-brain packs.
4. Flag true enables refunds globally in prod by mistake. Separate env projects; default false.
5. Logging flags but not pin hashes. You know the label, not the bytes.
6. Nesting ten flags for one feature. Collapse to a pin ID.
7. Using flags instead of kill switches for runaway loops. Kill switches must be faster and simpler - see HITL kill switch docs.
8. Forgetting mobile/async workers. All entrypoints must evaluate the same flags.

Alternatives

FAQs

Related

• What "Continuous Deployment" Means for a Prompt-Driven System - CD context
• CI/CD & Agent Lifecycle Basics - first percent router
• Versioning Prompts and Tool Schemas Alongside Code - what flags select
• Canary and Shadow Deployments for Agent Updates - traffic patterns
• Rollback Strategies When a Deployed Agent Regresses - flip back
• Kill Switches: Stopping a Runaway Agent Mid-Execution - emergency stop
• Privilege Rings: Scoping What an Agent Can Do Without Asking - autonomy gates
• CI/CD & Agent Lifecycle Best Practices - checklist

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.