Code Review Agents: Automated PR Feedback at Scale
Code review agents read a diff (and often surrounding context), check it against policies and risk patterns, and leave structured feedback on the pull request.
They scale first-pass review. They do not own merge accountability.
How to Use This List
- Skim tables before you design a bot workflow or buy a review product.
- Map each row to your stack (GitHub/GitLab, CI, monorepo paths).
- Prefer starting with comment-only mode before any blocking gates.
- Revisit after two weeks of PR noise metrics (false positives kill adoption).
A - What a Review Agent Actually Does
| Capability | What it looks at | Typical output |
|---|---|---|
| Diff comprehension | Changed hunks + nearby symbols | Summary of intent vs risk |
| Defect heuristics | Nulls, races, authz gaps, error swallowing | Line comments with rationale |
| Style / convention | Team rules, linters, naming | Nits separated from blockers |
| Security patterns | Secrets, injection sinks, unsafe APIs | High-severity findings first |
| Test adequacy | Diff vs test files | "Missing coverage on X" notes |
| Policy checks | OWNERS, license, migration rules | Pass/fail checklist items |
B - Inputs the Agent Needs
| Input | Why it matters | Minimum viable |
|---|---|---|
| Unified diff / PR files API | Core change set | Required |
| Base branch SHA | Correct merge base | Required |
| CI status / logs | Ground feedback in real failures | Strongly recommended |
| Repo conventions file | Encode "how we review here" | README / CONTRIBUTING / rules file |
| Historical PR comments | Reduce repeated nits | Optional |
| Severity taxonomy | Blocker vs nit language | Required for signal quality |
C - Feedback Modes (Pick Deliberately)
| Mode | Behavior | Use when… |
|---|---|---|
| Advisory comments | Posts notes; never blocks | First rollout |
| Checklist report | Single summary comment with scores | Noisy repos; prefer one thread |
| Blocking status check | Fails CI on severity ≥ threshold | Mature rules; low false positives |
| Auto-request changes | Uses review API "request changes" | Only with tight, tested policies |
| Silent shadow | Logs findings without posting | Calibration week |
D - Severity Rubric (Copy and Adapt)
| Severity | Examples | Human action |
|---|---|---|
| S0 Blocker | Auth bypass, secret commit, data loss risk | Must resolve before merge |
| S1 High | Likely prod bug, missing migration, unsafe default | Resolve or explicit waiver |
| S2 Medium | Edge-case bug, weak tests on critical path | Fix or track follow-up |
| S3 Low / nit | Style, naming, comment typos | Optional; limit volume |
| Info | Architecture suggestion | Never block |
E - Where Review Agents Excel vs Struggle
| Excels | Struggles |
|---|---|
| Mechanical policy (license headers, forbidden APIs) | Product intent and UX taste |
| Common bug classes in popular languages | Deep domain invariants with no tests |
| Consistent nit reduction for juniors | Cross-service design trade-offs |
| Secret and dependency red flags | Performance judgment without profiles |
| "Tests didn't touch changed code" | Flaky CI interpretation |
F - Pipeline Placement Cheatsheet
| Stage | Agent role | Tip |
|---|---|---|
| Pre-commit / IDE | Local quick checks | Keep under a few seconds |
| PR open | First-pass summary | Post once; update on push |
| CI parallel | Security + test-gap scan | Cache repo context |
| Before merge | Re-check latest commit only | Avoid commenting on outdated lines |
| Post-merge | Rare; prefer pre-merge | Use for audit sampling only |
G - Anti-Patterns (Avoid These)
| Anti-pattern | Why it hurts | Prefer |
|---|---|---|
| Wall of S3 nits | Authors ignore everything | Cap nits; separate severity |
| Blocking on style the linter already covers | Duplicate noise | Let linters own style |
| No waiver path | Emergency merges stall | Explicit override + audit log |
| Reviewing generated files | Noise and token waste | Path denylist |
| Hidden model prompts as "policy" | Unreviewable rules | Versioned rules in-repo |
| Auto-merge on agent LGTM alone | Accountability hole | Human for risky paths |
H - Metrics That Matter
| Metric | Healthy signal | Warning signal |
|---|---|---|
| Precision of S0/S1 | Authors agree findings are real | Constant "false alarm" replies |
| Time-to-first-feedback | Minutes after PR open | Hours later than human review |
| Comment volume per PR | Bounded, severity-sorted | Dozens of undifferentiated nits |
| Human override rate | Occasional, explained | Always ignored or always rubber-stamped |
| Escaped defects in agent-reviewed PRs | Flat or down | Up after bot adoption |
I - Minimal Rules File Outline
| Section | Content |
|---|---|
| Always flag | Secrets, eval, raw SQL concat, disabled auth |
| Never flag | Pure formatting already in CI, vendored code |
| Language packs | Python/TS/Go-specific footguns you care about |
| Path policies | payments/** always human-reviewed |
| Tone | "Suggest" vs "Require"; no shaming language |
| Output format | Severity, file:line, why, suggested fix |
FAQs
Do code review agents replace human reviewers?
No. They scale first-pass inspection. Humans still own design judgment, merge decisions, and accountability.
Should the agent block merges on day one?
Usually no. Run advisory or shadow mode until S0/S1 precision is trustworthy.
What context beyond the diff should it read?
Enough surrounding code to understand call sites and tests, plus project rules. Not the entire monorepo every time.
How do we stop comment spam?
Severity caps, nit budgets, update-in-place summaries, and denylists for generated paths.
Is lint + typecheck enough without an agent?
Those catch mechanical issues. Agents help with cross-hunk reasoning and policy narrative - overlapping but not identical.
Where should rules live?
In-repo, versioned, reviewable files. Prompt-only policy is invisible and drifts.
Can one agent review every language in a monorepo?
Yes with language-specific rule packs and path routing. One generic prompt over everything performs worse.
How do review agents interact with coding agents that open PRs?
Treat agent-authored PRs the same or stricter. Require human review on sensitive paths regardless of who wrote the diff.
What about secrets in the diff?
Always high severity. Prefer dedicated secret scanners in CI as a hard gate; the agent is a backup narrative layer.
Should findings include suggested patches?
Yes for clear mechanical fixes. Keep patches small and never auto-apply on protected branches without policy.
How do we evaluate a review agent before buying or building?
Label a set of historical PRs with known issues; measure precision/recall on S0/S1; measure author satisfaction for two sprints.
What is the biggest adoption killer?
High-volume low-value nits that train authors to ignore the bot.
Related
- How Coding Agents Changed Day-to-Day Software Development - workflow context for PR volume
- Coding & Developer-Tooling Use Cases Basics - packaging tasks and reviewing traces
- Autonomous Bug-Fixing Agents: From Issue to Verified Patch - when agents author the PR under review
- Documentation and Onboarding Agents for Codebases - related repo-grounded assistance
- Coding & Developer-Tooling Use Cases Best Practices - human review of risky parts
Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana
@solana/web3.js+@solana/spl-token.