Skip to content

Workflow · Coding

Build an AI Code Reviewer

An AI bot that reviews every PR — catches bugs, suggests improvements, blocks dangerous patterns. Like having a senior engineer on every team.

Intermediate2–4 hoursStack: Claude or GPT-5, GitHub Actions / API, Node.js / Python
Code ReviewAgentGitHubDevOps
Edited by The AIKnowHub team · Editorial team

Problem

PRs sit waiting for review. Reviewers miss things in long diffs. Junior PRs need basic checks before senior eyes. The same review comments get repeated across PRs — convention drift.

Final output

A GitHub Action that runs on every PR, posts a structured review comment within 60 seconds covering: correctness concerns, security issues, performance flags, convention drift, and a suggested checklist. Blocks merge on critical findings.

Architecture

PR opened / updated
  → GitHub Action triggered
  → Fetch diff + changed files + related files
  → Send to LLM with review prompt + repo conventions
  → Parse structured findings
  → Post review comment (or convert to inline comments)
  → Set status check (block merge if critical findings)

Step-by-step

  1. 01

    GitHub Action setup

    Trigger on pull_request events. Use github-script or a custom Node script with @octokit/rest. Required permissions: pull-requests: write, contents: read.

  2. 02

    Fetch diff + context

    Get the PR diff via Octokit. For each changed file, optionally fetch the full file + related files (imports, callers). Cap total context at ~50K tokens.

  3. 03

    Build the review prompt

    System prompt: senior engineer reviewer + repo conventions (CLAUDE.md / .cursor/rules). User prompt: 'review this diff for [list of categories]'. Require structured output.

  4. 04

    Call the LLM with structured output

    Use Anthropic / OpenAI with JSON mode. Schema: { findings: [{file, line, severity, category, message, suggestion}], summary }.

  5. 05

    Post review back to GitHub

    Convert findings into inline review comments. Post a summary comment. If any critical findings, fail the status check.

What you'll build

A GitHub bot that reviews every pull request automatically. Posts structured comments. Catches issues human reviewers miss. Never sleeps.

The reality of AI code review in 2026

It's good — at the right things:

  • Strong: bug detection on isolated functions, security flag-raising, convention adherence checks, missing-test detection.
  • Weak: architectural critique, business-logic correctness, taste / style judgment.

Designed correctly, the bot handles 60–80% of mechanical review and frees humans for the remaining 20–40% that requires actual thinking.

Tuning matters

The default LLM-on-diff approach posts noisy reviews. The difference between a bot engineers love and one they ignore is:

  • Repo conventions in the prompt — what's idiomatic in your codebase.
  • Severity tiers — critical vs nice-to-have.
  • Comment cap — max 10 per PR, prioritized by severity.
  • Category filtering — drop the categories your team consistently rejects.
  • Status check policy — block on critical only; everything else is informational.

A well-tuned bot has a 60–80% comment-accept rate and engineers actively look for its review.

Operating cost

A typical 100-PR-per-week team spends $50–$200/month on this depending on diff sizes and model choice. That's 0.05% of an engineer's salary for arguably the highest-leverage AI investment a team makes.

Beyond review

Once the basic bot ships, common extensions:

  • Auto-suggest commits — bot proposes the fix as a follow-up commit.
  • Pre-merge migration check — verify migrations are safe.
  • Test-coverage diff — flag PRs that ship code without tests.
  • Spec-implementation match — given a linked spec/issue, verify implementation aligns.

Each is a one-day extension on top of the base loop. The platform compounds.

Prompt examples

Copy any of these, replace the placeholders, run.

Reviewer system prompt

You are a senior software engineer reviewing a pull request.

Repository conventions:
{{repo_rules}}

Review the diff for:
- Correctness: bugs, race conditions, off-by-one errors
- Security: injection, authn/authz, secret handling
- Performance: N+1 queries, allocations on hot paths
- Maintainability: naming, structure, unclear sections, dead code
- Convention drift: anything violating the repo conventions above

Severity:
- critical: blocks merge (security, correctness bugs)
- high: should fix before merge
- medium: nice to fix
- low: optional / style

Be direct. Do not pad with "great work overall" if it isn't.

Output JSON:
{
  "findings": [
    { "file": "path", "line": number, "severity": "critical"|"high"|"medium"|"low",
      "category": string, "message": string, "suggestion": string }
  ],
  "summary": "1-3 sentences"
}

Cost estimate

Line itemApprox. cost
Per PR (typical 500-line diff, Sonnet)$0.15
Per PR (large diff with related context)$0.50
Per 100 PRs / week$15.00
GitHub Action minutes (negligible)$0.01
Total per run (approx.)~$15.66

Costs depend on model choice, content length, and how aggressively you cache.

Optimizations

  • Skip files matching ignore patterns (vendored deps, generated, lockfiles).
  • Use prompt caching for the conventions block — same text every PR.
  • Run the cheap model (Haiku / Mini) first; escalate to Sonnet only if findings warrant.
  • Per-language specialized prompts (Python vs TypeScript vs Go) — better recall on idiomatic issues.
  • Skip review on docs-only PRs.

Common mistakes

  • Posting hundreds of inline comments — overwhelming, ignored. Cap at 10 per PR.
  • No severity tiering — every comment looks equally important.
  • Reviewing without repo conventions — model invents standards that don't match yours.
  • Blocking merge on noisy false positives — engineers learn to ignore the bot.
  • Skipping logs — without metrics on accept rate, you can't improve the bot.

Frequently asked questions

No. Augments. Catches mechanical issues so humans can focus on architecture, design, and business logic. The best deployments treat the bot as a 'first pass' before human review.

Related on AIKnowHub