Building an LLM Code Review Bot People Do Not Mute

Every LLM review bot follows the same arc. Week one it finds a real bug and everyone is impressed. Week three it has left four hundred comments about naming conventions and someone has turned off notifications. Week six it is disabled.

The engineering problem is not getting a model to comment on a diff. That is a webhook, an HTTP call and a POST. The problem is precision — the fraction of comments a reviewer acts on — and almost everything in the design should be pointed at raising it.

The pipeline in outline

A review bot is four steps. Receive the pull request event, fetch the diff, ask a model about it, post comments back. Each has a decision in it that matters more than the plumbing.

Trigger on pull_request opened and synchronised, not on every push to every branch. On synchronise, review only the new commits since your last run rather than the whole diff again, otherwise every force-push regenerates the same comments and the thread becomes unreadable.

Fetch the diff in unified format with a small amount of context — three lines is the git default and usually too little for a model, ten is a reasonable compromise. Ask for the patch rather than the full files first; if a finding needs more context, fetch that file specifically in a second call rather than sending the whole repository up front.

Then post. Inline review comments anchored to a file and line are far more useful than a summary comment, because they appear where the reviewer is already looking. That means your prompt has to produce a structured result you can map onto positions, which brings us to output format.

Make the output structured, then validate it

Free-form prose cannot be turned into inline comments reliably. Ask for JSON and enforce a schema:

{
  "findings": [
    {
      "path": "src/auth/session.ts",
      "line": 142,
      "severity": "high",
      "category": "bug",
      "message": "Session token compared with == rather than a constant-time compare.",
      "confidence": 0.8
    }
  ]
}

Then validate before posting. Does the path exist in the diff. Is the line one that the diff actually touched. Is severity in your enum. Drop anything that fails, silently — a hallucinated file path posted as a review comment is the single fastest way to lose a team's trust in the bot.

The general mechanics of getting reliable JSON out of a model, including what to do when it returns prose anyway, are covered in structured outputs and JSON mode. For a review bot the important habit is treating the model output as untrusted input, parsed and checked like anything else that arrives over the network.

Filter aggressively before posting

This is where a tolerable bot becomes a good one. Take the findings the model returns and throw most of them away.

Drop everything below a confidence threshold. Drop entire categories — style, naming, formatting, missing comments — because a linter already handles those, deterministically and for free, and a model disagreeing with your linter is noise by definition. Cap the total at something small, five or six comments per pull request, ranked by severity. If the model found twenty things, posting five real ones beats posting twenty of which five are real.

Deduplicate against what is already on the pull request. On the second run the model will find the same issue again, and a bot that repeats itself reads as broken. Keep a fingerprint of each posted finding — file, line, a hash of the message — and skip matches.

Track the acted-on rate from the start. Whether a comment was resolved, replied to, or led to a change in the next commit is a measurable signal, and it is the only honest way to tune the threshold. The broader framing is in what to automate in code review and what to leave to humans.

What the model is genuinely good at here

Reviewers are inconsistent at exactly the things a model is consistent at, which is where the value sits.

Error handling gaps. A promise without a catch, an error swallowed into a log line, a resource opened in a branch that can return early. Missing edge cases at boundaries — empty collections, null after an optional lookup, an index that assumes non-empty. Inconsistency with a convention visible in the surrounding diff. And the mechanical security checks: string-concatenated SQL, a secret in a committed file, user input reaching a shell command.

What it is bad at is anything requiring knowledge outside the diff. Whether this abstraction fits the codebase, whether the change is the right approach, whether the test covers the behaviour that matters. Those need the whole repository and design context, and asking for them produces confident, plausible, wrong opinions. If you want a model to reason across a large codebase, that is a different tool with different requirements — see picking a model for large repositories.

Model choice and cost

Review is a bursty workload. Nothing for an hour, then eight pull requests in ten minutes because a team merged a stack. Latency matters less than it does in an editor — a comment appearing ninety seconds after the pull request opens is fine — so you can afford a stronger model than you would use interactively.

Diffs are usually small enough that context window is not the binding constraint. Where it becomes one is a bot that sends whole files, or a monorepo with generated code in the diff. Filter generated paths, lockfiles and vendored directories out before you count tokens; they are pure cost and produce no findings anyone wants.

Budget by pull request rather than by month, because that is the unit you can reason about. If review costs a few cents per pull request and your team opens forty a week, the arithmetic is easy and the answer is usually that it is worth it. Cost per pull request is a more useful metric here than cost per developer, precisely because it scales with the thing that generates the spend.

Tone, permissions and the failure modes

Instruct the model to report findings only, with no praise, no summary of what the change does, and no comment when it finds nothing. A bot that posts "This looks good overall" on every pull request adds a notification and no information.

Give it read access to the diff and write access to review comments, and nothing else. It should not be able to approve, request changes in a blocking way, merge, or push. An approval from a bot is worse than useless because it looks like review happened.

Handle the ugly cases explicitly. A pull request with two thousand changed files should be skipped with a note rather than truncated silently. A model call that fails should leave no comment rather than an error comment. A pull request from a fork should not run with credentials that a fork author can influence, which is a genuine prompt injection surface: the diff is attacker-controlled text, and a comment instructing the model to ignore its rules is a two-line change anyone can push.

A test before you ship it

Run the bot in shadow for two weeks. Have it write findings to a log instead of the pull request, then read them yourself against the merged outcome. Count how many you would have acted on.

If that number is above roughly one in three, ship it. If it is one in ten, the bot will be muted no matter how good the individual finds are, and the fix is a higher threshold and fewer categories rather than a better model.