Best Model for Pull Request Review: The False Positive Tax

An automated pull request reviewer has one failure mode that matters and it is not missing bugs. It is being ignored. A bot that posts eleven comments on every PR, of which two are useful, gets muted within a fortnight and never gets turned back on.

This is a different problem to whether a model can review code well. That question is covered in choosing a model for code review. This article is about the workflow the model sits inside, which is where most implementations fail.

Precision matters far more than recall

In most detection problems you trade off between catching everything and being right when you speak. PR review is unusually lopsided: precision dominates, because the cost of a false positive is paid by a human on every single PR and the cost of a false negative is diffuse.

Work through what a wrong comment actually costs. A developer reads it, thinks about it, decides it is wrong, and either replies or dismisses it. Say ninety seconds. At forty PRs a week with three spurious comments each, that is roughly three hours a week of your team's attention spent disagreeing with a bot.

Worse, it trains people to skim. Once the default assumption is that bot comments are noise, the genuinely important one gets skimmed too, and you have made review worse than having no bot at all.

So configure for precision aggressively. Instruct the model to comment only where it is confident there is a defect, cap the number of comments per PR at three or four, and make it rank by severity and drop the tail. A bot that says nothing on a clean PR is doing its job.

Style opinions are the main source of noise

Left unconstrained, models comment on naming, structure, whether a helper should be extracted, and general tidiness. Most of it is defensible and almost none of it is worth a review cycle.

Anything mechanical belongs to a linter and a formatter, which are deterministic, free, and do not need agreeing with. Configure those properly and then explicitly forbid the model from raising anything they cover.

What you actually want from the model is the class of defect that static analysis cannot reach: a null path the types do not capture, an off-by-one in a boundary condition, a missing await, a change that silently alters behaviour for an existing caller, a permission check that moved. Write that list into the system prompt as the scope, and say that everything else is out of scope.

The narrower the brief, the higher the precision, and the more likely anyone reads the output. Code review automation covers building the pipeline around this.

The diff is not enough context

The naive implementation passes the diff and asks for a review. It produces exactly the comments you would expect from someone who has only seen the diff: plausible, generic, and frequently about something the surrounding code already handles.

Give the model the full text of every changed file, not the hunks. Give it the PR title and description so it knows what the change is meant to do — most valuable review comments come from comparing intent against implementation. Where the diff calls a function defined elsewhere, pull that definition in.

A typical PR with full file context lands somewhere between 20K and 80K tokens, which is unremarkable for any current model. The one-million-token windows on Kimi K3, GLM-5.2, DeepSeek V4 and MiniMax M3 are irrelevant here — you are nowhere near the limit, and the effort belongs in choosing the right context rather than adding more.

Latency decides whether it is part of the workflow

A review that arrives in ninety seconds is part of the conversation. One that arrives in twelve minutes lands after the author has moved on, and gets read as an interruption rather than help.

This puts a real ceiling on how much reasoning you can afford. Long chain-of-thought traces on a heavily reasoning-tuned configuration will produce a slightly better review that nobody is waiting for. GLM-5.2's two reasoning effort levels are useful here precisely because you can pick the lower one for routine PRs and reserve the higher for changes touching sensitive paths.

Post as a review with inline comments on the relevant lines rather than one summary block. Inline comments are anchored to code and are actioned; summary blocks are collapsed. Low-latency model selection covers where the time actually goes.

Cost per PR, and where it goes wrong

Per-PR cost looks trivial and then surprises people at team scale. A 60K-token input with a 1.5K-token output on GLM-5.2 at roughly $1.40 in and $4.40 out per million is well under fifteen cents. On Kimi K3 at $3 in and $15 out it is closer to forty.

At two hundred PRs a month that is thirty dollars against eighty — neither of which is a real number for a team of engineers. The cost problem appears when you re-review on every push. A PR with fourteen commits reviewed on each one is fourteen times the bill for thirteen redundant reviews.

Review on PR open and on explicit request, not on every push. That one rule removes most of the spend and most of the noise simultaneously. AI cost per pull request works the arithmetic through properly.

Recommendation

For most teams, GLM-5.2 at the lower reasoning effort level is the right default: strong on structured code analysis, fast enough to land while the author is still present, and cheap enough that per-push cost discipline is a nice-to-have rather than a necessity.

For changes touching authentication, payments, data migrations or anything with a bad failure mode, route to Kimi K3. Its long-horizon reasoning is worth the extra latency and cost on the ten percent of PRs where a missed defect is expensive. A path-based rule in your CI config does this in a few lines.

Do not use the cheapest tier for review. DeepSeek V4 Flash is excellent for bulk reading, but review is precisely where false positives are expensive, and precision is what the cheap tier gives up.

Measure the thing that matters: of the comments posted last month, what fraction led to a code change? Below a third, tighten the scope or raise the confidence threshold. Above two thirds, you can afford to widen it. That single ratio tells you more than any benchmark.