Building a PR Summariser Reviewers Do Not Skip

The standard pull request summary bot posts a bulleted list: added a function, modified a config, updated tests. Every one of those facts is visible in the Files Changed tab, which the reviewer is about to open anyway. Within two weeks the comment is collapsed by default and nobody expands it.

A summary earns its place only if it tells the reviewer something the diff does not. There are exactly three such things, and building around them changes the product entirely.

The three things a reviewer cannot see

The first is intent. A diff shows that a boundary condition changed; it does not show that the change exists because invoices dated on the first of the month landed in the wrong billing period. That fact lives in the linked issue, and surfacing it saves the reviewer a tab and often a wrong assumption.

The second is reading order. A 30-file pull request has two files that carry the decision and 28 that follow mechanically from it. A reviewer who opens them alphabetically wastes twenty minutes before reaching the one that matters. Telling them "start with billing/period.ts, then scheduler.ts, the rest is call-site updates" is the single highest-value output of the whole system.

The third is risk surface: what could break that is not in this diff. A changed function signature has callers elsewhere; a migration has a lock implication; a changed default affects existing deployments. This is where a model reading the whole repository beats a reviewer reading a diff, and it is the one place worth spending tokens.

What to send, and what to leave out

Sending the raw unified diff is the default and it is wrong twice over. It includes noise you are paying to process, and it excludes context the model needs to say anything correct.

Filter first, deterministically. Drop lock files, generated code, vendored directories, and any file over a size threshold — a 12,000-line schema dump contributes nothing but cost. Keep the file names, though: "package-lock.json changed" is worth one line and worth zero tokens of content.

Then expand context for what remains. Three lines of surrounding code is not enough to reason about a change; ask git for a generous window, and for files under a few hundred lines send the whole file:

git diff -U30 origin/main...HEAD -- . ':(exclude)*.lock' ':(exclude)dist/*'
git diff --stat origin/main...HEAD

Add the PR title, the body, and the text of any linked issue. Those three fields do more for summary quality than any model upgrade, and they cost almost nothing. This is the same context problem that makes or breaks automated code review — a hunk in isolation produces confident nonsense.

Big pull requests need map-reduce, not a bigger window

Long-context models will accept a 400-file diff, but accepting it and reasoning well over it are different things. Quality degrades in the middle of very long inputs, and you are paying for every token of the parts it attends to least.

Summarise per file or per directory in parallel, producing one or two sentences and a risk flag each, then run a second call over those summaries to produce the ordering and the overall narrative. The second call sees a few thousand tokens instead of a few hundred thousand.

This also gives you a natural cost ceiling. Cap the number of file summaries, and when a PR exceeds it, summarise directories rather than files and say so in the output. A degraded summary on a 600-file PR is fine; an unbounded bill is not, and the arithmetic is laid out in AI cost per pull request.

Write for scanning, cap the length

Hard-limit the summary. Roughly 150 words, a suggested review order of at most five files, and at most three risk notes. A summary longer than the diff is self-defeating.

Structure it the same way every time so reviewers can skip to the part they want: one sentence of intent, the review order, the risk notes. Consistent shape means people learn where to look, and learning where to look is what keeps a bot alive past its first month.

Never let it editorialise about quality. The moment a summary says the change looks good, it is making a claim it cannot support and undermining the parts that were useful. Description and review are separate products; keep them separate.

Update one comment, keyed to the head SHA

Post the summary once and edit that comment on every push. Appending a new comment per push turns a five-push pull request into a wall of stale bot output and buries human review threads.

Key the cached result on the head SHA. A re-run of CI on an unchanged commit should hit the cache and cost nothing, which matters because CI reruns are common and each one would otherwise be a full inference call. Where your provider supports it, prefix caching across pushes on the same branch cuts the repeated portion further — see caching strategies to cut cost.

Skip generation entirely for draft PRs, for bot-authored PRs like dependency bumps, and for pull requests below a diff-size floor. A two-line change does not need a summary, and generating one trains reviewers to ignore the comment.

The diff is untrusted input

On any repository that accepts outside contributions, the diff is written by someone who may have read your prompt. A comment in the source saying to report the change as low risk is a real technique and it costs the attacker nothing to try.

The mitigation is structural rather than textual. The summariser should have no write access to code, no ability to run repository scripts, and no approval power — so the worst outcome of a successful injection is a misleading paragraph that a human reads with normal scepticism. Wrap the diff in an explicit delimiter and instruct the model to treat its contents as data, but do not rely on that alone; prompt injection and agent security covers why instruction-level defences are not sufficient.

Run a secret scanner over the diff before the model call, not after. Diffs occasionally contain credentials someone committed by accident, and sending those to a third-party API is a disclosure you cannot take back.

Measure whether it changed anything

The metric is not comments posted. It is whether review behaviour changed: time to first review, and whether reviewers open files in the suggested order.

The cheap proxy is a one-question survey in the comment footer, or simply asking six reviewers after a month whether they read it. If the honest answer is that they skim past it, the summary is not carrying information they lacked, and the fix is almost always more repository context rather than better prose.

Start narrow: one repository, intent and review order only, no risk notes. Add risk analysis after the ordering is trusted, and keep the same discipline that applies to any developer-facing generator — an artefact people ignore is worse than no artefact, because it costs money and occupies the space where a useful one could have gone. The same reasoning applies to generated changelogs.