Building a Changelog Generator People Actually Read

Most generated changelogs are a list of commit subjects with the conventional-commit prefix stripped and the first letter capitalised. That is not a changelog. It is the git log with worse formatting, and the reason nobody reads it is that it contains no information a reader could not already get from the repository.

A changelog is a translation, not a summary. It converts a set of internal changes into statements about what is different for someone outside the repository. Almost every design decision in a generator follows from taking that seriously.

Decide the audience before the pipeline

One repository usually needs three different changelogs and they cannot share a prompt. End users want behaviour changes and nothing else. API consumers want the contract diff: new endpoints, changed response shapes, deprecations, and the migration for each. Internal engineers want the full set including refactors, because they are the ones who will bisect a regression next month.

Trying to produce one document that serves all three gives you the median output: too much detail for users, not enough for integrators. Generate three artefacts from one classified change set instead. The expensive part — working out what each change actually did — is shared; only the writing pass differs.

Write the acceptance test as a sentence before you build anything. "A customer who has not read our code can tell from this file whether the release affects them." If the generated output fails that test, no amount of prompt tuning fixes it, because the input does not contain the answer.

Commits are poor input; pull requests are better

Commit subjects are written for the author, minutes after the change, in a context where everything is obvious. Half of them say "fix" or "address review comments" or "wip". Feeding those to a model gets you fluent restatements of nothing.

Pull requests carry far more: a title someone wrote deliberately, a description aimed at a reviewer, linked issues describing the original problem in user language, labels, and the diff itself. The issue text is often the single most valuable field, because it is the only place the user-facing symptom is stated.

Where a PR body is empty, fall back to the diff summary rather than the commit subject — file paths and the shape of the change tell you more than "fix". If your team writes decent commit messages you have a better starting point than most, and generating commit messages well is a cheaper investment than compensating for bad ones downstream.

Classify first, write second

The reliable architecture is two passes. The first pass takes each change and emits a structured record; the second pass takes the filtered, grouped records and writes prose. Combining them produces output where the model has silently decided what to omit, and you cannot audit that decision.

The classification record wants to be small and machine-checkable:

{
  "id": "PR-4821",
  "kind": "fix",
  "audience": ["end-user"],
  "user_visible": true,
  "breaking": false,
  "component": "billing",
  "summary": "Invoices dated on the first of the month were assigned to the previous period.",
  "evidence": "closes #3311, changes periodFor() boundary at billing/period.ts:44"
}

Ask for one record per change, in isolation, with a fresh context. Isolation makes the pass parallelisable and stops the model harmonising twenty changes into a narrative before you have had a chance to filter. Requiring the evidence field discourages invention — a change with no citable evidence for its claimed user impact is one to route to a human.

Then filter in code. Drop everything where user_visible is false for the customer changelog. Group by component. Sort breaking changes first. Your code decides the structure; the model only supplies the sentences, which is the same division of labour that keeps structured output pipelines maintainable.

Reverts, cherry-picks and the duplication problem

Every real release contains a change and its revert, a fix to a bug introduced earlier in the same release, and the same commit landed twice via a cherry-pick to a release branch. A naive generator reports all of them, which is how you get a changelog announcing a feature that does not exist.

Handle this deterministically before the model sees anything. Match reverts by the commit hash referenced in the revert message and cancel both. Detect cherry-picks by patch-id and keep one. Where a fix references a PR that is also in the same release window, collapse them into a single entry — the user never saw the bug, so it is not a fix, it is part of the feature.

Models are unreliable at this because it requires exact identity matching across a long list, and a long list is exactly where attention gets thin. Give the model a clean set and it writes well; give it a raw set and it will confidently list a reverted feature.

Regeneration must be stable

Someone will rerun the generator on the same tag. If the output changes, review becomes impossible and the file starts producing spurious diffs in every release PR.

Cache classification results by pull request number plus the head SHA. A rerun then re-uses every record for changes that did not move, so only genuinely new changes are re-classified. That is both cheaper and far more stable than reissuing every call.

Pin the model version and record it in the generated file header, for the reasons set out in pinning model versions. Setting a fixed seed helps a little but is not a guarantee, as LLM determinism and seeds explains — caching is the mechanism you actually rely on.

Ship it as a draft, never as a commit

The generator should open a pull request against the changelog file, or post the draft as a release-draft body. It should never push directly to a branch or publish a release.

The reason is not that the output is bad. It is that a changelog is a public communication, and the cases where a model gets the tone or the severity of a breaking change wrong are exactly the cases that cost you support load. A thirty-second human read before publishing removes that risk entirely, and it is the same argument that applies to any model output that leaves the building — see running LLMs in CI pipelines for how to wire the gate without slowing releases down.

Make the draft easy to edit: one entry per line, plain markdown, PR links intact. If editing the output is harder than writing it from scratch, people will write it from scratch.

Breaking changes deserve their own path

A breaking change is not a bullet point with a warning emoji. It needs the old behaviour, the new behaviour, how to tell whether you are affected, and the migration step, in that order.

Generate those from a template with required fields rather than as free prose, and fail the build if a change marked breaking has no migration text. This is the one place where making the pipeline strict is worth the friction, because a missing migration note generates support tickets for months.

If your team also produces upgrade guides, the same classified records feed them — which is worth considering alongside a broader approach to generated documentation.

A working order

Collect pull requests in the release window with their bodies and linked issues. Cancel reverts and cherry-picks deterministically. Classify each change in isolation into a structured record with evidence. Filter and group in code, per audience. Write prose per group. Cache by PR and SHA. Open a draft PR for a human to read.

The test for whether it works is not whether the file looks tidy. It is whether anyone changed their behaviour after reading it. If nobody upgraded differently, nobody read it, and the generator is producing a file to satisfy a convention rather than a reader.