Evaluating Models on Your Own Codebase

Every published benchmark measures a sample of tasks that somebody else chose. Your repository is a different sample: different language mix, different test coverage, different internal conventions, different amount of public discussion in any training corpus.

The gap between those two samples is why a model that leads a leaderboard can be mediocre on your work, and why an afternoon of local evaluation beats a week of reading tables.

Start from history, not imagination

The temptation is to write test tasks. Do not. Written tasks are unconsciously shaped to be solvable, they are cleaner than real work, and they carry no ground truth.

Your git history already contains everything you need. Closed issues where a fix was merged, incidents with a known root cause, refactors you completed, pull requests that were reviewed and changed. Each of those is a task with an answer attached, produced at zero marginal cost.

Pick tasks the way your work is actually distributed rather than the way it is interesting. If seventy percent of your model usage is small bug fixes, seventy percent of the set should be small bug fixes, however tempting the one gnarly architectural change is.

Include a few you expect every candidate to fail. A set where everything passes has no resolution, and the hard tail is where models separate.

What makes your codebase different

Language mix is the obvious one. Public coding benchmarks lean heavily toward Python with strong open-source test coverage, because that is what can be scored mechanically. If you write TypeScript services or Go infrastructure, the transfer is partial.

Convention density is the underrated one. A mature internal codebase has patterns nobody wrote down — how errors propagate, which helper is canonical, what the layering rules are. Public benchmarks measure fluency in widely published patterns, which is a different skill from inferring yours.

Test quality decides how you can score at all. Repositories with thin tests cannot be graded mechanically, which pushes you toward human review and changes the design of the whole exercise.

Then there is context shape. If your tasks routinely need six files and a schema, you are testing long-context retrieval as much as coding, and how you feed that context will move results more than the model choice does.

Choosing what to measure

Pass rate against your own tests is the primary metric where it is available, because it is mechanical and immune to persuasion. It is also incomplete: a patch that passes while duplicating logic or ignoring your layering scores identically to a clean one.

So record a second axis. Have a reviewer mark each passing patch as acceptable, acceptable with edits, or would-be-rejected. That single extra column catches the quality gap the test suite cannot see, and it takes seconds per task.

Record cost per task in tokens, not just in currency, so the number survives price changes. A model that resolves five percent more at three times the token spend is a different proposition depending on volume, which is the arithmetic behind cost per pull request.

Record wall-clock time too if anything sits in an interactive loop. Latency changes whether developers use a tool at all, and it never appears on a leaderboard.

How many tasks is enough

Fewer than you fear and more than you want. Fifteen to twenty-five real tasks is usually the point where the ordering of candidates stops flipping between runs, and it is small enough to build in a day.

The reason you need that many is variance. On ten tasks, a single lucky pass is ten percentage points, which is larger than the real gap between most adjacent models. Do not report a difference smaller than one task.

Run each task more than once if your budget allows, because sampling is stochastic and a single run conflates model quality with luck. Two or three runs per task per model is enough to see whether a result is stable — determinism and seeds covers what you can and cannot pin down.

Resist growing the set for its own sake. A maintained set of twenty tasks that reflects current code beats a neglected set of two hundred that reflects the repository you had two years ago.

Keeping the comparison fair

One harness, one prompt, one context assembly strategy, for every candidate. The moment you tune the prompt for one model you are measuring your tuning effort rather than the models, which is exactly the problem with vendor-reported figures.

Hide the identities while grading anything a human scores. Knowing which model produced a patch moves review judgement more than most reviewers believe, and blind comparison is cheap insurance.

Keep the set private. Tasks from a public repository can already be in a training corpus; tasks from your internal code cannot be, and that property is the whole reason the exercise is trustworthy.

Then treat the result as perishable. Re-run it when a candidate ships a new version, and store the outputs so you can diff behaviour rather than argue about it. Building a model eval harness covers turning this from an afternoon into something repeatable.