Choosing a Model for Non-English Codebases

Model comparisons assume English. English identifiers, English comments, English issue text, English documentation. Plenty of working codebases are not like that, and the assumptions built into a leaderboard quietly stop holding.

If your team writes commit messages in Japanese, comments in German, or issue descriptions in Portuguese, the model that wins the public benchmarks may not be the model that works for you — and the reasons are mechanical rather than mysterious.

Tokenisation is the first hidden cost

Tokenizers are trained on a corpus, and that corpus is mostly English. Text in a language that is well represented compresses efficiently; text that is not gets fragmented into many more tokens for the same meaning.

The effect is largest for non-Latin scripts. The same paragraph can cost noticeably more tokens in Japanese or Arabic than in English, and every one of those tokens is billed on input and counted against your context window on every turn.

That has two consequences people miss. Your effective context is smaller than the advertised figure — a 1M window holds less of your material than it holds of an English codebase — and your cost per task is higher for identical work. Tokenizer differences across models covers how much this varies between vendors.

Measure it rather than assuming. Take a representative file from your repository, run it through each candidate's tokenizer, and compare counts. Ten minutes of this can change which model is actually cheapest for you by a wide margin.

Where accuracy actually degrades

Code itself is largely language-neutral: keywords are English in almost every language, and syntax does not change. What changes is everything around the code.

Identifier comprehension is the first place things slip. A model that does not parse a compound identifier in Turkish or Vietnamese cannot infer what the function does, and it will fall back on the surrounding structure, which is a weaker signal.

Instruction following in the target language is the second. A model may understand a Korean bug report perfectly and still answer in English, or produce comments in English inside a file where every other comment is Korean. That is a consistency failure, not a comprehension failure, and it needs testing separately.

Third is domain vocabulary. Financial, medical and legal terms in a language other than English are exactly where training coverage thins out, and it is where a confidently wrong answer is most likely — a specific instance of why models hallucinate at the edges of their coverage.

Provenance matters more than the leaderboard

Training data mix follows the lab. A model built by a team whose primary market speaks a given language will have seen far more of it, and that shows up in fluency, idiom and technical vocabulary in ways no general benchmark reports.

Several of the strongest open-weight families — Moonshot's Kimi line, Z.ai's GLM, DeepSeek, Alibaba's Qwen, MiniMax — come from Chinese labs and are correspondingly strong on Chinese technical text. If your codebase is Chinese, that is a more relevant fact than any aggregate index position.

For languages without a large local lab, the picture is less predictable and the only reliable answer is measurement. Aggregate indices average across tasks that are almost entirely English, so a small gap in the index tells you nothing about a large gap in Finnish.

Read a model card for what it says about training data and languages, and note that most say very little. Reading a model card covers how to treat those omissions.

Testing it properly

Build your evaluation set from your own repository in your own language, not from translated English tasks. Translation produces unnaturally clean text, and the messiness of real comments is precisely what you are testing.

Include tasks with mixed-language content, because that is what most real repositories contain: English keywords and library names, local-language comments and issue text, and often a third language in vendored dependencies. Models handle the switch with varying grace.

Score two things separately: did the change work, and was the output in the right language with the right conventions. A patch that is correct but comments in English fails your review even though it passes your tests, and only a human column catches it.

Grade blind. Reviewers who are native speakers have strong opinions about naturalness, and those opinions are exactly what you want to capture — provided they are not anchored to a brand. Blind comparison is the protocol.

Practical mitigations

If the best model on your tasks writes comments in the wrong language, fix it in the system prompt before you switch models. An explicit instruction stating the language for comments, commit messages and documentation resolves most consistency failures without any capability cost.

If tokenisation is inflating your cost, look at what you send rather than which model you use. Trimming vendored files, excluding generated content and summarising history are all worth more here than in an English codebase — reducing token usage covers the levers.

If a smaller model handles your language well, use it for the routine work and escalate only the hard tasks. Language fit and raw capability are separate axes, and routing between models lets you have both.

The decision rule: measure tokens per file on your own text, evaluate on your own untranslated tasks, score language consistency as a separate column, and let provenance break ties. No published ranking answers this question for you.