Mistral Nemo Guide: The 12B Model That Outlived Its API

Mistral Nemo is two years old, has been retired from Mistral's own API, and still turns up in model lists, fine-tune repositories and provider catalogues. That is not nostalgia. A 12B model with permissive weights, a 128K window and a tokenizer that compresses non-English text well fills a gap that newer, larger releases do not.

This guide covers what Nemo actually is, what changed when it shipped, and the narrow set of jobs it still suits in late 2026.

Twelve billion parameters, sized for one card

Mistral released Nemo on 18 July 2024, built jointly with NVIDIA. It is a dense transformer: 12B parameters, 40 layers, a model dimension of 5,120, 32 attention heads with 8 key-value heads (grouped-query attention), SwiGLU activations and rotary embeddings with a theta of one million. The model card states it was trained with a 128K context window.

The size is the point. At roughly two bytes per parameter in BF16, the weights alone come to about 24 GB, which puts an unquantised copy within reach of a single 24 GB or larger card once you leave room for the KV cache. Mistral also says the model was trained with quantisation awareness so that FP8 inference runs "without any performance loss", which halves the weight footprint again. Treat that claim as something to check on your own prompts rather than a guarantee: quantisation losses tend to show up first in long outputs, exact numbers and rare languages, not in short chat replies.

Mistral pitched it as a drop-in replacement for Mistral 7B, because it uses a standard architecture. For anyone who had a 7B pipeline in 2024, that meant a capability step up without rewriting the serving stack.

Tekken: the tokenizer was the real upgrade

Nemo introduced Tekken, a new tokenizer based on tiktoken and trained on more than 100 languages, with a vocabulary of 2^17 entries (about 128K). Mistral claimed it compresses source code and several major languages around 30% more efficiently than the SentencePiece tokenizer in earlier Mistral models, about 2x for Korean and about 3x for Arabic.

Compression is not a vanity metric. Fewer tokens for the same text means a smaller bill per request, more real content inside the same context window, and faster generation for a given tokens-per-second rate. If your traffic is not English, the tokenizer can matter more than the benchmark table. Tokenizer differences across models walks through how to measure this on your own corpus instead of trusting the vendor's ratios.

The release listed English, French, German, Spanish, Italian, Portuguese, Chinese, Japanese, Korean, Arabic and Hindi as strong languages. The model card's multilingual MMLU figures sit around 59–65% across the languages it reports, against 68.0% for English MMLU (5-shot). That gap is typical for a model this size and is worth remembering if you plan to lean on it outside English.

Retired upstream, alive downstream

Mistral's model documentation lists Nemo 12B as deprecated on 22 May 2026 and retired on 31 July 2026, with Ministral 3 8B named as the replacement. So you can no longer call it on Mistral's own platform.

That does not remove it from the world. The base and instruct checkpoints are on Hugging Face under Apache 2.0, and third-party hosts continue to serve it. At the time of writing it is listed on OpenRouter at roughly three cents per million tokens for both input and output, with a 128K context window and JSON mode, structured outputs and tool calling listed as supported. Prices on aggregators move, so check before you budget against them.

The practical consequence is about risk rather than capability. A model that the vendor has retired gets no fixes, and the third parties serving it can stop at any time. If you depend on Nemo in production, either self-host the weights or have a tested fallback. Model deprecation and migration covers how to plan that before a provider forces the issue.

Where Nemo still makes sense

Nemo is not competitive with 2026 frontier or even mid-tier models on reasoning or agentic coding, and nothing here suggests otherwise. Its remaining strengths are structural:

• Fine-tuning base. Apache 2.0 weights, a standard architecture and a size that trains on modest hardware made Nemo a popular base for community fine-tunes. If you already have a tuned derivative that works, there is no urgency to move.
• Cheap, high-volume text tasks. Classification, extraction, routing and short rewrites where a small model is already good enough and the output is easy to check.
• Air-gapped or on-device work. A 12B model with FP8 in mind is easy to run where data cannot leave the building.

For new projects, the vendor's own suggestion is Ministral 3 8B, which also adds image input and a 256K window. If you need more capability than a small model offers, Mistral Small 4 is the current step up, though it is far larger to self-host.

Settings that matter

Two details from the model card trip people up.

1. Temperature. Mistral recommends 0.3, and notes that Nemo wants lower temperatures than earlier Mistral models. Copying a 0.7 default from another model produces noticeably looser output.
2. Tool call IDs. When using native function calling through transformers, tool call IDs must be exactly nine alphanumeric characters and must appear in both the call and the result. A wrapper that generates UUIDs will break silently.

The card also states the model has no moderation mechanisms. If you expose it to end users, the filtering is your job.

The decision rule is short. Keep Nemo where it already works and you control the weights. Do not start a new hosted integration on a retired model unless you are prepared to self-host it the day your provider drops it.