Qwen3.5-9B Guide: A Small Multimodal Model Worth Measuring

Small models are usually judged by how close they get to the flagship on a leaderboard. That is the wrong test. A 9B model earns its place on the jobs where the flagship was overkill to begin with: classification, extraction, document parsing, routing decisions, and anything that has to run on hardware you already own.

Qwen3.5-9B is the largest of the small Qwen3.5 models, which Qwen released on 2 March 2026 alongside 4B, 2B and 0.8B versions. It accepts text, images and video, carries the same 262,144-token native window as the rest of the family, and ships under Apache 2.0.

Where 9B sits in the line-up

The Qwen3.5 family arrived in three waves. The 397B-A17B flagship came first in February, then 122B-A10B, 35B-A3B and 27B on 24 February, then the four small models in March. The 9B is the top of the bottom tier: big enough to reason, small enough for a single consumer GPU once quantised.

Everything below it trades away capability quickly. On Qwen's own model card the 4B scores 79.1 on MMLU-Pro and 76.2 on GPQA Diamond, against 82.5 and 81.7 for the 9B. If your hardware fits the 9B, there is little reason to drop lower unless you are targeting phones or browsers.

What the card says about the architecture

The 9B uses the same hybrid layout as its larger siblings, scaled down: 32 layers in 8 blocks, each block three Gated DeltaNet linear-attention layers followed by one Gated Attention layer. The full attention layers use 16 query heads over 4 KV heads.

One oddity is worth flagging. The card's highlights mention a sparse mixture-of-experts design for the family, but the 9B's own overview lists a plain feed-forward layer with no expert count. Read it as a dense model, which is what the detailed specification describes. That also means all 9B parameters are active on every token, so there is no total-versus-active gap to reason about. Dense vs mixture-of-experts covers why that makes memory planning simpler.

The linear-attention layers matter more at this size than people expect. A small model running long prompts is often memory-bound by its KV cache, and three of every four layers here do not keep one.

The numbers against its bigger siblings

Comparing the 9B to the 397B flagship on the vendor's own figures shows where the size gap bites:

• Document parsing: OmniDocBench1.5 87.7 against 90.8. A small gap.
• Knowledge: MMLU-Pro 82.5 against 87.8. A moderate gap.
• Visual reasoning: MMMU 78.4 against 85.0.
• Competitive coding: LiveCodeBench v6 65.6 against 83.6. A large gap.

The pattern is the useful part. Perception and extraction survive the shrink well; multi-step code generation does not. The card also reports IFEval 91.5 and BFCL-V4 66.1 for the 9B, which suggests instruction following is solid and tool calling is usable but not something to lean on for long agent loops.

If you need coding ability at a small footprint, compare the Qwen3.6-35B-A3B, which activates only 3B per token but needs more memory to hold.

Running it yourself: rough memory arithmetic

The Hub lists about 10B parameters in the checkpoint, which includes the vision encoder. At 16-bit precision that is roughly 20 GB of weights before any cache. A 4-bit quantisation brings the weights to around a quarter of that. These are back-of-envelope figures, not measurements, and real usage depends on context length and runtime.

The card recommends keeping at least 128K of context available to preserve thinking quality, which pushes memory up again. On a 24 GB card you will likely be choosing between precision and context length. Quantization explained covers what you give up at each bit width.

vLLM, SGLang, KTransformers and Hugging Face Transformers all support it. If you never send images, vLLM's --language-model-only flag skips the vision encoder.

If you would rather not host it, OpenRouter lists it at $0.10 per million input tokens and $0.15 per million output at the time of writing, with a 32,768-token output limit.

Turn thinking off for the jobs it is best at

Like the rest of Qwen3.5, the 9B thinks by default. For a classifier or an extractor that is usually waste: you pay for hundreds of reasoning tokens to get a one-word label, and latency rises with them.

Set enable_thinking: false in the chat template arguments for these jobs and measure whether accuracy actually drops. Keep thinking on for multi-step questions over documents or images, where it tends to earn its tokens. The card's recommended thinking-mode sampling is temperature 1.0, top_p 0.95, top_k 20 and presence penalty 1.5.

When 9B is the wrong size

• Long agentic coding loops. The coding gap above is real. Use a larger model, or route only the easy steps here.
• Very long contexts at full quality. The window stretches to about 1M with YaRN, but a small model's grip on a very long prompt is weaker than its size class suggests.
• Anywhere a hosted flash model is cheaper than your electricity. If you are not using local inference for privacy or latency, run the numbers first.

The decision rule: use the 9B where the task is narrow, the input is visual or document-heavy, and you either need it on your own hardware or at very low cost. Escalate everything else, which is exactly the pattern model routing and fallbacks describes.