Ternary Bonsai 2 27B: A 27B Model in Under 6 GB

The usual rule of thumb for quantisation is that 4 bits per weight is safe, 3 is noticeable, and 2 is where models fall apart. Ternary Bonsai 2 27B is a direct challenge to the last part of that rule. PrismML takes Qwen3.8-27B, stores every weight as one of three values, and reports keeping about 98% of the full-precision model's average score.

If that holds on your work, it changes which machines can run a 27B reasoning model. If it does not, the reasons are worth understanding before you plan around it.

What ternary actually means

Each weight in Bonsai 2 is stored as -1, 0 or +1, with one FP16 scale shared by each group of 128 weights. Three states need about 1.58 bits of information; with the scales included, PrismML gives the true size as 1.72 bits per weight. The model card also says a Hadamard rotation is folded into storage, which spreads outlier values across a group so that rounding to three levels loses less.

Two packed file formats exist. PTQ1_0 stores dense trits at 1.75 bits per weight and comes to 5.95 GB. PQ2_0 uses 2-bit slots at 2.13 bits per weight and 7.21 GB, trading size for faster decoding on some GPUs. The FP16 original is about 54 GB.

The architecture underneath is unchanged. Bonsai 2 is Qwen3.8-27B with the same hybrid attention, 27.36B parameters including a 0.46B vision tower, and a 262K-token context window. The Qwen3.8 27B guide covers the base model itself. Quantization explained covers the general mechanics if bits per weight is a new idea.

What the quality numbers say

PrismML's model card averages 14 benchmarks in thinking mode:

• Qwen3.8-27B, FP16 — 54 GB, average 86.32
• Qwen3.8-27B, UD-Q4_K_XL — 17.6 GB, average 85.18
• Qwen3.8-27B, IQ2_XXS — 7.27 GB, average 72.59
• Bonsai 2 27B — 5.95 GB, average 84.78

The comparison that matters is the third line. A conventional 2-bit quantisation of the same model at a similar file size loses nearly fourteen points. Bonsai 2 loses about one and a half. That gap is the whole product: quantisation-aware work at extreme low bit widths beats rounding a finished model down after the fact.

The loss is not spread evenly. By PrismML's breakdown, coding and instruction following are flat or slightly up, maths drops half a point, agentic tool calling about 1.8, knowledge and reasoning about 5.7, and vision about 5.2. If your workload leans on recall of facts or on images, those are the categories to test first. Where quantised models actually lose quality covers why losses tend to be uneven across task types.

These are vendor numbers. The relative comparison against IQ2_XXS is the more trustworthy part, since the same team ran both, but run your own tasks before treating 98% as a promise.

What it takes to run

This is the catch. Stock llama.cpp will not load these files. The model card's quickstart uses PrismML's own llama.cpp fork, which supports CUDA, Metal and CPU. The card also lists Ollama, LM Studio, vLLM and Docker Model Runner, but the fork is the documented path, so check that whatever runtime you choose actually supports the ternary formats before assuming it does.

The model thinks by default, at an effort level of xhigh. medium gives shorter responses and low is not supported. Allow a large output budget; the card's example sets 16,384 tokens so the thinking trace has room to finish:

llama-cli -m Ternary-Bonsai-2-27B.gguf -ngl 99 -fa on -c 32768 -n 16384   --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.05

PrismML publishes decode speeds across hardware. An RTX 4090 runs the PTQ1_0 build at about 91 tokens per second; an RTX 5090 runs PQ2_0 at about 130; an Apple M5 Pro laptop manages about 28. The card notes that the dense PTQ1_0 format is slower than PQ2_0 on H100, A100 and Blackwell GPUs, so pick the format by hardware rather than by file size alone.

Image input needs an optional projector file of 0.63 GB at Q8_0 or 0.93 GB at BF16, loaded only when you send an image.

When this is the right model

The use case is narrow and real: you need a capable reasoning model on hardware with around 8 to 12 GB of free memory, or you want to fit several model instances on one GPU, or you are shipping to devices where a 17 GB download is unacceptable. In those situations a 6 GB 27B model that keeps its coding ability is a materially better option than a smaller dense model.

It is the wrong choice when memory is not the constraint. If you can fit the 17.6 GB Q4 build, you get a slightly better model with mainstream runtime support. And it is the wrong choice for knowledge-heavy work, where the measured loss is largest. Best model for self-hosting covers matching a model to the hardware you have.

Licence and hosted access

The model card gives the licence as Apache 2.0, which permits commercial use and redistribution without a revenue threshold. Check the base model's terms too if you plan to redistribute a derivative, since obligations can travel with weights.

If you only want to try it, OpenRouter listed it at $0.075 per million input tokens and $0.50 per million output at the time of writing, with tool calling, structured outputs and reasoning effort control. That is a cheap way to test whether the quality holds on your tasks before committing to the custom runtime. Compare it against the full-precision base on the same twenty prompts; if the answers you care about match, the ternary build is worth the setup.