Granite 4.2 8B Guide: IBM's Dense Reasoning Model, Three Modes

A small reasoning model is easy to misjudge in both directions. Read the maths scores and it looks like a bargain frontier model. Run it on a simple task with thinking enabled and it looks like a slow, chatty way to spend tokens. Granite 4.2 8B is both, depending on one flag, and getting value from it mostly means knowing when to flip that flag.

IBM released the Granite 4.2 family on 25 August 2026 in three sizes, 3B, 8B and 30B, with weights under Apache 2.0. The 8B is post-trained from Granite-4.1-8B-Base and IBM lists its best uses as reasoning, code generation, tool calling, agentic workflows and multilingual dialogue.

What changed from Granite 4.0

The Granite 4.0 generation was notable for its hybrid design, mixing Mamba-2 layers with a few attention layers to cut memory at long context. Granite 4.2 8B does not do that. The model card describes a dense, decoder-only transformer: 40 layers, grouped-query attention with 32 query heads and 8 KV heads, SwiGLU feed-forward blocks and RoPE positions.

The other change is the job. Granite 4.0's small models were instruction followers aimed at extraction, classification and fast tool calls. Granite 4.2 is a reasoning family with built-in chain of thought. IBM's card describes supervised fine-tuning on instruction, reasoning, tool-calling and agentic data, followed by reinforcement learning across maths, code, science, instruction following and tool use.

One small inconsistency to be aware of: the model card says 8B parameters, while the Hugging Face metadata reports 9B. It does not change how you deploy it, but it is worth knowing when a comparison table quotes one figure or the other.

Three modes in one checkpoint

The model wraps its reasoning in <think> tags and exposes three modes through chat template arguments:

• Thinking (the default) — enable_thinking: true. Full step-by-step reasoning before the answer.
• Non-thinking — enable_thinking: false. A direct answer, like an ordinary instruct model.
• Low-effort — enable_thinking: true with low_effort: true. Shorter reasoning for tasks that need some but not much.

On a vLLM or SGLang server with an OpenAI-compatible endpoint, these go in chat_template_kwargs:

{
  "model": "ibm-granite/granite-4.2-8b",
  "messages": [{"role": "user", "content": "Classify this ticket: ..."}],
  "chat_template_kwargs": {"enable_thinking": false}
}

IBM's card uses 8,192 new tokens for thinking mode and 2,048 for non-thinking, and recommends a temperature of 1.0 with top-p 0.95 in all modes. By default the chat template strips earlier reasoning from multi-turn history, which keeps long conversations from filling the 128K window with old thoughts. Hosted providers may expose these modes differently, sometimes as a reasoning-effort setting, so check how yours maps them.

The benchmark table, read carefully

IBM publishes a broad table for all three sizes. Selected 8B results, all vendor-run:

• AIME25 — 86.67; HMMT Feb25 — 78.33
• GPQA — 64.14; MMLU-Pro — 74.04
• LiveCodeBench v6 — 73.24; SciCode — 36.09
• SWE-bench Verified — 47.67; SWE-bench Pro — 19.11
• Terminal-Bench 2.1 — 20.56; BFCL v4 — 52.39
• IFBench — 79.33; RULER at 128K — 71.41

The shape is clear. Competition maths and single-function coding are strong for the size. Long-horizon agentic work is where it drops: SWE-bench Pro and Terminal-Bench 2.1 scores are a fraction of what large open models post. Long-context recall also falls off, with RULER going from 80.99 at 64K to 71.41 at 128K.

Note too that BFCL v4 is essentially the same for the 3B (52.41) and the 8B (52.39). If structured function calling is the whole job, the larger model is not buying you better tool calls by IBM's own measure. Best model for tool calling covers why reliability matters more than raw intelligence there.

Cheap tokens, but many of them

OpenRouter lists Granite 4.2 8B at about $0.06 per million input tokens and $0.25 per million output at the time of writing. That is low, but reasoning changes the arithmetic: a thinking-mode answer can spend thousands of output tokens before the visible reply begins, and those are billed as output.

For a classifier, non-thinking mode may produce twenty tokens where thinking mode produces two thousand. That is a hundredfold difference on the expensive side of the bill for what may be the same label. The token cost of reasoning models works through this in detail. The practical rule is to default to non-thinking or low-effort and turn full thinking on only for task types where you have measured a gain.

Self-hosting it

An 8B dense model in bfloat16 fits comfortably on a single data-centre GPU, and quantised builds bring it within reach of a workstation card. The weights are on Hugging Face, the card points to quantised builds for llama.cpp, Ollama and LM Studio, and it lists agentic coding harnesses such as OpenCode, Pi and OpenHands running against a vLLM or SGLang server.

Apache 2.0 means you can fine-tune, redistribute and serve it commercially without a separate agreement. IBM also recommends pairing it with Granite Guardian for safety-sensitive deployments, which is worth taking seriously if the model faces end users. Open weights vs open source explains why a permissive licence matters more as a product grows.

Picking a Granite model

• Granite 4.0 H-Micro for high-volume extraction, classification and edge deployments where memory is tight and reasoning is unnecessary. The H-Micro guide covers its hybrid design.
• Granite 4.2 8B for maths, code generation and multi-step reasoning on hardware you control, with thinking switched on per task.
• Granite 4.2 30B when the 8B falls short on agentic coding; IBM reports 57.00 on SWE-bench Verified and 29.24 on Terminal-Bench 2.1 for it.

Whichever you test, run each task type in both thinking and non-thinking mode and record accuracy and output tokens together. The mode choice will move your cost more than the model choice does.