Granite 4.0 H-Micro: IBM's 3B Hybrid Mamba Model Explained

Small models are usually judged on one question: how close do they get to a big one? Granite 4.0 H-Micro is more interesting for a different reason. It is a 3B model in which almost every layer is not a transformer layer at all, and that changes what it costs to run far more than its benchmark scores suggest.

IBM released the Granite 4.0 family on 2 October 2025 under the Apache 2.0 licence. H-Micro was the smallest hybrid in that launch line-up, sitting below H-Tiny (7B total, about 1B active) and H-Small (32B total, about 9B active). IBM aims it at low-latency, edge and local deployments, and at fast sub-tasks such as function calling inside larger agent workflows.

Why a 3B model has 36 Mamba layers

According to the model card, H-Micro is a dense decoder with 40 layers: 36 Mamba-2 layers and just 4 attention layers. It uses grouped-query attention in those four, SwiGLU feed-forward blocks, RMSNorm, and no positional embeddings at all (NoPE). Context length is 128K.

The reason for the mix is memory. A transformer's attention layers keep a key-value cache that grows with every token in the context, and with every concurrent request. Mamba-2 is a state-space layer that carries a fixed-size state instead, so its memory does not grow with sequence length in the same way. Replace most attention layers with Mamba-2 and the KV cache shrinks to whatever the remaining four layers need. KV cache explained covers why that cache, not the weights, is what fills a GPU at long context.

Where the memory saving shows up, and where it does not

IBM's announcement says the hybrid Granite 4.0 models can offer over 70% reduction in RAM needed to handle long inputs and multiple concurrent batches. Read that sentence carefully, because the qualifiers carry it.

The saving is largest exactly when the KV cache would dominate: long prompts, many simultaneous sessions, or both. For a single short request, a 3B model's weights are the main memory cost and the hybrid design buys you little. For a server handling dozens of retrieval-augmented requests with long contexts, it can be the difference between one small GPU and two.

So the question to ask is not whether H-Micro is fast in isolation but what your concurrency and prompt lengths look like. If you run batch extraction over long documents, test it. If you run short classification calls one at a time, a conventional 3B transformer will behave much the same.

The dense twin and the runtime question

IBM shipped a second 3B model alongside it, Granite-4.0-Micro, which is a conventional transformer. The announcement is explicit about why: some platforms and communities do not yet support hybrid architectures, and the dense Micro exists so they can still run Granite 4.0.

At launch IBM listed full optimised support in vLLM 0.10.2 and Hugging Face Transformers, and support in llama.cpp and MLX with throughput optimisation still in progress. Local runners built on older llama.cpp releases may refuse to load a Mamba-based model. Before choosing H-Micro for an on-device product, confirm the exact runtime version you will ship can load it and run it at the speed you need. If it cannot, the dense twin is the fallback with the same parameter count and context length.

Benchmarks: compare it with its twin, not the frontier

The model card publishes the two 3B models side by side, which is the useful comparison. A selection of IBM's own figures:

• IFEval (average) — 84.32 for H-Micro against 82.31 for dense Micro
• ArenaHard — 36.15 against 25.84
• MMLU (5-shot) — 67.43 against 65.98
• GSM8K (8-shot) — 81.35 against 85.45
• BFCL v3 — 57.56 against 59.98
• HumanEval (pass@1) — 81 against 80

The pattern is mixed rather than a clean win: the hybrid is better at instruction following and chat preference, slightly worse at grade-school maths and function calling. Neither is a reasoning model. These are vendor-run numbers, and a 3B model will fall well short of large models on open-ended tasks; its job is narrow, cheap and fast work.

What it is built to do

The card lists summarisation, classification, extraction, question answering, retrieval-augmented generation, code tasks, function calling and multilingual dialogue across 12 languages, plus fill-in-the-middle code completion. Tools are declared in OpenAI's function definition schema.

That list describes a worker model. Good fits are routing a request to the right handler, pulling fields from a document, tagging tickets, or acting as the first stage in a cascade where a larger model handles only what the small one cannot. Model routing and fallbacks covers how to structure that.

Two governance details matter to enterprise buyers. IBM says Granite is the first open model family to receive ISO/IEC 42001:2023 accreditation, and every Granite 4 checkpoint on Hugging Face is cryptographically signed with a model.sig file you can verify. If your procurement process asks where weights came from, that is a concrete answer.

H-Micro or Granite 4.2 8B

IBM's later Granite 4.2 line went in a different direction: dense transformers with built-in reasoning modes. The Granite 4.2 8B guide covers that model.

• Pick H-Micro for high-concurrency, long-input extraction and classification, for edge devices with tight memory, or wherever you want the smallest model that follows instructions reliably.
• Pick Granite 4.2 8B when the task needs multi-step reasoning, maths or agentic coding, and you can afford more parameters and more output tokens per answer.

The deciding test is cheap to run. Take two hundred real inputs from the task, run both, and measure accuracy, peak memory under your real concurrency, and tokens generated per answer. A reasoning model that is a few points more accurate but writes ten times the tokens may not be the better choice for a classifier. Best model for self-hosting has more on fitting a model to the hardware you have.