Vision-Language Models: What to Test Before You Commit

Native image input has moved from a differentiator to a checkbox in the open-weight field, and the model cards have become correspondingly useless. Almost every frontier-class release now claims multimodality, and the published multimodal benchmarks predict very little about developer workloads.

This article is about what to test instead, anchored on the two open-weight models whose image capability is clearly stated rather than implied.

The two with clearly stated image support

Kimi K2.6, released by Moonshot on 20 April 2026, is a vision-language model. It is 1T total parameters with roughly 32B active, built from 384 experts — eight routed plus one shared — using multi-head latent attention, with a 256K context window. Moonshot lists it at around $0.95 per million input tokens and $4.00 output.

MiniMax M3, released 1 June 2026, is described as natively multimodal rather than a text model with vision attached. It ships a 1M-token context window and reports 59.0 percent on SWE-bench Pro as a vendor figure, landing near 44 on the Artificial Analysis Intelligence Index. Its pricing differs between sources by roughly a factor of two, so quote it from the endpoint you would actually use.

Both have proper individual write-ups — the K2.6 guide and the M3 guide — and a direct comparison in Kimi K2.6 versus MiniMax M3. What follows applies to any candidate, including whichever ships next month.

Why the published benchmarks do not help

Multimodal benchmarks overwhelmingly test description and question answering over photographs, diagrams and charts drawn from academic datasets. Developer image workloads look nothing like that.

The images you actually feed a model are screenshots at whatever resolution the user's display happened to be, with small monospaced text, subtle colour differences that carry meaning, and the critical detail in a corner. A model can score well on chart question-answering and still fail to read a stack trace in a terminal pane.

There is also a scoring problem. "Describe this image" has many acceptable answers, so multimodal benchmarks lean on judge models or multiple-choice formats, both of which introduce their own biases. Why benchmarks disagree covers why two published scores for the same model routinely diverge.

None of this means the numbers are fabricated. It means they measure a different task than yours, and the correlation with your workload is unknown until you check it.

Three probes that actually separate models

The small-text probe. Take a real screenshot of your own application in a failure state, at native resolution, with a console or log pane visible. Ask the model what went wrong. This tests whether it resolves fine text or merely summarises the layout — the single most common failure in developer image work, and the one that most often goes unnoticed because the answer sounds plausible.

The spatial probe. Give it a design mock or a wireframe and ask for the layout as structured markup. This tests whether the model represents relative position and containment rather than just recognising components. Models that pass the first probe frequently fail this one, listing every element correctly in the wrong arrangement.

The extraction probe. Feed it a chart from a real specification PDF and ask for the underlying numbers. This tests whether it reads axes and gridlines or estimates from the shape of the plot. An estimated answer looks confident and is wrong by ten percent, which is worse than a refusal.

Run each probe five times on the same image. Consistency is part of the result — a model that gets it right twice out of five is not usable in an unattended pipeline, whatever the average suggests.

Images are expensive in context terms

An image does not arrive as an image. It is cut into patches, each patch becomes an embedding, and those embeddings occupy positions in the same context the text uses. How images become tokens covers the mechanism.

The practical consequence is that a high-resolution screenshot can consume as much context as several pages of text, and models differ in how aggressively they downsample before encoding. Aggressive downsampling is exactly what makes small text unreadable, so there is a direct trade between token cost and the capability you most need.

This bites hardest in agent loops. If every turn attaches a fresh screenshot, context grows far faster than a text-only loop and you hit the window ceiling much earlier. K2.6's 256K and M3's 1M sound like very different budgets until you start attaching images, at which point both shrink quickly. Context window cost trade-offs covers the arithmetic.

The mitigation is to crop before sending. A cropped region containing the relevant pane costs a fraction of the full screenshot and usually improves accuracy, because the model is not spending attention on chrome.

When a vision model is the wrong tool

If your images are documents with a consistent layout — invoices, forms, standard reports — a dedicated extraction pipeline is often more accurate, far cheaper, and easier to validate than a general vision model. Vision models earn their place on heterogeneous inputs, not uniform ones.

If the underlying data exists in a structured form somewhere, use that instead. Asking a model to read numbers off a rendered chart when the CSV behind it is available is a self-inflicted accuracy problem.

And if only a small fraction of your traffic contains images, do not choose your primary model on vision capability. Route the image requests to a vision model and keep a cheaper, stronger text model for everything else. Model routing and fallbacks covers wiring that without spreading model names through your code.

A short evaluation protocol

Collect twenty real images from your actual workload — not curated examples, whatever your users would genuinely send. Include the ugly ones: bad crops, wrong aspect ratios, dark mode, compression artefacts.

Write down the correct answer for each before you run anything. This matters more than it sounds, because vision model output is fluent and confident, and grading after the fact leads you to accept answers you would have rejected.

Run each image five times per candidate model, score strictly on whether the specific fact you needed was correct, and record the token count alongside. Then compute cost per correct extraction rather than per call.

Re-run it quarterly. Vision capability is improving faster than text capability across recent releases, and a conclusion from two quarters ago is more likely to be stale here than anywhere else in your model stack.