The Artificial Analysis Index Explained: What It Does Measure

Composite indices exist because nobody wants to read fourteen benchmark tables. The Artificial Analysis Intelligence Index collapses a set of evaluations into one figure, and that figure now anchors most public arguments about which model is best.

It is a genuinely useful instrument, used for the thing it measures. The trouble starts when a number designed to rank the field gets treated as a prediction about your workload.

What the index actually is

The index is an aggregate. Several independent evaluations — reasoning, mathematics, coding, knowledge — are run under a common harness, normalised, and averaged into a single score on a shared scale. Nothing about that is exotic; it is the same construction as a stock index or a composite credit rating.

The value of running them under one harness is real and often understated. Vendor-reported numbers come from different prompt formats, different sampling settings, different scaffolding and different retry policies. A third party running every model through identical machinery removes a large class of incomparability.

Current figures give a sense of the spread. Kimi K3 sits around 57. GLM-5.2 sits around 51. DeepSeek V4 Pro and MiniMax M3 both sit around 44. Those gaps are meaningful at the top and mushy in the middle, for reasons worth understanding.

Why the middle of the table is noisy

An aggregate score inherits the noise of every component. Each underlying benchmark has sampling variance, and averaging reduces but does not eliminate it. Two models three points apart on a composite are not reliably distinguishable on any task you care about.

The weighting is also a choice rather than a fact. Deciding that coding counts as much as knowledge recall, and that both count as much as mathematics, encodes an opinion about what intelligence means. Change the weights and the middle of the table reorders.

DeepSeek V4 Pro and MiniMax M3 both landing near 44 illustrates this. They are not the same model. V4 Pro reports 80.6 percent on SWE-bench Verified; M3 is natively multimodal with a vendor-claimed 59.0 percent on SWE-bench Pro. Identical composite scores, entirely different shapes. Why benchmarks disagree covers the mechanics of that divergence.

Contamination and the age of a benchmark

Every public benchmark degrades. Once the questions are on the internet, they are in training corpora, and a model can score well by having seen the answers rather than by solving the problem.

This does not make old benchmarks worthless, but it does mean scores drift upward for reasons unrelated to capability. Newer, harder evaluations are added to composites specifically to counter this, which is also why the index is re-based periodically and historical comparisons need care.

The practical consequence: a composite score is most trustworthy as a comparison between models measured at the same time, and least trustworthy as a claim about progress across years. Treat it as a ranking, not a measurement. SWE-bench explained walks through how one widely-cited component is actually scored.

What the index deliberately omits

Price is not in it. Kimi K3 at $3 in and $15 out per million tokens is roughly seven times the input cost of DeepSeek V4 Pro at $0.44. The index has nothing to say about whether the capability gap justifies that, because that is not what it is for. Cost-adjusted model scoring is where that question belongs.

Latency is not in it either. A model that thinks for ninety seconds and a model that answers in three can score identically, and one of them is unusable in an editor. Latency-adjusted scoring handles that axis.

Nor is anything about deployment. Licence terms, weight availability, context window, tool-calling reliability, structured output discipline, and how the model behaves at 400K tokens of context are all outside the composite, and all of them routinely decide real selections.

Using it correctly

Use the index as a filter, not a decision. It is very good at telling you which five models are worth evaluating and which thirty are not. That is a substantial saving, and it is the whole job.

Once you have a shortlist, the index has done its work and should be set aside. From there the question is how each candidate performs on your prompts, your context lengths, your tool schemas and your failure modes. Benchmarking models on your own work covers assembling that set.

Be especially wary of using the index to justify an upgrade. A three-point composite gain will not be visible in your product, and the migration cost — prompt retuning, re-evaluation, new failure modes — is certain. When to switch models sets a threshold worth crossing.

A reasonable reading protocol

When you see a composite score quoted, ask three questions. Which evaluations went into it, since a coding-heavy composite and a knowledge-heavy one rank differently. When was it measured, since both models and benchmark sets move. And what is the gap, since anything under about five points on a normalised scale is inside the range where your own testing should decide.

Then check the axes the index does not cover before committing. Price per million tokens, time to first token, licence, and context window are all published facts that take minutes to look up and frequently overturn a ranking-based preference.

The index is a map of the field at a moment. It is not a map of your workload, and it was never claiming to be.