A/B Testing Two Models Without Fooling Yourself
Comparing two models on live traffic sounds simple and usually is not. Sample sizes, paired designs, and the metrics that actually settle the question.
ReadPractical writing for developers building with large language models — how they work, how to pick one, and how to keep the bill predictable.
Comparing two models on live traffic sounds simple and usually is not. Sample sizes, paired designs, and the metrics that actually settle the question.
ReadA 2.8T model and a 27B model can be two-to-one apart on the figure that governs thinking. How to read parameter counts across the 2026 field.
ReadWhen an agent does something surprising, the log is the only account of what happened. What a usable agent audit record contains, and what it must not.
ReadAgents resend the whole transcript every turn and re-read the same files repeatedly. Three caching layers fix that, and each has different keying rules.
ReadLong agent runs die halfway. What to checkpoint, where the boundaries belong, and why external side effects break the snapshot model entirely.
ReadUnbounded agent spawning turns a fast run into a retry storm. Worker pools, semaphores, resource locks and the fairness problem nobody plans for.
ReadA provider bill tells you the total and nothing else. How to attribute agent spend to a team, a task and a tool call so you can act on it.
ReadAgents fail in about eight recognisable ways, and each one needs a different fix. A field guide to spotting them from a trace and knowing what to change.
ReadFan-out looks free until the orchestrator stops reading results properly. The four ceilings that cap parallel agents, and how to find yours before production does.
ReadThe filesystem is an agent's most consequential tool surface. Path containment, read and write asymmetry, and why files are an injection vector.
ReadEvery handoff between agents is a compression step. Four patterns for transferring control, what each one drops, and how to build a handoff packet.
ReadEvery coding agent is the same short loop. Understanding its structure tells you where they fail and which parts are worth engineering.
ReadShowing 1–12 of 404 articles