FundamentalsGPU Memory for Inference: What Actually Fills the Card
Weights are only the first line of the memory budget. Where the rest goes, why concurrency runs out before compute does, and how to size a deployment.
ReadNotes on models, agents, and what inference costs
Practical writing for developers building with large language models — how they work, how to pick one, and how to keep the bill predictable.
How large language models actually work, in plain terms.
Fundamentals
FundamentalsGrouped-Query Attention: Why Long Context Got Affordable
GQA shrinks the KV cache by sharing keys and values across query heads. What it costs in quality, and why it is on the spec sheet of nearly every model.
Read
FundamentalsInference vs Training Cost: Which One Is Actually Your Problem
Training is a one-off capital cost you never pay. Inference is a recurring cost you pay per request. How the two differ and why the distinction shapes pricing.
Read
FundamentalsInstruction Tuning: How a Text Predictor Becomes an Assistant
A pretrained model continues text; it does not answer questions. Instruction tuning is the small, cheap stage that turns one into the other.
ReadFundamentalsKV Cache Explained: The Memory Behind Long Context
The KV cache is why generation is fast and why long context is expensive in memory rather than compute. What it stores and what it costs you.
Read
FundamentalsLayer Normalisation: The Part That Keeps Training Stable
Why deep transformers need normalisation, what pre-norm changed, and why RMSNorm won. The connection to quantisation and low-precision serving.
Read
FundamentalsLoad Balancing in MoE: Stopping Experts Going Idle
An MoE model only delivers its promised capacity if traffic spreads across experts. Collapse during training and skew during serving are both real risks.
ReadFundamentalsLogits and Softmax: What a Model Really Outputs
A model does not emit words, it emits a score for every token in its vocabulary. What softmax does to those scores and why probabilities are not confidence.
ReadFundamentalsLong Context Degradation: How Quality Falls Before the Limit
Accuracy does not hold flat and then fall off a cliff at the context limit. It declines gradually, and the decline starts far earlier than most teams assume.
Read
Fundamentalsmax_tokens and Truncation: Why Your Output Stops Mid-Sentence
A response that ends mid-word is almost never a model failure. It is a cap you set, or a default you never set. How truncation actually happens.
Read
FundamentalsModel Parallelism: How Huge Models Are Split to Run
Tensor, pipeline and expert parallelism split a model across devices in different ways. Each moves a different cost onto the network, and that decides latency.
ReadFundamentalsModel Size vs Capability: Why Bigger Stopped Meaning Better
Parameter counts once tracked capability closely and no longer do. What size still predicts, what it never predicted, and what to check instead.
ReadShowing 13–24 of 72 articles