FundamentalsKV Cache Explained: The Memory Behind Long Context
The KV cache is why generation is fast and why long context is expensive in memory rather than compute. What it stores and what it costs you.
ReadNotes on models, agents, and what inference costs
Practical writing for developers building with large language models — how they work, how to pick one, and how to keep the bill predictable.
Fundamentals
ModelsLatency-Adjusted Model Scoring: When Fast Beats Smart
Benchmarks measure what a model answers, never how long it took. How to weight latency into model selection for interactive and agentic workloads.
Read
FundamentalsLayer Normalisation: The Part That Keeps Training Stable
Why deep transformers need normalisation, what pre-norm changed, and why RMSNorm won. The connection to quantisation and low-precision serving.
Read
GuidesLlamaIndex Setup on a Custom OpenAI-Compatible Endpoint
Configure LlamaIndex against a non-OpenAI base URL, split the LLM from the embedding model, and avoid the defaults that silently call OpenAI anyway.
Read
GuidesLLM Classification: Fix the Taxonomy Before the Model
Most classification failures are label definition failures. How to design a taxonomy, handle abstention and imbalance, and check the model beats a cheap baseline.
ReadGuidesLLM Data Extraction: Design the Schema Before the Prompt
Extraction failures are usually schema failures. How to model absent, ambiguous and multi-valued fields, attach provenance, and validate what comes back.
Read
GuidesLLM Summarisation: Omission Hurts More Than Invention
Summaries fail by leaving things out, not by making things up. How to control what gets kept, pick a chunking strategy, and evaluate faithfulness cheaply.
Read
GuidesLLM Translation: Placeholders, Glossaries and Register
Machine translation is solved enough. What breaks in software localisation is interpolation syntax, terminology drift and missing context, not language quality.
Read
GuidesLLM-Powered Search: The Model Is Not the Retriever
Adding a model to search usually means query rewriting and reranking, not generation. Where each stage helps, what it costs in latency, and how it fails.
Read
FundamentalsLoad Balancing in MoE: Stopping Experts Going Idle
An MoE model only delivers its promised capacity if traffic spreads across experts. Collapse during training and skew during serving are both real risks.
ReadGuidesLogging LLM Requests Safely Without Building a Liability
The fields that make a model request debuggable are mostly not the fields that make it risky. How to log traffic that stays useful and minimal by default.
Read
FundamentalsLogits and Softmax: What a Model Really Outputs
A model does not emit words, it emits a score for every token in its vocabulary. What softmax does to those scores and why probabilities are not confidence.
ReadShowing 157–168 of 404 articles