LLM-Powered Search: The Model Is Not the Retriever

Most search systems that get worse after adding a language model got worse for the same reason: the model was asked to find things. Models do not find things. They rank, rewrite and explain things that a retrieval system found, and a system built on that division of labour beats one built on generation.

The useful mental model is a pipeline with four stages, each of which can independently be the reason your search is bad. Understanding which stage is failing is most of the work.

Retrieval sets the ceiling

Nothing downstream can recover a document that was never retrieved. If the right answer is not in the top hundred candidates, no amount of reranking or synthesis will produce it, and the model will confidently answer from whatever was retrieved instead.

So measure recall at your candidate depth first, before touching anything else. Take fifty real queries, mark the documents that should have been found, and check how often they appear in the candidate set. If recall at 100 is 70 percent, your search has a 30 percent ceiling and your reranker is polishing the wrong problem.

Hybrid retrieval is the reliable default. Lexical search handles exact identifiers, error codes, product SKUs and rare words that embeddings blur together. Dense retrieval handles paraphrase and synonymy, where the user says "cannot log in" and the document says "authentication failure". Run both, merge with reciprocal rank fusion, and you get the union of their strengths without tuning a weight. What embeddings actually capture, and where they lose information, is worth knowing before you rely on them, and how embeddings represent meaning as vectors covers the mechanics.

Query rewriting is where the model earns its place first

Real queries are short, ambiguous and full of context that lives outside the query string. "Does it support SSO" means nothing without knowing what "it" is. Rewriting is a cheap, high-yield use of a model.

Three rewrites that consistently help. Resolving pronouns and ellipsis against conversation history, so the retrieval system sees a self-contained query. Expanding into multiple query variants covering different phrasings, running them all and fusing the results, which raises recall substantially on paraphrase-heavy corpora. And extracting structured filters from natural language — a date range, a product, a document type — so they become database predicates rather than fuzzy matching.

The failure mode is over-rewriting. A model asked to improve a query will happily replace a precise term with a more common synonym, which destroys lexical matching on exactly the queries where lexical matching was your advantage. Always search the original query as well as the rewrites, and never let a rewrite fully replace the user's words.

Reranking is the highest-yield stage

A cross-encoder or an LLM reranker reads the query and a candidate together, rather than comparing two independently computed vectors. That joint view is why it consistently reorders better than the retriever, and it is the single change most likely to make your search visibly better.

It is also the expensive stage, because it scales with the number of candidates. Reranking 100 candidates means 100 scoring passes per query. The practical shape is retrieve broadly and rerank narrowly: 100 to 200 candidates from hybrid retrieval, rerank the top 50, return the top 5 to 10.

If you use a general-purpose model as the reranker, score candidates independently rather than asking it to order a list. Independent scores are parallelisable, are not affected by list position bias, and degrade gracefully when one call fails. Asking for a ranked list of fifty items in one call is cheaper and produces ordering that shifts when you shuffle the input, which is a signal you are measuring position rather than relevance.

Chunking decides what relevance even means

A chunk is the unit that gets retrieved and the unit that gets shown. Get it wrong and both retrieval and answers degrade, for different reasons.

Chunks that are too small lose the context that makes them interpretable — a paragraph that says "this is not supported in the free tier" is useless without knowing what "this" is. Chunks that are too large dilute the embedding, because a vector averaging six topics is close to nothing in particular, and they waste context budget on irrelevant text.

Split on document structure rather than on a fixed token count where you can: headings, sections, function boundaries, table rows. Then attach the ancestry — document title, section path — to each chunk so it stays interpretable on its own. Retrieving a small chunk and expanding to its surrounding section before showing it to the model is a cheap trick that gets you precise retrieval with sufficient context.

Answer synthesis is optional, and often wrong

The instinct is to have the model write a paragraph answering the query. Sometimes that is right. Frequently it is worse than showing the results.

Synthesis destroys the user's ability to verify. A ranked list with snippets lets someone see why a result matched and judge it in a second. A generated paragraph is a claim they have to trust, and it will sometimes merge two documents into a statement neither of them made. For navigational queries, where the user wants a specific document, synthesis is pure overhead.

Where synthesis genuinely helps is multi-document questions: comparisons, aggregations, and anything requiring the reader to combine three sources. Even then, cite spans and link them, so verification stays one click away. Whether you should be retrieving at all rather than putting the corpus in a long context window is a real architectural fork, laid out in when retrieval beats a million-token context window.

Latency budget, stage by stage

Search is interactive, and the pipeline described above adds two model calls to a request that used to take 50 milliseconds. Budget explicitly or you will ship something correct and unusable.

Rewriting sits on the critical path before retrieval, so it wants your fastest model with the shortest output. Reranking parallelises across candidates, so its wall-clock cost is roughly one call as long as you fan out. Synthesis is the longest stage and should stream, because time to first token is what users experience. The distinction between throughput and perceived latency matters here, and what actually determines time to first token is the right background.

Cache aggressively. Query rewrites are highly repetitive across users, rerank scores for a query and document pair are stable until the document changes, and both cache well on a simple key.

Instrument the stages separately

When search is bad, the question is which stage. Log the original query, the rewrites, the candidate IDs with retrieval scores, the rerank scores, and the final set. With that log you can replay a bad query and see whether the document was missing, retrieved but ranked low, or ranked high and then ignored during synthesis.

Without it, every complaint turns into an argument about the model. Most of the time the model is not the problem, and the fix is a chunking change or a missing lexical index. Retrieved documents also become untrusted input to whatever runs next, which matters if a model acts on them, and why retrieved content should be treated as hostile is worth reading before you wire search into anything with tools.