Building a Code Search Tool That Beats grep

Before writing a line of a code search tool, run ripgrep on your repository and time it. On a million-line codebase it returns exact matches in well under a second, with zero index, zero cost, and no possibility of a wrong answer.

That is the baseline you have to beat, and it is a high one. A semantic search tool that is slower, costs money per query, and sometimes returns the wrong file is not an improvement — it is a regression with better marketing. The only justification for building one is the specific set of queries where exact matching fails.

Know which queries you are actually solving

Lexical search fails when the searcher does not know the vocabulary. "Where do we check that a user is allowed to see this project" finds nothing, because the code says canAccess, or hasScope, or a decorator named guard. "The thing that turns a webhook payload into an order" finds nothing, because the function is called handleInbound.

It also fails on conceptual queries with no anchor term at all: find every place that retries a network call, find code that writes to the filesystem outside a temp directory, find the two implementations of the same rate limiting logic.

Write twenty of these down, taken from real questions people asked in your team chat. That list is your evaluation set and it is the only thing that will tell you whether the tool works. Build it before the index, not after, because a search tool always feels good to its author.

Chunking is the whole game

Fixed-size chunking works acceptably for prose and badly for code. Split a file every 500 tokens and you get chunks that begin mid-function, that separate a function from its signature, and that place a class's fields in one chunk and its methods in another. The embedding of half a function is not a useful representation of anything.

Split on syntax instead. Parse with tree-sitter and emit one chunk per function, method or class, with the enclosing context prepended as a header: file path, package or module, class name, and the imports the symbol actually uses. A chunk should be readable on its own by a person who has never seen the file.

For symbols too large to fit a single chunk, split at statement boundaries within the body and repeat the signature on each piece. For files that are mostly configuration or data, one chunk per file is usually right, and often it is better to skip embedding them entirely.

Prepend a short natural-language line to each chunk before embedding — either the docstring, or a generated one-sentence summary. Embeddings of code and embeddings of an English query live in awkwardly different regions of the space, and giving each chunk some English is the cheapest way to close that gap. The mechanics of why are worth reading in embeddings explained.

Hybrid retrieval, always

Pure vector search is worse than grep for the queries grep is good at, which are most of them. A developer searching for an exact symbol name expects the definition first, and a nearest-neighbour search over embeddings will happily return five similar-looking functions instead.

Run both. Take the top 50 from a lexical index and the top 50 from the vector index, merge with reciprocal rank fusion, and you get a list that is never worse than either input on its own. Fusion needs no tuning and no training data, which makes it the right default.

Then rerank the merged candidates, either with a cross-encoder or with a cheap model call that scores each candidate against the query. Reranking 50 candidates down to 8 is where most of the quality comes from, because the retrieval stage only has to get the answer into the top 50 rather than into the top 3.

Ranking signals that a general search engine does not have

Code has structure that text does not, and ignoring it wastes the main advantage you have over a generic retrieval stack.

Boost definitions over usages — a query for a symbol almost always wants the definition, and you know which chunk contains it because you parsed the file. Boost files that changed recently, because active code is more likely to be the subject of a question than code untouched for four years. Demote tests, generated code, vendored dependencies and lock files, unless the query mentions them.

The call graph is the strongest signal most tools leave on the table. If a chunk is called by many others, it is likely to be the canonical implementation rather than one of several wrappers. A cheap static import graph gets you most of this without a full language server.

Incremental indexing or it will not survive

A full reindex of a large repository is slow and, if you are calling a hosted embedding API, not free. If reindexing takes twenty minutes, the index goes stale, results start pointing at deleted code, and people go back to grep.

Key every chunk by a content hash. On each commit, parse only the changed files, compute hashes, and embed only the chunks whose hash is new. A branch switch or a rebase then costs a handful of embeddings rather than a full rebuild, because most chunk contents are unchanged.

Store the file path and a line range rather than the file content in your result payload, and read the current file at query time. That way a slightly stale index still returns correct code — the worst case is a miss, not a wrong snippet, which is a much better failure mode.

Serving humans and serving agents are different products

A human wants a ranked list of locations to open, quickly, with the query terms highlighted. Latency matters more than completeness, because they will refine the query themselves.

An agent wants a small number of complete, self-contained chunks that fit inside a context budget, because it cannot open a file and skim. It also benefits from the surrounding definitions, which a human already has in their editor. This is the difference discussed in RAG versus long context: retrieval quality matters most precisely when you cannot afford to send everything.

If you are building this as an agent tool, cap the response size hard and return paths plus line ranges plus the chunk text, never whole files. An unbounded search tool is the fastest way to fill a context window with material the agent did not need, and the consequences are laid out in agent memory and context management. On very large repositories, the retrieval quality bar rises further — see choosing a model for large repositories for how that interacts with model selection.

Measure against grep, not against nothing

Run your twenty evaluation queries through ripgrep and through your tool, and record whether the correct location appears in the top five. Publish both numbers internally.

You will likely find that your tool loses on eight of the twenty and wins decisively on six. That is a good result — it means the right product is a search box that runs both and merges, not a replacement. It also tells you exactly which query shapes to route where.

Track the rate at which people click a result versus reformulating the query. A reformulation is a failure, and the reformulated query usually tells you what the original was missing.

Where to start

Build the evaluation set first. Then a tree-sitter chunker, a lexical index, and rank fusion with no embeddings at all — that alone beats naive grep on a surprising share of queries and costs nothing per query.

Add embeddings only for the queries the lexical path measurably fails, and add reranking only if retrieval recall is already good. If your top 50 does not contain the answer, no reranker will save you, and a better chunker will do more for accuracy than any model upgrade.