Building a Docs Chatbot That Refuses to Guess

The failure that kills a documentation chatbot is not that it cannot find the right page. It is that when the answer is not in the documentation at all, it produces a fluent, well-formatted, entirely fabricated answer, complete with a configuration flag that does not exist.

Users cannot tell the difference. They will file a bug report about the flag, your support team will spend an hour on it, and after the third occurrence someone will suggest turning the bot off. Everything below is aimed at that single failure.

The corpus sets the ceiling

Retrieval quality cannot exceed corpus quality. If the documentation does not say what the rate limit is, no amount of reranking will find it, and the model will invent one because a plausible number is more probable than a refusal.

Before building anything, take the top 100 support questions from the last quarter and check, by hand, whether each is answered somewhere in the docs. In most organisations the honest number is between 50 and 70 percent. That figure is your realistic ceiling, and it reframes the project: half the value is going to come from writing the missing pages, not from the retrieval stack.

Include the sources people actually answer from, not just the published site. Release notes, the API reference, the FAQ, and — carefully, with permission — resolved support tickets, which frequently contain the only written explanation of a behaviour. Exclude drafts, internal runbooks and anything you would not want quoted back to a customer.

Chunk on headings, and carry the hierarchy

Documentation has an explicit structure and you should use it rather than splitting on token counts. Chunk at the smallest heading level that produces a self-contained unit, usually h2 or h3, and split further only when a section is too long.

Prepend the full heading path to every chunk before embedding: product, page title, section, subsection. A chunk that begins "Set this to false to disable" is meaningless in isolation and retrieves for nothing; the same chunk headed with its path is unambiguous.

Keep code blocks and tables intact within a chunk. Splitting a configuration example in half is worse than a slightly oversized chunk, because a truncated example is what the model will paste into its answer. The general reasoning behind embedding behaviour here is covered in embeddings explained.

Version awareness is not optional

If your product has versions, this is the feature that decides whether the bot is trusted. Answering a question about v2 with v4 documentation is indistinguishable from hallucination from the user's side, and it happens constantly with naive retrieval because the same section exists in every version and they all embed nearly identically.

Store the version as metadata on every chunk and filter on it before the vector search, not after. Resolve the version from the user's account where you can, from an explicit selector otherwise, and default to the latest with a visible label saying so.

Deprecated content is the related trap. A page describing a removed feature still retrieves well, because it matches the query perfectly. Tag deprecation explicitly and either exclude it or force the answer to lead with the deprecation notice. Never let a removed feature be described in the present tense.

Hybrid retrieval, then force citations

Run lexical search alongside vector search and fuse the results. Documentation queries contain exact identifiers constantly — error codes, flag names, endpoint paths — and pure semantic search is worse than keyword matching at those. The same argument applies here as in building a code search tool: fusing two mediocre rankings beats either alone and needs no tuning.

Then make citation structural rather than requested. Number the retrieved chunks, require the answer to reference them by number, and validate after generation that every referenced number exists and that no sentence carrying a factual claim lacks one. If validation fails, do not post the answer — regenerate once, then abstain.

This is the mechanism that produces abstention reliably. Prompting a model to say it does not know works perhaps half the time. Refusing to render an answer whose claims are not attributable works every time, because it is your code making the decision rather than the model.

Make the retrieval decision explicit

Before generating anything, score whether the retrieved set is good enough to answer from. A cheap classifier call, or a similarity threshold on the top result, is enough to catch the case where nothing relevant came back.

When it does not clear the bar, say so plainly and give the user somewhere to go: the closest three pages by title, and a link to open a support ticket with the question pre-filled. A bot that says it cannot answer and hands you a ticket is a useful product. One that guesses is a liability. The reasons a model will fill that gap by default are worth understanding — see why LLMs hallucinate.

Resist the instinct to raise coverage by loosening the threshold. Every point of coverage bought that way is bought with wrong answers, and wrong answers cost far more support time than an abstention does.

Follow-up questions need query rewriting

The second turn of a conversation is where naive retrieval collapses. "Does it work with SSO?" retrieves nothing useful, because the subject is in the previous turn.

Rewrite the query before retrieving. A cheap model call that takes the last few turns and emits a standalone question — "Does the audit log export work with SAML single sign-on?" — fixes almost all of it. Log both the original and the rewritten query, because a bad rewrite is a common and otherwise invisible cause of a bad answer.

Keep the conversation state server-side, and cap how much history goes into the rewrite. Two or three turns is plenty; sending twenty adds cost and drags the rewrite off topic. The broader mechanics of conversation state, idempotency and cost controls are covered in building a chatbot from scratch and apply here unchanged.

Unanswered questions are the roadmap

The most valuable output of a docs bot is not the answers. It is the log of questions it could not answer, clustered by topic.

That list is a ranked, evidence-backed documentation backlog, generated by your actual users rather than guessed at in a planning meeting. Review it weekly, write the missing pages, and watch coverage rise for reasons that have nothing to do with the model. This feedback loop is the main reason the project is worth doing at all, and it pairs naturally with a broader approach to generated documentation.

Also track the rate at which users open a ticket immediately after an answer. That is your false-confidence signal: an answer the user did not believe, or that was wrong. It is more informative than a thumbs-down widget, which almost nobody clicks.

Cost, abuse and the boring operational parts

A public docs bot is an inference endpoint on the open internet. Rate limit per IP and per account, cap input length, cap output length, and set a daily spend ceiling with an alert. The realistic threat is not a clever attacker but one script in a loop.

Cache aggressively. Documentation questions cluster hard — a small number of questions account for a large share of traffic — so a normalised-query cache with a short expiry removes a lot of load. Embeddings for unchanged chunks should never be recomputed; key them by content hash and only embed what changed on each docs deploy.

Rebuild the index as part of the documentation publish pipeline, not on a nightly cron. A bot answering from last night's docs on release day is the same failure as answering from the wrong version.

The order to build it in

Audit corpus coverage against real support questions. Chunk on headings with version metadata. Fuse lexical and vector retrieval. Rewrite follow-up queries. Force numbered citations and validate them in code. Abstain below a threshold and hand off to support. Log the unanswered questions and write the missing pages.

The decision rule for shipping: if you cannot show that the bot abstains on a question the documentation genuinely does not answer, it is not ready, however good the answers look on the questions it can handle.