Why retrieval projects stall

Retrieval-augmented generation demos beautifully and stalls reliably. The demo works because someone hand-picked ten documents that answer ten questions. Production fails because the real corpus contains four versions of the same policy, two of which are wrong, none of which are dated, and a handful of PDFs that are photographs of paper.

The model is almost never the bottleneck. The retrieval layer is, and the retrieval layer is mostly a content and metadata problem dressed up as an infrastructure one.

The four failures, in the order they appear

• Stale and conflicting sources. The corpus contains the old policy and the new one, and nothing in the text says which is current. The system answers confidently from whichever chunk scored higher.
• Chunking that destroys meaning. A table split across two chunks becomes two lists of numbers with no headers. A clause separated from its exception becomes a wrong answer with a citation.
• No permission model. Retrieval returns whatever the index holds, so the moment a document with restricted content enters the corpus, the agent becomes a leak with a chat interface.
• No freshness path. The index is built once, and nobody owns re-indexing when the source changes, so accuracy decays silently from the day it launches.

A retrieval system inherits the quality of your documentation. If your knowledge base is a graveyard, you have built a search engine for a graveyard.

What the retrieval layer actually needs

Before anything is embedded, the corpus needs structure that the retrieval step can use as more than text. That means each document carries its source system, its owner, its effective date, its supersession status, and its access classification. Those five fields do more for answer quality than any change of embedding model, because they let you filter before you rank rather than hoping the ranking gets it right.

Then the retrieval step should be hybrid rather than purely semantic. Exact identifiers, product codes, and policy numbers are where vector search is weakest and keyword search is strongest, and real questions contain both prose and identifiers. Ranking the union of the two beats either alone in most operational corpora.

Make the system show its work

Every answer should carry the chunks it was built from, with their dates and owners visible to the user. This serves two purposes. It lets the reader verify an answer without leaving the interface, and it turns every wrong answer into a content bug with an address, which is the only way the corpus ever improves.

It also changes the review conversation. Without citations, a wrong answer is an argument about whether AI works. With citations, it is a ticket assigned to whoever owns the document that was wrong, and those tickets get closed.

The honest sequencing

If the corpus is unowned and undated, do not start with retrieval. Start with a narrow, well-maintained slice of it, prove the pattern there, and use the visible value to fund the cleanup of everything else. Teams that try to index everything at once ship a system that is plausibly wrong across the whole business, which is worse than a system that is reliably right about one part of it.