Your RAG isn't broken. Your chunks are.

The report is always the same. It answers most things well, but sometimes it makes things up, and we can't work out when.

And the first instinct is always to go at the prompt. Add "only answer from the provided context." Add "say you don't know." Turn the temperature down. Try a bigger model.

Almost none of that helps, because almost none of those systems were failing at generation. They were failing at retrieval, and a model handed the wrong three paragraphs will do exactly what you'd do in the same position: answer from them anyway.

So before touching the prompt, answer one question.

Was the right passage even in the context?

Not "was the answer good". Was the source that contains the answer present in what you handed the model?

This is measurable and most teams have never measured it. You need thirty to fifty real questions — from your actual users, not invented — and for each one, the document that contains the answer, identified by a human. Then run retrieval alone and check whether that document came back in the top k.

That number is your ceiling. If it's 60%, then 40% of your questions cannot be answered correctly no matter what you write in the system prompt, and every hour spent on prompt wording is an hour spent on the wrong floor of the building.

In every system I've been brought into, this number was the problem. Usually badly — the sort of gap where fixing retrieval moved answer quality more than any model upgrade did.

The chunking is usually the culprit

The default in every tutorial is to split on a token count with some overlap. It is fast to implement and it is wrong for most real documents, for a reason that's obvious once you look at what it produces: it cuts through the middle of things.

A table split across two chunks becomes two chunks of numbers with no header. A clause split mid-sentence becomes a fragment that means the opposite of what it said. A policy whose conditions are in paragraph one and whose exceptions are in paragraph four becomes four chunks, none of which is correct on its own.

What works better is boring: split on the document's own structure first, and only fall back to length when a structural unit is genuinely too big. Headings, sections, list items, table rows with the header re-attached to each one. Then, when a chunk is a fragment of something larger, carry the context down into it:

text = f"{doc_title} — {section_heading}\n\n{chunk_text}"

That one line, prepending the document and section title to every chunk before embedding, is the highest-yield change I know in this whole area. It costs nothing and it fixes the pathology where a chunk reading "must be filed within 30 days" is unfindable because nothing in it says what must be filed.

Queries and chunks don't live in the same neighbourhood

There's a mismatch nobody warns you about. Your chunks are long, specific, and written in the formal register of whoever wrote the document. Your queries are eight words, vague, and in the register of somebody in a hurry.

You embed both into one space and hope they land near each other. Often they don't — not because the retrieval is bad, but because "can I claim this" and a paragraph titled Eligibility of Expenditure under Clause 4.2 genuinely are far apart as text.

Two things help, and both are cheap.

Hybrid retrieval. Run keyword search alongside vector search and merge the results. Dense retrieval is reliably bad at exactly the things people search for most: product codes, policy numbers, acronyms, surnames, anything where the string itself is the query. IFC PS6 has a meaning to your users and effectively none to an embedding model. BM25 finds it instantly. Run both, merge with reciprocal rank fusion, move on.

Rerank the shortlist. Retrieve fifty, then score each candidate against the query with a cross-encoder and keep the best five. It's a second model call over a small candidate set and it is consistently the largest single quality jump available, because a cross-encoder actually reads the pair together instead of comparing two vectors that were computed in ignorance of each other.

Filter before you search, not after

If your query is scoped — one fund, one year, one document type — apply that as a filter inside the vector search, not as a pass over the results afterwards.

Filtering afterwards means your top 50 was chosen from the whole corpus, and after filtering you may be left with four, none of which was the best match within the scope you cared about. Every serious vector store supports pre-filtering. Use it.

Negation, and knowing when to stop

Some things retrieval simply will not do, and it's worth knowing them so you stop trying.

Embeddings are weak at negation — "projects not requiring an environmental assessment" retrieves passages about projects requiring one, because that's what the sentence is mostly about. They're weak at counting and aggregation, because "how many funds closed in 2024" has no single passage that contains the answer. And they cannot do arithmetic across documents.

Those want a different tool: a query against structured data, exposed to the model as a function it can call. Trying to solve them with better chunking is a long walk to a wall.

An order of operations

If I'm handed a system that's underperforming, this is the order, and I rarely get past the third step:

1. 1Measure retrieval on its own. Fixed question set, known-correct sources, recall@k. Write the number on the wall.
2. 2Look at twenty failures by hand. Not summary metrics — read them. The pattern is usually visible in the first ten and it's usually chunking.
3. 3Fix ingestion. Structural chunking, titles prepended, tables handled deliberately.
4. 4Add hybrid retrieval, if identifiers and acronyms appear in the failures.
5. 5Add reranking.
6. 6Then, and only then, look at the prompt.

The order matters because every step above the prompt changes what the prompt is working with, and tuning wording against a moving retrieval layer is how teams spend three weeks going in a circle.

A RAG system that feels good and one that is correct diverge quietly, and the only thing that catches it early is a fixed set of questions you re-run every time you change the ingestion.

Set that up on day one. It is thirty questions in a file. It will save you a month.