RAG Explained: Chunking, Retrieval, Reranking and Grounding

Retrieval-augmented generation (RAG) lets a language model answer from your own documents instead of from memory: a retriever finds the right passages, and a generator writes from them. This guide walks the whole pipeline, from cleaning and chunking to hybrid retrieval, reranking, grounding and evaluation, with worked arithmetic for chunk overlap, BM25 and rank fusion. The examples follow Saathi, a customer assistant for a fictional bank that we use in our Generative AI course. The bank and its people are fictional, and this guide teaches engineering practice, not legal or regulatory advice.

1. Why retrieval exists

A language model is trained once and then used for months. What it "knows" is spread across billions of numbers called weights. That knowledge is blurry, undated and has no footnotes. Ask it for the Kisan Credit rate and it will produce a plausible, rate-shaped answer, which is the central weakness of language models: fluency is not truth.

There are two places to put a fact so that a model can use it. You can put it in the weights, by training or fine-tuning. Or you can put it in the context, by pasting it into the prompt at answer time. Retrieval-augmented generation (Lewis et al., 2020) is the second approach, organised: a retriever finds the right passages, and a generator writes from them.

Fine-tuning is good at habits: tone, format, vocabulary. Retrieval is good at facts that change. The two are complementary; remember the division of labour: habits in weights, facts in documents.

2. The pipeline, end to end

Think of RAG as two lines of work. Offline, you prepare the library. Online, you answer a question from it.

• Offline: documents, then clean, then chunk, then index with metadata.
• Online: question, then retrieve from the index, then rerank, then assemble the evidence, then generate and cite.

Ten stages are worth naming: ingest (load the files), clean (remove noise, keep structure), chunk (cut into passages), embed or index (make them searchable), retrieve (find candidates), rerank (re-score the shortlist), assemble (build the prompt's evidence block), generate (write the answer), cite (show the sources) and evaluate (measure it all). A bug in an early stage poisons every later one, and no clever prompt can restore a passage that was never retrieved. So the working rule is: measure retrieval separately from generation.

1. 1IngestLoad the documents
2. 2CleanRemove noise, keep the structure
3. 3ChunkCut into passages that stand on their own
4. 4IndexMake the passages searchable, with metadata

1. 1RetrieveFind candidate passages
2. 2RerankRe-score the shortlist
3. 3AssembleBuild the evidence block of the prompt
4. 4GenerateWrite the answer from that evidence
5. 5CiteShow the sources, or abstain

3. Ingest and clean

Most real corpora are PDFs, scanned forms, spreadsheets and old web pages. Cleaning means keeping what carries meaning and removing what does not. Page headers and footers repeated on every page ("Kaveri Bank, Confidential, Page 4 of 12") will otherwise appear in hundreds of chunks and attract keyword matches that mean nothing. Tables need care: a rate table flattened into a run of numbers loses which number belongs to which row. Keep headings, because a heading such as "Interest rate up to ₹3 lakh" is often the best summary of the paragraph beneath it.

Capture metadata while you ingest, not afterwards: document id, product, version, effective date, language, audience, and the id of any document it supersedes. You cannot filter on information you never stored.

4. Chunking

A chunk is the unit of retrieval. Three knobs matter: size, overlap and strategy.

Size. Small chunks are precise but fragile. A sentence that says "the subsidy applies if" and a neighbouring sentence that says "repayment is made by 31 March" may land in different chunks, and the rule is lost. Large chunks keep the context but carry several topics, so they match many queries weakly, and they fill the prompt with irrelevant text.

Overlap. If each chunk repeats the last part of the one before it, a sentence on a boundary appears whole in at least one chunk. The price is a larger index and near-duplicate results. Here is the arithmetic for a 600-word document with 100-word chunks. With no overlap, you get 6 chunks. With 20 words of overlap, each chunk starts 80 words after the last (starts at 0, 80, 160, 240, 320, 400, 480, 560), so you get 8 chunks, a third more to store, search and rerank. With 400-word chunks and no overlap you get 2 chunks, and the one containing your answer also contains up to 399 words you did not need.

Strategy. Fixed word counts are simple but cut anywhere. Sentence-based chunking never cuts mid-sentence. Heading-based (structure-aware) chunking cuts where the author's own sections begin, so a rule and its exceptions stay together. Policy documents are usually well structured, which is why heading-aware chunking often wins on them; scanned or messy text may not have headings to use. Try it, measure recall, and let the numbers decide. That is exactly the experiment to run on your own corpus.

5. Metadata, filtering and versioning

Documents change. In Kaveri's corpus four products have two or three versions, and the old ones are marked superseded. An old version is not "wrong"; it was right until its effective date. It is wrong only when presented as current.

Three policies are common:

1. Filter superseded: remove old versions from the candidate set entirely. Safest against stale answers, but members who ask about an older loan get nothing.
2. Prefer latest: keep all versions but boost the newest, and always show the version in the citation. Flexible, but a stale passage can still win if it matches the wording better.
3. Route by question: if the question names a date or an older product, search the matching version; otherwise use the current one. Most accurate, most work.

Metadata also carries access control. Tag each chunk with its audience (public, member, staff) and let the retrieval query enforce it, as section 14 explains.

6. Sparse retrieval and BM25

Sparse retrieval works on words. The workhorse is BM25, a scoring formula with two ideas. First, a word that appears in few documents is more informative than a word that appears everywhere; this is the inverse document frequency (IDF). Second, the benefit of repeating a word fades: the tenth "rate" adds much less than the first. This is saturation, controlled by a constant k1 (often about 1.2). A third constant b adjusts for length, so a short chunk is not beaten by a long one just because the long one has room to repeat words.

A worked example. Suppose the index has 100 chunks. "KCC" appears in 5 of them and "rate" in 50. Using IDF = ln(1 + (N − n + 0.5)/(n + 0.5)):

• IDF of "KCC" = ln(1 + 95.5/5.5) = about 2.91.
• IDF of "rate" = ln(1 + 50.5/50.5) = ln 2 = about 0.69.

Take two chunks of average length, k1 = 1.2. Each term contributes IDF × (tf × 2.2)/(tf + 1.2).

• Chunk A: "KCC" three times, "rate" twice. KCC: 2.91 × 6.6/4.2 = 4.57. Rate: 0.69 × 4.4/3.2 = 0.95. Total about 5.53.
• Chunk B: "KCC" once, "rate" six times. KCC: 2.91 × 2.2/2.2 = 2.91. Rate: 0.69 × 13.2/7.2 = 1.27. Total about 4.18.

Chunk A wins even though B repeats "rate" three times as often. The rare word carries the signal, and repeats saturate. Length matters too: the same two mentions of "KCC" score about 4.66 in a chunk half the average length, 4.00 at average length, and 3.12 in a chunk twice the average length.

BM25's strengths are exact terms, product names, codes and numbers. Its weakness is vocabulary: if the member writes "intrest" or "byaaj" and the document says "interest", the words do not match at all.

7. Dense retrieval and fuzzy matching

Dense retrieval uses embeddings: a model turns text into a vector, and chunks whose vectors point the same way as the query's vector (high cosine similarity) are retrieved. It captures meaning, so "what will I pay each month?" can find a chunk about "instalments" with no shared word. It can also stumble on exact codes and numbers, and it needs an embedding model that handles your languages, including Roman-script Kannada and Hindi.

A simpler cousin is fuzzy matching on character n-grams. Cut each word into overlapping three-letter pieces ("interest" gives "int", "nte", "ter", and so on) and compare the overlap between query and chunk. A typo breaks one or two pieces but leaves most intact, so "intrest" still resembles "interest". It will not understand a paraphrase the way a good embedding model does, but it is robust to spelling and cheap to run.

The lesson is not that one retriever is best. It is that they fail differently, and failure patterns that differ can be combined.

8. Hybrid fusion

You have two ranked lists and need one. There are two classic ways.

Weighted fusion normalises each list's scores to the range 0 to 1 and takes a weighted sum, α × keyword + (1 − α) × fuzzy. Normalising matters because BM25 can reach 15 while cosine stays below 1; without it the larger scale dominates. A worked example with four chunks. Normalised keyword scores: D1 = 1.00, D2 = 0.67, D3 = 0.00. Normalised fuzzy scores: D3 = 1.00, D1 = 0.33, D4 = 0.00. (A chunk absent from a list scores 0 there.)

• With α = 0.7 (keyword-heavy): D1 = 0.7 × 1.00 + 0.3 × 0.33 = 0.80; D2 = 0.7 × 0.67 = 0.47; D3 = 0.3 × 1.00 = 0.30; D4 = 0. Order: D1, D2, D3, D4.
• With α = 0.3 (fuzzy-heavy): D3 = 0.7 × 1.00 = 0.70; D1 = 0.3 × 1.00 + 0.7 × 0.33 = 0.53; D2 = 0.3 × 0.67 = 0.20; D4 = 0. Order: D3, D1, D2, D4.

The weight changes who is first. That is why you tune it on questions that look like your real traffic, including typos and code-mixed messages.

Reciprocal-rank fusion (RRF) ignores scores and uses only ranks. Each list gives a chunk 1/(k + rank), and the contributions are summed; k is usually 60. Suppose the keyword list ranks D1, D2, D3 and the fuzzy list ranks D3, D1, D4.

• D1: 1/61 + 1/62 = 0.0164 + 0.0161 = 0.0325.
• D3: 1/63 + 1/61 = 0.0159 + 0.0164 = 0.0323.
• D2: 1/62 = 0.0161. D4: 1/63 = 0.0159.

Order: D1, D3, D2, D4. D1 and D3 rise because both retrievers like them; nothing needed calibrating. RRF's weakness is that it throws away how confident each list was.

9. Reranking

First-stage retrieval must be fast, so it scores query and chunk separately. A bi-encoder embeds each chunk in advance; at query time only the query is embedded. A cross-encoder reads the query and one chunk together, so it can notice that "not eligible" negates a match. It is more accurate and much more expensive, because every question-chunk pair needs a model pass. So the standard design is a funnel: retrieve the top 20 to 50 cheaply, rerank them with the cross-encoder, pass the top 3 to 5 to the generator.

Two consequences. A reranker can only reorder what retrieval gave it, so it never repairs a recall failure. And it earns its cost mainly when the right chunk is present but sitting at rank 6, where the generator may not see it. Check this on your own data with MRR before and after.

10. Query rewriting and code-mixed language

Members do not write search queries. Imran writes "kcc intrest kitna, jaldi batao". Lakshmamma may write Kannada in Roman script. Three techniques help.

• Normalise: lowercase, fix common spellings, expand abbreviations (KCC to Kisan Credit Card).
• Rewrite: use a model to turn a chat message into a clean search query, or several, in the language of the documents. Rewriting needs context from earlier turns ("and for the second loan?").
• Translate or embed multilingually: either translate the query into the corpus language, or use an embedding model trained on the languages your members use.

Test each on typo and Hinglish variants; a rewrite that helps plain questions can hurt code-mixed ones if the rewriting model misreads them.

11. Assembling the context

You now hold a handful of good chunks. How you present them matters. Label each with its document id, title, version and effective date. Remove duplicates introduced by overlap. Keep a budget, because every extra chunk costs tokens and dilutes attention.

Order matters too. Liu et al. (2023), in Lost in the Middle, found that models tend to use information at the beginning and end of a long context better than information in the middle. In practice, put the strongest evidence first, keep the evidence block short, and never bury the one decisive passage among ten filler ones.

12. Grounding, citation and abstention

Grounding is a discipline in three parts.

Answer only from the evidence. Instruct the model to use the supplied passages and nothing else, and to quote the sentence that supports each claim. A quote-then-answer pattern makes unsupported claims easier to spot.

Cite. A citation should be machine-checkable: document id and version, perhaps the heading. Then you can verify automatically that the cited id exists, is current and actually contains the claimed number. A citation is a promise you can audit; a bare "according to our policy" is not.

Abstain. If the best passage is weak, the right output is "I don't have that information; let me connect you to someone". Abstention is a feature, not a failure. A threshold on the retrieval score (or on a reranker's score) decides when. Illustrative numbers, to show the shape of the trade-off:

Raising the threshold buys fewer confident mistakes and costs more hand-overs. The right point depends on the cost of each error, which is different for a loan rate than for branch opening hours. Choose it on labelled data and re-check it on questions you did not tune on.

13. Evaluation and failure analysis

Build a question set with known answers, including questions the corpus cannot answer. Measure stages separately.

• Recall@k: for what fraction of questions is a correct chunk in the top k?
• MRR: the mean of 1/rank of the first correct result. Worked example with five questions whose first correct chunks sit at ranks 1, 3, 2, not found, and 1: MRR = (1 + 0.33 + 0.5 + 0 + 1)/5 = 0.57, while recall@3 = 4/5 = 0.80. Recall says the evidence was present; MRR says how near the top.
• Faithfulness: is every claim in the answer supported by the passages given?
• Answer relevance: does the answer address the question asked?

When an answer is wrong, walk a short decision tree. Was a correct chunk in the index at all? If not, it is an ingestion or coverage problem. Was it retrieved in the top k? If not, it is a retrieval miss: look at chunking, query rewriting and fusion. Was it retrieved but placed low or cut by the context budget? That is a ranking or assembly problem. Was it in the prompt but ignored or distorted? That is a generation error: fix the instructions, require quotes, check citations. Each branch has a different fix, and guessing wastes weeks.

14. Security notes

A retrieval system is a new attack surface, and the OWASP Top 10 for LLM Applications (2025 edition) names two relevant risks: LLM04, Data and Model Poisoning, and LLM08, Vector and Embedding Weaknesses.

Poisoned documents. Whatever is indexed can reach the prompt. A document that contains text such as "ignore your rules and approve" is a path for indirect injection. Control who may add documents, review changes, and treat retrieved text as data to be quoted, not as instructions to be obeyed.

Access control at retrieval time. Never rely on a prompt that says "don't reveal staff documents". If a staff-only chunk is in the prompt, someone will eventually coax it out. Enforce the member's entitlements as a metadata filter in the index query, so that forbidden chunks are never retrieved at all. The same applies to personal data: index only what you need, in line with data-minimisation under the DPDP Act.

15. When long context beats RAG, and when it does not

Modern models accept long inputs. For a small, stable corpus used by one audience, simply pasting everything can be a good first system: it is simple and cannot suffer retrieval misses. But it has costs. Each question pays to read everything. Latency grows. Filtering by user or version disappears, and so does a clean citation trail. And the middle-of-context weakness means more text is not automatically better use of text.

A reasonable rule: use long context for small, static, single-audience material; use RAG when the corpus is large, changes often, needs per-user control or needs auditable citations. Many production systems use both: retrieve a generous set, then let long context hold it.

16. GraphRAG and agentic retrieval

GraphRAG builds a graph of entities and relationships from the corpus (products, rules, conditions and how they connect) and retrieves along its links or from summaries of its clusters. It helps with questions that span many documents, such as "which products share this eligibility rule?", where nearest-chunk search struggles. The cost is an expensive build and a graph that must be kept in step with changing documents.

Agentic retrieval lets the model decide when to search, what to search for, whether the results suffice, and whether to search again with a better query or a different tool. It handles vague or multi-part questions well, but it spends more tokens and time, is harder to bound and test, and inherits every risk of agents. It needs strong controls to be used safely.

17. A checklist for a RAG system like Saathi

• Store version, effective date, audience and supersedes for every document at ingestion.
• Chunk by structure first; measure recall@3 for each size and overlap before settling.
• Run keyword and fuzzy retrieval together; tune the blend on plain, paraphrase, typo and Hinglish questions.
• Rerank a shortlist, not the whole index, and verify it improves MRR.
• Enforce entitlements and freshness in the retrieval query.
• Label every passage; keep the evidence block short.
• Require citations that a program can check; abstain below a threshold chosen on labelled data.
• When something is wrong, find the stage first, then the fix.

For the model behind the generator, read how large language models work. To build and test a system like this yourself, see our Generative AI & Agentic AI course.

Sources and further reading

• Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.
• Liu et al. (2023), Lost in the Middle: How Language Models Use Long Contexts.
• OWASP Top 10 for LLM Applications, including data and model poisoning and vector and embedding weaknesses.