RAGnosis and the context engineering lesson

RAG advice gets vague fast.

Use query rewriting. Add a reranker. Improve your chunks. Try a better embedding model. Enrich your documents. Retrieve smaller pieces and expand to larger context. All of that sounds right, but it is hard to know which part actually matters until you put numbers behind it.

That is why I built RAGnosis.

RAGnosis is a self-contained applied research project for benchmarking retrieval-augmented generation techniques over a synthetic healthcare database. The database is not clinical data. It is a fake clinic system with patients, doctors, departments, appointments, medical records, prescriptions, and billing rows.

That shape made it useful for testing RAG because the questions were not all simple lookups. Some asked for direct facts. Some joined multiple tables. Some needed visit-level details. Some asked for counts, totals, rankings, or summaries across the whole corpus.

A normal vector search baseline can sometimes find a patient or a doctor. It struggles when the answer is spread across several records, or when the answer is an aggregate that does not exist as a single sentence anywhere.

The project started with a basic RAG pipeline: create documents, split them with recursive character chunking, embed the chunks, retrieve the top-k matches, and answer from that context.

Then I tested the kinds of improvements people usually talk about:

• query rewriting
• dual retrieval with the original and rewritten query
• cross-encoder reranking
• document enrichment with generated titles and summaries
• Small-to-Big retrieval
• rollup documents for aggregate questions
• a final reranker ablation with Jina

Every run had retrieval evals and answer evals. Retrieval used MRR, nDCG, and context keyword coverage. Answers were judged on accuracy, completeness, and relevance, with answer keyword coverage tracked separately as a deterministic signal.

The first baseline was useful because it failed in predictable ways.

The best Basic run reached 0.406 retrieval MRR and 2.856 answer overall. Direct facts were much easier than numerical, spanning, or holistic questions. That made the problem clearer: the model did not only need better ranking. It needed better evidence.

Query rewriting and BGE reranking helped, but not enough. The best run with rewriting and reranking reached 0.425 retrieval MRR and 3.056 answer overall. That is a real improvement, but it did not fix the underlying issue.

Reranking can reorder candidates. It cannot make missing context appear.

Document enrichment was also mixed. Generated titles and summaries gave the embedding model more searchable text, but the results stayed in the low 3s for answer quality. Enrichment helped in some configurations and did not in others. It made the documents nicer to search, but it still did not change the unit of retrieval enough.

The first big jump came from Small-to-Big retrieval.

Instead of searching arbitrary chunks from large patient records, the system searched small visit-level child documents. Each child kept the patient identity in the text and pointed back to a full parent patient record. At answer time, the system retrieved precise visit-level matches and expanded them to the larger patient context.

That changed the job.

Retrieval was still precise, because the child chunks were small. The answer had enough context, because the parent record came back into the prompt. Run 6 reached 0.614 retrieval MRR and 4.044 answer overall.

The improvement was especially obvious for spanning questions, where the answer needed several details about one entity. The system could find the right visit, then answer from the whole patient record instead of a chopped-up fragment.

But Small-to-Big retrieval did not solve aggregate questions.

That failure was useful. A question like "Which doctor has the largest appointment load?" is not a lookup problem. It is an aggregate computation. If no document says the answer, vector search has nothing fair to retrieve.

So Run 7 added rollup documents: precomputed counts, rankings, totals, and summaries that made aggregate facts explicit. Patients by city. Bills by payment status. Doctors by appointment load. Patients ranked by total billed amount.

That was the second big jump.

The best Run 7 result reached 0.779 retrieval MRR, 0.818 context keyword coverage, 4.622 answer overall, and 4.500 hard-question answer overall. Numerical answers rose from 3.333 in Run 6 to 4.667. Holistic answers rose from 1.444 to 5.000.

The lesson was blunt: RAG cannot reliably retrieve an answer that was never written down.

The final run swapped the reranker from BGE to jinaai/jina-reranker-v3 while keeping the rollup architecture. It improved retrieval with the stronger embedding setup, reaching 0.792 MRR and 0.831 context keyword coverage, the highest retrieval numbers in the archive. But the best answer overall, 4.522, was slightly below the best Run 7 result.

That does not mean the reranker was bad. It means the architecture change mattered more.

For this dataset, the practical recipe was:

• rewrite and rerank the query
• retrieve small visit-level child chunks
• expand matches to larger parent records
• create rollup documents for aggregate facts
• evaluate retrieval and answers separately

The most important part was not picking the biggest model. It was making the corpus answerable.

That is the part I would carry into future RAG work.

If a question needs a patient visit, retrieve at the visit level and answer from the patient level. If a question needs a count or ranking, write that aggregate down before retrieval. If a question needs relational context, do not cut the identity away from the chunk. If a reranker is not helping much, check whether the right evidence exists before blaming the reranker.

The project has caveats.

The dataset is synthetic. The evaluation set has 30 questions. Each configuration ran once. The answer judge is a small local model, so the absolute 1 to 5 scores are calibration-dependent. The per-category results are directional because some categories have only a few questions.

Still, the direction was clear enough to be useful.

The score moved most when the retrieval corpus changed shape. Query rewriting and reranking helped. Better embeddings helped. Document enrichment sometimes helped. But Small-to-Big retrieval and rollup documents changed what the system could know.

That is the core RAGnosis lesson for me:

RAG quality is not just a retrieval problem. It is a data representation problem.

The model can only answer from the evidence you make retrievable.

Source notes

• Interactive report: https://ragnosis.samarthmn.com/
• GitHub repository: https://github.com/samarthmn/RAGnosis
• RAGnosis README and RESEARCH.md were used for the run descriptions, evaluation setup, limitations, and reported metrics.