RAGnosis

Constraints

• Synthetic clinical data only; this is not a medical benchmark.
• A deliberately small set of 30 questions: 10 easy, 10 medium, and 10 hard.
• Mostly local model execution and a bounded experiment budget.
• Each configuration was run once, without error bars or significance testing.

1. 01

   Question

   A bundled evaluation question enters the selected pipeline.

2. 02

   Query processing

   The advanced path can rewrite the question and run dual retrieval.

3. 03

   Embedding

   An OpenAI-compatible embedding model encodes the query.

4. 04

   Vector retrieval

   Chroma returns candidate child chunks from the indexed corpus.

5. 05

   Parent or aggregate expansion

   Visit-level children expand to full patient records or rollup documents.

6. 06

   Reranking

   BGE or Jina reranks candidate evidence before context assembly.

7. 07

   Context assembly

   The pipeline builds the evidence passed to the answer model.

8. 08

   Generation

   A selected chat model answers only from the assembled context.

9. 09

   Evaluation

   Retrieval proxies, keyword coverage, and LLM judging score the run.

Data and models

The synthetic dataset contains 1,200 patients, 120 doctors, 12 departments, 2,200 appointments, 700 medical records, 400 prescriptions, and 368 billing rows. The implementation supports Ollama-served and OpenAI-compatible models, Chroma, BGE reranking, and Jina reranking.

Key engineering decisions

Evaluation methodology

Retrieval uses MRR, nDCG, and context keyword coverage. Answer quality combines deterministic keyword coverage with a 1–5 score from a small local LLM judge. The report compares eight run groups and preserves the underlying configurations.

Results

• Retrieval MRR increased from 0.406 in the best baseline run to 0.792 in the strongest documented retrieval configuration.
• The best documented answer-quality score was 4.622/5.
• The largest architectural gains followed small-to-big retrieval and aggregate rollup documents; model and reranker changes produced smaller, configuration-dependent shifts.

Failure modes

• A relevant-looking child chunk can omit the parent identity or relational evidence required for synthesis.
• Aggregate questions fail when the requested count or ranking is never represented in retrievable text.
• Reranking cannot recover facts that are absent from the candidate corpus.
• A local LLM judge can introduce calibration bias into answer scores.

Limitations

• Synthetic data; results must not be interpreted as clinical performance.
• Only 30 evaluation questions, with 3–5 questions in several categories.
• Single-run configurations without repeats, error bars, or statistical significance testing.
• Best-of-N selections make several representative scores upward-biased.
• Answer-quality levels depend partly on one small local LLM judge.

Reliability and security controls

• No real patient or protected health information is used.
• Run configurations and raw evaluation outputs are archived for review.
• A deterministic keyword-coverage signal complements model-based judging.
• The report states licensing trade-offs for the Jina reranker.

Latency and cost observations

The study was bounded by local hardware and budget. The repository notes that a full run rebuilds embeddings and that the default local embedding stage takes a few minutes; no portfolio-level latency or cost claim is published.

Deployment

The source, raw synthetic dataset, question bank, run archive, and interactive HTML report are public. The report is deployed separately and linked rather than embedded into this portfolio.

What I would improve next

• Repeat configurations and report variance or confidence intervals.
• Expand and preregister the evaluation set.
• Add human review for a subset of answer judgments.
• Measure latency, memory, and per-run cost alongside quality.