Implementing Two-Phase RAG Systems for Unstructured Medical Documents

A discharge summary, a lab panel, a scanned consent form, and a 40-page payer policy all arrive as "a PDF." They share a file extension and nothing else. The lab panel is a table where one cell flips a clinical decision. The policy is prose with cross-references. The consent form is a scan that OCR turned into half-readable soup. One embedding pipeline cannot serve all four, and pretending it can is where most of these projects quietly fail.

So I split the work in two. Phase one decides what a document is and pulls out the structured spine. Phase two answers questions against it. The phases run on different schedules and fail in different ways, and that separation is the whole point.

Classify and extract before you ever embed

The first pass never touches the vector store. It routes. A cheap classifier, often just the document's own headers plus a small model call, tags each file: discharge summary, medication list, imaging report, administrative. That tag picks the extractor. Lab tables go through a layout parser that preserves rows and units, so "creatinine 2.1 mg/dL" stays one fact instead of three floating tokens. Prose goes through section-aware chunking that keeps a heading attached to its body. Scans get an OCR-confidence score, and anything under threshold gets flagged for review instead of silently entering the index as garbage.

The output of phase one is not embeddings. It is structured records with hard metadata: document type, patient ID, encounter date, section label, source page, OCR confidence. Then I embed. The metadata earns its keep, because phase two leans on it far more than on vector similarity.

In one system, a query like "what was the patient's potassium on admission" kept missing the right value, which sat two rows down in a parsed table. Cosine similarity put a discharge potassium first; both chunks read almost identically to an embedding model. The fix was not a better embedding. It was a metadata filter applied before the vector search ran: restrict to the encounter date, prefer the "admission labs" section. Retrieval here is filter-then-rank, not rank-everything.

Why I reject the single pipeline

The tempting approach is a single pass: dump every document through a uniform chunker, embed it all, and trust a strong retriever plus a reranker to sort it out at query time. Less code. One chunker. Trust the reranker. I still reject it, for a specific reason.

A uniform chunker destroys the structure that makes medical data answerable, and that loss is unrecoverable downstream. Split a lab table at 512 tokens and the unit lives in one chunk while the value lives in another; no reranker can reassemble a fact that was never stored intact. Strip the section heading and "negative" has no idea whether it modifies the cancer screen or the allergy history. The model at query time cannot recover information you threw away at ingestion. A reranker reorders what retrieval found. It cannot resurrect what chunking shredded.

The single pipeline also hides its own failures. A bad OCR scan becomes a few low-quality vectors among millions, invisible until it surfaces a wrong dose in an answer. The two-phase version stops that document at the gate, with a confidence number attached, before anyone trusts it.

Phase one is more pipeline to build: more parsers to maintain, more document types to enumerate. But you spend that effort once per document, at ingestion. The alternative spends it at every query, in answers you cannot fully trust. For medical data, that is the wrong place to economize.