RAG pipeline over large clinical PDFs

HAI-DEF
1

Context:
We are building a tool that retrieves clinical protocols from large (500+ page) PDFs and shows clinicians relevant protocols in real time.

Questions:

  1. What splitting/parsing methods are best for large clinical protocol documents (semantic chunking, splitting based on structure, etc.)?

  2. Are there pre-processing steps we should consider that meaningfully improve embedding quality?

  3. What should we consider when choosing a text embedding model for a medical/clinical domain?

@fmahvar @Steven_Liao

1 Like
2

@fmahvar Would love to hear your thoughts on this - do you or your team have any experience parsing large clinical documents?

1 Like
3

You face three challenges: layout-aware parsing, semantic context-aware chunking, and high-precision smart retrieval.

While you can experiment with the GCP RAG Engine, I believe a tailored architecture is required to preserve the structural boundaries of 500+ page clinical protocols. Here is how I would go about it:

1. Layout-Aware Parsing

You cannot treat PDFs as flat text strings; doing so destroys clinical data tables and sections. For an open-source solution, I’d look into Docling. For an enterprise-managed option, I’d go with GCP Document AI, especially because it’s covered under the GCP BAA for processing HIPAA-regulated workloads.

2. Parent-Child Semantic Chunking

Standard fixed-token chunking fragments critical clinical boundaries. Instead, I’d implement a hierarchical structure and use semantic chunking via an open solution such as SemanticChunker:

  • Child Chunks: Small, granular blocks (100–256 tokens) optimized for rapid vector distance calculation.

  • Parent Chunks: Full document sections or entire data tables that preserve complete clinical context.

3. Hybrid Storage & Asymmetric Embedding

I’d embed the child chunks and store them alongside their parent ID mappings in a vector-enabled database like AlloyDB, and offload the raw text of the large parent chunks to a document store like Cloud Firestore.

To embed the child chunks, I’d look into managed services like GCP’s Text Embeddings API or open solutions like EmbeddingGemma. There may also be a custom solution based on MedGemma by extracting context-aligned hidden layers and condensing the generated embeddings. But for that to work, you’ll probably have to add a query prefix to help with semantic search recall:

instruction_prefix = "Represent this clinical protocol passage for semantic medical retrieval: "

formatted_chunk = f"{instruction_prefix}{raw_child_chunk_text}"

4. Two-Stage Smart Retrieval & Parent Swap

For real-time clinical performance, I’d implement a “Retrieve Wide, Rank Narrow” pipeline:

  • Recall (Stage 1): Compute the vector distance of the incoming user query against the child chunks in AlloyDB to surface the top 15–20 candidates.

  • Rerank (Stage 2): Pass those 15–20 child strings through a ranker such as GCP’s Ranking API to evaluate precise linguistic negation and clinical nuances. There are several open solutions as well.

  • The Swap: Filter down to the top 3–5 winning child IDs, use their relational keys to pull their full parent sections out of Firestore, and inject those complete context blocks into your LLM prompt for grounded, hallucination-free generation.