How to build accurate template-agnostic PDF/invoice extraction using Gemini API?

Gemini API
, , ,
1

I’m building a production document price-intelligence system using Django and the Gemini API.

The system must extract structured commercial data from highly variable PDFs, scanned quotations, invoices, purchase orders, price lists and email documents.

Required fields include:

  • supplier

  • document date

  • currency

  • product/service name

  • item code / part number

  • quantity

  • UOM

  • unit price

  • line total

  • line-level specifications

The main issue is accuracy and consistency. New document layouts frequently cause:

  • missed line items

  • wrong product-to-price binding

  • incorrect dates/suppliers

  • decimal errors

  • descriptions attached to the wrong row

  • previously working formats breaking after prompt changes

I currently use Gemini multimodal extraction with a strict JSON schema, page-by-page PDF extraction, adjacent-page context and Python validation/post-processing.

For people who have built production-grade invoice/quotation/document extraction systems:

  1. Is direct Gemini vision + structured output reliable enough, or should I use OCR/Document AI first and Gemini only for semantic mapping?

  2. What architecture gives the best line-item accuracy across unknown document formats?

  3. How do you prevent hallucination and wrong row/price binding?

  4. Do you use a two-stage pipeline: source-faithful transcription first, then schema mapping?

  5. How do you regression-test prompt/model changes against a golden dataset?

  6. Have you achieved high accuracy without writing supplier-specific rules for every new layout?

My goal is not just to support a fixed set of invoice templates. I need a scalable approach where new supplier/document layouts do not require continuously adding prompt rules or Python patches.

I’m especially interested in real production experience rather than simple demo invoice extraction.