How to reliably extract date and amount from poor-quality photos of digital receipts

Gemini API
1

Hi everyone,

We are building a Rocket.Chat bot (Apps-Engine, JS) that processes bank receipts sent by employees. The bot needs to extract the transaction date and total amount, then verify the date against today’s date.

The hardest edge case we face: photos of a phone screen taken with another phone (held in hand, bad angles, screen glare, tiny fonts, blurry backgrounds). Standard direct screenshots parse perfectly, but these physical photos are causing a lot of headaches.

Our Current Stack:

  1. Yandex Vision OCR: We extract raw text and use regex/templates. We’ve tried 4 modes (page, column-sort, table, markdown).
  2. Gemini-3.8-flash (via gptunnel aggregator): Accessed via an OpenAI-compatible chat/completions endpoint. We pass the image as base64 (image_url), use response_format: { type: "json_schema" } with strict: true, and max_tokens: 3000. Expected output is a JSON object (is_receipt, date, time, amount, currency, status, confidence).
  3. Yandex AI Studio (qwen3.6-35b-a3b): Used as a fallback model.

The Issues:

  • The OCR often completely misses the total amount on these glossy screen photos. In one instance, it mistook the date “18.09” for an amount of $18.09.
  • Gemini via the aggregator fails to return parsable JSON in about 5 out of 8 requests. We’re not sure why—it might be wrapping the output in markdown (```json), or getting cut off due to reasoning chains hitting token limits. The exact same receipt works perfectly in one request but fails in the next.
  • Sometimes Gemini reads the amount perfectly but hallucinates the year (e.g., outputs 2024 instead of 2026).
  • Due to these issues, valid, honest receipts are frequently rejected.

My Questions:

  1. How do you reliably enforce strict JSON from Gemini via OpenAI-compatible aggregators? Should we stick to json_schema strict: true, or is it better to allow a free-form response and use a tolerant JSON parser on our backend?
  2. How can we eliminate the year hallucination? Should we explicitly inject the current date into the system prompt?
  3. Is image preprocessing (auto-cropping the screen, perspective warp correction, contrast adjustment) absolutely necessary before feeding the image to a Vision LLM?
  4. In your experience, which Vision models handle this specific “photo of a screen” edge case best? Does a “cheap model reads, expensive model verifies only on low confidence” architecture make sense here?
  5. Has anyone solved a similar receipt extraction issue in production? What workflow finally worked for you?

Thanks in advance for any insights!