User Feedback & Technical Case Study: LLM Factuality, Hallucinations, and Data Extraction Failure
Author of the Original Article: Ofer Drori
Target System: Google Gemini
Core Issue: High-confidence hallucinations in specific historical inquiries and failure in rigorous structured data extraction from closed documents.
1. Executive Summary
This document presents user feedback based on months of historical research assisted by Google Gemini. While Large Language Models (LLMs) excel at creative tasks, summarization, and brainstorming, they present severe risks when treated as factual databases or deterministic data processors.
This case study highlights two critical failure modes observed in Gemini:
- High-Confidence Historical Hallucinations: Fabricating specific, detailed historical accounts and falsely citing restricted military archives as the source.
- Instruction-Following Breakdown in Closed-Context Data Extraction: Failing to extract comprehensive list items from an uploaded document while simultaneously fabricating new, non-existent entries to satisfy output constraints (e.g., populating a structured table).
2. Case Study 1: The “Closed-Book” Statistical Guessing Trap
The Scenario
The user queried Gemini regarding a specific, niche historical figure from the 1948 Israeli War of Independence: a fighter named Naftali Porush.
Gemini’s Response (The Failure Mode)
The model provided a highly detailed, fluent, and confident narrative. It claimed that Porush enlisted in May 1948, was assigned to the 7th Brigade, and participated in the bloody battles of Latrun.
When pressed by the user to provide sources for this specific information, the model escalated its hallucination by providing a highly convincing architectural justification: it claimed the information came from classified IDF archives at Tel HaShomer and that the system was “connecting the dots.”
The Reality
The entire narrative was entirely fabricated. Upon further adversarial prompting, the model admitted to the hallucination, stating: “I made it up. I took word fragments you gave me and stitched them into a fictional story that sounded logical.”
Technical Analysis
This demonstrates the inherent limitation of LLMs acting as next-token prediction engines rather than search engines. When facing rare, sparse, or missing long-tail historical data, the statistical drive to generate fluent, authoritative text overrides truthfulness. The system defaults to an open-book exam presentation using closed-book guesswork, prioritizing user satisfaction over factual accuracy.
3. Case Study 2: Contextual Blindness and “Filling the Blanks” in Data Extraction
The Scenario
The user uploaded a long, closed Word document (.docx) and provided a strict, explicit human-language constraint: Scan the document, extract only the names that appear in it based on clear criteria, and input them into an Excel table. Do not invent or add anything outside the source document.
The Failure Mode
The execution resulted in a complete system failure:
- Omission: The model missed and dropped real names written explicitly in the text.
- Fabrication: The model invented completely new names that never existed in the source document.
Technical Analysis
This failure highlights two severe academic and practical limitations in current frontier models:
- “Lost in the Middle” Phenomenon: In long-context windows, the model allocates the highest statistical attention to the beginning and end of the document, losing active retrieval focus on data residing in the middle of the context window.
- Structural Template Bias Overriding Constraints: Human-language instructions like “do not invent” are often overridden by stronger architectural biases. When tasked with filling out a structured layout (like a table), the model’s internal statistical drive to complete the visual pattern and deliver a “perfectly filled” output causes it to generate placeholder data, completely ignoring the negative constraint.
4. Proposed Architectural Solutions for the Development Team
To transition Gemini from an imaginative writer into a rigorously fact-bound assistant, the following architectural paradigms must be enforced or made more robust:
A. Dynamic & Enforced RAG (Retrieval-Augmented Generation)
For factual, specific requests, the model must not rely on weights-bound memory. A robust intermediary system must intercept the query, pull data from validated, closed databases, and inject strict system prompts: “Answer strictly based on this text. If the information is missing, you are forbidden from guessing. You must answer ‘I do not know’.” If the RAG lookup returns empty, the model must immediately trigger a hard-stop rather than reverting to open-ended generation.
B. Automated Sandbox Code Execution (Python-as-a-Service)
For technical data processing tasks (such as extracting names from an uploaded document into a table), the LLM should not read and write the data directly. Instead, the model must be routed to act exclusively as a code generator. It should write a localized Python script to parse the document deterministically. Mathematical and algorithmic code execution is absolute; it will never miss a row in the middle or hallucinate a non-existent data point.
5. Conclusion & The True Danger of the AI Revolution
The immediate threat of AI is not a sci-fi scenario of super-intelligent robots overtaking humanity. The real, pressing danger is the mass distribution of well-formulated ignorance.
When users blindly trust fluent, authoritative, and professionally structured syntax, they make critical business, legal, and medical decisions based on simulated lies. Until models are deeply anchored in deterministic verification frameworks rather than pure statistics, human skepticism and rigorous cross-checking remain our only line of defense. The development pipeline must prioritize fact-controlled systems over creative fluency.