Gemini Tool Calling - Completely broken

Gemini API
,
1

Not sure if something has changed in the last month but my tool calling has completely broken in my app. Even thought multiple functions are called and responses are given back to gemini e.g.
```
{

"role": "model",

“parts”: [

  {

“functionCall”: {

“name”: “list_all_payment_methods”,

“args”: {

“chat_id”: “f4519ac8-ca78-4d9c-8987-89e56a057f75”,

“eob_document_id”: 3006

      }

    }

  }

\]

},

{

“role”: “user”,

“parts”: [

  {

“functionResponse”: {

“name”: “list_all_payment_methods”,

“response”: {

“output”: “[{\“id\”:5,\“name\”:\“Cash\”},{\“id\”:7,\“name\”:\“Direct Deposit\”},{\“id\”:8,\“name\”:\“Personal Check\”},{\“id\”:9,\“name\”:\“Insurance Check\”},{\“id\”:10,\“name\”:\“Account Credit\”},{\“id\”:78,\“name\”:\“Card\”}]”

      }

    }

  }

\]

},
```
it still says: I have previously called list_all_dentists, list_all_insurance_providers, and list_all_payment_methods. These calls returned empty objects, which indicates that I need to ask the user for help to get the necessary IDs.
What is going on?? Can someone help

1 Like
3

We’re seeing the same issue. Were you able to resolve this?

1 Like
4

I encountered the same issue and filed a separate bug.

Are you analyzing PDF files as part of your calls to Gemini? To me this only happens when a PDF is part of the prompting.

1 Like
5

Yes, we’re analyzing PDFs as well - I hadn’t put together that that was a factor, but it aligns with what we’ve seen.

1 Like
6

yeah pdf + tool calls is broken. facing the same issue

8

Hi @Saj_Arora,

Thanks for reaching out!

Most likely this is happening due to thought signatures.
For newer models, you need to pass the Thought Signatures during function calling. Gemini Thought Signatures (Refer to initial note)

Also you can try passing JSON Object(Struct) directly instead of string and see if it works.

Please let us know if you face any challenges on these.

10

This bug created lots of product issues for our company. the only workaround I found is using vertex AI auth.

Gemini Token Hallucination Bug

Issue

Gemini 2.5 models do not correctly pass tool response values to subsequent tool calls when PDF attachments are present. Instead of using the actual returned values, the model hallucinates values.

Root Cause

Bug in Gemini API endpoint (NOT Vertex AI) - confirmed by:

  1. Intercepting raw HTTP requests (SDK sends correct data)
  2. Testing across model versions (2.0 works on Gemini API, 2.5+ fail)
  3. Testing Vertex AI - ALL models PASS!

Proof

Raw request sent by SDK:

{
  "functionResponse": {
    "name": "capture_supplier_name",
    "response": {
      "token_a": "abc12345"
    }
  }
}

Model returns: token_a = "capture_supplier_name_response" (hallucinated)

Key Findings

1. PDF Attachment Triggers Bug

  • Without PDF: Token passing works correctly (PASS)
  • With PDF: Token passing fails - model hallucinates values (FAIL)

2. Gemini API vs Vertex AI

Model Gemini API Vertex AI
gemini-2.0-flash :white_check_mark: PASS :white_check_mark: PASS
gemini-2.5-flash :cross_mark: FAIL :white_check_mark: PASS
gemini-2.5-pro :cross_mark: FAIL :white_check_mark: PASS
gemini-3-flash-preview :cross_mark: FAIL :white_check_mark: PASS
gemini-3-pro-preview :cross_mark: FAIL :warning: (function format error)

3. Gemini API Failure Patterns

Model Hallucination Pattern
gemini-2.5-flash Claims “token_a not returned”, sees {"capture_supplier_name_response": {}}
gemini-2.5-pro Hallucinates random UUID, sometimes infinite loop
gemini-3-pro-preview Hallucinates placeholder tokens (token_a_12345, token_b_67890)

Test Scripts

  • scripts/gemini_pdf_memory_test.py - Full agent test with GeminiAgent
  • scripts/gemini_token_test_raw.py - Raw API test with HTTP interception

Test PDF

Located at /Users/mingyangjin/Downloads/test.pdf with content:

Supplier: Acme Corporation
Product: Enterprise Widget Pro
Start Date: 2024-01-15
End Date: 2025-01-14

Hallucination Patterns Observed

Depending on response format, model hallucinates different values:

Response Format Hallucinated Value
{"result": {...}} token_a_value
{"token_a": "..."} capture_supplier_name_response
{"output": {...}} token_a

Impact

Any workflow relying on passing values between sequential tool calls with PDF attachments is unreliable on Gemini 2.5. This affects Hermione document processing workflows.

Workarounds

  1. Use Vertex AI instead of Gemini API - all models work correctly
  2. Use gemini-2.0-flash on Gemini API - only 2.0 works correctly

Vertex AI Setup

from google import genai

client = genai.Client(
    vertexai=True,
    api_key=os.environ.get("VERTEX_API_KEY"),
)