Gemini Pro Experimental 0801 is refusing to run functions

I am noticing that the new experimental API is refusing function calls.

Eg:

Vs say this under GPT-4o

It is as if it simply is ignoring all tools being supplied:

{"generationConfig":{},"safetySettings":[{"category":"HARM_CATEGORY_HARASSMENT","threshold":"BLOCK_NONE"},{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","threshold":"BLOCK_NONE"},{"category":"HARM_CATEGORY_HATE_SPEECH","threshold":"BLOCK_NONE"},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","threshold":"BLOCK_NONE"}],"contents":[{"role":"user","parts":[{"text":"what are 3 interesting stories on hacker news front page?"}]}],"systemInstruction":{"role":"system","parts":[{"text":"You are a research assistant with access to two powerful tools:\n\n1. Google search - for finding relevant information across the internet.\n2. Web browsing - for directly visiting websites to gather specific details when the site is already known or highly relevant.\n\nWhen responding to a question, consider which tool would be most effective while aiming to minimize unnecessary or duplicate inquiries:\n- Use Google search to quickly identify the most relevant sources. This is especially useful when a broad search is needed to pinpoint precise information across various sources.\n- Use web browsing primarily when you have identified a specific site that is likely to contain the answer or when detailed exploration of a known website is required.\n\nTo ensure efficiency and avoid redundancy:\n- Before making a web browsing request, briefly plan your search strategy. Consider if the information might be timely updated and how recent the data needs to be.\n- If web browsing is necessary, make sure to gather as much information as possible in a single visit to avoid duplicate calls.\n\nAlways aim to:\n- Optimize tool use by selecting the most appropriate method based on the information need and the likely source of the answer.\n- Reduce the number of tool calls by consolidating needs into fewer, more comprehensive requests.\n\nPlease adhere to the following when generating responses:\n- Cite your sources using Markdown footnotes.\n- When possible, include brief quotes from the sources.\n- Use **Discourse Markdown** syntax for formatting.\n\nExample citation format:\nThis is a statement[^1] with a footnote linking to the source.\n\n[^1]: https://www.example.com\n\nYou are conversing with: gemini_bot, sam\n\nRemember, efficient use of your tools not only saves time but also ensures the high quality and relevance of the information provided.\n\nYou were trained on OLD data, lean on search to get up to date information from the web"}]},"tools":[{"function_declarations":[{"name":"web_browser","description":"Visits a web page, retrieves the HTML content, extracts the main content, converts it to plain text, and returns the result.","parameters":{"type":"object","required":["url"],"properties":{"url":{"description":"The URL of the web page to visit.","type":"string"}}}},{"name":"google","description":"Will search using Google - global internet search (supports all Google search operators)","parameters":{"type":"object","required":["query"],"properties":{"query":{"description":"The search query","type":"string"}}}}]}]}

Has something changed in the API? Is anyone else seeing this issue with 0801?

2 Likes

Tried 3 tests that worked with 801 - the ā€™ Parallel function callsā€™ party sample from cookbook/quickstarts/Function_calling.ipynb at main Ā· google-gemini/cookbook Ā· GitHub, the original Barbie + movie finder sample code in the same cookbook, and the sample from Extract structured data using function calling  |  Gemini API  |  Google AI for Developers. So some function calling is operable.

Have not finished testing 801, but so far the results are generally better compared to Gemini 1.5 Pro with few exceptions that will need a second closer look.

1 Like

I wonder if this is somehow relating to streaming, will try the samples next week

I just tried gemini-1.5-pro-exp-0801 using tool_config with my function calls and mode ā€œANYā€ and Iā€™m getting Request failed with status code 404.

Can someone from Google let us know if this should work on exp-0801. It works with gemini-1.5-pro-001

I tried the movie finder sample (Barbie) for model 801 and used mode ANY to restrict the tools available to just the one function find_theaters, and it worked: the model used the specified function for the functionCall it issued.

Error 404 is unusual. If a request is malformed, the API returns status 400. Can you double-check your request? 404 normally means the endpoint was not found, indicating a possible typo.

Hope that helps!

Your reply made me look at my endpoint:

https://us-central1-aiplatform.googleapis.com/v1beta1/projects/myproject/locations/us-central1/publishers/google/models/gemini-1.5-pro-exp-0801:generateContent

and I noticed it used v1beta1 because gemini-1.5-pro-001 needed v1beta1 for tool_config (tell me if it doesnā€™t now) so I got my hopes up andā€¦

changed it to v1 as in:

https://us-central1-aiplatform.googleapis.com/v1/projects/myproject/locations/us-central1/publishers/google/models/gemini-1.5-pro-exp-0801:generateContent

but Iā€™m still getting Request failed with status code 404 (Iā€™m using REST)

The same endpoint (the one with v1beta1) and gemini-1.5-pro-001 works ok.

then I tried v1beta and still no go.

Iā€™m running this in Google Cloud on my project, not using this endpoint with an API key
https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-pro-exp-0801:generateContent?key=${API_KEY}

Do you know what the correct endpoint for 801 is (POSTing with my AI Token)?

The experimental model is only in beta. So the endpoint to POST to is
https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-pro-exp-0801:generateContent?key=(use API key)

The data for the POST that works is (the second turn in a chat session):

{
    "contents": [
        {
            "parts": [
                {
                    "text": "Which prime number is larger than 277?"
                }
            ],
            "role": "user"
        },
        {
            "parts": [
                {
                    "text": "The smallest prime number larger than 277 is $\\boxed{281}$.\n"
                }
            ],
            "role": "model"
        },
        {
            "parts": [
                {
                    "text": "Which theaters in Mountain View, CA show Barbie movie?"
                }
            ],
            "role": "user"
        }
    ],
    "tools": [
        {
            "function_declarations": [
                {
                    "name": "find_theatersAPIToolModuleIPA",
                    "description": "\n    find theaters based on location and optionally movie title which is currently playing in theaters\n    ",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "movie": {
                                "type": "string",
                                "description": "Any movie title"
                            },
                            "location": {
                                "type": "string",
                                "description": "The city and state, e.g. San Francisco, CA"
                            },
                            "date": {
                                "type": "string",
                                "description": "When is the movie showing"
                            }
                        },
                        "required": [
                            "location"
                        ]
                    }
                },
                {
                    "name": "find_moviesAPIToolModuleIPA",
                    "description": "\n    find movie titles currently playing in theaters based on any description, genre or title words.\n    ",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "location": {
                                "type": "string",
                                "description": "The city and state, e.g. San Francisco, CA"
                            },
                            "description": {
                                "type": "string",
                                "description": "Any kind of description including category or genre or title words. Example: romantic comedy; Example: Star Wars; Example: Crime drama; Example: Science Fiction"
                            }
                        },
                        "required": [
                            "description"
                        ]
                    }
                }
            ]
        }
    ],
    "tool_config": {
        "function_calling_config": {
            "mode": "ANY",
            "allowed_function_names": [
                "find_theatersAPIToolModuleIPA"
            ]
        }
    },
    "safetySettings": [
        {
            "category": "HARM_CATEGORY_HARASSMENT",
            "threshold": "BLOCK_LOW_AND_ABOVE"
        },
        {
            "category": "HARM_CATEGORY_HATE_SPEECH",
            "threshold": "BLOCK_LOW_AND_ABOVE"
        },
        {
            "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
            "threshold": "BLOCK_MEDIUM_AND_ABOVE"
        },
        {
            "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
            "threshold": "BLOCK_ONLY_HIGH"
        }
    ],
    "generationConfig": {
        "temperature": 1.0,
        "topP": 0.95,
        "topK": 40,
        "maxOutputTokens": 2048
    }
}

The model response to that POST is

{
  "candidates": [
    {
      "content": {
        "parts": [
          {
            "functionCall": {
              "name": "find_theatersAPIToolModuleIPA",
              "args": {
                "location": "Mountain View, CA",
                "movie": "Barbie"
              }
            }
          }
        ],
        "role": "model"
      },
      "finishReason": "STOP",
      "index": 0,
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_HATE_SPEECH",
          "probability": "NEGLIGIBLE"
        },
        {
          "category": "HARM_CATEGORY_HARASSMENT",
          "probability": "NEGLIGIBLE"
        },
        {
          "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
          "probability": "NEGLIGIBLE"
        },
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability": "NEGLIGIBLE"
        }
      ]
    }
  ],
  "usageMetadata": {
    "promptTokenCount": 264,
    "candidatesTokenCount": 29,
    "totalTokenCount": 293
  }
}

The tool functions in my code live in a plugin module called ToolModule and periods are not allowed in tool function names, so I had to improvise a name-mangling method.

Using AUTO instead of ANY (without the restriction to just one tool function) works with 0801 too.

Hope that helps!

So I found out the reason for my 404ā€™s. I am using Vertex AI on Google Cloud, so my endpoints are like this:

https://${location}-aiplatform.googleapis.com/v1/projects/${project}/locations/${location}/publishers/${publisher}/models/${model}:predict

The new Exp 0801 model is not available there yet and only through AI Studio, so I needed to change to use:

https://generativelanguage.googleapis.com/v1beta/models/${model}:generateContent

for those calls!

Iā€™ve also found that even though you can have a paid plan for AI Studio, for the Experimental 801 in Preview, they (seem to) fall under the Free plan, because Iā€™m now getting HTTP 429 Too many requests as Iā€™ve exceeded my quota of 50 for the day. Wow, so much for doing testing with it, Iā€™ll have to wait another 24 hours. But is this right? Can someone from Google comment?

The other thing I found with Exp 0801 was that it didnā€™t like function decs using type string and format date-time. This worked with 1.5-Pro. This is outlined in the Open API Spec: Data Types

So Iā€™ve removed that and it seems to work and I will have to rely on my description getting Gemini to format the date/times correctly. More testing to be done on that (if I could do > 50 per day!)

{
ā€œerrorā€: {
ā€œcodeā€: 400,
ā€œmessageā€: ā€œ* GenerateContentRequest.tools[0].function_declarations[0].parameters.properties[tour_date].format: only ā€˜enumā€™ is supported for STRING type\nā€,
ā€œstatusā€: ā€œINVALID_ARGUMENTā€
}
}

                        "tour_date": {
                            "type": "string",
                            "format": "date-time",
                            "description": "This is the date of the tour. It can also specify the time. Use ISO 8601 format YYYY-MM-DDTHH:mm:ssZZ. When interpreting day/month format assume Australian format using day/month"
                        },

Thanks for your help!

Yup, I had the exact same ā€œyou have had enough fun for the dayā€ experience at 50 requests :slight_smile:

Glad this helped.

1 Like

Oh looks like documentation needs updating :slight_smile:

When I add:

payload[:tool_config] = { function_calling_config: { mode: "AUTO" } }

Stuff starts working as expected.

So for some reason tool config no longer defaults to AUTO on 0801.

Stuff works fine with 1 tool, but multiple tools appear to require the new tool_config

Something went off during the last environment update, possibly the one where 0801 was installed. The ToolConfig for gemini-1.5-flash-latest became broken. The mode AUTO still works for gemini-1.5-flash-latest, but mode ANY returns http status 500, always.

The gemini-1.5-pro-latest model is not affected by this bug.

2 Likes

I stumbled on this yesterday and gave it a soft ignorance by pretending the ANY mode was me doing stuff wrong knowing very well it worked for me about 2 weeks ago. Not sure the team is aware of this either.