Gemini 2.0 experimental function calling doesn't return the function most of the time

Gemini API
, ,
1

Maybe I’m making a mistake with the implementation, but I added a very simple open URL function. It always executes correctly the first time it’s called and the second time. However, if asked to do the same task again, it will not do so. It doesn’t matter how many times or in how many different ways I ask it; it will not run the same function multiple times. It will say verbally that it has completed the task but no function is actually called.

I modified the multimodal Gemini real-time demo to add this simple change.

import { type FunctionDeclaration, SchemaType } from "@google/generative-ai";
import { useEffect, useRef, useState, memo } from "react";
import vegaEmbed from "vega-embed";
import { useLiveAPIContext } from "../../contexts/LiveAPIContext";
import { ToolCall } from "../../multimodal-live-types";

const declaration: FunctionDeclaration = {
  name: "render_altair",
  description: "Displays an altair graph in json format.",
  parameters: {
    type: SchemaType.OBJECT,
    properties: {
      json_graph: {
        type: SchemaType.STRING,
        description:
          "JSON STRING representation of the graph to render. Must be a string, not a json object",
      },
    },
    required: ["json_graph"],
  },
};

function handleRunFunction(name: string, data: { text: string }) {
  console.log(`handleRunFunction: ${name}`, data);
  // For example, open URL in a new tab:
  window.open(data.text, "_blank");
}

const openUrlDeclaration: FunctionDeclaration = {
  name: "open_url",
  description: "Open a specified URL in a new browser tab.",
  parameters: {
    type: SchemaType.OBJECT,
    properties: {
      url: {
        type: SchemaType.STRING,
        description: "A valid URL to open.",
      },
    },
    required: ["url"],
  },
};

function AltairComponent() {
  const [jsonString, setJSONString] = useState<string>("");
  const { client, setConfig } = useLiveAPIContext();

  useEffect(() => {
    setConfig({
      model: "models/gemini-2.0-flash-exp",
      generationConfig: {
        responseModalities: "audio",
        speechConfig: {
          voiceConfig: { prebuiltVoiceConfig: { voiceName: "Aoede" } },
        },
      },
      systemInstruction: {
        parts: [
          {
            text: 'You are my helpful assistant. Any time I ask you for a graph call the "render_altair" function I have provided you. Dont ask for additional information just make your best judgement.',
          },
        ],
      },
      tools: [
        // there is a free-tier quota for search
        { googleSearch: {} },
        { functionDeclarations: [declaration, openUrlDeclaration] },
      ],
    });
  }, [setConfig]);

  useEffect(() => {
    const onToolCall = (toolCall: ToolCall) => {
      console.log(`got toolcall`, toolCall);
      const fc = toolCall.functionCalls.find(
        (fc) => fc.name === declaration.name,
      );
      if (fc) {
        const str = (fc.args as any).json_graph;
        setJSONString(str);
      }
      const fcUrl = toolCall.functionCalls.find(
        (fc) => fc.name === openUrlDeclaration.name,
      );
      if (fcUrl) {
        const providedUrl = (fcUrl.args as any).url || "";
        const url = providedUrl.startsWith("https://")
          ? providedUrl
          : "https://" + providedUrl;
        handleRunFunction("open url", { text: url });
      }
      // send data for the response of your tool call
      // in this case Im just saying it was successful
      if (toolCall.functionCalls.length) {
        setTimeout(
          () =>
            client.sendToolResponse({
              functionResponses: toolCall.functionCalls.map((fc) => ({
                response: { output: { sucess: true } },
                id: fc.id,
              })),
            }),
          200,
        );
      }
    };
    client.on("toolcall", onToolCall);
    return () => {
      client.off("toolcall", onToolCall);
    };
  }, [client]);

  const embedRef = useRef<HTMLDivElement>(null);

  useEffect(() => {
    if (embedRef.current && jsonString) {
      vegaEmbed(embedRef.current, JSON.parse(jsonString));
    }
  }, [embedRef, jsonString]);
  return <div className="vega-embed" ref={embedRef} />;
}

export const Altair = memo(AltairComponent);

Maybe my implementation is incorrect, which causes the Gemini real-time API to not receive the correct signal. I’m not sure. It always responds with voice, saying that it has executed the function, but if you look at the output, it has not.

It will always run the function correctly the first time, sometimes the second time, but after that, it refuses to execute the function no matter what I say, as if it’s unable to do so after a certain point.

This is a critical feature that must be very consistent. If I ask it to open a URL, it should be able to do so 100% of the time. However, it notoriously fails to repeat the same task over and over again.