How to pass image data from a tool result?

Gemini API
,
1

Hi, I am trying to use Gemini 2.5 Flash to use a web browser - that returns a base 64 image. I want to pass that back to Gemini after the tool call, but:

  • if I just pass it in functionResponse the LLM call can’t use that as an image, which is expected
  • if I pass it as a fileData after the tool call, I get strange results like it doesnt understand the context anymore of why I passed this image.

What is the good way to pass an image result from a tool call so it’s understood by the LLM as the result of the tool? Thanks!

3

@Marco_Schwartz,

welcome to the community,
did you try using the files api ? i.e upload that image using the files api in the function and give the file object as part of the response,
after the function call, you can add the file object as part of “contents” while calling the llm along with the function response

please check the following on how to leverage files api

5

Hi, thanks for your answer! I do use the Files API for the project, but it doesnt solve the issue here. The problem I have is how to do I pass the file reference to the functionReponse:

// Create a function response part
const function_response_part = {
  name: tool_call.name,
  response: { result }
}

// Append function call and result of the function execution to contents
contents.push({ role: 'model', parts: [{ functionCall: tool_call }] });
contents.push({ role: 'user', parts: [{ functionResponse: function_response_part }] });

If I pass it in result - the model doesnt have the information of what is in the image.