Issue: Multimodal Function Calling Error - display_name not matching despite correct reference in $ref

Gemini API
1

image
image2210×706 102 KB
I I am implementing a multimodal function calling flow where the tool returns a JSON response referencing an image. I’ve strictly followed the official documentation using the "$ref" syntax, but I keep encountering a 400 INVALID_ARGUMENT error.

ApiError: {“error”:{“message”:“{\n “error”: {\n “code”: 400,\n “message”: “The referenced name image_0.png in function_response.response does not match to a display_name in the function_response.parts.”,\n “status”: “INVALID_ARGUMENT”\n }\n}\n”,“code”:400,“status”:“Bad Request”}}

3

Hello :waving_hand:

I think the problem the issue is that $ref: ‘image_0.png’ to reference an image from a previous function call, but the Gemini API can’t find a matching displayName in the current function_response.parts.
The root cause:
When implementing multimodal function chaining (where one function returns an image and another function needs to reference it), the $ref value must exactly match a displayName that exists in the function_response.parts of that same response object.

Try these options first and let us know if it works.

  1. Ensure the image returned from the first function has the exact displayName: ‘image_0.png’.

  2. When referencing it in the second function call, the $ref must match that display name precisely.

  3. The reference needs to be in a context where that image is still available in the parts array
    This is a validation error from Google’s API—it’s stricter about multimodal references than they expected.

You may need to verify the function response structure includes the image part with the matching displayName before attempting the $ref.

Good luck :crossed_fingers: hope it helps.

~Cmd.Proton🖖

5

Hi Cmd.Proton,

Thanks for the reply, but I think there is a misunderstanding regarding my scenario. I am not using “function chaining” or trying to reference an image from a previous call.

I am implementing a single-turn multimodal function response, exactly as shown in the official documentation. In this single response, I provide the JSON data (with the $ref) and the image data (in the parts array) simultaneously. They are packaged together in one go.

The issue is that even though the reference and the data are provided together in the same functionResponse object, the API still returns a 400 error claiming it cannot find the matching display_name.

Since I am strictly following the “all-in-one” response pattern from the docs, why would the API fail to perform this internal lookup? Could this be a validation bug on the API side when handling the $ref within a single functionResponse?

Thanks!

6

Ah ok I’m sorry about the misunderstanding that is a api bug indeed I suggest filing a bug report as this is not a user error issue.

  1. File a bug report with Google using this exact scenario and screenshots—you’re following the documentation and getting an API error, which is actionable feedback.
  2. Include: The exact documentation you’re following, the complete request payload, and the error response.
  3. Mention: That this works as a workaround if split it into function chaining (previous call returns image, next call references it), but the “all-in-one” pattern from the docs fails.

Sorry that I can not assist further thank you for informing us of the issue here anyway.

~Cmd.Proton🖖

8

Hi @ddb, welcome to the community!

To analyze the issue and help debug, please share the part of your code where you construct the parts array for the functionResponse.

Thank you!