Image understanding does not seem to work using the Openai compatible API

Gemini API
,
1

We are unable to get image understanding to work using the openai api for gemini models.
We are following the docs here OpenAI compatibility  |  Gemini API  |  Google AI for Developers

Here is an example prompt:

{
  "model": "gemini-2.5-flash-preview-05-20",
  "temperature": 0.7,
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "what is in the image"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/png;base64,{{base_64_data}}"
          }
        }
      ]
    }
  ]
}

The error we get is this

{code: 400, message: 'Unsupported file uri: data:image/png;base64,…ABLPLmMAAAACXBIWXMAAA7EAAAOxAGVKw4bAAAgAE...', status: 'INVALID_ARGUMENT'}

Is this a known issue or are we doing something wrong?

We have tested that it is valid base64 data and we can convert it back into an image.

3

Hi @Abrie , Welcome to the forum.

Could you please try sending the payload like the one below and check again?

payload = {
    "model": "gemini-2.5-flash-preview-05-20",
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is in this image?"},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}
                }
            ]
        }
    ]
}
5

Hi @GUNAND_MAYANGLAMBAM

The issue was on our side, the base64 was html encoded in a separate step in our pipeline. Sorry for wasting your time.

1 Like