OpenAI compatibility bug in response object

Gemini API
,
1

The chat completion response object’s choices[0].message.content field should be string or null according to the docs.

In some calls it consistently returns an array of two text objects where the real OpenAI returns null.

        "content": [
          {
            "text": "\n",
            "type": "text"
          },
          {
            "text": "\n",
            "type": "text"
          }
        ],

Here is a reproducer:

PROJECT=<your project>
LOCATION=us-east4
GEMINI_API_KEY=$(gcloud auth print-access-token)
MODEL_ID=gemini-1.5-flash
ENDPOINT="https://${LOCATION}-aiplatform.googleapis.com/v1beta1/projects/${PROJECT_ID}/locations/${LOCATION}/endpoints/openapi/chat/completions"

echo '{
  "messages" : [ {
    "role" : "system",
    "content" : "You are really bad at math, but you are really good at using tool calls to do math."
  }, {
    "role" : "user",
    "content" : "What is 2 + 2?"
  }, {
    "role" : "user",
    "content" : "What is 3 + 5?"
  }, {
    "role" : "user",
    "content" : "What is 1 + 9?"
  } ],
  "model" : "google/gemini-1.5-flash",
  "max_tokens" : 4000,
  "seed" : 0,
  "stream" : false,
  "temperature" : 0.0,
  "tools" : [ {
    "type" : "function",
    "function" : {
      "name" : "add",
      "description" : "Add two numbers",
      "parameters" : {
        "type" : "object",
        "properties" : {
          "left" : {
            "type" : "integer",
            "description" : "The number to the left of the addition operator."
          },
          "right" : {
            "type" : "integer",
            "description" : "The number to the right of the addition operator."
          }
        },
        "additionalProperties" : false
      },
      "strict" : false
    }
  } ],
  "tool_choice" : "auto"
}' | curl -sS -X POST \
  -H "Authorization: Bearer $GEMINI_API_KEY" \
  -H "Content-Type: application/json" \
  -d @- $ENDPOINT

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "logprobs": null,
      "message": {
        "content": [
          {
            "text": "\n",
            "type": "text"
          },
          {
            "text": "\n",
            "type": "text"
          }
        ],
        "role": "assistant",
        "tool_calls": [
          {
            "function": {
              "arguments": "{\"left\":2,\"right\":2}",
              "name": "add"
            },
            "id": "add",
            "type": "function"
          },
          {
            "function": {
              "arguments": "{\"left\":3,\"right\":5}",
              "name": "add"
            },
            "id": "add",
            "type": "function"
          },
          {
            "function": {
              "arguments": "{\"left\":1,\"right\":9}",
              "name": "add"
            },
            "id": "add",
            "type": "function"
          }
        ]
      }
    }
  ],
  "created": 1735763828,
  "id": "2025-01-01|12:37:08.384753-08|7.69.23.47|167929181",
  "model": "google/gemini-1.5-flash",
  "object": "chat.completion",
  "system_fingerprint": "",
  "usage": {
    "completion_tokens": 11,
    "prompt_tokens": 70,
    "total_tokens": 81
  }
}
1 Like
3

This bug :beetle: is breaking :broken_heart: the OpenAI compatibility :disappointed_relieved:

Google Staff, could you take a look at this issue, please?

1 Like