Gemini OpenAI compatibility: Does it support the "stream_options" parameter?

Gemini API
, , ,
1

Hello everyone,

I’m currently using the Gemini API through the OpenAI compatibility endpoint, and it’s been great for a smooth transition. I have a specific question regarding streaming responses.

In the OpenAI Chat Completions API, to receive the usage object containing token counts in the final chunk of a stream, we must explicitly set the parameter stream_options: { "include_usage": true }.

My question is: Does the Gemini OpenAI compatibility endpoint support or require this stream_options parameter?

From my testing, it appears that the final stream chunk from the Gemini compatibility endpoint includes the usage object by default, even when I do not pass the stream_options parameter.

Here is a conceptual code snippet illustrating the parameter in question:

import OpenAI from 'openai';

// Assuming the client is configured for the Gemini endpoint
const client = new OpenAI({
  apiKey: 'YOUR_GEMINI_API_KEY',
  baseURL: 'https://generativelanguage.googleapis.com/v1beta', // or similar
});

async function main() {
  const stream = await client.chat.completions.create({
    model: 'models/gemini-1.5-pro-latest',
    messages: [{ role: 'user', content: 'Write a short story.' }],
    stream: true,
    // This is the parameter in question
    stream_options: { include_usage: true }, 
  });

  for await (const chunk of stream) {
    // Process stream...
  }
  // The goal is to reliably get the final chunk with the 'usage' object.
}

main();

To be perfectly clear, my questions are:

  1. Does the Gemini OpenAI compatibility endpoint officially recognize the stream_options: { "include_usage": true } parameter?
  2. Is including the usage object in the final stream chunk the default (and intended) behavior, making this parameter unnecessary when targeting Gemini?
  3. If it is the default behavior, is it safe to rely on this, or would it be best practice to pass the parameter anyway for forward compatibility?

I couldn’t find specific documentation on the compatibility of this particular parameter. Any clarification on the official stance or a link to relevant docs would be much appreciated.

Thank you

3

Hi @hayaoki827 , Welcome to the forum.

Yes, Gemini OpenAI compatibility now supports the stream_options parameter as well as the include_usage feature. You can find the documentation at this link.

If you dont include stream_options: { include_usage: true } in your request, the response will not contain usage information (it will return usage=None). Therefore, it is recommended to always include this parameter to reliably receive usage details and maintain compatibility with future updates. Note that this option is not enabled by default.

Also, a quick reminder: please use the 2.5 model, as this functionality works specifically with the 2.5 model.