Output order in structured outputs

Gemini API
,
1

I have the following classes which define the output schema:

class SlideTags(enum.Enum):
EQUITY_STORY = “Equity Story”
SNAPSHOT = “Snapshot”
BUSINESS_MODEL = “Business Model”
OTHERS = “Others”

class Categorization(typing.TypedDict):
reasoning: str
primary_categories: list[SlideTags]

I have a prompt which says on which criteria the categorization shoul happen. So the input to the LLM is prompt + slide image.

model = genai.GenerativeModel(“gemini-1.5-flash”)

response = await model.generate_content_async(
contents=[
{“role”: “user”, “parts”: [
{“text”: categorization_prompt},
{“inline_data”: {“mime_type”: “image/png”, “data”: image_b64}},
]}
],
generation_config=genai.GenerationConfig(
response_mime_type=“application/json”, response_schema=Categorization
),
stream=True
)

I want that the “reasoning” key should get filled first and then the “primary_categories”. But when i stream the response, i am observing that the opposite is happening.

I want reasoning key to be filled first so that i can get the benefit of chain of thought as compared to the model justifying the categories afterwards. So my question is:

  1. How is the output order decided? Do the keys undergo lexical sorting? Because when i replaced “reasoning” with “explanation”, then “explanation” was getting filled first.
  2. In gemini is the structured outputs always guaranteed (like openai does by using cfg)? If some grammar rules are being used, i dont think the order can be controlled by prompting.
1 Like
2

Hi,
You could consider adding a system instruction to your prompt(s) specifying the order of information expected.

4

@jkirstaetter system instruction won’t work. For whatever reason, gemini API uses alphabetical order for fields in structured generation. In the vertex api, you can specify properyOrdering, as a superset of the OpenAPI schema definition. In the AI studio api, there is no such option.

Here’s my rubber-ducking in the bug tracker when I ran into this issue. Has some links to relevant docs.

https://issuetracker.google.com/issues/383595390

My hope is that the devs will change the default behavior here at some (non-breaking) point , however this ordering still seems to be the case for gemini-2.0-flash-exp in AI studio UI. IMO like a terrible idea to re-order fields alphabetically for structured generation considering the auto-regressive nature of generation.

One very ugly workaround is to prefix your field names with something to help with sorting. For example “a_reasoning” and “b_primary_categories”. Or just rename “reasoning” to “analysis” to hack it. My use case has many fields though, so it gets a little crazy to use these workarounds

1 Like
6

Just tried out Gemini 1206 with a prompt used in OpenAI models in our production. I just noticed the same, structured outputs seems to order properties alphabetically and it totally wrecks the output. I hope this gets fixed, not much point in evaluating switching LLM provider with this huge flaw. Structured outputs are foundational for our products.