Gemini does not respect the order of the properties in a schema

,

Let’s imagine a situation where you want an answer and a rationale from the LLM. You want it to be in a JSON format like

{
  "choice": 4,
  "rationale": "The answer is 4 because..."
}

If the LLM completes the choice property first, then the next rationale can be made up to justify the choice.
However, if the rational comes first like

{
  "rationale": "The answer is 4 because...",
  "choice": 4
}

Then, the choice will be determined by rationale. It is similar to CoT.

It is important that the LLM follows the given order of the properties in an object. However, I found that Gemini SDK ignores it.

    properties: MutableMapping[str, "Schema"] = proto.MapField(
        proto.STRING,
        proto.MESSAGE,
        number=3,
        message="Schema",
    )

I think this one better to be a repeated, not a map field, to keep the order of the properties.

On the other hand, GPT does it well.
https://platform.openai.com/docs/guides/structured-outputs/key-ordering

2 Likes

+1 this is having a huge effect on my results!

It seems that the ordering is alphabetical (when using typing.TypedDict) so a temporary solution is to name the keys so that their alphabetical ordering is how you want it.

+1
I also reported this in the following issue and I believe this an important matter: