JSON Mode - Internal Server Error 500

Gemini API
1

When using JSON schema mode and the schema object is particularly big an Internal Server Error message is returned. This happens using the API and the AI Studio Playground. I have tested against all models and they all fail apart from gemini-1.5-flash-8b-exp-0827. To test you can use the following schema and the prompt ‘generate values’. The schema fails, however if you remove item_9 then the schema works as intended. This is just a sample schema but it also fails using schema generated in my production app:

{
    "type": "object",
    "properties": {
        "items": {
            "type": "object",
            "properties": {
                "item_1": {
                    "type": "object",
                    "properties": {
                        "item_A": {
                            "type": "number",
                            "description": "A number for item_A"
                        },
                        "item_B": {
                            "type": "number",
                            "description": "A number for item_B"
                        },
                        "item_C": {
                            "type": "number",
                            "description": "A number for item_C"
                        },
                        "item_D": {
                            "type": "number",
                            "description": "A number for item_D"
                        },
                        "item_E": {
                            "type": "number",
                            "description": "A number for item_E"
                        },
                        "item_F": {
                            "type": "number",
                            "description": "A number for item_F"
                        },
                        "item_G": {
                            "type": "number",
                            "description": "A number for item_G"
                        },
                        "item_H": {
                            "type": "number",
                            "description": "A number for item_H"
                        },
                        "item_I": {
                            "type": "number",
                            "description": "A number for item_I"
                        },
                        "item_J": {
                            "type": "number",
                            "description": "A number for item_J"
                        }
                    }
                },
                "item_2": {
                    "type": "object",
                    "properties": {
                        "item_A": {
                            "type": "number",
                            "description": "A number for item_A"
                        },
                        "item_B": {
                            "type": "number",
                            "description": "A number for item_B"
                        },
                        "item_C": {
                            "type": "number",
                            "description": "A number for item_C"
                        },
                        "item_D": {
                            "type": "number",
                            "description": "A number for item_D"
                        },
                        "item_E": {
                            "type": "number",
                            "description": "A number for item_E"
                        },
                        "item_F": {
                            "type": "number",
                            "description": "A number for item_F"
                        },
                        "item_G": {
                            "type": "number",
                            "description": "A number for item_G"
                        },
                        "item_H": {
                            "type": "number",
                            "description": "A number for item_H"
                        },
                        "item_I": {
                            "type": "number",
                            "description": "A number for item_I"
                        },
                        "item_J": {
                            "type": "number",
                            "description": "A number for item_J"
                        }
                    }
                },
                "item_3": {
                    "type": "object",
                    "properties": {
                        "item_A": {
                            "type": "number",
                            "description": "A number for item_A"
                        },
                        "item_B": {
                            "type": "number",
                            "description": "A number for item_B"
                        },
                        "item_C": {
                            "type": "number",
                            "description": "A number for item_C"
                        },
                        "item_D": {
                            "type": "number",
                            "description": "A number for item_D"
                        },
                        "item_E": {
                            "type": "number",
                            "description": "A number for item_E"
                        },
                        "item_F": {
                            "type": "number",
                            "description": "A number for item_F"
                        },
                        "item_G": {
                            "type": "number",
                            "description": "A number for item_G"
                        },
                        "item_H": {
                            "type": "number",
                            "description": "A number for item_H"
                        },
                        "item_I": {
                            "type": "number",
                            "description": "A number for item_I"
                        },
                        "item_J": {
                            "type": "number",
                            "description": "A number for item_J"
                        }
                    }
                },
                "item_4": {
                    "type": "object",
                    "properties": {
                        "item_A": {
                            "type": "number",
                            "description": "A number for item_A"
                        },
                        "item_B": {
                            "type": "number",
                            "description": "A number for item_B"
                        },
                        "item_C": {
                            "type": "number",
                            "description": "A number for item_C"
                        },
                        "item_D": {
                            "type": "number",
                            "description": "A number for item_D"
                        },
                        "item_E": {
                            "type": "number",
                            "description": "A number for item_E"
                        },
                        "item_F": {
                            "type": "number",
                            "description": "A number for item_F"
                        },
                        "item_G": {
                            "type": "number",
                            "description": "A number for item_G"
                        },
                        "item_H": {
                            "type": "number",
                            "description": "A number for item_H"
                        },
                        "item_I": {
                            "type": "number",
                            "description": "A number for item_I"
                        },
                        "item_J": {
                            "type": "number",
                            "description": "A number for item_J"
                        }
                    }
                },
                "item_5": {
                    "type": "object",
                    "properties": {
                        "item_A": {
                            "type": "number",
                            "description": "A number for item_A"
                        },
                        "item_B": {
                            "type": "number",
                            "description": "A number for item_B"
                        },
                        "item_C": {
                            "type": "number",
                            "description": "A number for item_C"
                        },
                        "item_D": {
                            "type": "number",
                            "description": "A number for item_D"
                        },
                        "item_E": {
                            "type": "number",
                            "description": "A number for item_E"
                        },
                        "item_F": {
                            "type": "number",
                            "description": "A number for item_F"
                        },
                        "item_G": {
                            "type": "number",
                            "description": "A number for item_G"
                        },
                        "item_H": {
                            "type": "number",
                            "description": "A number for item_H"
                        },
                        "item_I": {
                            "type": "number",
                            "description": "A number for item_I"
                        },
                        "item_J": {
                            "type": "number",
                            "description": "A number for item_J"
                        }
                    }
                }, "item_6": {
                    "type": "object",
                    "properties": {
                        "item_A": {
                            "type": "number",
                            "description": "A number for item_A"
                        },
                        "item_B": {
                            "type": "number",
                            "description": "A number for item_B"
                        },
                        "item_C": {
                            "type": "number",
                            "description": "A number for item_C"
                        },
                        "item_D": {
                            "type": "number",
                            "description": "A number for item_D"
                        },
                        "item_E": {
                            "type": "number",
                            "description": "A number for item_E"
                        },
                        "item_F": {
                            "type": "number",
                            "description": "A number for item_F"
                        },
                        "item_G": {
                            "type": "number",
                            "description": "A number for item_G"
                        },
                        "item_H": {
                            "type": "number",
                            "description": "A number for item_H"
                        },
                        "item_I": {
                            "type": "number",
                            "description": "A number for item_I"
                        },
                        "item_J": {
                            "type": "number",
                            "description": "A number for item_J"
                        }
                    }
                },
                "item_7": {
                    "type": "object",
                    "properties": {
                        "item_A": {
                            "type": "number",
                            "description": "A number for item_A"
                        },
                        "item_B": {
                            "type": "number",
                            "description": "A number for item_B"
                        },
                        "item_C": {
                            "type": "number",
                            "description": "A number for item_C"
                        },
                        "item_D": {
                            "type": "number",
                            "description": "A number for item_D"
                        },
                        "item_E": {
                            "type": "number",
                            "description": "A number for item_E"
                        },
                        "item_F": {
                            "type": "number",
                            "description": "A number for item_F"
                        },
                        "item_G": {
                            "type": "number",
                            "description": "A number for item_G"
                        },
                        "item_H": {
                            "type": "number",
                            "description": "A number for item_H"
                        },
                        "item_I": {
                            "type": "number",
                            "description": "A number for item_I"
                        },
                        "item_J": {
                            "type": "number",
                            "description": "A number for item_J"
                        }
                    }
                },
                "item_8": {
                    "type": "object",
                    "properties": {
                        "item_A": {
                            "type": "number",
                            "description": "A number for item_A"
                        },
                        "item_B": {
                            "type": "number",
                            "description": "A number for item_B"
                        },
                        "item_C": {
                            "type": "number",
                            "description": "A number for item_C"
                        },
                        "item_D": {
                            "type": "number",
                            "description": "A number for item_D"
                        },
                        "item_E": {
                            "type": "number",
                            "description": "A number for item_E"
                        },
                        "item_F": {
                            "type": "number",
                            "description": "A number for item_F"
                        },
                        "item_G": {
                            "type": "number",
                            "description": "A number for item_G"
                        },
                        "item_H": {
                            "type": "number",
                            "description": "A number for item_H"
                        },
                        "item_I": {
                            "type": "number",
                            "description": "A number for item_I"
                        },
                        "item_J": {
                            "type": "number",
                            "description": "A number for item_J"
                        }
                    }
                },
                "item_9": {
                    "type": "object",
                    "properties": {
                        "item_A": {
                            "type": "number",
                            "description": "A number for item_A"
                        },
                        "item_B": {
                            "type": "number",
                            "description": "A number for item_B"
                        },
                        "item_C": {
                            "type": "number",
                            "description": "A number for item_C"
                        },
                        "item_D": {
                            "type": "number",
                            "description": "A number for item_D"
                        },
                        "item_E": {
                            "type": "number",
                            "description": "A number for item_E"
                        },
                        "item_F": {
                            "type": "number",
                            "description": "A number for item_F"
                        },
                        "item_G": {
                            "type": "number",
                            "description": "A number for item_G"
                        },
                        "item_H": {
                            "type": "number",
                            "description": "A number for item_H"
                        },
                        "item_I": {
                            "type": "number",
                            "description": "A number for item_I"
                        },
                        "item_J": {
                            "type": "number",
                            "description": "A number for item_J"
                        }
                    }
                }
            },
            "required": [
                "item_1",
                "item_2",
                "item_3"
            ]
        },
        "costs_1": {
            "type": "array",
            "description": "Array of costs",
            "items": {
                "type": "object",
                "properties": {
                    "description": {
                        "type": "string"
                    },
                    "cost": {
                        "type": "number",
                        "description": "A cost value"
                    }
                },
                "required": [
                    "description",
                    "cost"
                ]
            }
        },
        "costs_2": {
            "type": "array",
            "description": "Array of costs",
            "items": {
                "type": "object",
                "properties": {
                    "description": {
                        "type": "string"
                    },
                    "cost": {
                        "type": "number",
                        "description": "A cost value"
                    }
                },
                "required": [
                    "description",
                    "cost"
                ]
            }
        }
    },
    "required": [
        "items",
        "costs_1",
        "costs_2"
    ]
}
1 Like
2

I’ve noticed the same thing – happened across Gemini on vertexAI (though the error is a bit different) and AI studio, and it affects the call regardless of if schema is set as response schema object, or using function calling with schema for args. There has to be some hard limit for combined schema size or something as there’s no repeatable rhyme/reason to what specific schema settings will cause this.

3

Interesting! I will escalate this bug.

4

Keep in mind that the output context size is limited to 8K

5

That’s absolutely correct, and if the model generates output over the specified or default limit, finishReason is set to MAX_TOKENS and http status is still HTTP_OK (200).

1 Like
6

For reference I’m nowhere near that - currently ~2k input and ~1500 output and still seeing LOTS of these errors (sometimes 500, sometimes 400 … even for the exact same back-to-back request) about 20% of request seem to succeed. Ultra frustrating

7

Just an update – this was happening to me with Gemini in VertexAI as well but they seem to have fixed it in the last few days – but still happening here (regardless of if schema is passed in as function response schema or general model response schema)

1 Like
9

I am also having this issue and it’s still happening almost 4 months later