Issue with structured output

Gemini API
1

I have tried this many many times, for some reason in structured output, object property doesn’t get returned, I have tried updating prompt, making it required in the schema nothing seems to fix it, I have more properties as well, all properties other than object type properties are been given properly, but no object properties show. Any fix?

3

Hey @Muhammad_Zafar, Could you share a sample prompt and schema you are using so I can try to replicate the issue?

1 Like
5

Sure, here’s the exact one being used, “coat” is the object that doesn’t get returned, I don’t know if temperature would make a difference just now came to mind, for me it’s 0.3:

export const dogGeminiSchema: ObjectSchema = {
  type: SchemaType.OBJECT,
  properties: {
    breed: { type: SchemaType.STRING },
    confidence: { type: SchemaType.NUMBER },
    notableFeatures: {
      type: SchemaType.ARRAY,
      items: { type: SchemaType.STRING },
    },
    altBreeds: {
      type: SchemaType.ARRAY,
      items: { type: SchemaType.STRING },
    },
    sizeCategory: {
      type: SchemaType.STRING,
      format: "enum",
      enum: ["Small", "Medium", "Large", "Unknown"],
    },
    coat: {
      type: SchemaType.OBJECT,
      properties: {
        length: { type: SchemaType.STRING },
        texture: { type: SchemaType.STRING },
        colorPattern: { type: SchemaType.STRING },
      },
    },
    ageGroup: {
      type: SchemaType.STRING,
      format: "enum",
      enum: ["Puppy", "Adult", "Senior", "Unknown"],
    },
    emotion: { type: SchemaType.STRING },
    spayedOrNeutered: { type: SchemaType.STRING },
    healthNote: { type: SchemaType.STRING },
    funFacts: {
      type: SchemaType.ARRAY,
      items: { type: SchemaType.STRING },
    },
  },
  required: [
    "breed",
    "coat",
    "confidence",
    "notableFeatures",
    "altBreeds",
    "sizeCategory",
    "ageGroup",
    "emotion",
    "spayedOrNeutered",
    "healthNote",
    "funFacts",
  ],
};

export function dogPrompt(options?: any): string {
  return `You are an expert canine behaviorist, dog show judge, and veterinarian.

Identify the dog breed (or best-guess mix) in the image. Return only a single valid JSON object using the following schema, dont miss any fields, you have been missing the coat field please dont miss it:

{
  "breed": string,                       // Common breed name (e.g., "Golden Retriever") or best-guess mix
  "confidence": number,                  // Integer from 0 to 100 indicating confidence in the identification

  "notableFeatures": string[],           // Visual traits that helped identify the breed

  "altBreeds": string[],                 // Up to 2 visually similar breeds
  "sizeCategory": string,                // One of: "Small", "Medium", "Large", or "Unknown"
  "coat": {
    "length": string,                    // e.g., "Short", "Medium", "Long", or "Unknown"
    "texture": string,                   // e.g., "Smooth", "Wiry", "Curly", "Double", or "Unknown"
    "colorPattern": string               // e.g., "Brindle", "Solid", "Bicolor", "Tricolor", "Spotted", etc.
  },

  "ageGroup": string,                    // One of: "Puppy", "Adult", "Senior", or "Unknown"
  "emotion": string,                     // Inferred emotion (e.g., "Happy", "Alert", "Relaxed", "Anxious", or "Unclear")
  "spayedOrNeutered": string,            // "Unknown" (typically), unless visually obvious
  "healthNote": string,                  // Any visually obvious issues (e.g., underweight, skin problem), or "No visible issues"
  "funFacts": string[]                   // Up to 3 short, interesting facts about the breed (or best-guess mix)
}

Rules:
- Only return a JSON object. No text before or after it.
- If the image is unclear, reduce the confidence score and explain in "notableFeatures".
- If any trait cannot be visually determined, explain that or use "Unknown".
- All strings should be clean, human-readable, and factually accurate.`;
}
1 Like
7

Hey, I tested the same schema using the python SDK, and the object property “coat” was returned correctly in the structured output. It seems to work as expected when the prompt clearly requires it.

Here is the colab gist file for reference: Colab Link

Let me know if I missed anything in the repro steps.

Thanks

9

That’s really weird. can it be soemthing related to the js sdk?

10

I tried again, here’s the result I got:

result:  {
  "breed": "Golden Retriever",
  "confidence": 95,
  "notableFeatures": [
    "Classic golden coat color",
    "Well-proportioned head and body",
    "Friendly expression"
  ],
  "altBreeds": [
    "Labrador Retriever",
    "Flat-Coated Retriever"
  ],
  "sizeCategory": "Large",
  "ageGroup": "Adult",
  "emotion": "Happy",
  "spayedOrNeutered": "Unknown",
  "healthNote": "No visible issues",
  "funFacts": [
    "Golden Retrievers are known for their gentle and friendly temperament.",
    "They were originally bred in Scotland to retrieve waterfowl.",
    "Golden Retrievers are often used as therapy and service dogs."
  ]
}