Possible to return BOTH structured (application/json) and unstructured (markdown) output?

Gemini API
,
1

Is it possible for Gemini request to return BOTH structured (application/json) and unstructured (markdown) output in a single API call? The use case:

(1) Structured output for backend business logic process and DB persistence.

(2) Unstructured (markdown) to show it on the UI, maybe for human-in-the-loop validation of the data returned by the model.

1 Like
3

This can’t be done in a single call, as you set the response type (json) and schema in the generation request. However, we get around it by having this in our schema:

    message: {
        type: Type.STRING,
        description: "The response text to display to the user"
    },

And comments like this in our system instructions:

  • You can use markdown in the ‘message’ field to highlight important words.
  • You can use emojis in the ‘message’ field to make it more engaging.
  • For any ‘friendly chatter’, ‘tips’ or ‘suggestions’ that the user should read from you, that are not part of the game design document, put them into the ‘message’ field of the responseSchema.

And this works fine for us. We then parse the ‘message’ field and display the content in the chat stream and use the rest of the object for the actual data.

1 Like
5

This is a good solution if you could have a comment/details/summary of the data returned. However, it requires baking a free-text field/property/attribute into the schema which is not used in the backend business logic but only to show to the user, maybe as part of human-in-the-loop workflow. For example, I have the following string / object returned:

{"message":"{\n  \"date_str\": \"28-07-2017\",\n  \"vendor\": \"Walmart\",\n  \"currency\": \"USD\",\n  \"items\": [\n    {\n      \"name\": \"PET TOY\",\n      \"amount\": 1.97\n    },\n    {\n      \"name\": \"FLOPPY PUPPY\",\n      \"amount\": 1.97\n    },\n    {\n      \"name\": \"SSSUPREME S\",\n      \"amount\": 4.97\n    },\n    {\n      \"name\": \"2.5 SQUEAK\",\n      \"amount\": 5.92\n    },\n    {\n      \"name\": \"MUNCHY DMBEL\",\n      \"amount\": 3.77\n    },\n    {\n      \"name\": \"DOG TREAT\",\n      \"amount\": 2.92\n    },\n    {\n      \"name\": \"PED PCH 1\",\n      \"amount\": 0.50\n    },\n    {\n      \"name\": \"PED PCH 1\",\n      \"amount\": 0.50\n    },\n    {\n      \"name\": \"COUPON 23100\",\n      \"amount\": 1.00\n    },\n    {\n      \"name\": \"HNYMD SMORES\",\n      \"amount\": 3.98\n    },\n    {\n      \"name\": \"FRENCH DRSNG\",\n      \"amount\": 1.98\n    },\n    {\n      \"name\": \"3 ORANGES\",\n      \"amount\": 5.47\n    },\n    {\n      \"name\": \"BABY CARROTS\",\n      \"amount\": 1.48\n    },\n    {\n      \"name\": \"COLLARDS\",\n      \"amount\": 1.24\n    },\n    {\n      \"name\": \"CALZONE\",\n      \"amount\": 2.50\n    },\n    {\n      \"name\": \"MM RVW MNT\",\n      \"amount\": 19.77\n    },\n    {\n      \"name\": \"STKOBRLPLABL\",\n      \"amount\": 1.97\n    },\n    {\n      \"name\": \"STKOBRLPLABL\",\n      \"amount\": 1.97\n    },\n    {\n      \"name\": \"STKO SUNFLWR\",\n      \"amount\": 0.97\n    },\n    {\n      \"name\": \"STKO SUNFLWR\",\n      \"amount\": 0.97\n    },\n    {\n      \"name\": \"STKO SUNFLWR\",\n      \"amount\": 0.97\n    },\n    {\n      \"name\": \"STKO SUNFLWR\",\n      \"amount\": 0.97\n    },\n    {\n      \"name\": \"BLING BEADS\",\n      \"amount\": 0.97\n    },\n    {\n      \"name\": \"GREAT VALUE\",\n      \"amount\": 9.97\n    },\n    {\n      \"name\": \"LIPTON\",\n      \"amount\": 4.48\n    },\n    {\n      \"name\": \"DRY DOG\",\n      \"amount\": 12.44\n    }\n  ],\n  \"tax\": 4.59,\n  \"total\": 98.21\n}"}

and I try to use showdown javascript library to convert it to markdown and makeHtml but to no avail. What I get on the UI is only an unformatted json string and it is difficult to verify if the model has missed anything or introduced any hallunication in it’s reasoning.