I’ve been noticing a drop in the output quality of the Gemini Flash model recently. For example, in my prompt, I explicitly specify the output format as: “Using this JSON schema: Title = {“A”: str, “B”: str, “C”: str}. Return a list[Title].”
However, the model sometimes returns outputs like {'title': text} or {'D': text}, which don’t align with the requested format.
I previously experienced a similar issue when multiple users sent requests simultaneously, so I initially thought that might be the cause. However, now the problem seems to occur even with single-user requests.
Does anyone know how to address this issue? Would increasing the RPM/TPM quota help?
The API is stateless. That ensures that all requests are atomic: they do not depend on any other requests, concurrent or previous.
In short, simultaneous requests cannot have been the source of the error. You have been observing the same error source all along. Flash is ‘temperamental’ or even ‘erratic’ (I know, that’s an anthropomorphic analogy). Tweaking your prompt will likely reduce the probability that the model fails to follow instructions. I do not have a recipe that ensures full compliance (zero probability of the model not following instructions). Pro is significantly better at instruction following, and Pro costs more.