I’ve been noticing a drop in the output quality of the Gemini Flash model recently. For example, in my prompt, I explicitly specify the output format as: “Using this JSON schema: Title = {“A”: str, “B”: str, “C”: str}. Return a list[Title].”
However, the model sometimes returns outputs like {'title': text} or {'D': text}, which don’t align with the requested format.
I previously experienced a similar issue when multiple users sent requests simultaneously, so I initially thought that might be the cause. However, now the problem seems to occur even with single-user requests.
Does anyone know how to address this issue? Would increasing the RPM/TPM quota help?
The API is stateless. That ensures that all requests are atomic: they do not depend on any other requests, concurrent or previous.
In short, simultaneous requests cannot have been the source of the error. You have been observing the same error source all along. Flash is ‘temperamental’ or even ‘erratic’ (I know, that’s an anthropomorphic analogy). Tweaking your prompt will likely reduce the probability that the model fails to follow instructions. I do not have a recipe that ensures full compliance (zero probability of the model not following instructions). Pro is significantly better at instruction following, and Pro costs more.
We are currently working with the Gemini 1.5 Flash API in conjunction with the SQL Query Agent. However, we are encountering significant inconsistencies in its behavior.
Unpredictable Performance: While the API occasionally responds correctly, it often takes an excessive amount of time—sometimes exceeding five to 15 minutes to provide a response.
Failure Scenarios: There are instances where the API fails outright, providing no output or returning incomplete results.
Output Variability: Even when the API functions, the responses lack consistency, with noticeable variations in the results across multiple requests made under similar conditions.
These issues are impacting our ability to effectively use the Gemini 1.5 Flash API for its intended purpose. We would appreciate guidance or solutions to address these challenges to ensure smoother integration and operation.