Summary
Calling the Gemini API (via LiteLLM) with a strict JSON responseSchema and a large, mostly-static system prompt repeatedly returned narrative text matching a different, earlier API call rather than reflecting the actual input of the current call — including specific numeric figures that were demonstrably not present anywhere in the current call’s prompt. This happened across two different models (gemini-3.1-pro-preview and gemini-3-flash-preview), on two different tickers, on multiple occasions, and after I deliberately changed the prompt text between calls.
I’ve ruled out every client-side explanation available to me (see “What I ruled out” below). I can’t rule out a server-side cause, and I’m filing this as a request for Google to investigate from your side, since I have no visibility into the serving infrastructure.
Supporting evidence
Full raw request/response logs, structured trace summaries, and a related follow-up test are available here: API Bug - Google Drive
That folder includes a README.md indexing each file. File paths have been redacted throughout (C:\Users\<user>\...); no API keys are present in any file (LiteLLM’s own debug logging redacts the request header). The two files most relevant to the concrete example below are evidence_BHRPB_run1_fresh_debug_log.txt and evidence_BHRPB_run2_DUPLICATED_debug_log.txt.
Environment
- API surface: Google AI Studio / Generative Language API (
https://generativelanguage.googleapis.com/v1alpha/models/{model}:generateContent), reached via LiteLLM’sgemini/provider prefix (not Vertex AI, no explicitCachedContentobject ever created). - Models:
gemini-3.1-pro-previewandgemini-3-flash-preview. - Client library:
litellm==1.93.0(Python), callinglitellm.completion(model="gemini/...", messages=[...], response_format={"type": "json_schema", "json_schema": {...}}). - No explicit caching used anywhere — I never call any
CachedContent/context-cache API.litellm.cacheis confirmedNonein my process. Google’s implicit caching (automatic, no developer action required — documented on Google’s own “Vertex AI context caching” blog post) is the only caching feature that could be in play, since I take no action to enable or disable it. temperature: never set by my code (Nonepassed through). Confirmed via raw request logging that the API defaults this to1.0for both models (not a determinism/low-temperature issue).
What I observed (concrete example)
Two separate synthesis calls, ~1h50m apart, same-day, same ticker (BHR-PB, a preferred stock), each with its own freshly-generated research dossier (produced by an upstream, unconstrained, no-schema call that legitimately varied between the two runs — different web search results, different phrasing, different total length).
The two dossiers fed to synthesis had genuinely different figures, not paraphrases of the same numbers:
| Anchor | Run 1’s dossier | Run 2’s dossier |
|---|---|---|
| Preferred stock price | $14.23 | $14.26 |
| Common stock price | $2.14 | $2.16 |
| Common equity market cap | $147.0M | $148.3M |
| Net debt | $957.3M | $964.2M |
| Dividend yield | 9.7% | 9.64% |
| Field naming / anchor count | 12 anchors, e.g. "Preferred Stock Price (BHR-PB)" |
9 anchors, differently named, e.g. "BHR-PB Price" |
The synthesis outputs for run 1 and run 2 were byte-identical on 20 of 24 output fields (Business, Financials, Competitive Landscape, Investment Thesis, Risks, Risk Level, Risk Level Rationale, Hold Period, Hold Period Rationale, Thesis List, Thesis List Rationale, Profile Rationale, etc.) — including a Financials field that quotes Preferred Stock Price (BHR-PB): USD 14.23 and the exact field label "Preferred Stock Price (BHR-PB)", both of which are run 1’s values and run 1’s naming convention, not anything present in run 2’s own dossier (14.26, "BHR-PB Price").
This is not a “same HTTP response returned twice” cache hit. I compared the raw API response metadata directly:
| Run 1 | Run 2 | |
|---|---|---|
responseId |
fUBpao_ZGqjgz7IP6PuEsQE |
b1ppaquVHcuIz7IPioDViAU |
usageMetadata.promptTokenCount |
11,897 | 15,067 |
usageMetadata.thoughtsTokenCount |
6,855 | 4,982 |
usageMetadata.cachedContentTokenCount |
absent | absent |
Two different response IDs, two different prompt sizes, two different (non-trivial) internal-reasoning token spends, and neither response reports a cache-hit (cachedContentTokenCount is the field Google’s API uses for that; it’s absent both times). Whatever produced run 2’s output did real, distinct, billed work — and still returned content anchored to run 1’s specific facts.
I confirmed, by directly inspecting the raw outbound request body in my own debug logs, that run 2’s prompt genuinely contained run 2’s own dossier data (the different figures above) and did not contain any of run 1’s figures anywhere in its context window.
Pattern across a wider set of tests (same day)
| Ticker | Gap between the two calls | Result |
|---|---|---|
CRGY (a different, unrelated preferred/equity ticker) |
3 days (matched a run from 3 days earlier, on the same day’s first attempt) | Duplicated |
CRGY |
26 minutes (between two other calls) | Not duplicated — both fresh, distinct from each other |
CRGY |
44 minutes | Duplicated |
BHR-PB |
1h 50m | Duplicated |
There is no consistent time-window pattern (a 26-minute gap was clean; a 3-day gap duplicated), which argues against a simple short-TTL cache explanation and suggests — if this is caching at all — some form of fuzzy/entity-level matching rather than exact-prefix, time-bounded caching.
No cross-ticker bleed was observed. A third ticker (SERE) was run in between several of the CRGY calls above (sharing the identical several-thousand-token system prompt) and came back clean and distinct every time — it never picked up CRGY or BHR-PB content. Whatever is happening appears to track the specific entity being profiled, not just “a similar-looking prompt was seen recently.”
What I ruled out on my side
- My own code: no
functools.lru_cache, no global mutable state, no file-based response cache anywhere in my pipeline. Verified by direct code inspection. - LiteLLM’s cache:
litellm.cacheconfirmedNoneat runtime; no cache-related environment variables set; no litellm config file present. - Temperature/determinism: confirmed
1.0on every call via raw request inspection (never explicitly set by me; this is Gemini-3-family’s server-applied default when omitted). - Stale prompt / code not picking up my edits: I changed the system prompt between two calls that both showed this behavior and confirmed, by string-searching the raw outbound request body, that the new prompt text was genuinely transmitted in the call that still returned old content.
- JSON-schema constrained decoding generally suppressing output diversity: tested directly — an upstream call in the same pipeline (
research_extract) uses the identical schema-constrained (response_format/responseSchema) mechanism and, on the same two runs, returned 0 of 14 fields identical (fully fresh both times). Whatever is happening is not “any schema-constrained call converges,” since a sibling call using the same feature did not exhibit it.
What I have not been able to determine
- Whether this is implicit context caching specifically, some other server-side memoization/entity-matching behavior, or something else entirely on Google’s infrastructure. I have no visibility into your serving stack and cannot confirm a mechanism, only that the symptom is real and reproducible on my end.
- I’m aware of a similar-sounding report on the AI Developers forum (“Critical bug: Vertex API with context cache leaks prompt state between generateContent calls”), but that thread describes explicit context caching with image-containing prompts, and a Google engineer replied unable to reproduce it. My case uses no explicit caching and no images/files — text-only prompts — so I don’t believe it’s the same bug, though the general category (cross-request content leaking into an unrelated response) looks similar.
Impact
For my use case (an automated research-profile pipeline), this means a synthesis call’s output cannot currently be trusted to reflect the actual data given to that specific call, particularly for an entity that has been profiled more than once. I’ve added a client-side mitigation (checking that a sample of the current call’s own input figures appear somewhere in its own output, and rejecting/flagging the response if not), but this is a symptom-level guard, not a fix, and it can’t recover a correct answer — only detect that the one returned is untrustworthy.
What would help me (and future reporters of this class of bug)
- Confirmation of whether implicit caching can, under any circumstance, affect generated content rather than only billing for repeated-prefix tokens.
- Any diagnostic I could attach to a request (a header, a request ID I should report immediately after a suspected incident) to let your team correlate this from server-side logs, since by the time I notice via output comparison, the request is long past.