Can't get the implicit caching to work

Gemini API
,
12

Hi, I noticed implicit caching behavior changed after gemini 3.6 flash was released. I used to get >90% cache rate but after the release, the cache rate fluctuated between 0% to ~65% for my app. I did a thorough investigation using lots of claude quota and lot of diagnostic data from my chat history. I collect the in/out token and cache token count as diagnostic on each turn including the data for each tool call made by the main chat AI. Here’s what I learned based on diagnostic logs from a chat session with gemini 3.6 flash.

  1. Only the first 8,192 or so tokens are cached even though my conversation has >17k token and the prefix had not changed to break implicit caching.

  2. The minimum threshold to activate implicit caching was ~10k tokens. Any input below that does not activate implicit caching. The kicker is that even if the 10k threshold was over, it still occasionally miss implicit caching.

  3. Here is a snippet of the report from one of several analysis on my diagnostic data. Notice the actual cache amount was was only ~8k token regardless of input token size from 10k and up. This mean the max cache rate is only ~80% and it only gets worst as the conversation lengthens. Also notice how there is a 10k token floor that has to be met before caching hits.

    image
    image841×685 53 KB

  4. Google says minimum token limit is only 4096 for 3.5 flash and 6144 for 3.0 flash preview which I also noticed the behavior had changed. Context caching overview  |  Gemini Enterprise Agent Platform  |  Google Cloud Documentation

This has been very frustrating because a change in implicit caching behavior can impact the cost of a chat bot by over 100% and I could have been losing a lot of money if my app is in production. What made this worst is that the change was unannounced. If developers has more knowledge on what is going on, we could have pivoted a lot quicker on the way we handle caching in the app. I think this lack of clarity and communication from Google is driving customers away. Even though I prefer the chat responses from gemini, I had to pivot part of my app to gpt because of the cost from the poor implicit caching. I hope Google can restore the previous implicit caching behavior and at least be transparent with any changes be it temporary or permanent.