We would like to use the Vertex API for our requests, but we’re seeing terrible implicit caching rates. The Gemini API backend, however, appears to manage the implicit caching really well. It appears that when we use US region, we get a higher cache hit rate on Vertex, but that’s not available for 3.6 Flash (and the performance is still worse than I’d expect). My guess is that the routing is not working properly behind the scenes, so subsequent requests don’t get routed to the warm cache.
The docs state that implicit caching should work with the “global” region, but we’re just not seeing that to be the case. And obviously, the cost implications of broken caching are pretty significant.
Is this expected? A bug? User error?