Issues with inconsistent caching between Vertex endpoints and Gemini API endpoints

Gemini API
, ,
1

We would like to use the Vertex API for our requests, but we’re seeing terrible implicit caching rates. The Gemini API backend, however, appears to manage the implicit caching really well. It appears that when we use US region, we get a higher cache hit rate on Vertex, but that’s not available for 3.6 Flash (and the performance is still worse than I’d expect). My guess is that the routing is not working properly behind the scenes, so subsequent requests don’t get routed to the warm cache.

The docs state that implicit caching should work with the “global” region, but we’re just not seeing that to be the case. And obviously, the cost implications of broken caching are pretty significant.

Is this expected? A bug? User error?

2

I am getting 0 cached tokens with Vertex and decent cace perfomance via the Gemini API.

With the latest Gemini models they are only allowed on global endpoints, so cant even tie to us-central1 to debug further.

The price savings is mind boggling, nearly 10x difference. We just migrated our entire stack off Vertex.

Please Google debug why.

3

Hi, For issues related to the Vertex API, please contact your sales representative if you have one. Otherwise, you can find more relevant assistance on the Google Cloud Forum.