Does Batch API in Vertex AI support caching?

Gemini API
,
1

I am using the following format

{'cachedContent': 'projects/*******/locations/us-central1/cachedContents/******',
 'contents': [{'parts': [{'text': '.'}], 'role': 'user'}],
 'generationConfig': {'candidateCount': 1,
  'maxOutputTokens': 65534,
  'temperature': 0,
  'topP': 0.95},
 'safetySettings': [{'category': 'HARM_CATEGORY_SEXUALLY_EXPLICIT',
   'threshold': 'OFF'},
  {'category': 'HARM_CATEGORY_HATE_SPEECH', 'threshold': 'OFF'},
  {'category': 'HARM_CATEGORY_HARASSMENT', 'threshold': 'OFF'},
  {'category': 'HARM_CATEGORY_DANGEROUS_CONTENT', 'threshold': 'OFF'}]}

But I get this error

Bad Request: {"error": {"code": 400, "message": "Model gemini-2.5-flash-001 does not support cached content with batch prediction.", "status": "INVALID_ARGUMENT"}}
2

Hi @Shreyansh_Bardia,

Batch prediction does not support explicit caching. Batch prediction is optimized for high-throughput, asynchronous processing of a large number of prompts at a reduced cost, reference (refer to “Why use batch prediction?”); while explicit context caching is designed for real-time or near-real-time scenarios where you want to manually manage and reuse a large context across multiple individual requests to reduce latency and cost.

3

Okay, but It would be great if we could include caching in Batch Prediction as well, we have a prompt of around 10k tokens which we would like to run with different inputs. Caching would help in such scenarios

5

Sure @Shreyansh_Bardia,

I will raise this as a feature request to the concerned team. Thanks for providing your use-case as an example. If you would like to elaborate on this example based on your use-case and cost saving estimates with explicit context caching with batch, it will help the team.

7

Hey Krish_Varnakavi1, sorry to jump in, but my team actually has a perfect use case for this. We process a lot of documents by sending the same doc with several different prompts. Because of this, our input-to-output token ratio is heavily skewed at about 10:1. We don’t need real-time responses, so the Batch API should be a great fit. However, without Context Caching, it just doesn’t make financial sense. Saving 50% on our tiny outputs via Batch isn’t worth losing the caching discount on our massive inputs, since we’d end up paying full price to tokenize the same document over and over.

8

Hey Krish_Varnakavi1,
I also have a perfect use case for this, similar to the one reported by Rostyslav_Dembitskiy.
I also have an input-to-output token ratio of around 10:1, but in my case we use the same system prompt to process hundreds of thousands of different requests for parsing and analyzing images.
With explicit caching we have an estimated saving of 50%, same as the batched requests themselves. In theory, we don’t need the requests to be processed right away, but if we have the same savings on the non-batched calls because the batched requests don’t support explicit caching, it doesn’t make sense to use the batched requests.

I would also like to note that even though we send batches of around 200 requests at a time, all with the same system prompt, our percentage of requests that actually get the system prompt cached implicitly is around 1%.