Gemini Live API Caching

Gemini API
1

Production EdTech app (English school): a real-time voice tutor for ~100 students. Of course, I want to escale this to 1000 students.

  • Model: gemini-3.1-flash-live-preview (Developer API, ephemeral tokens minted by a FastAPI backend, React Native client)
  • Audio-only sessions (soon I will connect video stream), typically 5min - 15 min
  • Tier 2. AI Studio dashboard for “Gemini 3 Flash Live”: RPM unlimited, RPD unlimited, TPM 400k

Measured usage (real production data, from usageMetadata)

Average ~16,400 tokens/min per session (last 8 active days, 142 min, 2.33M tokens):

Component tokens/min share
Audio input ~8,500 52% mostly conversation history re-billed as audio each turn
Text input ~7,100 43% ” system prompt + injected context, re-counted each turn
Audio output ~750 5%

At 400k TPM that puts my effective ceiling at ~24 concurrent sessions, which is tight for 100 students.

What I’m already doing

  1. Context window compression (sliding window, trigger_tokens=6000, target_tokens=3000) this alone cut cost ~23% and flattened per-minute cost
  2. audioStreamEnd on mic mute and long pauses; half-duplex mode on speaker playback
  3. VAD tuned (silenceDurationMs=300)
  4. thinkingLevel at default minimal
  5. Trimming the system prompt (it’s re-counted every turn)
  6. Planned next: session recycling close the session every ~5 min andy as text transcript via send_client_content +initial_history_in_client_content in history_config, so accumulated history stops being re-billed as audio tokens

Questions

  1. Is there any caching mechanism for the Live API? The caching docs cover implicit/explicit caching for generateContent and the Interactions API, but don’t mention Live. Since
    the per-turn context is identical between turns, implicit caching would be in the roadmap?
  2. Beyond context compression and text re-seeding, are there other supported techniques to reduce per-turn context re-billing (e.g., any config to keep history as text server-side instead of audio)?
  3. Is TPM the only concurrency constraint on the Developer API, or is the current-sessions limit per project? The dashboard only shows RPM/TPM/RPD for the Live model.
  4. For a production workload like this (spiky usage, school hours), what’s the recommended path to a TPM increase on Tier 2 the standard rate-limit increase form, or is there a
    better channel for Live API workloads? Vertex maybe? :sweat_smile:

Thanks, happy to share more measurements if useful.

2

Hello @Mikael_FlowMika ,

  1. The Gemini Live API does not currently support explicit or implicit context caching. For more on how Live API sessions are configured and billed, see the Gemini Live API Overview and pricing documentation.
  2. Audio input and output can be transcribed server-side via input_audio_transcription / output_audio_transcription, but the transcripts are returned to the client for logging or captions — they don’t replace the accumulated audio tokens in the session’s context window. For long sessions you therefore need context window compression (to get past the 15-minute audio-only limit) and session resumption (to survive the ~10-minute connection reset), as described in the Session Management guide.
  3. Live API workloads are constrained by your project’s spend-based usage tier and Tokens Per Minute (TPM) limits as outlined in the Gemini API Rate Limits Guide. We currently do not guarantee support for concurrent sessions. As an FYI you can read more details in Vertex AI documentation
  4. You can request higher TPM allocations for high-concurrency Live API workloads by submitting the rate limit request form or upgrading your project’s spend tier as explained in the Rate Limit Increase Options. As mentioned earlier you can also read more details in Vertex AI documentation