Gemini Live API Caching

Gemini API
1

Production EdTech app (English school): a real-time voice tutor for ~100 students. Of course, I want to escale this to 1000 students.

  • Model: gemini-3.1-flash-live-preview (Developer API, ephemeral tokens minted by a FastAPI backend, React Native client)
  • Audio-only sessions (soon I will connect video stream), typically 5min - 15 min
  • Tier 2. AI Studio dashboard for “Gemini 3 Flash Live”: RPM unlimited, RPD unlimited, TPM 400k

Measured usage (real production data, from usageMetadata)

Average ~16,400 tokens/min per session (last 8 active days, 142 min, 2.33M tokens):

Component tokens/min share
Audio input ~8,500 52% mostly conversation history re-billed as audio each turn
Text input ~7,100 43% ” system prompt + injected context, re-counted each turn
Audio output ~750 5%

At 400k TPM that puts my effective ceiling at ~24 concurrent sessions, which is tight for 100 students.

What I’m already doing

  1. Context window compression (sliding window, trigger_tokens=6000, target_tokens=3000) this alone cut cost ~23% and flattened per-minute cost
  2. audioStreamEnd on mic mute and long pauses; half-duplex mode on speaker playback
  3. VAD tuned (silenceDurationMs=300)
  4. thinkingLevel at default minimal
  5. Trimming the system prompt (it’s re-counted every turn)
  6. Planned next: session recycling close the session every ~5 min andy as text transcript via send_client_content +initial_history_in_client_content in history_config, so accumulated history stops being re-billed as audio tokens

Questions

  1. Is there any caching mechanism for the Live API? The caching docs cover implicit/explicit caching for generateContent and the Interactions API, but don’t mention Live. Since
    the per-turn context is identical between turns, implicit caching would be in the roadmap?
  2. Beyond context compression and text re-seeding, are there other supported techniques to reduce per-turn context re-billing (e.g., any config to keep history as text server-side instead of audio)?
  3. Is TPM the only concurrency constraint on the Developer API, or is the current-sessions limit per project? The dashboard only shows RPM/TPM/RPD for the Live model.
  4. For a production workload like this (spiky usage, school hours), what’s the recommended path to a TPM increase on Tier 2 the standard rate-limit increase form, or is there a
    better channel for Live API workloads? Vertex maybe? :sweat_smile:

Thanks, happy to share more measurements if useful.