Hi everyone,
We run a production voice AI platform (outbound sales/survey calls) integrating directly
with the Gemini Live API via Vertex AI (native C module + libwebsockets, no proxy in
between). We’re seeing audio crackling/stuttering mid-call in ~50-60% of calls and have
exhausted client-side troubleshooting. Details below — would appreciate input from anyone
who’s hit this, or from a Googler who can point us at the right place.
Setup:
- Project: my-asist-ai, region: europe-west4 (moved from us-central1 today)
- Model: gemini-live-2.5-flash-native-audio
- Endpoint: wss://europe-west4-aiplatform.googleapis.com/ws/google.cloud.aiplatform.v1.LlmBidiService/BidiGenerateContent
- generationConfig.responseModalities=[“AUDIO”], speechConfig.voiceConfig.prebuiltVoiceConfig set,
inputAudioTranscription + outputAudioTranscription enabled - Auth: service account JWT → OAuth bearer token
What we ruled out on our end (all verified/measured, not guessed):
- Network path: ~10ms RTT, 0% packet loss (mtr)
- Server CPU/RAM: idle headroom, no starvation
- Real-time process scheduling: SCHED_RR, high priority confirmed
- Fixed 20ms/640-byte audio frame pacing, absolute-deadline scheduling (clock_nanosleep)
- Client-side jitter buffer: A/B tested 250ms vs 400ms — 400ms did NOT fix the crackling
- TCP_NODELAY: confirmed enabled by default in libwebsockets
- Resampling: custom phase-continuous 24kHz->16kHz converter with anti-aliasing FIR
(exact 3:2 ratio, not a generic library) - Connection region: moved us-central1 → europe-west4, cut connection+setup time from
~1150ms to ~430ms average (measured via direct WS handshake test, 4 repeated trials) —
helped opening latency, did NOT fix mid-call crackling
What we measured that we can’t explain from our side:
On a real post-migration call (6 turns, ~2 min), our own playback instrumentation showed:
- 6 buffer-underrun pad events (audio queue ran dry) across 6 turns
- 45 “late” frame emissions (>30ms gap) and 34 “burst” emissions (<10ms gap) in ~2 minutes
- Receive queue depth spiking to ~19.4 seconds of buffered-but-unplayed audio at one point
- Max gap between consecutive received audio chunks: 24.4s (caveat: this specific metric
isn’t yet scoped per-turn on our end, so it may include normal listening silence between
turns — we’re adding per-turn instrumentation to nail this down precisely)
Given items 1-9 above are already correctly configured, the remaining candidate is the
pacing/consistency of audio chunk delivery over the WS connection during generation itself.
Question: is bursty/irregular chunk delivery for streaming audio responses a known
characteristic of BidiGenerateContent, or would this indicate something specific to our
setup/region that’s worth escalating? Any guidance on recommended consumption patterns
(e.g. expected inter-chunk timing) would help us narrow this down further.
Happy to share full logs / session IDs if useful.