NOTE: This is a temporary solution intended to stabilize production apps until the core “ghosting” issues in the native-audio stream are patched.
Hi everyone,
I’ve been testing a workaround for the native-audio-preview instability, and I can confirm that moving the VAD (Voice Activity Detection) to the client-side resolves the “ghosting” and tool-use failures.
The Fix: Instead of relying on the Realtime API’s open stream to detect “End of Turn” (which seems to be the point of failure), I implemented a client-side buffer.
-
Local VAD: App listens to the mic → Detects Silence → Cuts the recording.
-
Batch Send: Uploads the audio buffer as a single input to the
gemini-2.5-flash-native-audiomodel.
Test Results:
• Success Rate: 100% response (No ghosting).
• Baseline Latency: ~3.5s (average round-trip on default settings).
• Accuracy: The model correctly identifies raw audio nuances because it receives the full context at once.
Optimization Possibility (~1.75s - 2s): My initial tests were conservative. You can likely get the latency down to under 2 seconds if you:
• Force 16kHz Sampling: The Flash model is optimized for 16kHz. Recording at this rate reduces the payload size by ~33% compared to 24k/44k, speeding up the upload significantly.
• Aggressive VAD: Tightening the silence timeout (e.g., from 1.5s down to 0.6s) makes the “turn” feel much snappier.
Why do this? While this introduces a small delay compared to true streaming, it is infinitely better than the “3-5 retries” or total silence currently plaguing the streaming endpoint. It forces the model to treat the input as a complete thought, which drastically improves tool-calling reliability.
If you are stuck before the Dec 9th deprecation, try buffering your audio locally and sending it as a “committed” turn until the devs fix the main cause.
Hope it helps, ~Cmd.Proton