I’m building a voice agent using the Gemini Live API
(gemini-live-2.5-flash-native-audio) and noticed a confusing
behavior with input_audio_transcription.
Setup:
LiveConnectConfig with input_audio_transcription=AudioTranscriptionConfig()
response_modalities=[“AUDIO”]
Listening to server_content.input_transcription.text for displaying
user speech in the UI
Issue:
The transcription text does not match what the model actually understood.
Were you able to find any workaround or fix for this issue? I’m currently using the latest version of Gemini 3.1 Flash Live Preview, and I’m facing a similar issue. @Sathurshanath@Jaydipsinh_Zala
The Gemini Live 3.1 preview also has the same issue.
My guess is the same: text transcription and speech are processed through different channels.
If you have the Live API use a tool to relay the request, you’ll see a relatively accurate result.
For example, if I speak a sentence in Chinese, the text transcription might produce Japanese or Korean. However, if I request that the user’s speech be passed to the tool, it will produce the correct Chinese.
@kaven I don’t think we should introduce an additional STT model just for transcripts. We’re already investing significantly in the S2S model and the voice-agent infrastructure. If transcript quality from the S2S stack isn’t meeting expectations, I’d rather see improvements in the core model than compensate with another model.
One of the key promises of S2S is reducing the complexity of the traditional sandwich architecture. Adding a separate STT model back into the stack starts to defeat that purpose and adds more cost, latency, and operational complexity.
If we’re already paying a premium for S2S and its infrastructure, I would expect transcript quality to be at least comparable to what we had with the traditional architecture. Otherwise, we’re effectively reintroducing parts of the old stack while still paying for the new one.
Of course, this is just my opinion, but once we’re already making a substantial investment in the S2S stack, I think we should hold it to a higher standard rather than patching over its shortcomings with additional models.
I have Gemini Live act solely as a conversational tool. Its task is to “correct the text based on context and generate the user’s complete intent text to send to the tool.” The prompts differ here. If only “transmit the text” is requested, the received content will be almost identical to the recorded text. However, if the prompt is “correct the text based on context before transmitting,” the corrected content will be received, improving accuracy, although the text description may differ slightly.
The tool receives the text and forwards it to the text model for processing. After processing, the text model replies to the Live API, which generates the dialogue and sends it back to the user.
The resulting latency depends on the text model’s speed; faster models typically only add about one to two seconds of latency.
Here’s a trick: after the text model uses the tool, some tools can avoid generating a response text before sending it to the Live API. The response can be directly passed to the Live API to generate the reply text, significantly reducing latency.
This way, I don’t need to use very long prompts on the Live API. The Live API can execute very lightweightly.
@KuanMing_Chen Interesting approach, but I think this discussion changes once you move from a hobby project to a production environment.
At production scale, it’s not just about adding 1–2 seconds of latency. You also have to consider the broader operational complexity and infrastructure overhead that comes with running voice systems reliably.
Even at a relatively modest volume-say 500 calls/day with an average duration of 2 minutes—the costs add up quickly. You’re already paying a premium for the S2S model and maintaining the surrounding infrastructure. Adding another STT layer and an additional text model means more complexity, more failure points, more things to monitor, and more billing at the end of the month.
One of the promises of S2S architectures is reducing stack complexity. If we end up reintroducing STT + text reasoning layers to compensate for shortcomings, we’re effectively rebuilding a sandwich architecture while still paying S2S prices.Of course, this is just my opinion, but I think we should evaluate these trade-offs from a production perspective rather than from the standpoint of a small-scale project.
This architecture is the solution I devised based on the current state of the Gemini Live API.
In my tests, this approach is more cost-effective, but stability issues cannot yet be verified in a production environment.
The cost reduction is due to the Gemini Live API’s lack of a Context Cache and its higher price.
Adding complex features would cause the system’s prompt length to accumulate multi-turn conversation tokens rapidly.
Furthermore, I cannot manually trim the dialogue history or use hidden prompts to update the bot’s progress before each user input.
These problems can be solved with a text model.
This is actually just a temporary solution. Currently, the Live API model’s performance still has room for improvement. I think the Live API could consider adding hooks to allow the system to inject additional prompts at different stages, and the correct transcription of input text should be internalized within the system.
Context Cache is also an area that needs improvement; at the very least, System Instructions and tool definitions should be able to hit the cache.
If these could be achieved, my method wouldn’t be necessary.
@KuanMing_Chen I agree that your approach makes sense given the current limitations of the Live API. My only concern is that, from a production perspective, every additional model introduces more cost, complexity, and operational overhead. One of the promises of S2S is reducing the complexity of the traditional sandwich architecture. If we keep adding layers to compensate for missing features, we’re essentially rebuilding a hybrid stack while still paying S2S prices. Hopefully, future improvements to Live API will make these workarounds unnecessary.
I totally agree that the model should be improved. For now, I decided to ask the model to use a tool to save the needed text. I consider the transcript a FYI only for now.