Live API (gemini-3.1-flash-live-preview): audio+video sessions die with 1011 "Internal error encountered" ~2 min in;

Gemini API
1

Summary

Bidirectional Live sessions that stream both continuous audio and 1 FPS JPEG video frames are terminated by the server with WebSocket close code 1011 “Internal error encountered”, typically 90–120 seconds into the session, with no client protocol violation. Worse, resuming such a session via sessionResumption handle poisons the restored context: every resumed session then dies within ~0.5s of the next user utterance (right as generation starts), producing an unrecoverable disconnect/reconnect loop until the client abandons the handle and starts a fresh context.

Audio-only sessions with the identical config run for many minutes without issues.

Environment

  • Model: gemini-3.1-flash-live-preview
  • Endpoint: wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContentConstrained (ephemeral auth token, uses: 0)
  • Clients reproduced: iOS URLSessionWebSocketTask (iPhone 13 Pro, iOS 26) and Node.js (macOS) — same server behavior; iOS surfaces the server-side termination as NSPOSIXErrorDomain Code=57 "Socket is not connected" on the pending receive, Node shows the clean close frame 1011 "Internal error encountered".
  • Session config: responseModalities: [AUDIO], Puck voice, inputAudioTranscription + outputAudioTranscription, ~20–30 functionDeclarations, automaticActivityDetection (START/END sensitivity LOW, prefixPaddingMs 20, silenceDurationMs 300), sessionResumption: {}, contextWindowCompression: { triggerTokens: "10000", slidingWindow: { targetTokens: "512" } }, systemInstruction ~2–8 KB.

Reproduction

  1. Connect with the config above, send setup, receive setupComplete.
  2. Stream microphone audio continuously (16 kHz PCM, 40 ms chunks, base64) and JPEG frames at 1 FPS (~60–150 KB each, 640–768 px).
  3. Speak one turn early (works fine — model answers about the video). Keep streaming.
  4. Around 90–120 s into the session (often right after the next user utterance is transcribed, before any response part arrives), the server closes with 1011.

The failure is probabilistic per session on a minimal client, but highly reproducible on our production app (larger system prompt + longer conversations): sessions died at ~117 s consistently. In one minimal-client run the session died at 87.7 s with no user action at all (only silence + 1 FPS frames after an early completed turn).

Timeline excerpt (device, wall clock):

09:03:05 setupComplete                       (camera on, 1 FPS frames ~60–150 KB, sends 1–3 ms)
09:03:14 → 09:04:57  8 full voice exchanges about the video — all fine, barge-in fine
09:05:02 inputTranscription of next utterance
09:05:02 +0.7s  socket dead (iOS: POSIX 57; Node equivalent: close 1011 "Internal error encountered")

The resumption chain (worse than the initial death)

After the 1011 death, reconnecting with the last sessionResumptionUpdate.newHandle restores the context — and the restored session dies deterministically within ~0.5 s of the first user utterance (again before any response part). We observed 3+ consecutive resumed sessions dying this way; a fresh session (no handle) immediately worked again for another ~2 minutes. So the corrupted context appears to survive inside the resumption state.

09:05:05 reissue token with handle → setupComplete 09:05:07
09:05:13 inputTranscription → +0.5s dead
09:05:15 reissue with new handle → setupComplete 09:05:17
09:05:22 inputTranscription → +0.6s dead
(repeats until the client drops the handle)

Ruled out (all verified against the live endpoint)

  • Wire format / frame size / rate: identical payloads run 6+ minutes when no generation happens late in the session.
  • Session duration or connection lifetime limits: audio-only sessions with the same config run far past 2 minutes.
  • Context compression window crossing alone (turn at 50 s and at 130 s after compression trigger — survived in most runs).
  • Tool call generation during video (forced a real get_weather functionCall mid-video — survived).
  • Client network: reproduced on Wi-Fi, LTE, and desktop ethernet; also reproduced with the SDK-equivalent Node client receiving the explicit 1011 close frame.
  • Abrupt client death + resume (SIGKILL mid-generation, then resume) with a short context — survived, so resumption itself is fine; it is the post-1011 context that stays poisoned.

Expected

  • Audio+video sessions should not hit internal errors as a function of session age/content, or should at least fail with an actionable error.
  • A session that died from a server-internal error should either be resumable cleanly or the resumption handle should be invalidated — restoring a context that deterministically kills the next generation makes sessionResumption unusable for video sessions.

Impact

Voice+video assistant sessions (accessibility use case: blind user relying on camera descriptions) are capped at ~2 minutes and then enter a reconnect death-loop unless the client discards the handle and loses conversation context.

Minimal repro (Node 22+, self-contained)

npm i @google/genai
export GEMINI_API_KEY=...        # AI Studio key
# optional real assets (silence + gray frame also repro; real ones match prod timing):
#   say -o utt.wav --data-format=LEI16@16000 "What do you see in my camera?"
#   sips -z 768 768 -s format jpeg any.png --out frame.jpg
node repro.mjs                   # session A only
RESUME=1 node repro.mjs          # also resume the handle → watch the re-death

// Minimal reproduction: Gemini Live audio+video session dies with close 1011
// "Internal error encountered" ~90-120s in; resuming the handle then kills the
// next session within ~0.5s of the following utterance.
//
// Node 22+ (built-in WebSocket) + @google/genai.  No app code, no local paths.
//
//   npm i @google/genai
//   export GEMINI_API_KEY=...            # AI Studio key
//   # optional real assets (recommended — silence + gray frames still repro,
//   #   but real speech/frames match the production timing more closely):
//   #   say -o utt.wav --data-format=LEI16@16000 "What do you see in my camera?"
//   #   sips -z 768 768 -s format jpeg /path/to/any.png --out frame.jpg
//   node public-repro.mjs
//
// It runs session A (audio+1fps video, one spoken turn), then keeps streaming
// silence+frames and logs the close code. Set RESUME=1 to also reconnect with
// the last sessionResumption handle and observe the deterministic re-death.

import { readFileSync, existsSync } from "node:fs";
import { GoogleGenAI, Modality, StartSensitivity, EndSensitivity } from "@google/genai";

const MODEL = "gemini-3.1-flash-live-preview";
const RUN_SECONDS = Number(process.env.RUN_SECONDS || 180);
const t0 = Date.now();
const log = (m) => console.log(`[${((Date.now() - t0) / 1000).toFixed(1)}s] ${m}`);

const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });

// Real assets if present, else 16kHz silence + a gray 768x768 JPEG placeholder.
const FRAME_BYTES = 1280; // 640 samples * 40ms @16kHz Int16
const silence = Buffer.alloc(FRAME_BYTES);
const utt = existsSync("utt.wav") ? readFileSync("utt.wav").subarray(44) : null;
const frameJpeg = existsSync("frame.jpg")
  ? readFileSync("frame.jpg").toString("base64")
  // 1x1 gray JPEG fallback (repro is weaker; a real ~100KB frame is closer to prod)
  : "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAP//////////////////////////////////////////////////////////////////////////////////////wAALCAABAAEBAREA/8QAFAABAAAAAAAAAAAAAAAAAAAAAP/EABQQAQAAAAAAAAAAAAAAAAAAAAD/2gAIAQEAAD8AfwD/2Q==";

async function mintToken(resumeHandle) {
  const token = await ai.authTokens.create({
    config: {
      httpOptions: { apiVersion: "v1alpha" },
      uses: 0,
      liveConnectConstraints: {
        model: MODEL,
        config: {
          responseModalities: [Modality.AUDIO],
          speechConfig: { voiceConfig: { prebuiltVoiceConfig: { voiceName: "Puck" } } },
          systemInstruction: { parts: [{ text: "You are a helpful voice assistant. Keep answers short." }] },
          inputAudioTranscription: {},
          outputAudioTranscription: {},
          realtimeInputConfig: {
            automaticActivityDetection: {
              startOfSpeechSensitivity: StartSensitivity.START_SENSITIVITY_LOW,
              endOfSpeechSensitivity: EndSensitivity.END_SENSITIVITY_LOW,
              prefixPaddingMs: 20,
              silenceDurationMs: 300,
            },
          },
          sessionResumption: resumeHandle ? { handle: resumeHandle } : {},
          contextWindowCompression: { triggerTokens: "10000", slidingWindow: { targetTokens: "512" } },
        },
      },
    },
  });
  return token.name;
}

function runSession(label, token, speakAt) {
  return new Promise((resolve) => {
    const url = `wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContentConstrained?access_token=${token}`;
    const ws = new WebSocket(url);
    let handle = null, buf = null, off = 0, audioTimer, videoTimer;

    const done = (outcome) => {
      clearInterval(audioTimer); clearInterval(videoTimer);
      resolve({ handle, outcome });
    };

    ws.addEventListener("open", () => ws.send(JSON.stringify({ setup: { model: `models/${MODEL}` } })));
    ws.addEventListener("close", (e) => { log(`${label}: CLOSE code=${e.code} reason="${e.reason}"`); done(`close ${e.code}`); });
    ws.addEventListener("message", async (ev) => {
      const text = typeof ev.data === "string" ? ev.data : Buffer.from(await ev.data.arrayBuffer()).toString("utf8");
      let m; try { m = JSON.parse(text); } catch { return; }
      if (m.setupComplete !== undefined) {
        log(`${label}: setupComplete — streaming audio + 1fps video`);
        audioTimer = setInterval(() => {
          if (ws.readyState !== WebSocket.OPEN) return;
          let c = silence;
          if (buf) { c = buf.subarray(off, off + FRAME_BYTES); off += FRAME_BYTES; if (off >= buf.length) buf = null; if (!c.length) c = silence; }
          ws.send(JSON.stringify({ realtimeInput: { audio: { data: c.toString("base64"), mimeType: "audio/pcm;rate=16000" } } }));
        }, 40);
        videoTimer = setInterval(() => { if (ws.readyState === WebSocket.OPEN) ws.send(JSON.stringify({ realtimeInput: { video: { data: frameJpeg, mimeType: "image/jpeg" } } })); }, 1000);
        setTimeout(() => { buf = utt; off = 0; if (utt) log(`${label}: (speaking)`); }, speakAt);
      }
      if (m.sessionResumptionUpdate?.newHandle) handle = m.sessionResumptionUpdate.newHandle;
      if (m.serverContent?.inputTranscription?.text) log(`${label}: inputT="${m.serverContent.inputTranscription.text}"`);
      if (m.serverContent?.turnComplete) log(`${label}: turnComplete`);
    });

    setTimeout(() => { log(`${label}: reached ${RUN_SECONDS}s without dying`); try { ws.close(); } catch {} done("survived"); }, RUN_SECONDS * 1000);
  });
}

log("=== session A (audio+video) ===");
const a = await runSession("A", await mintToken(null), 2000);
log(`A ended: ${a.outcome}; handle=${a.handle ? a.handle.slice(0, 12) + "…" : "none"}`);

if (process.env.RESUME === "1" && a.handle) {
  log("=== session B (resume A's handle) ===");
  const b = await runSession("B", await mintToken(a.handle), 2000);
  log(`B ended: ${b.outcome}  <-- resumed sessions die within seconds of the next utterance`);
}
process.exit(0);