Since Oct 29, about 6.5% of our paid image generation requests have stalled and failed with a deadline error. We reproduced it outside our production app. We’ve read the 429/503 guidance and are adding backoff and retries. This post is about the stalls and what changed around Oct 29.
Error
- API response: HTTP 503
UNAVAILABLE, “Deadline expired before operation could complete.” - AI Studio logs:
504 - DEADLINE_EXCEEDED,[original: RPC::DEADLINE_EXCEEDED] (qos=CRITICAL), 0 tokens
Production 504s (from AI Studio logs)
| Date | 504s | Requests | Rate |
|---|---|---|---|
| Oct 14–28 | 0 | 578 | 0% |
| Oct 29 (first at ~19:36 UTC) | 17 | 707 | 2.4% |
| Oct 30 | 52 | 702 | 7.4% |
| Nov 1 | 30 | 463 | 6.5% |
On Nov 1, the rate was similar on both models we used: gemini-nano-banana-2.1 at 6.4% and gemini-3.1-flash-image at 6.8%. The config and system instruction matched in the production requests we sampled, before and after the failures began. The status page’s Billed tier view shows no API incident on these dates.
Reproduction
We replayed the request 90 times with a standalone script and a 600 s client timeout: generateContent with two inline JPEGs, a system instruction, responseModalities: ["IMAGE"], thinkingLevel: MINIMAL, aspectRatio: 3:4, imageSize: 1K.
- 88 requests succeeded in under 14 s
- 1 succeeded after 115.3 s
- 1 failed after 127.5 s (logged Nov 1, 22:46:58 UTC, with no other traffic in the project’s logs during that run)
That image pair came from a failed production request, and it succeeded on 59 of 60 replays, so the input doesn’t appear to be the trigger.
Questions
- Did something change for this model around Oct 29, or is this expected overload?
- Why do these requests stall for ~2 minutes instead of failing quickly?
- What client timeout do you recommend, given that most requests finish in under 15 s?
- Should we treat the API’s 503 and the logged 504 as the same event?
We can share our project ID and log IDs privately.
Thank you for your help!