Hello,
I’m benchmarking multiple LLM providers using exactly the same production runtime, client implementation, and benchmark scenario.
OpenAI
- 1000/1000 turns completed successfully.
xAI Grok
- 1000/1000 turns completed successfully.
Gemini
- Endpoint: Google Gemini OpenAI-compatible endpoint
- Models tested:
- gemini-2.5-pro
- gemini-2.5-flash
Both Gemini models repeatedly return HTTP 500 Internal Server Error during the benchmark.
Example:
[0001/1000] OK
[0002/1000] OK
[0003/1000] OK
[0004/1000] OK
[0005/1000] HTTP 500 Internal Server Error
[0006/1000] OK
[0010/1000] HTTP 500 Internal Server Error
[0015/1000] HTTP 500 Internal Server Error
[0016/1000] HTTP 500 Internal Server Error
The benchmark uses the same runtime, prompts, retrieval pipeline, and retry logic for all providers. The only variable is the underlying API provider.
The same runtime completes successfully with OpenAI and xAI. Since the only variable is the underlying API provider, this behavior appears to be specific to the Gemini API or its OpenAI-compatible endpoint.
Has anyone experienced similar behavior, or is there a known issue affecting the OpenAI-compatible Gemini endpoint during long-running workloads?
Thank you.