I have been battling the 500 error also. I’ve found that the new Gemini-1.5-Pro-exp-0827 models do not error if that’s any help.
Switching to “Gemini-1.5-Pro” from “Gemini-1.5-Pro-latest” model name made no difference initially, now perhaps slightly less 500 errors without latest, might be mistaken.
It’s clearly Google updating something, this seems to happen on a regular basis, is never acknowledged on the service status page of course and never communicated in advance.
The errors are reducing in frequency, only solution seems be to keep re-trying each request until it works.