Google releasing a ‘developer’ update prior to IO hoping that developers could use it to build more, which actually made us unable to build.
Didn’t they test the model before release?
While the newer version does follow instructions better. The march one was the first truly objective and critical model ive had encountered.
It literally didn`t handhold and gave critical honest feedback, even under wrong assumptions. Where all other models were just parroting what they thought the user wants to hear.
Google please keep the old models around. Always after finetuning for the masses “Modified by moderator” GPT4 shortly after release was better than what we got now. Claude 3.5 also better than 3.7 in many instances. Its just repeating over and over again. Humans always pick the feedback that makes em feel good and in return you get an emoticon pissing sycophant ai.
@Michael_Six, while I’m trying to keep this thread laser-focused on Google’s policy for dated endpoint names, your observations on model performance are too important not to comment on.
I’m right there with you on 03-25. It was the first model that genuinely pushed back in a way that felt truly meaningful. I’ve been a heavy user of LLMs personally and professionally since OpenAI’s earliest public models back in 2022, run billions of tokens through them, and I will never forget the first time a model made me question my own assumptions; that was 03-25. I pushed back, expecting the usual apology and course correction you get from every other model. It didn’t apologize. It doubled down, and it shocked me into realizing I was the one who was wrong. That kind of interaction is incredibly valuable, something I hadn’t seen before.
And you’re 100% right with this whole A/B testing and fine-tuning cycle. We’ve seen it AGAIN and AGAIN. The current chatgpt-4o-latest is, frankly, a mess compared to the base 4o model because of it. You’re not getting reinforced feedback from actual “experts” in reasoning, like OpenAI did with their ‘o’ series models; you’re getting reinforcement from what general users “liked” because it was agreeable and didn’t challenge them. I’d hate to see Gemini go down that same path. Fine-tuning, IMO, should almost never be for personality; let users handle that with a system message. Trying to fine-tune personality at this scale has massive, unforeseen side effects every single time.
On your point about instruction following, I have to somewhat disagree that 05-06 is better. My personal testing seems to lean toward it being worse over longer contexts. Now, this is totally anecdotal, and I have zero eval data to back up that personal claim, so take it for what it is worth, but I can point directly to the public benchmark for long fiction recall. That’s, in my opinion, an excellent indicator of coherent understanding over extended contexts. The 03-25 model hitting 90% at 120k tokens was groundbreaking and a real industry first. I confirmed it myself that it was just a benchmark score; it translated to real-world understanding at long context that was remarkably good, only really falling apart around 150k to 200k tokens.
If you check the chart, you’ll see gemini-2.5-pro-preview-03-25 and gemini-2.5-pro-preview-05-06 now show nearly identical performance. That’s because, as we know, the 03-25 endpoint got silently redirected to the 05-06 checkpoint. But as you can see, the 03-25-exp still shows the original, much higher performance, because it is still the original model. I’ve personally verified this by checking the response object from the Gemini API; that specific experimental endpoint is NOT redirected and still points to the actual March 25 checkpoint.
So the data clearly shows 05-06 is a significant regression in long-context understanding, dropping from an exceptionally outstanding model to something more in line with current SOTA averages.
I believe understanding over context translates directly to the ability to follow instructions over long context as well, to my point.
Apart from long-context problems, as you’ve identified, we’re also seeing substantially longer response times, combined with an alarming ability to ‘not think’, which all lead to increased hallucinations and far worse overall results.
We do have internal evals, and they’re now consistently failing in lots of cases. “Modified by moderator” They giveth with one hand while taking away with another 
A quick but important update: Logan Kilpatrick has announced a “temporary pause” on Gemini 2.5 Pro free tier API access, citing high demand.
This is significant because it effectively removes API access to gemini-2.5-pro-exp-03-25, which many of us confirmed was the actual, non-redirected March 25th checkpoint.
It’s hard not to connect this pause with the likely surge in usage on that specific exp-03-25 endpoint, once word spread that it was the true 03-25 model, especially after the main preview endpoint was redirected.
This continues to show the urgent need for Google to clarify its policy on dated endpoints. We still have no official word on their immutability or the conditions for redirection or removal, which continues to hurt developers’ trust and our ability to build reliably.
Hoping for clarity from Google soon.
Thank you @H_Express for this thread. I will keep this brief as I am not using the API at the moment, but I was planning to start building on it after witnessing the capabilities of Gemini Pro 2.5 (03-25).
For the past few weeks I have been using Gemini for machine learning research and its ability to push back on ideas and provide genuine technical insight struck me as a major leap ahead – for me the second “uh-oh” moment, the first one being with the original gpt-4. Gemini was acting as a competent collaborator in fairly technical fields (not just for coding). Since the 05-06 update, most of that is gone, as mentioned by @Michael_Six. The new model’s thoughts patterns and writing are not as sharp, and sometimes just confused in a way reminiscent of lesser models, with glaring attentional mistakes that muddle together information from different parts of the context or its own thought chain, something 03-25 would not do.
I started looking around and found this thread (together with others). Hopefully there will be clarity about what is happening, and 03-25 will be made available again.
Dear @Govind_Keshari
Are you able to provide any acknowledgement of this issue? Has this thread been attended to by the internal team?
Many Thanks
This is very unfortunate. I spent a lot of time working on our use cases in the area of legal text comparison with gemini-2.5-pro-exp-03-25. The results were great. Now things are broken. I hope there’s a way to get back API access to 03-25.
Yes, 100%
For myself, and for many others, the quality of the 3-25 model is what drove us to adopt it over other models we’d been using.
Redirecting silently to the FAR inferior 5-06 version, requiring significant rewriting of system prompts in order to restore some of the innate functionality of the 3-25 model, although still falling short, is beyond frustrating.
Please restore access to 3-25!
Since Google (and @Logan_Kilpatrick) continue to ignore this thread and refuse to clarify anything related to their new apparent policy of redirecting clearly dated endpoints, several of us have been forced to investigate alternative solutions on our own time.
I wanted to share a quick update about what we’ve found that might help others whose applications were broken by Google’s backdoor model switch (now that even the exp endpoint has been disabled on the Gemini API per Logan’s announcement).
The good news: We’ve confirmed that the original March 25th checkpoint IS still accessible, but only through the Vertex AI API, not the Gemini API.
The response object from Vertex does give you a model name identifier that seemed to confirm there is NO redirect in place, but instead of trusting it (since that is what this whole issue is about), I decided to do research to make absolutely certain.
Using a fixed seed number and temperature=0 in the API for as deterministic an output as currently possible, we ran tests comparing the output “signatures” (content, logprobs, token counts) of different dated endpoints on Vertex AI. Our findings:
-
NOT Redirected on Vertex: On Vertex AI, the
gemini-2.5-pro-preview-03-25endpoint produces completely different output signatures than thegemini-2.5-pro-preview-05-06endpoint. They are definitely NOT the same model. -
The
gemini-2.5-pro-exp-03-25endpoint on Vertex (when you can access it) produces output signatures IDENTICAL to the original March 25th model we all came to rely on. However, this endpoint appears severely rate-limited.
The result you, as a dev, care about: The preview-03-25 endpoint on Vertex is NOT redirected to 05-06, like on the Gemini API. exp-03-25 is rate-limited but confirmed to be IDENTICAL in signature to preview-03-25. Therefore, developers who need to use the original March model can switch to Vertex AI as a temporary solution, confidently using preview-03-25 in place of the exp endpoint.
@wowitsjack, @Ed_Godshaw - if you switch your apps to Vertex AI instead of Gemini API (straightforward in most SDKs), this could be your path back to stability until Google either fixes the 05-06 model issues or (hopefully) provides an official way to access the original 03-25 model again.
Is this ideal? Absolutely not. We shouldn’t have to do detective work to find out where Google hid a model that was working perfectly. The breach of trust regarding dated endpoints remains unaddressed, and Google’s continued silence on this thread speaks volumes.
But at least there’s a potential path forward for those whose applications are currently broken due to this sudden switch. I hope this helps at least one frustrated and confused developer out there.
i fully agree. I based my judgment of its skills purely on whether it could stop adding comments to code. The March version just couldn’t help itself - commented everything, drove me insane. Otherwise the best critical & visual understanding AI ive seen to this day.
Lets hope google reacts and keeps even older APIs accessable
Gemini with agentic workflows now is completely useless. Cant even get it to complete a mostly simply git commit. It’s utterly unusable.
I had to take my company over to Claude for the day, expensive as but not as expensive as not working for the day.
If Gemini can’t fix this before the end of the week we will not be using Gemini again, it was a big departure from Claude to come to it in the first place and now Claude is offering significant and appropriate pricing tiers that just fit us better if we are going to be dealing with this crap.
Its honestly such a Google thing to do. Make it bad in every iteration. The new 2.5 pro model just “Modified by moderator” big time. It can’t even do the basic tasks right now.
Yeah. And there I thought Google had done something worth seeing at last. Well, going back to Claude 3.7 now.
@John_Zealand-Doyle, @Anant_Prasad, @A_V - I get your frustration. The regression in the 05-06 model has been painful for a lot of us, especially those who built workflows and applications around the capabilities of the 03-25 checkpoint.
BUT hear this: I would encourage you to see this as a bump in the road, rather than a reason to abandon ship entirely. As someone else here pointed out, Google is operating on the bleeding edge, and these kinds of mistakes happen. The base model for the 2.5 series is truly groundbreaking; we’ve all seen this in both 2.5 Flash and the original 03-25 checkpoint. That isn’t debated by any of us.
Keep in mind that 05-06 is just a fine-tune, NOT a new model series. So remember what that means in the big picture. Google has at LEAST half a dozen competing fine-tunes running internally right now (visible on LM Arena). See this chart:
“Claybrook” (now known as 05-06) was likely chosen as the “I/O event model” because it was tuned for “single shot” UI generation and app building, things that look impressive in demos and benchmarks but don’t necessarily translate to real-world developer needs or user satisfaction.
I’m confident that six months from now, Google and Gemini could still be leading the pack with these mistakes corrected. They have the technical capability to iterate and deliver versions that not only resolve these issues but push the industry to new heights.
What I’m less confident about, though, is whether they’ll take responsibility for the poor developer relations aspect of this whole thing.
When questioned on X about the intention for the Gemini API as a serious API, @Logan_Kilpatrick said, “We are super focused on having a world-class, production-scale, reliable API service. Lots of work to do still.”
If Logan or any Google employees are reading this: If you want us to treat the Gemini API as a truly world-class service we can rely on, and not the ugly stepchild of Vertex AI, you need to commit to treating it seriously and following standard practices and basic expectations like proper versioning. If dates don’t mean anything in endpoint names, what else is up for change? The model names too? Could gemini-2.5-pro suddenly become something else entirely? It seems absurd, but so did the idea of a fixed date becoming another date, and yet that just happened.
So my advice: stick with Google and the Gemini 2.5/2.0 series for now. Don’t jump ship yet. If 03-25 worked for you before, run it via Vertex AI as I outlined in my post above. It’s relatively painless to switch. The model issues will likely be fixed in future iterations. What you should watch for, to know whether to jump ship or not, is not the model changes themselves, but how Google handles developer concerns like this “RedirectGate” situation, and whether they give us confidence that we can take the Gemini API seriously moving forward, or if it’s just an “anything goes” platform we can’t trust or reliably build upon.
H_Express
I am already using Qroq, which I used before moving to Google, but sadly, suddenly, they are changing many rules, and now they are even deleting messages they do not like.
Hey everyone, thank you for taking the time to share your detailed feedback and concerns on this thread. We’ve heard your feedback on dated model stability and versioning and it will be taken into account moving forward.
@Vishal, thank you so much for acknowledging the community’s concerns here. After a week of silence, your response is genuinely appreciated and feels like a step in the right direction.
While I appreciate that your statement, “it will be taken into account moving forward,” represents some internal progress, this still leaves the developer community in a difficult position regarding how to build reliably on the Gemini API. We simply can’t confidently deploy applications without knowing whether the same situation might happen again next week, next month, or a year from now.
What would tremendously help restore developer confidence is a simple clarification, which I can distill into one key question:
Going forward, is it Google’s policy that dated model endpoints (like
gemini-x.x-xxx-preview-MM-DD) will remain stable pointers ONLY to the checkpoint from that specific date?Yes (dated endpoints always point to their namesake date’s checkpoint)
No (dated endpoints may be redirected to different dated checkpoints at Google’s discretion)
This is not about the past redirection itself anymore; we understand that is done. It is solely about establishing clear expectations for future development.
If that policy varies by tier (Experimental vs. Preview vs. GA), that is completely fine; just knowing the rules for each would give us the clarity needed to make informed decisions about which endpoints to rely on for different use cases.
That said, I can appreciate the fact that, as a product engineer, you may not be in a position to make a definitive policy statement on this. Even if you cannot provide a complete answer immediately, acknowledging this specific question would help us understand that our core concern has been properly communicated, while we wait for a formal clarification from someone who has that authority.
On a personal note of encouragement: you and your team have really built impressive technology with the Gemini 2.5 family. I think the strong developer reaction to this situation comes exactly because so many of us are excited about building with these models, but we simply want to do so with confidence, as we can with the clear, definitive policies of competitors like OpenAI, Anthropic, etc.
If we receive no definitive answer to the fundamental question above, the developer community will have no choice but to interpret that silence as a strong signal that Google does not intend to create an API in the same serious class as those competitors, and we will make our architectural and investment decisions accordingly.
Thanks again for engaging with the community on this.
Signed,
A Developer Building on Gemini
Yeah same here for coding at least, 2.5 Pro 03-25 was nearly as smart as o3 and much faster, I was using it 90% of the time for coding but in the last few days I had to move back to OpenAI models and stop using Gemini entirely. I wonder what the usage figures on Cursor look like now!
I was able to use exp this morning on Vertex and now I can’t. Is it gone?