This is not going to sound nice, but it’s because they really wants you to use Vertex AI.
You can compare the gemini 1.0 pro on makersuite vs gemini-1.0-pro-002 on Gcloud.
Hi Harry, welcome to the forum. I think the issue of the newest model generation as of a few days ago causing specific performance problems to specific applications is equally prevalent in the Vertex environment.
Specifically, the previous version gemini-1.5-pro-preview-0409 is really gemini-1.5-pro-preview-0409 (points to and uses gemini-1.5-pro-preview-0514 ) according to the documentation
Simply put, the users on Vertex are experiencing the same problem, unavailable accessible previous version.
Yeah, Vertex AI at the end of the day is another medium to call the same thing.
lol, you can say that again! I’ll admit, it’s bizarre to see docs actually talk about things ahead of the development pace. Usually it’s the other way around.
have you made any progress here? Honestly, this is what I would do for right now. I would just go down the list of images, make descriptions for each (ideally as separate single shot prompts), and see if it’s “off”, or if it’s relatively on the mark. We could use this data in a few ways. If its accuracy mirrors something closer to the 42%, that essentially confirms the model performance problem. If it’s higher and closer to the original %, then we may be able to adjust other aspects of the prompts and data to get that accuracy back up. If it’s completely off from either or, we might have consider a fundamental re-design of your approach to this problem in order to accommodate whatever is going on.
Thank you for sharing us
I’ve tried it and seems like Gemini does have problems seeing the content in each video frame. For example, in the screenshots attached, Gemini was asked to describe the speaker’s gesture in a few frames (randomly picked). When the video frame does not show any gesture/hand, Gemini tends to make something up, which is very weird…
Please see the screenshots below:
hey @May2024 would you be able to retest this with gemini-1.5-pro-latest again? And also the 0801 experimental version?
I am curious to see results
I tested it again in July, but no change. I can retest it again. Has it been updated recently?
The -0801 experimental model is quite new. And in a possibly related task, it did much better than Gemini 1.5 (both flash and pro) when prompted about left, right, up, down. It would be interesting if that carried over to better gesture recognition.
I believe they do test changes under the “-latest” suffix based on unusual response style changes I’m seeing every once in a while
I think it would be worth testing, and ofcourse for 0801 experimental version too
I am seeing mixed results from my testing with the 0801 version. it’s definitely better for Video, audio, and image. But, I am seeing some consistent regressions in Text in some areas.
Hey @Exo-1 @OrangiaNebula , I tested both, and the results are better. The accuracy increased to 0.526 (1.5-pro-latest) and 0.596 (1.5-pro-exp-0801).
However, it seems 0801 has strict limit even for paid users (50 per day). Do you know when they will increase the limit for 0801?
Here’s the screenshot:
2 Likes
sadly they only increased the limits in aistudio for 0801 experimental version
maybe we’ll get an update on things next week. heard something is happening but not sure what exactly…
ok… let’s wait and see. thx for letting me know!!
I use Gemini for translations because I need extensive context. I’ve noticed that Gemini has become much less accurate. If it used to consistently translate certain parts, now it’s impossible to get the desired results. I hope they roll back the update.
A few thoughts for all of these reports:
- Stating exactly which model you’re using will help them identify if there are problems. And by this I mean the exact string you’re using when creating the model. Something like “gemini-1.5-flash-001”. Saying “Gemini Flash” helps less than you think.
- You should be using a specific model number. Again, “gemini-1.5-flash-001” rather than “gemini-1.5-flash”. In theory - once it has a trailing number, it shouldn’t change.
- Examples that illustrate the issue go a long way to identifying what is going on. There isn’t much they can do with a generic report. But if you show examples of behavior that was good before (with details) and bad now (again, with details), they have at least a starting point.
All reports are good - but the more information you can provide, the better the chance that they can actually do something about it.
1 Like
This page Model versions and lifecycle | Generative AI on Vertex AI | Google Cloud theoretically states the stable versions numbers and this matches the list models call. Last stable version is 001, release May 24, 2024, which seems like a long time between drinks.
List models from curl https://generativelanguage.googleapis.com/v1beta/models?key=XXX
“name”: “models/gemini-1.5-pro-latest”, “version”: “001”,
“name”: “models/gemini-1.5-pro-001”, “version”: “001”,
“name”: “models/gemini-1.5-pro”, “version”: “001”,
“name”: “models/gemini-1.5-pro-exp-0801”, “version”: “exp-0801”,
“name”: “models/gemini-1.5-pro-exp-0827”, “version”: “exp-0827”,
1 Like
I think they’ve changed things for -latest to point to latest stable, I think at one point it was getting latest beta