I want know how different the new experimental model is, I did some testing but couldn’t tell if it was better or how it was different. Is it a iteration on the 1.5 pro model or is it more closer to the flash models? Please let me know what you guys think if you’ve tested it out.
The Gemini-exp-1114 model is definitely Pro class, not flash class. Based on the limited testing (no one has had more than 100 interactions with the model two days after it was released because of the 50 rpd limit) it performs well on “reasoning” tasks. The model evaluates its utterances and reflects on them, which often results in self-correcting behavior. You will see that introduced as a new section that starts with “However…” or something equivalent — that’s the model dumping a solution path that wasn’t working and trying something else.
Gemini-exp-1114 is the first Gemini model that managed to correctly map the 2-input logical expression ¬p ∨ q using only NAND gates. That doesn’t sound like a hard task, but every Gemini up to and including Gemini-1.5-pro 002 fails at this task. There are several other tasks which Gemini-1.5-pro 002 could only complete with a second prompt to nudge it towards a solution, and which Gemini-exp-1114 handles one-shot.
The models/gemini-exp-1114 entry has very recently appeared in the list_models output, so we will be able to use the API with proprietary data now (the first couple of days it was only possible to access the model through AI Studio and that means the data used for testing becomes eligible to be used for next generation model training).
Hope that helps a bit.
5 Likes
Hm, to start with it has a very limited input context of 32k tokens only.
Yeah i’ve noticed that, that’s a big downside. Hopefully they extend the token limit so i can test more of my longer prompts