I'm not having fun. An internal error has occurred

Hello, everyone. I want to talk about a recently encountered issue with an internal error. I use Gemini 1.5 Pro for working with large contexts, and it has always been a step ahead of its competitors. Its capacity of 2 million tokens enables the execution of a wide variety of tasks, which was what initially attracted me to this neural network.

During my work with Gemini, there were occasional issues, such as blocking certain normal foreign words (e.g., 디자인이, 이유나, and others) or the meaningful context threshold (800,000 tokens, beyond which severe hallucinations would occur). However, using certain techniques, these problems could be circumvented, and Gemini’s advantages outweighed its shortcomings. Unfortunately, this happy time has come to an end.

One gloomy morning, I discovered a new update to AI Studio. Some experimental models had disappeared, Gemini Pro 1.5 002 was downgraded to just Gemini Pro 1.5, and other changes were made. As usual, I submitted my standard prompt (600,000 tokens of context) using Gemini Pro 1.5, only to receive the message: “An internal error has occurred.” The exact same prompt had worked consistently up until that day, but now it no longer does. I tried tweaking the parameters, but it didn’t help. I switched to Gemini Flash 1.5 and encountered the same result.

I thought to myself, “Well, it’s time to switch to the next generation.” So, I moved to Gemini Flash 2.0, and surprisingly, the request went through. However, with every second of output, my mood kept sinking lower and lower. Unfortunately, Gemini 2.0 delivered very poor results, with numerous hallucinations and a poorly structured response. According to metric measurements, Gemini 2.0 has lost its main feature: working with large contexts. (Metric: MRCR, I’ll attach an image below).

Gemini Pro 1.5 had a result of 82.6%, while Gemini Flash 2.0 scored only 69.2%.

I don’t know what happened or how it works, but I’m asking for this to be fixed. Gemini’s main advantage has been lost. ((((

2 Likes

It is incredible that were getting reasonable responses at 600k tokens, I have scarcely been able to get beyond 200k from back and forth iterative work with Gemini 1.5 before it simply errors out let alone begins feeding me back hallucinations. Any time I have attempted to submit anything larger than 100k, it just spits me back out an error right away (API or in AI Studio). This eventually conditioned me such that I don’t often try such large single bursts of tokens, if I need to feed it something anywhere near 100k, I manually break the code up into different prompts after using code2prompt to render the codebase as a single markdown document first. Doing this, things usually work or at least I don’t get immeadiate errors this way.

I very much like the idea of the large context window and overall it is possible with Gemini to get coherent responses on large token values of information, but its not something that I have had the experience of it being fully realize at present, though still better than the competition on this front and its reasoning/coding ability far surpass anything else I have gotten my hands on thus far. Also radically easy to work with its API in python, but I would really appreciate the token window to be more realized in the future as my cursory testing of 2.0 on this front just now indicates the same errors seem to emerge.

Hello!
@vadimor , Which is the Prompt that previously worked on 1.5 and on 2.0 returns prompt results?

@Thomas_Leon_Highbaug , how much larger is your expectation of the token window to currently be?

1 Like

@nguadiana

https://aistudio.google.com/app/prompts?state={“ids”:[“1xQEZ6ndHD-GMnMmUQSjYN_unau9i0Uf3”],“action”:“open”,“userId”:“115821622235527330732”,“resourceKeys”:{}}&usp=sharing

Gemini Pro 1.5 002 was downgraded to just Gemini Pro 1.5

I think this was just a rename. They removed 001, and started calling “1.5-002” 1.5.

Thanks for sharing the link… It looks like that query works now. It doesn’t give an internal error at least, I can’t confirm the accuracy.

Yes. Seems like problem was solved.