We were billed for a very large amount of Gemini API usage, but we cannot identify any activity on our side that would explain this traffic. Starting on August 12th, the usage gradually increased and then on August 17th skyrocketed and it reached nearly $5,000 per day ($40,000 in total).
We have various models in the billing statistics that we have never used. Mostly expensive models!
The only way we were able to stop the charges was by completely disabling the Gemini API.
This looks like it may have been caused by an issue on Gemini’s side rather than our usage. Can you please investigate and provide clarity on what generated these charges?
Thank you for your urgent assistance.
We can provide all what is needed.
I apologize for the loss.. As you have mentioned that there are various models in billing statistics, it looks to me that one of your API keys associated with your GCP project might have leaked and the hacker must have increased usage which explains models listing in your billing statistics.
Architectural Patterns to prevent such issues in future
Establish OAuth and frequently refreshing API keys pattern (simple) OR completely avoid access to API key access to project users by using use OAuth + Proxy + token + refresh token pattern.
These approaches help you gain user level tracking with authentication and prevents exposing API keys thereby completely avoiding leaks.
Just to give you a glimpse of all this mess… As you can see there are tons of tokens used for input on gemini pro 2.5 but not just input… the long input!
There is no output related to this amount of this much input data… We need a clear explanation of this. You have the tool (API endpoints), it’s your job to make sure you can explain key leak or whatever you say it is.
Isn’t that pretty basic? Google certainly turns off your ads when you have exhausted your Google AdWords spend. Seems like a very similar situation.
I would suggest that anything which has the ability to bill you due to automated processes needs a few safety features. It should support:
Spend cannot exceed $X in Y time (could be dollars a day, but it also could be an hour or a minute – better to catch problems early)
Spend cannot exceed monthly total of {total].
Messages to the account holders email and smart devices should be triggered at at least these caps, and ideally would be configurable to alert you before you reach the cap.
IP addresses should be reported.*
*Privacy concern? Not really. No one else has permission to use the key. If they are trying to hack my account, I think we can all agree that they have given up their right to privacy. In the case where it turns out to be someone in my company, then the agreement signed with Google should make it clear that this data can and will be revealed in the event of a charge dispute.
These things are so obvious, and yet if they are not being done, as the OP said, it makes you wonder if it isn’t on purpose.
So what would be the solution for ALL of us? Be able to set a hard daily limit on the API instead of a soft budget reminder. Can you give us a reason why this wouldn’t be considered as a standard feature in the near future.
Come on Google - you need to help out your community here. You want us developers to be trying out new features and functionality without this kind of fear. Please - a simple - cash value cap on developer account - a kill switch at a value determined by me - eg $100 per month as a default - but customisable up or down as preferred by the developer. This would not be difficult to implement. I understand the many arguments against this in prod - but for a dev account - developing - and by implication venturing into the unknown / likely to make some mistakes / misconfigurations, and even for some not being aware enough of how to protect their API keys. I think Google has a massive opportunity right now - with the expansive and truly top notch products across the whole stack (including AI). To make it easy for developers to standardise on your stack (and hence turn that into more paying customers down the line). This one doesn’t seem like rocket science to me. Interested on thoughts of other developers currently learning. experimenting - and hence choosing their tech stack for this AI revolution.
This is definitely a serious bug. That has huge business implications. I think they reverted some costs in August, but it has again started.
We are just testing our concept on a single machine, still you can still check out that the number of hours they calculated is way higher and unrealistic to comprehend.
In these screenshots, you can see the latest screen is not showing up with high charges for 7th and 8th Sept anymore.
