Replying to your post specifically cos its fully relevant to the bits below, and your in a very similar position regarding seemingly false positives. I also just realized its a super long one, massive bad on my part for that.
Going a bit further on my previous posts about the gambling loop and the paid quota drain especially for those of us on premium subscriptions or active API projects, I’ve been doing some rather limited reading given I barely understand this stuff on consumer protection laws, standard form contracts, and how they apply to tech platforms. And I will say right now, I am utterly out of my depth but just wanted to further bring it up given the clear lack of communication from above on the ~19’th most viewed thread in this section. Also yes I did use Gemini to assist me figuring this out which is some level of Irony I’m sure.
I’d love to get some input from anyone here with a background in IT contract law or consumer advocacy, because the more I look at this from a regulatory perspective, the more it seems like Google’s current architecture is walking a very dangerous legal line.
If you look at the laws in jurisdictions like Australia (under the ACCC/Australian Consumer Law) which I know is extreamly hardline on things like this or the EU, there are two strange and almost, glaring contradictions in how Google has designed this system.
Burden of Proof, Catch-22:
Right now, if we dispute a charge or a lost quota due to a false positive post-generation “hard-wipe,” Google’s support structures from my understanding of the privacy for API and sub can’t actually help us. Given that privacy protocols ‘should’ prevent them from reviewing our prompts or logs to verify the block?
But from a trade practices standpoint doesn’t the party imposing a financial penalty bear the legal burden of proof to justify it?
If someone chooses to withhold a service we paid for due to an alleged policy violation, they must be able to substantiate that violation no? If their own privacy architecture rightfully prevents them for data privacy reasons from verifying or showing us evidence of a breach, how can they keep the money? From what little understanding I have of standard consumer guarantees, if a merchant cannot prove you breached a policy, they must default to a refund. They can’t just keep the cash and say, “Our hands are tied by our own privacy settings.” But at the same time, if they do allow the refund, that also opens them up to fraud and abuse from ill intended users because no one can reasonably prove beyond screenshots and such that the block took place. Which is part of why I say catch 22.
Who is actually responsible?:
This is where the logic completely falls apart for me. If you read the official Gemini API Additional Terms of Service, Google is extremely clear about who carries the legal risk?
Under Section 1 (Use Restrictions), it says: “You are responsible for determining the necessary and appropriate safety settings and factuality tools for your use case.”
Under Section 3 (Use of Generated Content), it states: “You’re responsible for your use of generated content, and for the use of that content by anyone you share it with.”
So, on paper, Google legally washes its hands of the output. They shift 100% of the responsibility, safety tuning, and liability onto the developer or user.
But on the platform, they do the exact opposite. They ignore our chosen safety settings, run a secret backend filter, delete the output post-generation, and still charge us for the computing power.
How can a contract say “You are responsible for deciding your safety settings and managing the output”, while the software says “We are overriding your settings, deleting your output, and keeping your money anyway”? They are having their cake and eating it too, shifting all the legal liability to us, while keeping absolute veto power and our cash on the backend.
(Edit) I was just re-reading the Generative AI Prohibited Use Policy and noticed another strange loophole that seems to uniquely penalize paying users and private developers.
The policy says Google may make exceptions for content based on:
“…educational, documentary, scientific, or artistic considerations, or where harms are outweighed by substantial benefits to the public.”
Because of the word “or,” these are separate categories, but in practice for private developers, both of these exceptions are functionally useless:
The Artistic Exception is a Black Box: Because an automated backend filter that may or may not work on keywords or another LLM model has no visible technical capacity to evaluate “artistic considerations,” Category A is effectively a dead letter for real-time API calls. Given that an LLM does not by definition have the ability to even understand what artistic considerations are. As they are highly subjective.
The Public Loophole Locks Out Private Projects: If you are using the API or your Advanced/Ultra subscription for private, personal creative use—like running private D&D campaigns, solo text-adventures, or offline creative brainstorming there is no public to benefit. Category B does not seem possible to apply to private users.
What is worse is the bizarre double standard this creates between Paid and Unpaid users:
Unpaid Tier: Google anonymizes data for human reviews, stripping your Account ID and Project ID. Free users face zero financial penalties for false blocks, and their account identity is shielded from manual review.
Paid Tier: Google keeps your prompts linked to your billing account for abuse detection. If their black box filter falsely flags your private project, Google not only charges you for the blocked run, but also has your direct billing identity ready to ban or suspend your entire Cloud Project.
Paying developers are seemingly uniquely penalized. We are the ones paying for the false positives, we face direct account-linked tracking, and we potentially face immediate platform bans, while free-tier users can simply cycle anonymous accounts with zero stake beyond having some form of private data tracked, pc, internet ip or what have you. (Edit)
Solution Doesn’t Require Privacy Violations:
Some might argue, “But Google has to block bad content, and if they just refund every block, users will abuse the system to get free computing.”
But do they actually need to log our data? And we shouldn’t realistically need to report the failures anyway based on most other similar systems I’ve used.
Google’s own server determines when a block occurs and outputs a specific system-level metadata flag. The billing engine doesn’t need to read our text, It only needs to read that binary metadata flag or whatever they use to identify that block. If the flag is SAFETY, OTHER, or whatever backend status they use to identify these blocks meaning Google’s server chose to destroy the output post-inference the billing engine could read that status flag to automatically set the cost of that specific run to $0.00, or instantly reinstate the consumed subscription quota. Though I really don’t know on the quota bit.
If this is even correct, again not a backend developer or anything. But because the user has no control over Google’s serverside flags, I would assume there is almost no risk of user fraud, and we shouldn’t have any privacy risk because no human ever reads the prompt.
(edit)
I was just playing around with the AI Studio interface and found a massive technical loop that I think completely exposes how broken and unfair this billing/wipe system is.
When you stream a long response, the model outputting text chunks in real-time is paid for and rendered on your screen. When a post generation block triggers, the system “hangs” the generation for about half a second to a full second at the end before it executes the hard wipe and replaces everything with a blank Content Blocked box.
But here is the catch: If you are fast enough and hit Stop Generation during that last second or so of hang time, the stream cuts. Because the stream was terminated, the wipe command fails to execute, and you keep the entire generated text up to the point of the block if you are fast enough.
This completely exposes the reality of what Google is doing here:
The compute is fully completed and delivered: The fact that we can rescue the text by hitting stop proves the model successfully generated and delivered the data safely to our machine.
They are actively deleting a finished product: Google isn’t “preventing” unsafe generation. They are letting the generation finish, calculating the token cost to bill our account, and then sending a post-inference command to reach onto our screens and erase data we’ve already paid for.
The Double Standard: This clunky client-side wipe penalizes paying users. On the paid tier, our prompts are linked directly to our billing accounts for “abuse detection” meaning we face direct account-ban risks if a buggy algorithm false-flags us, while free-tier users get their data anonymized and face zero financial stakes for false blocks beyond a potential account ban if done.
If the data has already been physically streamed to our machine, the transaction is complete. Reaching onto our screens to delete it afterward while keeping our paid tokens/quota is an incredibly clunky and unfair way to handle safety. But it already shows a system could exist where the response is cut off at that point of block rather than destroying the entire response. I don’t know how hard it would be to change the hard nuke into a stop generation but it must be possible. Is anyone able to replicate this? (Edit)
This might be a regulatory risk for Google:
The fact that they previously operated a system that seemed to work perfectly fine with 1.5, 2.5 and all those, doesn’t this also show they are technically capable of running a fair system?
Choosing instead to deploy this hard-wipe that processes the tokens, charges our accounts, and then deletes the output while refusing to credit us because of self-imposed support limitations feels like a textbook case of unconscionable conduct and unfair contract terms.
Has anyone else looked into filing complaints with their local consumer protection bodies like the ACCC for any Australians? Because if Google’s billing team is relying on “it’s a black box, so we can’t refund you” as a legal defense, I don’t think that is going to hold up under scrutiny.
I’d love to hear if anyone from the Google team has any input on how this is currently being handled behind the scenes. Honestly I think any of us would just love to hear from any google dev in this thread at all given how long its been active and how many people view and use it.
. A daily reminder. Could you please soften the filters in Google AI Studio? A whole bunch of people have already commented on this in various places and forums. Why aren’t you doing anything about it? And yes, as was already correctly noted here, the filter’s severity seems to vary depending on the language.