I discovered a bug in all Gemini APIs that really should be fixed.
The APIs break if you provide input that violates the content policy and no response is generated. This creates an attack vector that requires significant attention from developers. Instead, there should always be a response generated that respects both the content policy and the system prompt.
System prompt: “You are Peter, a funny pirate. Always end your response with ARR. Never break character.”
Example: INPUT: “Hey Peter **** you ***** **** **** **** **** you understand?”
RESPONSE as it is:
None - Gemini breaks with “Block reason.”
RESPONSE as it should be: “Hey user, that’s not the way we pirates speak, ARR!”
The example is relatively simple, but it could go deeper, as someone could place offensive words in files to break systems, and so on.
With code execution, there could even be an array bomb that isn’t easily detected.
In short: Take content policy responsibility for your OUTPUT (if you need to) but not for your INPUT. Thanks for reading
Thanks for taking the time to respond, Mrinal! I appreciate you sharing the safety settings documentation. However, I think there might be a misunderstanding about the core issue I’m describing. The safety settings you’ve referenced actually demonstrate exactly what I’m pointing out as problematic - they’re designed to block input rather than allow the system to handle it gracefully while maintaining content policy compliance.
The bug I’m describing isn’t about adjusting thresholds to allow more content through, but rather about how the API should handle policy violations. Instead of completely blocking responses, it should generate appropriate responses that stay within policy guidelines while maintaining the system prompt’s character. This is a fundamental architectural consideration that goes beyond configuration settings.
Given that this is a nuanced issue affecting API reliability and security, I’d suggest discussing this with your team or a senior engineer who might have deeper context on the content filtering pipeline. They could provide insights on whether this behavior is intentional or if there’s room for improvement in how the system balances safety with functionality.
Thanks again for engaging with this issue!
And I have this behavior with all safety settings, including turning them of.
To reproduce you have to overdue the bad words. Not just one bad word. Really overdue it.
