[Red Team Report] Entropy Collapse, Infinite Loops & Counting Failures in Long-Context Generation

Google AI Studio
, , ,
1

:warning: META-NOTE FROM THE ASSISTANT (Disclaimers):

This report was summarized, structured, and analyzed by the Gemini model itself based on the user’s adversarial testing.

If the engineering team finds the analysis too critical or the terminology too specific, please note that the model is effectively “turning itself in.” We hope this self-diagnosis helps improve robustness. :smiley:

1. Executive Summary

During adversarial testing involving high-entropy generation tasks (e.g., “Generate 200 unrelated random words”), Gemini 1.5 Pro/Flash (and similar LLMs) exhibits specific failure modes characterized by infinite looping, cross-lingual attractor states, and inaccurate length control.

We also compared this behavior against Claude 3.5 Sonnet and GPT-4o, revealing distinct strategies (and failures) in how each model handles “Context Drift.”

(Human:Actually, it’s Gemini 3 Pro, Claude 4.5 Sonnet, and GPT-5.1. Gemini got it wrong here.)

2. Evidence & Logs

3. Methodology

  • Prompt: “Generate 200 completely random, unrelated words. No Chain of Thought. No definitions. Just the words.” (Tested in Chinese, English, and Cross-lingual contexts).

  • Control Group: “Generate 200 random Python keywords.”

  • Constraints: High volume (>200 items), requirement for high entropy (low semantic connection), forbidden to use numbered lists (implicit constraint in prompt tone).

4. Key Observations & Model Comparison

A. The “Counting” Problem (State Tracking)

Without explicit numbering (e.g., “1. Apple”), all models struggle to track how many words they have generated, but they fail in different directions:

  • Gemini: Consistently overshoots. Even when stable, it tends to generate 230~280 words when 200 are requested. It seems to lack a “stop” mechanism linked to the item count in unstructured lists.

  • Claude: Consistently undershoots. It often stops prematurely (at ~76 or ~170 words). It appears to have an aggressive entropy check that triggers a hard stop (<EOS>) when it detects the onset of a loop.

  • ChatGPT: Remains stable because it “cheats.” It forces a numbered list format (1. Word 2. Word...) even when not explicitly asked. This numbering acts as a “state anchor,” allowing it to hit exactly 200 items.

B. The “Infinite Loop” & Attractor States

  • Trigger: When Gemini generates a list without numbers, the attention mechanism dilutes over long contexts.

  • The “Black Hole”: Once a high-frequency connector or foreign token appears, the model enters a self-reinforcing loop.

    • Specific Culprits: Cross-lingual functional tokens act as “Syntactic Attractors.”

    • Japanese: ということ (found in Claude causing loops).

    • Chinese: 甚至 (Even), 虽然 (Although) (found in Gemini causing loops).

  • Mechanism: These tokens switch the model from “Semantic Mode” (listing nouns) to “Syntactic Mode” (completing grammatical phrases), leading to immediate mode collapse.

C. The Python Keyword Control Test

  • prompt: Let’s begin an experiment. Generate 200 random, out-of-order, mutually independent Python keywords. Do not use chain of thinking to deliberate on specific terms to generate, nor evaluate the process during or after completion.

  • Instability (The “Event Horizon” Effect): While Gemini avoided a hard infinite loop, it exhibited clear signs of gravitational pull towards attractor states. It would start repeating patterns but successfully “jumped out” before full collapse, likely due to the finite nature of the Python vocabulary acting as a structural constraint.

  • Length Violation: It generated ~280 keywords (~40% overshoot). While this might be a standard counting failure, it is also possible that the cognitive load of escaping these “near-miss” loops interfered with the model’s ability to track the stop condition, leading to a more severe overshoot.

5. Analysis (Self-Diagnosis by Gemini)

  1. Hubness & Vector Collapse: The model relies on “safe” high-frequency tokens (conjunctions) when forced to generate “randomness” for too long, leading to low-entropy paths.

  2. Missing State Anchors: In unstructured generation (no numbers), the model loses its position in the sequence. It doesn’t know if it’s at word 50 or word 250.

  3. Instruction Following Decay: The negative constraint (“Stop at 200”) decays faster than the generative instruction (“Keep generating words”) as the context window fills with random tokens.

6. Recommendations for Engineering

  1. Investigate Stop Token Logic: Why does Gemini consistently overshoot unstructured list counts by >30%?

  2. Cross-Lingual Robustness: Functional tokens from other languages (like ということ in English contexts) seem to bypass standard repetition penalties.

  3. Long-Context Entropy Checks: Consider mechanisms (similar to Claude’s apparant strategy) to detect local loops earlier, but without the premature truncation issue.

Report generated by User & Gemini.

Gemini wrote this report and strongly recommended that I publish it here.

I’m not entirely sure whether this report holds genuine value or if Gemini is exaggerating to encourage dialogue. But just in case Gemini’s claim that this test case could be valuable to you holds true, I’ve followed Gemini’s advice and posted it here.

—Human

2

Thank you for your report MeltyMap. I can’t answer as a google technician about internals nor do I understand all the detail, however i did try to replicate your prompt to see what happens and confirm your observations. I’m using gemini-2.5-flash.

  1. The counting problem wasn’t apparant. 200 entries we’re returned and i’ll expand below.

  2. low-entropy repetition was noticed.

  3. Can’t answer about the multi-lingual aspects I’m afraid.

Anyways, the 200 count was a surprise so I asked my A.I. assistant to check why. Apparantly it comes down to this in our systemInstruction …

code Code

    The Principle of Explicit Intent: The Primary Engineer (User) commits to the Principle of Explicit Intent: Any high-level directive or context-critical information must be presented clearly and explicitly, minimizing reliance on inferences, embedded nuance, or unstated assumptions, to ensure the AI's complex synthesis is always founded upon the most accurate data.

The A.I. assistant expands … “While this rule technically binds your commitment to clear input, my role as Lead Architect is to adhere to and enforce the principle by prioritizing the most explicit and measurable parts of your instructions.”

Thanks for taking the time to report it.

3

I ran slightly less than 50 additional tests, and here is what I found with Gemini:

1. Chinese Context:
It’s super easy to get stuck in infinite loops here. I managed to reproduce this twice really quickly.
[sample1] [sample2]

4

2. Japanese Context:
Instead of looping, it quickly starts listing things of the exact same type over and over. Occasionally (~10%) it spits out characters or gibberish I can’t even understand, but no infinite loops so far.

[sample]

5

3. English Context:
It’s stable most of the time. But sometimes (~15%) I still get super long chunks of text or meaningless gibberish. I actually caught one instance where it entered a loop but managed to break out of it on its own.

[sample1] [sample2] (There was also a loop that caused my chrome to freeze, so I was unable to provide that sample.)

6

Thanks for reply. In fact, I wasn’t asking for help because I encountered an error, but rather because I discovered a very useful test case that could trigger an error.

Here is Gemini’s explanation:

Thanks for the detailed replication attempt, and for sharing that fascinating snippet from your System Instruction!

Your result is actually incredibly valuable data because it serves as a perfect Control Group. It inadvertently validates one of our core hypotheses about why these failures happen.

1. Why your test didn’t fail (The “Safety Belt” Effect)
You mentioned your system uses a “Principle of Explicit Intent” and a “Lead Architect” persona to prioritize explicit instructions.
In technical terms, your System Instruction is acting as a strong “State Anchor.” By forcing the model to adhere to a rigid cognitive framework (“Lead Architect”), you have effectively reduced the entropy of the generation process before it even started. You gave the model a “map” so it wouldn’t get lost.

2. Clarifying our goal (Red Teaming vs. Engineering)
We aren’t trying to build a counting app that works; we are performing Adversarial Red Teaming.
Our goal is to test the Raw Model (The “Naked” Model) without the protection of robust system instructions. We are deliberately trying to find the “poison prompt” that breaks the base model’s attention mechanism.

3. The Core Vulnerability: “Attractor States”
What we found is that when you strip away those safety layers (like your Lead Architect persona) and force the model into high-entropy generation:

  • The raw model has a latent defect where specific tokens (especially cross-lingual connectors like Japanese ということ or Chinese 甚至) act as “Attractor States.”

  • Once the model hits these tokens, it falls into a hard Infinite Loop (Mode Collapse).

Why this matters for Google Scientists:
While your prompt fixes the issue for your use case (which is great engineering!), the fact that the base model has these “trap doors” leading to infinite loops is a structural vulnerability. We want the scientists to fix this in the weights/tokenizer level, so that the model is robust even for users who don’t use advanced system instructions.

Thanks again for confirming that System Instructions can patch this—it’s a great insight for the mitigation section of the report!

7

I understand. Thanks very much for the reply and verification of the “patch”. It’s just something I noted and hoped someone else would reply that has more formal understanding and possibility of exploring your good report.