Why do AI models sometimes fail to follow instructions accurately, especially in image generation?

Gemini API
, ,
1

I’d like to open a discussion about a growing frustration when working with AI, especially in image generation.

These tools are incredibly powerful — but at the same time, the experience can be surprisingly frustrating.

Quite often, we give clear instructions, request specific changes, refine details carefully… and yet the AI:

  • Ignores parts of the request

  • Repeats mistakes that were already pointed out

  • Changes things that were not asked to be modified

  • “Drifts” toward a different style or interpretation

  • Doesn’t seem to respect previous iterations

After several rounds, it can feel like you are fighting the tool instead of collaborating with it. What should be a precise refinement process turns into trial-and-error, guesswork, and repeated corrections — which is mentally exhausting, especially in professional workflows where details truly matter (design, architecture, branding, visualization, etc.).

There’s a specific kind of frustration in clearly explaining a change… and seeing the AI confidently do something else.

So I’m curious about several things:

:magnifying_glass_tilted_left: Technical & workflow questions

  1. Is this mainly a technical limitation of current models?
    Does it relate to how language is interpreted versus how images are generated?

  2. How much does prompt structure actually matter?
    Does organizing instructions by priority, constraints, or “do not change” rules significantly improve results?

  3. Do AI systems handle iteration poorly?
    Sometimes it feels like every new image is almost starting from scratch, even when we are refining an existing result.

  4. Is this a creative vs. technical control issue?
    Are these models designed to interpret more “artistically” rather than with strict precision?

  5. Have you found techniques that improve model obedience?
    For example:

    • Layered prompts

    • Separating structure from style

    • Repeating constraints

    • Using visual references

Right now, AI feels less like a precise technical instrument and more like a creative collaborator that sometimes decides to improvise — even when you really need accuracy.

:thinking: A more critical/system-level angle

There’s also a broader question that adds to the frustration.

As users, we often end up:

  • Spending more time than expected

  • Making many more attempts

  • Using more credits or resources

So I wonder — not in a conspiratorial way, but from a product design and system perspective:

Are these limitations purely technical, or could there also be design choices that prioritize exploration, engagement, and prolonged usage over strict control and predictability?

In other words:

Are AI systems optimized more for ongoing interaction than for behaving like fully deterministic, precision tools?

I’d really like to hear others’ experiences.

Is this something that will improve with more advanced models?
Is it mostly about user skill and prompt design?
Or is this unpredictability an inherent trait of generative AI?

Because while the potential is amazing, the frustration during the process is very real.