Gemma4: How to prevent looping output in thinking?

Gemma
,
1

Hi, i’m running Gemma4 locally on Apple M2 with 32GB of Unified RAM. I am serving it with llamma.cpp. I am trying to use it with coding agents. No matter which coding agent I test with (Kilo Code, Pi, Continue, Aider), it almost always devolves into looping thoughts. I realize in the config below I am running an unsloth variant, but this issue happens on the original versions of the model as well, and on other variations and sizes that I have tried.

I’m reluctant to raise the repeat-penalty arg because I don’t want to reduce the quality of the output. Are there any recommended configs I am missing that would fix this?

This is my llamma config:

~/.unsloth/llama.cpp/build/bin/llama-server \
  -m /Users/cnrudd/.cache/huggingface/hub/models--unsloth--gemma-4-12B-it-qat-GGUF/snapshots/7102bdea62863acff919c945405ef29973113d66/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \
  --port 50305 \
  --ctx-size 262143 \
  --n-predict 32768 \
  --parallel 1 \
  --flash-attn on \
  --no-context-shift \
  --fit on \
  --jinja \
  --reasoning on \
  --chat-template-kwargs '{"preserve_thinking": true}' \
  --spec-draft-n-max  1 \
  --mmproj /Users/cnrudd/.cache/huggingface/hub/models--unsloth--gemma-4-12B-it-qat-GGUF/snapshots/7102bdea62863acff919c945405ef29973113d66/mmproj-F16.gguf \
  --temp 1.0 \
  --top-k 40 \
  --top-p 0.95 \
  --repeat-penalty 1.0 \
  --log-verbose \

Here is a screengrab of the looping:

Screenshot 2026-07-01 at 7.09.39 AM
Screenshot 2026-07-01 at 7.09.39 AM993×1105 228 KB

2

Try lowering temperature to 0.85

3

Thanks for the suggestion, but --temp 0.85 did not fix the issue.

~/.unsloth/llama.cpp/build/bin/llama-server \
  -m /Users/cnrudd/.cache/huggingface/hub/models--unsloth--gemma-4-12B-it-qat-GGUF/snapshots/7102bdea62863acff919c945405ef29973113d66/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \
  --port 50305 \
  --ctx-size 262143 \
  --n-predict 32768 \
  --parallel 1 \
  --flash-attn on \
  --no-context-shift \
  --fit on \
  --jinja \
  --reasoning on \
  --chat-template-kwargs '{"preserve_thinking": true}' \
  --spec-draft-n-max  1 \
  --mmproj /Users/cnrudd/.cache/huggingface/hub/models--unsloth--gemma-4-12B-it-qat-GGUF/snapshots/7102bdea62863acff919c945405ef29973113d66/mmproj-F16.gguf \
  --temp 0.85 \
  --top-k 40 \
  --top-p 0.95 \
  --repeat-penalty 1.0 \
  --log-verbose \

Screenshot 2026-07-03 at 9.19.45 AM
Screenshot 2026-07-03 at 9.19.45 AM1211×1152 281 KB

4

Hmm, that’s weird then, reasoning in gemma is extremely weird, are you using quantized model or not?

EDIT: Just checked, looks like there’s q4 in the name, that means it’s 4-bit quantization, it might be the problem. Also try putting temperature on 0.7

5

You can also try decreasing top-K to 30 and/or top-P to 0.9. I have no idea if it helps, just better to experiment with parameters

6

Thanks @Koeqaife
Before your latest suggestions, I ventured into adjusting repeat-penalty, and found that that this config is working well:

# Only non-default values are set here.


~/.unsloth/llama.cpp/build/bin/llama-server \
-m /Users/cnrudd/.cache/huggingface/hub/models--unsloth--gemma-4-12B-it-qat-GGUF/snapshots/7102bdea62863acff919c945405ef29973113d66/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \
--port 50305 \
--ctx-size 262143 \
--n-predict 32768 \
--n-gpu-layers all \
--no-context-shift \
--reasoning on \
--chat-template-kwargs '{"preserve_thinking": true}' \
--spec-draft-n-max  1 \
--mmproj /Users/cnrudd/.cache/huggingface/hub/models--unsloth--gemma-4-12B-it-qat-GGUF/snapshots/7102bdea62863acff919c945405ef29973113d66/mmproj-F16.gguf \
--temp 1.0 \
--top-k 40 \
--top-p 0.95 \
--repeat-penalty 1.08 \
--repeat-last-n 4096 \
--log-verbose

The key changes from before are:
–repeat-penalty 1.08
–repeat-last-n 4096

Same prompt as in before attempts returned a real answer, no looping. I’ve tested with several new prompts and haven’t seen the looping come back yet. I’ll follow up if I come up with more refinements.

7

Small update: I’m having quite a bit of success running this configuration on my M2 MacBookPro with 32GB of RAM:

llama-server \
  -hf unsloth/gemma-4-12b-it-GGUF:UD-Q8_K_XL \
  --host 127.0.0.1 \
  --port 50305 \
  --ui-mcp-proxy \
  --ctx-size 262143 \
  --n-predict 32768 \
  --n-gpu-layers all \
  --no-context-shift \
  --reasoning on \
  --chat-template-kwargs '{"preserve_thinking": true}' \
  --mmproj ~/.cache/huggingface/hub/models--unsloth--gemma-4-12b-it-GGUF/snapshots/3249fa54d5efa384afc552cc6700ad091efd5c39/mmproj-BF16.gguf \
  --temp 1.0 \
  --top-k 40 \
  --top-p 0.95 \
  --repeat-penalty 1.08 \
  --repeat-last-n 4096 \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  --log-verbose

I have not run into any looping in either Aider or VS Code’s Copilot Chat, where I set unsloth/gemma-4-12b-it-GGUF:UD-Q8_K_XL as the model with this config in
~/Library/Application Support/Code/User/chatLanguageModels.json

[
	{
		"name": "Local Gemma 4 12B Q8_K_XL",
		"vendor": "customendpoint",
		"apiType": "chat-completions",
		"apiKey": "llama-cpp-local",
		"models": [
            {
                "id": "unsloth/gemma-4-12b-it-GGUF:UD-Q8_K_XL",
                "name": "unsloth/gemma-4-12b-it-GGUF:UD-Q8_K_XL",
                "url": "http://127.0.0.1:50305/v1/chat/completions",
                "toolCalling": true,
                "vision": true,
                "maxInputTokens": 262144,
                "maxOutputTokens": 262144
            }
        ]
	}
]

I’m sure
“maxInputTokens”: 262144,
“maxOutputTokens”: 262144
being the same and both the max that gemma4 can handle is a little bogus, but I haven’t run into any problems yet.

There is some latency when I am loading large context, but it is not intolerable. I have been using it to edit python and typescript. The VSCode Copilot CHAT is doing well using the model in agent or ask mode so far.

I’ll report back again as I test out plan mode, and then executing those big plans.

8

I’m running the same model on a M1 MacBook Pro with 32GB of ram. I made an account. just so I could say thank you for sharing this! It’s dramatically changed things for me, thank you again.

9

i had a somewhat similar phenomenon. a looping was caused by a latency. i tried a smaller quantization so that the model and the context fit into my vram. the looping stopped. prior to this i also tried out some penalty and temperature adjustments. but the smaller model fixed the problem.

10

@SCP1762a That’s awesome! Thanks for letting me know, and providing confirmation that my configuration is not “just randomly working for me”. Good to know too that an M1 can have functional local AI with Gemma4.

11

Hello
How r u
Its temporally to create

12

I have moved to running this configuration so that I have a reasoning model and a tab completion model available in VSCode.

#!/bin/bash

# llama-server-start.sh

llama-server \
  --models-dir ~/.cache/huggingface/hub \
  --models-preset ~/workspace/mcp-server-workspace/presets.ini \
  --port 50305 \
  --log-verbose \
  --metrics \
  --ui-mcp-proxy

# presets.ini

# Presets for the Reasoning Model
[unsloth/gemma-4-12b-it-GGUF:Q8_K_XL]
alias = local-reasoning
n-predict = 32768
n-gpu-layers = all          # Offload all layers to GPU
flash-attn = on             # Saves VRAM on long contexts
no-context-shift = true
reasoning = true
chat-template-kwargs = "{\"preserve_thinking\": true}"
temp = 1.0
top-k = 40
top-p = 0.95
repeat-penalty = 1.08
repeat-last-n = 4096
spec-type = draft-mtp 
spec-draft-n-max = 4
load-on-startup = true

# Presets for the Tab Completion Model
[unsloth/Qwen2.5-Coder-1.5B-Instruct-GGUF:F16]
alias = local-tab-completions
load-on-startup = true
n-gpu-layers = all       # Keep in VRAM for instant response
flash-attn = on
temp = 0.2
top-k = 40
top-p = 0.9
repeat-penalty = 1.1

I have found that the Github Copilot Chat extension does not allow a user to use a local model for tab completions. So, I installed the Continue extension, and use it exclusively for the tab completion feature, and use Github Copilot Chat for the Ask/Agent/Plan with the local reasoning model.

This is my chatLanguageModels.json which lets the Github Copilot Chat extension know only about the reasoning model.

 [
	{
		"name": "Local LLama Cpp",
		"vendor": "customendpoint",
		"apiType": "chat-completions",
		"apiKey": "llama-cpp-local",
		"models": [
            {
                "id": "unsloth/gemma-4-12b-it-GGUF:Q8_K_XL",
                "name": "local-reasoning",
                "url": "http://127.0.0.1:50305/v1/chat/completions",
                "toolCalling": true,
                "vision": true,
                "maxInputTokens": 262144,
                "maxOutputTokens": 262144,
                "thinking": true
            }
        ]
	}
]

This is my Continue config.yaml file, which lets Continue know only about the tab-completions model:

%YAML 1.1

---

name: Local Config

version: 1.0.0

schema: v1




models:

# We only declare this model, because we are using "continue" for 

# tab completions only.  

# The copilot chat extension is using the reasoning model, 

# configured here: ~/Library/Application Support/Code/User/chatLanguageModels.json

 - name: local-tab-completions
   provider: llama.cpp
   model: unsloth/Qwen2.5-Coder-1.5B-Instruct-GGUF:F16
   apiBase: http://127.0.0.1:50305
   roles:
      - autocomplete
   autocompleteOptions:
     disable: false
     maxPromptTokens: 2048
     debounceDelay: 250
     modelTimeout: 150
     maxSuffixPercentage: 0.2
     prefixPercentage: 0.3
     onlyMyCode: true
     useCache: true
     useImports: true
     useRecentlyEdited: true
     useRecentlyOpened: true

Completions are working, but they are not great. Sometimes good, often not. I’m still testing completions config and other models.

13

Thank you, this thread is helpful, I will try the config. I am using unsloth gemma4 26b QAT UD-Q4_K_XL, it loop on response a lot.