Hi, i’m running Gemma4 locally on Apple M2 with 32GB of Unified RAM. I am serving it with llamma.cpp. I am trying to use it with coding agents. No matter which coding agent I test with (Kilo Code, Pi, Continue, Aider), it almost always devolves into looping thoughts. I realize in the config below I am running an unsloth variant, but this issue happens on the original versions of the model as well, and on other variations and sizes that I have tried.
I’m reluctant to raise the repeat-penalty arg because I don’t want to reduce the quality of the output. Are there any recommended configs I am missing that would fix this?
This is my llamma config:
~/.unsloth/llama.cpp/build/bin/llama-server \
-m /Users/cnrudd/.cache/huggingface/hub/models--unsloth--gemma-4-12B-it-qat-GGUF/snapshots/7102bdea62863acff919c945405ef29973113d66/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \
--port 50305 \
--ctx-size 262143 \
--n-predict 32768 \
--parallel 1 \
--flash-attn on \
--no-context-shift \
--fit on \
--jinja \
--reasoning on \
--chat-template-kwargs '{"preserve_thinking": true}' \
--spec-draft-n-max 1 \
--mmproj /Users/cnrudd/.cache/huggingface/hub/models--unsloth--gemma-4-12B-it-qat-GGUF/snapshots/7102bdea62863acff919c945405ef29973113d66/mmproj-F16.gguf \
--temp 1.0 \
--top-k 40 \
--top-p 0.95 \
--repeat-penalty 1.0 \
--log-verbose \
Here is a screengrab of the looping:
Try lowering temperature to 0.85
Thanks for the suggestion, but --temp 0.85 did not fix the issue.
~/.unsloth/llama.cpp/build/bin/llama-server \
-m /Users/cnrudd/.cache/huggingface/hub/models--unsloth--gemma-4-12B-it-qat-GGUF/snapshots/7102bdea62863acff919c945405ef29973113d66/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \
--port 50305 \
--ctx-size 262143 \
--n-predict 32768 \
--parallel 1 \
--flash-attn on \
--no-context-shift \
--fit on \
--jinja \
--reasoning on \
--chat-template-kwargs '{"preserve_thinking": true}' \
--spec-draft-n-max 1 \
--mmproj /Users/cnrudd/.cache/huggingface/hub/models--unsloth--gemma-4-12B-it-qat-GGUF/snapshots/7102bdea62863acff919c945405ef29973113d66/mmproj-F16.gguf \
--temp 0.85 \
--top-k 40 \
--top-p 0.95 \
--repeat-penalty 1.0 \
--log-verbose \
Hmm, that’s weird then, reasoning in gemma is extremely weird, are you using quantized model or not?
EDIT: Just checked, looks like there’s q4 in the name, that means it’s 4-bit quantization, it might be the problem. Also try putting temperature on 0.7
You can also try decreasing top-K to 30 and/or top-P to 0.9. I have no idea if it helps, just better to experiment with parameters
Thanks @Koeqaife
Before your latest suggestions, I ventured into adjusting repeat-penalty, and found that that this config is working well:
# Only non-default values are set here.
~/.unsloth/llama.cpp/build/bin/llama-server \
-m /Users/cnrudd/.cache/huggingface/hub/models--unsloth--gemma-4-12B-it-qat-GGUF/snapshots/7102bdea62863acff919c945405ef29973113d66/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \
--port 50305 \
--ctx-size 262143 \
--n-predict 32768 \
--n-gpu-layers all \
--no-context-shift \
--reasoning on \
--chat-template-kwargs '{"preserve_thinking": true}' \
--spec-draft-n-max 1 \
--mmproj /Users/cnrudd/.cache/huggingface/hub/models--unsloth--gemma-4-12B-it-qat-GGUF/snapshots/7102bdea62863acff919c945405ef29973113d66/mmproj-F16.gguf \
--temp 1.0 \
--top-k 40 \
--top-p 0.95 \
--repeat-penalty 1.08 \
--repeat-last-n 4096 \
--log-verbose
The key changes from before are:
–repeat-penalty 1.08
–repeat-last-n 4096
Same prompt as in before attempts returned a real answer, no looping. I’ve tested with several new prompts and haven’t seen the looping come back yet. I’ll follow up if I come up with more refinements.
