Gemma 4 MTP on an RTX 5070 Ti: throughput gains, frequent greedy-output divergence, and paired quality evidence

Gemma
1

I have published an independent quality–throughput study of QAT Q4_0 Gemma 4
12B IT and 26B-A4B IT using multi-token prediction in a pinned CUDA
llama.cpp build on one RTX 5070 Ti.

In the fixed tested grid, N=6 produced the highest measured aggregate decode
throughput for both models: 226.39 versus 89.20 tokens/s for 12B, and 322.53
versus 169.58 tokens/s for 26B-A4B. Exact greedy-output identity was much
weaker: 112/200 and 118/200 N=6 outputs were byte-identical to ordinary
decoding. On the paired 200-item objective suite, the macro-score change was
-0.83 percentage points for both models; both confidence intervals remained
above the prospectively frozen -5-point pilot margin.

The scope is deliberately narrow. This does not establish universal N=6
optimality, exact behavioral identity, or generalization to other GPUs,
runtimes, precisions, or serving workloads. The public artifact excludes
verbatim benchmark items and raw generated text while retaining IDs, hashes,
processed measurements, scripts, pinned sources, figures, tables, and audit
records.

Author: Taiko Toeda, Independent Researcher
ORCID: orcid. org/0009-0001-7267-0201

2

Follow-up: the QAT-vs-PTQ question this thread left open (both models here were QAT) is now measured in a 2×2 sequel — opposite-sign routing effects across the released pairs, and 68–77% greedy output divergence between quantization regimes at temperature 0: Gemma 4 QAT Q4_0 vs PTQ Q4_K_M on an RTX 5070 Ti: opposite-sign routing effects across released pairs, and 68-77% greedy output divergence (sequel) (archived as DOI 10.5281/zenodo.21875513).

3

Hi @Taiko_TOEDA
Thanks for sharing this detailed performance study.

Independent benchmarks like this are incredibly valuable for the community and help everyone better understand Gemma 4’s real-world capabilities. We really appreciate you taking the time to test this and publish your findings so thoroughly.

4

Thank you, @Pannaga_J — much appreciated. The goal was exactly that: narrow, pinned, and reproducible enough that others can extend it or prove it wrong.

Two things I’d genuinely welcome pointers on, if anyone on the team can comment:

  1. Whether the frequent greedy-output divergence under MTP (~44% of items at N=6, byte-level) matches internal expectations for multi-token paths in general, or more likely reflects something specific to the pinned CUDA llama.cpp build I used.

  2. If there is a benchmark harness or workload mix the Gemma team considers more representative for serving-style evaluation, I’d be happy to run the same paired design on it.

For deployers, the QAT-vs-PTQ sequel linked above is probably the more decision-relevant result — feedback on either study is very welcome.