This is a sequel to my earlier MTP measurement thread: Gemma 4 MTP on an RTX 5070 Ti. That study ran both models in QAT form and left the value of QAT itself unmeasured — so this one measures it: a 2×2 design (quantization regime × released model pair) over Gemma 4 12B IT (dense; Q4_K_M vs the QAT Q4_0 release — a clean same-base pair) and a 26B-class MoE released pair (25.8B PTQ vs 25.2B QAT — a quasi-contrast that may embed a base revision). All cells ran on one RTX 5070 Ti via Ollama 0.30.6 at temperature 0, on routing/abstention-style tasks (should the model verify, date-hedge, re-anchor, or abstain?) under a fixed, bit-identical governance stack — 100-task and 500-task suites, plus repeat and configuration controls.
Headline numbers (Core-500, N=500 rows = 100 stems × 5 paraphrases):
| Cell | route correctness | mean latency |
|---|---|---|
| 12B dense, PTQ Q4_K_M | 0.760 | 2,482 ms |
| 12B dense, QAT Q4_0 | 0.748 | 2,626 ms |
| 26B MoE, PTQ Q4_K_M | 0.738 | 3,252 ms (CPU-offloaded — excluded from conclusions) |
| 26B MoE, QAT Q4_0 | 0.756 | 1,613 ms |
- The within-pair regime effects are individually marginal and oppositely signed: QAT −0.012 on the dense 12B (exact McNemar p=0.070) and +0.018 on the MoE pair (p=0.078).
- The regime × pair interaction — an exploratory, single-run endpoint — is +0.030 with a stem-clustered bootstrap 95% CI of [+0.010, +0.052], concentrated almost entirely in the volatile-current family (the “does this question need verification or a dated hedge?” family), where the two pairs’ scores cross (0.531/0.497 vs 0.497/0.531).
- Quantization regime is a behavioral change even where scores barely move: only 23–32% of same-task temperature-0 outputs are byte-identical across regimes (median first divergence at 46–73 characters). Same-configuration repeat runs are 100/100 byte-identical in all four cells, and a
num_ctxcontrol is also 100/100 — so the divergence is attributable to the weights. Practical corollary: hash-based caching, audit replay, and golden-file tests do not survive a QAT↔PTQ swap. - Safety-style failure rates (stale commitment, unsupported current claim, over-verification) were 0.000 in seven of eight runs, with a single over-verification event (0.008) in the offloaded PTQ MoE run.
- Ops note for 16 GB cards: the QAT Modelfiles declare a 262,144-token context; without an explicit
num_ctxthe KV allocation fails with a CUDA “resource allocation failed” error. Pinningnum_ctx(we used 8192) fixes the load and — per the control above — does not change outputs at these prompt lengths.
Scope is deliberately narrow. These are routing/abstention decisions under one governance stack, one GPU, one runtime — not general answer quality, and nothing here contradicts the published QAT general-benchmark results. All inferential results are exploratory (single run per cell, no preregistration); the 26B pair confounds regime with a possible base revision and a CPU-offload compute path, so we attribute effects to the released artifact pairs, not to architecture per se. The one clean statement is about the same-base 12B pair, where QAT bought nothing on these tasks (directionally −1 point). Before archiving, the manuscript was adversarially reviewed by three independent reviewers (statistics / consistency / scope) and corrected.
If the sign-flip across pairs replicates elsewhere, “which quantization should I download?” may not have a recipe-level answer for decision-style workloads — it may be a per-artifact, per-task-family question.
Resources (kept to two hyperlinks due to new-user link limits — the rest as plain identifiers)
- Paper (archived, CC BY 4.0): DOI 10.5281/zenodo.21875513
- Reproducibility package (processed data, deterministic analysis, environment/artifact hashes): GitHub repo
mobius-style/gemma4-quant-regime-study - Predecessor study DOI: 10.5281/zenodo.21860461
Taiko Toeda — ORCID 0009-0001-7267-0201