With gemini-2.5-flash retiring Oct 16 we auditioned replacements for our production workload (12 distinct “reading lens” prompts — literary close reading, Socratic questioning, EN→KO translation with a register constraint, vocab coaching — served via Firebase AI Logic).
Method: same stimulus (the closing paragraph of Melville’s “Whiteness of the Whale” chapter — chosen adversarially for archaic vocab, a giant periodic sentence, and heavy figurative language), all 12 production prompts verbatim, temp 0.4, thinkingBudget 0 on the Gemini models, reasoning-on for Gemma (serving default). One generation per cell, 36 total, usage metadata recorded, every output read by hand + LLM judge (Claude Fable 5).
Results:
| Model | Median | $/1M in·out | Measured billed ¢/reply |
|---|---|---|---|
| gemini-2.5-flash | 1.7 s | 0.30 · 2.50 | 0.057¢ |
| gemini-3.1-flash-lite | 2.2 s | 0.25 · 1.50 | 0.038¢ |
| gemma-4-26b-a4b-it | 49.6 s* | 0.15 · 0.60 | 0.150¢ |
*shared endpoint w/ reasoning on; compare quality, not speed
Three observations:
- 3.1-flash-lite beat 2.5-flash on output quality for us, not just price — consistently the sharpest specific observations on the literary tasks. If you’re assuming flash-lite is the downgrade tier, test it before deciding.
- Reasoning tokens dominated Gemma’s real cost. Lowest sticker price on the bench, ~4× flash-lite per reply delivered — thousands of hidden thinking tokens per answer, billed as output. Worth checking usageMetadata.thoughtsTokenCount before comparing models on list price.
- Prompt contracts were fully portable: 36/36 outputs held format (word caps, fixed structures, question-only constraints, KO formal register 해라체) across all three models.
Full write-up with all 36 unedited transcripts: ioreader.app/blog/the-whiteness-test
(Disclosure: our app, our blog.) Interested whether others migrating off 2.5-flash are seeing flash-lite hold up — and how you’re accounting for thinking-token billing in model comparisons.