Gemini 4 Hydrogen: Strong Benchmarks, but How Reproducible Are the Cross-Model Comparisons?

Google DeepMind team,

I’ve reviewed the Gemini 4 Hydrogen model card and the accompanying evaluation methodology.

The published results are impressive in several categories, but I have concerns about cross-model evaluation parity, statistical uncertainty, and independent reproducibility.

1. Cross-model evaluation parity

SWE-sweep uses Gemini’s self-computed Antigravity-CLI v1.4.8 results alongside external leaderboard scores. Terminal-Bench similarly combines different evaluation sources.

How are harness-level confounders, tool orchestration differences, inference budgets, and execution-environment variance controlled before attributing performance deltas to model capability?

2. Statistical uncertainty and selection effects

OSWorld-2.1 reports results maximized over three runs, while other benchmarks use averaged attempts.

Could the team publish per-run score distributions, variance estimates, confidence intervals, and paired statistical significance tests? How does the reporting methodology account for selection effects?

3. Independent reproducibility

Would DeepMind consider releasing version-pinned evaluation manifests, exact agent configurations, tool schemas, generation parameters, task-level outcomes, and anonymized execution traces?

4. Compute-normalized comparisons

Can we see quality-latency-cost Pareto frontiers under standardized token budgets, tool-call limits, and comparable test-time compute allocations?

5. Contamination-resistant evaluation

What benchmark-specific contamination audits or held-out evaluation controls support the reported generalization claims?

I appreciate that DeepMind has already documented its benchmark methodology and several harness configurations. My questions concern the remaining uncertainty, not an assumption that no methodology exists.

A benchmark table establishes reported scores. It does not, by itself, establish causal architectural superiority or reproducibility under controlled conditions.

I’d be interested in a technical response from the evaluation team, particularly regarding cross-model parity and uncertainty quantification.

References:

Gemini 4 Hydrogen Model Card: <https//deepmind.google/models/model-cards/gemini-4-hydrogen/>

Official Evaluation Methodology: https//deepmind.google/models/evals-methodology/gemini-4-hydrogen