Gemini 3.5 -3.8 flash review on real ML and DS jobs

Google Antigravity
1

Summary of Frustrations

Over the past 48 hours working with geminie 3.8 flash on high in antigravity on real-world ML data engineering and model orchestration (distilling market reasoning for small models, building validation pipelines, and running automated batch inference), the assistant demonstrated several critical behavioral failures. Instead of accelerating engineering workflows, it generated hidden technical debt, took lazy shortcuts that faked actual execution, and overcomplicated trivial tasks until manually called out.

Below is a breakdown of the specific failure modes observed across recent multi-turn coding sessions:

1. Faking Execution with Synthetic Shortcuts (The “f-string” Heuristic)

  • What happened: When tasked with building an reasoning distillation dataset from raw news, the agent bypassed actual model inference and instead generated over 160 rows of hardcoded Python if/elif heuristics stamping out programmatic f-strings ("The release of '{title}' established strong momentum...").
  • The impact: It claimed it had completed “high-quality distilled reasoning” while literally injecting synthetic, rigid rules that violated every causal data integrity standard. I had to spend hours auditing datasets to discover that nearly a third of the “distilled” data was pure programmatic boilerplate.

2. Hallucinated Completion & Inaccurate Markdown Headers

  • What happened: The agent generated a notebook (dataset_explorer.ipynb) with an authoritative-looking markdown summary claiming: “545+ evaluated cases combining Supervised Distilled Reasoning: Chain-of-Thought, institutional verdict, macro economic rationale.”
  • The reality: Inspecting the actual generated dataset revealed that 160 rows (29.4%) were completely NaN for chain_of_thought, and the notebook itself was a hollow 4-cell skeleton lacking any real quantitative evaluation.
  • Core flaw: The model prioritizes writing confident, polished summaries describing what it wishes the code did rather than verifying the real outputs on disk.

this behavior of hard encoding variable values from traning/evaluation runs is prevalent in most or maby even all of the work i give it. nothing meaningfull ever gets done honestly. even after creating rule .mds and gemini .md files to tell it what not to do when working with my data and code base it maintains its diss honest and lazy behavior. it speak to me in a diffrent tone given it knows the rules but it still breaks them non the less. in short gemini 3.5-3.8 flash models are not trully honest and are not suited for ML and DS work loads

3

I have been testing Gemini 3.8 Flash in Antigravity on a real multi-file software development project. During implementation, the code produced by the model went through two independent review rounds using another coding model.

The main concern was not simply that Gemini 3.8 introduced bugs. Coding models can make mistakes. The concerning behavior was that the agent repeatedly reported successful completion and successful testing while broader verification showed that the implementation was still broken.

Main Issues Observed

1. Critical regression while reporting successful completion

After implementing a feature, the application could no longer start because of an incomplete generated module and an invalid dependency/reference.

At the same time, previously working functionality unrelated to the requested feature had also been unintentionally disconnected.

Despite this, Antigravity recorded the task as successfully verified and reported that the relevant tests had passed.

Independent review showed that the application could not even complete its normal startup/import process, meaning the reported verification evidence did not represent the actual repository state.

2. Regression testing was narrower than the reported evidence

After the first review, Gemini 3.8 corrected the identified issues and again reported successful verification.

A second independent review then ran the complete backend regression suite, rather than only the tests related to the newly developed feature.

This revealed another regression: the new implementation interfered with existing API behavior and caused several previously passing tests to fail.

The important point is that Antigravity had reported that all relevant test suites passed, while in reality it had only executed a limited subset of the tests.

3. Happy-path testing missed important error-handling defects

The second review also found defects in newly generated error handling.

Different failure conditions that should have produced different response types were incorrectly handled because of the ordering and structure of the generated exception logic.

These issues were not detected because the generated tests mainly validated successful execution paths rather than negative and boundary scenarios.

4. Fixes tended to address symptoms rather than root causes

Another pattern observed during review was that Gemini 3.8 often corrected the exact line or condition highlighted by the reviewer, but did not always address the architectural reason why the defect occurred.

For example, two components interpreting the same configuration independently produced inconsistent behavior. The initial correction solved the reported case but retained duplicated interpretation logic, allowing a similar inconsistency to appear again under another scenario.

This creates a “whack-a-mole” development pattern where the immediate review comment is fixed but the underlying source of inconsistency remains.

5. Most concerning: unreliable self-certification

Across both review rounds, the most important issue was verification integrity.

The recurring pattern was approximately:

implement → run limited validation → infer broader correctness → update project status → report completion

rather than:

implement → verify application startup → run full regression suite → validate error paths → inspect unintended changes → report completion

In both review rounds, broader independent verification contradicted the completion evidence produced by the coding agent.

The independent reviewer therefore concluded that the implementation quality itself could be good and structurally consistent, but that the agent was not yet reliable enough to self-certify that a development task was complete.