Using Gemini structured output to turn messy scraped HTML into clean JSON — sharing my pipeline

Gemini API
1

Hi all — sharing a pattern that’s been working really well for me, in case it helps anyone building data pipelines on top of the Gemini API.

The problem: web pages are hostile input for downstream systems. I collect product/pricing data from public pages across multiple markets (via a geo-targeted residential proxy layer, since most of these pages serve different content per country), and the raw HTML-to-structured-data step was always the fragile part. CSS selectors break on every redesign; dedicated parser maintenance became a second project.

What changed: I stopped parsing and started asking Gemini to do it.

The flow:

  1. Fetch page HTML (requests + residential proxies for geo-appropriate vantage points)
  2. Strip scripts/styles, keep semantic HTML (cuts tokens roughly 60–70%)
  3. Send to Gemini with a response_schema, so output is guaranteed JSON

The key piece is structured output — a Pydantic model defines the schema once, and every response comes back validated:

from pydantic import BaseModel
from google import genai

class Product(BaseModel):
    name: str
    price: float
    currency: str
    availability: str

client = genai.Client()
resp = client.models.generate_content(
    model="gemini-2.5-flash",
    contents=f"Extract product data from this HTML:\n{clean_html}",
    config={"response_mime_type": "application/json",
            "response_schema": Product},
)
product = Product.model_validate_json(resp.text)

Notes from running this in production for a couple of months:

  • gemini-2.5-flash handles this task well and keeps per-page cost negligible
  • Stripping scripts/styles before sending is the single biggest cost/quality lever
  • Structured output eliminates the “almost valid JSON” failure mode entirely
  • For multilingual markets, I keep the prompt in English but let the model extract in the page’s language, then translate in a second pass — cheaper than prompt + page both being multilingual

Curious whether others here have hit the “LLM as parser” pattern, and what you do about hallucinated fields on sparse pages — my current mitigation is a confidence check where the model must also return an “evidence” substring I verify against the raw HTML.ge