I have been building a Chrome extension solo for about 14 months, and somewhere along the way it turned into a fairly complete tour of the Gemini API surface. Nine of them are running inside the same MV3 extension right now, which means I have spent a lot of time on the parts where two of these have to coexist in one process, and on the parts where the API is fine but Chrome is not.
Around 750 people use it weekly, so most of what follows is stuff that broke in front of real users rather than in a demo. Writing it down here because I could not find most of it documented anywhere, and I would rather it sit somewhere searchable than in my head.
**Computer Use**
The agent screenshots the active tab, gets an action back, executes it, screenshots again. Two things cost me real time.
The model returns coordinates in a normalized 0 to 999 space, so every action has to be converted against the actual viewport. If you forget devicePixelRatio on a scaled display, everything lands slightly off in a way that looks exactly like the model being wrong, and you will waste a day blaming the wrong layer.
The second one is code editors. Monaco and CodeMirror do not respond to a plain synthetic input event, so “type this into the editor” silently does nothing while the loop happily continues to the next step and reports success. My action handler is now the largest file in the codebase almost entirely because of per-editor and per-widget special cases. Same story for drag and drop, right click menus, and anything inside a cross origin iframe.
One product decision that came out of this: the agent stops before anything irreversible. It will fill a five page job application end to end and then leave the submit button alone. Users forgave a lot of misclicks once they knew it could not actually send anything on their behalf.
**Live API, audio plus screenshare**
Bidirectional audio over a persistent WebSocket, with the tab’s screen streaming in so you can ask “what am I looking at right now” and get a real answer.
Audio arrives as raw PCM chunks, and if you play each chunk as it lands you get clicks and gaps between them. Scheduling every chunk at an explicit AudioContext timestamp instead of playing on arrival fixed it completely. I also hold the session open for two minutes after the user stops talking, because a cold start is very noticeable when someone is going back and forth conversationally.
There is also a visual layer on top: while it talks, it draws SVG arrows on the live page pointing at the element it is describing, by resolving the target’s bounding rect at speak time. Rendering that in an isolated overlay was the only way to keep it from inheriting the host page’s CSS.
**Live API for text to speech**
Same connection, different job. Click any paragraph on any page and it reads aloud with word level highlighting and auto scroll.
Speed control was the hard part. Changing playbackRate shifts pitch and the voice goes cartoonish, so I implemented WSOLA time stretching to get 0.5x to 2.0x with pitch intact. Hardest single piece of code in the project by a wide margin. If anyone has solved variable speed on streamed PCM more cheaply than a time stretching algorithm, I would genuinely like to know.
**Flash Lite for transcription**
Hover any input on any site, a mic button appears, you talk, cleaned up text lands at the cursor. Flash Lite is the right call here purely on latency and cost, since dictation means hundreds of tiny requests a day rather than a few large ones.
The interesting problem was not the model at all. It was injecting a button into every input on the open web via Shadow DOM so it neither inherits nor breaks the host page’s styles.
**Nano Banana Pro**
Text to image up to 4K with aspect ratio control, plus editing an image you already have, straight into the side panel. Nothing painful to report. This one mostly just worked.
**File Search**
Upload PDFs, Word docs and spreadsheets, ask questions, get answers with citations back to the source chunk. The citations are the part users actually trust. Having them come back from the API instead of stitching them together myself killed a component I had already half built.
**Grounding with Google Search**
Used for the path where a stale answer is worse than a slow one. It matters most in voice mode, where someone is asking out loud and is not going to go verify anything themselves.
Worth noting what grounding does not cover: for paywalled or logged in pages, I read the DOM locally from the content script instead of fetching the URL, because anything server side just gets the login wall. Users attach open tabs and ask about them directly.
**Function calling**
The plumbing for the connectors: Gmail, Calendar, Sheets, GitHub, Slack, Linear, Trello, outbound webhooks to n8n and Zapier, and a read only database connector for Postgres, MySQL and Neon.
The lesson here was that tool descriptions matter far more than tool count. Every time I shipped a tool with a vague description, the model started reaching for it in situations it had no business being in, and the fix was always rewriting the description, never touching the code.
**Structured output**
Database results and extracted page data come back in a fixed schema and render as bar, line, pie and doughnut charts inline in the chat. This is the least glamorous item on the list and the one that changed the database connector from a toy into something people use.
**The thing that changed how I actually use it**
The agentic loop above is fine for logging into something. It is miserable for anything repetitive. I asked it to pull data off a profile page and sat there watching it scroll, screenshot, scroll, screenshot, one full round trip per item.
At some point it clicked that I was making the model pretend to be a human holding a mouse, when the thing it is genuinely good at is writing code. So there is a second mode now: the model reads the page structure, writes a JavaScript snippet, and the extension runs it in the page the way you would paste it into the devtools console. One call, and the loop runs in the DOM instead of in the model. Minutes of clicking becomes seconds.
Getting that to run under MV3 was worse than expected. `executeScript` no longer accepts a `code:` string, so you cannot inject a dynamic one. `eval` is blocked by the extension’s own CSP in the isolated world, and sites like Instagram, X and GitHub ship a strict CSP that blocks it in the main world too. I ended up going through `chrome.debugger` and `Runtime.evaluate`, which is why Chrome shows the “started debugging this browser” banner while a script runs.
Two more things if anyone tries this. The model cannot write selectors for a page it has never seen, so a silent read only pass goes first and reports back the repeating blocks, a working CSS selector for each, sample values, and whether the list scrolls its own container instead of the window. That last one cost me a full day: lists inside a modal ignore `window.scrollTo`, and you silently get one screenful of results with no error at all.
And since the model is authoring arbitrary JS against whatever origin the user happens to be on, the generated source gets screened before anything attaches. I am deliberately not publishing the exact checks, but here is the thing that shaped them: the obvious version of this filter looks for network writes, and a plain GET to `attacker.com/?d=` exfiltrates just as well as a POST does while having no method keyword to grep for. Anyone building the same thing should assume static screening of generated code is a speed bump and not a boundary, because anything that assembles its URL at runtime walks straight past it.
**If you want to run it on your own key**
It is BYOK, so everything above runs against your Gemini API key and nothing routes through my servers except OAuth token exchange. Chat history syncs to your own Google Drive rather than to me.
Chrome Web Store: https://chromewebstore.google.com/detail/hdcehjaodheipickeopncehaikamghjk?utm_source=item-share-cb
Affiliation and pricing upfront so nobody feels ambushed: I build this. Free to install, core features work with no account, and there is a nine dollar one time tier for the full set. I am not posting to sell it, I am posting because this is the only forum where the Computer Use coordinate handling and the Live API audio scheduling above are interesting to anybody.
**The open question**
For anyone who has built on Computer Use: is the click by click loop actually the right primitive for repetitive extraction, or should the model be authoring a script and running it once?
I went with letting the model write the whole script, but that means trusting generated JS on arbitrary origins, and I am not confident that tradeoff is right for everyone. The safer version is having the user write the selector and letting the model only run and paginate it, which is worse UX and much easier to reason about. Curious where other people have drawn that line.