I built Autopilot as a parity implementation of Claude in Chrome, Anthropic’s extension that lets Claude drive your browser. Then I spent a day driving both against the same Chrome, same profile, same sites, three runs each, alternating which went first. This is what the numbers said, what broke on each side, and the one thing I took away that was not on the scoreboard.

the setup

Two bridges drove one Chrome 152 on one signed-in Windows profile: Claude in Chrome 1.0.90 and Autopilot at extension version 0.1.7. Coverage was a 24-scenario local fixture site, eight public test fixtures, fourteen real sites, six bot-detection probes, a timing suite measured three times with the order alternated, GIF recording, file upload, tab management, and a static read of the official extension’s minified bundles.

Ten agent runs, one at a time, because both bridges share one browser. Every timing is wall clock bracketed with shell timestamps, so it includes the agent’s own turnaround for both bridges. That means only ratios carry meaning, and the report says so on every table. The three-run rule held for the timing primitives and the real-site groups and is labelled where it did not.

the numbers

Medians of three:

Roughly 3.5x on per-call latency in this run, 5.7x in an earlier session on the same machine, and 12x on screenshots. The row where they tied was the realistic form flow, 18.3 s against 17.4 s, where both bridges spent their time recovering from their own failure modes. That row is the honest one.

The per-call gap has a structural cause. The official bridge routes every call through a hosted service and back. Autopilot’s path is local: stdio to a named pipe to native messaging to the tab. The find gap is the same story: the official tool runs a nested model call to resolve a description to an element, 5 to 12 seconds, where Autopilot runs a lexical match in under a millisecond and no tokens.

the five worst things in theirs

• Ref clicks report success and fire no event. Proven with a DOM listener on a fixture page: find returned an exact reference, the click reported success, the listener’s flag stayed false. Seen on at least eleven distinct elements in one group, three of three on a wiki table-of-contents link, three of four on a workspace app’s sidebar. Not deterministic per element. The workaround that always worked was a screenshot followed by a click by coordinate.
• Screenshots time out for thirty seconds. Ten of thirty calls across three runs, and three of three on tabs left idle for 45 seconds or more. A javascript call on the same tab answered immediately after each failure, so the tab was not frozen. The capture path was.
• Enter does not submit a plain HTML search form. Four sites, both text-entry methods.
• The javascript content filter withholds ordinary values. A long repeated string came back as a blocked base64 placeholder.
• The interactive page read under-reports with no notice. Down to 63 nodes on a 9,000-element page, and to zero on another.

the five worst things in mine

• A debugger attach refusal killed a tab three of three times on one social site, because the page embeds another extension’s frame and Chrome refuses to attach across extensions. Fixed since with a recovery ladder: strip the foreign frames, retry, detach and re-attach, replace the tab once per cause. The cap exists because the uncapped version produced six replacement tabs across three runs and never a working one.
• Two clicks reported success and did nothing. Two against their twenty or more, and two is not zero. This is the finding that reshaped the project, below.
• Page text came back empty on two major sites. Still partial.
• The debugger session dropped silently after a navigation that made the tab one the extension may not debug. The attachment map went stale and the next call failed while the extension believed it was attached. Fixed by listening for the detach event.
• Bot detection flagged it on all three runs, with isAutomatedWithCDP as the only flag set. Every spoofable signal read clean. The one change that moved it was making Runtime.enable opt-in per tab, issued only when a console read is first requested. After that, a trace of a session that never reads the console carries no Runtime.enable across 56 calls, and the same detector reports clean on three runs. The debugger banner stays visible, and the README lists that as a limit rather than hiding it.

ahead by design, behind by product

Autopilot is ahead where the design rule put it: background operation against the signed-in profile with nothing brought forward, a tree that walks open shadow roots and same-origin iframes with live-measured offsets, form input through the native value setter with input then change in the right order so React-controlled fields take the value, uploads through the DevTools file API at 25 MB against 10 MB of base64 in tool arguments, an on-disk action journal per browser per day, and diagnostics that name a cause: a refused attach lists the tab’s frames and targets, a covering cross-origin frame is announced in the tree, a batch names the step it stopped at. Nothing leaves the machine. The official extension ships Segment, Sentry, Datadog RUM, and an event-logging endpoint.

It is behind by product surface. find resolves by wording, so a query sharing no word with the element fails unless the client offers MCP sampling for escalation, and none did. No site-specific skills, no scheduling, no workflow recording, no chat surface. No performance traces or heap analysis. Cross-origin iframes are leaves. The official keepalive is triple redundant, including an offscreen document specifically to defeat the worker idle kill, and its scroll path verifies the scroll actually moved and falls back to an injected scroll, which mine does not do. Their reconnect had a result queue and a generation counter before mine did. Both have since been built.

the rule i took away

Both tools have a signature failure mode that neither set of documentation mentions, and it is the same one: a call that returns ok while the page did not move. Theirs did it twenty-plus times, mine twice. Neither was caught by a test, because a test that asserts on the tool’s own return value cannot see it.

A tool that fails is fine. The agent reads the error and tries something else. A tool that reports success and did nothing sends the agent forward on a false premise, and everything after it is wasted. So an automation tool’s first job is to be legible about failure. That became a result contract on every input call, which is its own post.

One caveat the report carries and I will repeat: the campaign ran against extension 0.1.7. The fixes it produced landed over the following week and were verified in five live passes with 155 numbered checks, but the head-to-head was never re-run on the new build. The numbers above are the numbers from that day.