As an Agent Tool

TL;DR

BUA is packaged as a built-in subagent (subagent_type: "browser") whose private toolkit is the browser tool — one tool with a Zod discriminatedUnion of 13 actions (navigate, click, type, …). The main agent does not see the browser tool directly; it delegates via the standard task tool. Inside the subagent’s isolated context, the browser loop (extract → reason → click → extract → …) runs against the user’s logged-in Chrome via CDP.

One delegation in (from main agent), one summary out (from subagent). The N-step inner loop never touches the main conversation.

Three tool-config shapes

The codebase uses three shapes. Pick the one whose characteristics match the operation family.

Why BUA picked the third:

Counter-pressure (why filesystem is a group): grep returns an array of matches, read_file returns file content, ls returns a list of entries. Heterogeneous outputs → separate tools help the model pick correctly.

Decision hint: if tracing shows the model conflating variants inside a single + actions schema, split. If it’s confused picking between group siblings, merge into actions. Prompt-engineering, not API-design.

Delegation via built-in subagent

The browser tool is registered in the normal tool registry but stripped from the main agent’s loadout via INTERNAL_ONLY_TOOLS in packages/backend/src/agent/subagents/index.ts. The main agent reaches BUA through the standard task tool, where a built-in subagent type "browser" is pre-registered (see packages/backend/src/agent/subagents/browser-subagent.ts).

Why wrap in a subagent? BUA’s useful unit is a loop, not a single call. If the main agent ran the loop directly, each of the 10–20 browser actions would re-send the main dialogue (cost ≈ M × N). Wrapping it in a subagent means only the subagent’s own prompt + inner loop context grows with N — the main thread pays just one tool_call + one summary return.

Isolation invariants (enforced in task.tool.ts):

• Fresh RuntimeContext — carries only { taskId, userId, sandbox, workspace, timezone }; no inherited messages, todos, reminders, writer
• browserBridge reaches the subagent through toolServices: { browserBridge, kanbanService } (captured from the parent’s deps at construction), not on the fresh runtime context
• taskId preserved (so task_milestone events route to the same popup session)
• The subagent shares the parent’s workspace + sandbox 1:1 — no chroot, no .tasks/{toolCallId}/ subdirectory isolation (irrelevant for pure-extraction BUA tasks anyway; the browser subagent’s loadout writes no files today)

Tier gating stays on the familiar allowedTools list — if a tier’s allowedTools contains "browser", the subagent is registered; otherwise it isn’t. The tier list gates the subagent, not direct tool access.

Rollback: set ENABLE_BUILTIN_BROWSER_SUBAGENT=false to suppress the built-in subagent and restore direct browser tool access on the main agent (emergency use).

Anatomy

Three code pieces in packages/backend/src/tools/tools/browser.tool.ts plus the shared schema.

Schema — @zapvol/common

// packages/common/src/schemas/browser-bridge.ts
export const browserActionSchema = z.discriminatedUnion("type", [
  z.object({ type: z.literal("navigate"), url, tabId: tabIdSchema.optional() }).describe("..."),
  // element-targeting actions take EXACTLY ONE of `uid` or `selector` (a .refine enforces the xor).
  // `uid` (e0, e1, … from the last extract's `elements`) is preferred; `selector` is the fallback.
  z.object({ type: z.literal("click"), selector: sel.optional(), uid: uid.optional() }).describe("..."),
  z.object({ type: z.literal("type"), selector: sel.optional(), uid: uid.optional(), text }).describe("..."),
  z.object({ type: z.literal("extract"), selector: sel.optional() }).describe("..."), // → { text, markdown, elements }
  // …
  // `open_tab` auto-creates a session and (by default) lands the new tab in the minimized BUA window.
  // `focus: true` overrides — opens in the user's focused window.
  z.object({ type: z.literal("open_tab"), url, focus: z.boolean().optional() }).describe("..."),
  // 13 variants total: navigate, click, type, press_key, scroll, screenshot, extract,
  //   wait_for, hover, evaluate, get_tabs, open_tab, close_tab
]);

Zod’s discriminatedUnion compiles to a JSON schema with oneOf + type literals. The model picks a branch by writing the literal; AI SDK validates the rest.

Registration — backend

// packages/backend/src/tools/tools/browser.tool.ts
export const browserServerConfig: ServerToolConfig = {
  name: TOOL_NAME_BROWSER,

  instructions: async (deps: ToolBuildDeps) => (deps.browserBridge ? BROWSER_TOOL_INSTRUCTIONS : ""),

  createTools: async (deps: ToolBuildDeps) => {
    if (!deps.browserBridge) return {};
    const bridge = deps.browserBridge;
    return {
      browser: tool({
        description: "Drive the user's logged-in Chrome tab — pick one action from: navigate, click, type, ...",
        inputSchema: zodSchema(browserActionSchema),
        execute: async (input) => {
          const result = await bridge.request(input as BrowserAction);
          return result.ok
            ? { ok: true, action: input.type, result: result.result }
            : { ok: false, action: input.type, error: result.error };
        },
      }),
    };
  },

  compact: ({ input, output }) => {
    /* trim screenshot bytes, truncate extract text */
  },
  toClientOutput: (output) => {
    /* preserve dataUrl for <img>, text for UI */
  },
};

Observation: execute is action-agnostic — it forwards the whole input to bridge.request(). The extension switches on input.type, not the backend. Backend is a thin pass-through.

Tier gating

browser is not in ALL_TOOL_KEYS — admin must opt in per tier. When opted in, createTools(deps) and instructions(deps) run eagerly on every task start; when not, the whole tool is absent from the model’s view.

End-to-end: one click call

Participants: Model (the LLM), execute (backend tool’s execute function), bridge (BrowserBridge per-user instance), Pool (BrowserBridgePool singleton), Ext (extension’s action-dispatcher), CDP (chrome.debugger).

The model saw one tool call; the backend saw one request/response; the extension issued one CDP command. The discriminated union collapses what would be 13 AI SDK tools into one.

Cancellation: AI SDK passes an AbortSignal to each execute(input, { abortSignal }). The browser tool forwards it to bridge.request(action, signal), and the pool listens for abort on every pending request — on trigger the pending promise resolves with internal_error "aborted by caller" and the timer is cleared. No waiting for the 30s pool timeout when the parent agent is cancelled mid-tool.

How the model and the DOM talk

The diagram above shows how a click flows. It doesn’t answer why the element the model targets matches the real DOM. A recurring confusion: “the backend must send DOM-specific instructions, right?” Actually, no layer between the model and the extension touches DOM. The backend is a string relay. The LLM reads a digest of the page and decides which element to target — primarily by a short uid handle, falling back to a CSS selector.

What extract returns

extract does not return raw HTML. It returns three fields:

• markdown — the page converted to Markdown with nav / footer / overlays stripped. Cheap enough to read in bulk; this is what the model reads to understand the page.
• elements — the accessibility tree’s interactive nodes, each tagged with a short uid (e0, e1, …). This is the model’s handle vocabulary for the next action.
• text — the raw innerText fallback.

Element-targeting actions (click, type, hover, wait_for) take exactly one of uid or selector (a Zod .refine enforces the xor). The model is instructed to prefer uid — it is shorter, token-cheap, and survives class-name churn — and reach for a CSS selector only when no uid fits (e.g. an attribute-state predicate).

Who has access to what

The backend is still a string relay with no shared DOM data model. The one piece of state is a uid → backendNodeId map that lives in the extension: rebuilt on each extract, cleared on navigation (Page.frameNavigated / a navigate action). So there is a lightweight binding — but it lives at the extension edge, not in the backend.

A concrete round-trip

1. Model emits browser({ type: "extract" }) (no selector → whole page)
2. Extension walks the a11y tree + DOM, returns { markdown, elements, text }. The elements list looks like:

   e4  button  "View"   (row: Jane Doe)
   e5  button  "View"   (row: John Smith)
   e6  input   "Search candidates"

   and markdown renders the list as readable text (- Jane Doe … [View]).
3. Backend relays the payload back to AI SDK; it lands in the model’s context window.
4. Model decides to open Jane’s panel. It emits browser({ type: "click", uid: "e4" }) — no selector authoring needed.
5. Backend relays the action unchanged. Extension’s debuggerController.click resolves e4 → backendNodeId from the map built in step 2, gets the box model, and dispatches Input.dispatchMouseEvent at its center.

The binding between step 2 and step 4 is the uid map. If the model instead needs a state predicate the a11y list can’t name (e.g. “the detail panel once aria-hidden flips to false”), it falls back to a CSS selector.

Why uid handles + Markdown (with selector fallback)

The earlier design fed raw HTML and asked the model to author CSS selectors. The current design borrows from two neighbours:

• Element-index / handle approach (how Browser Use does it): number every interactive element and let the model reference it by handle. Robust to class-name drift, token-cheap. BUA’s uid (e0, e1, …) is exactly this.
• Accessibility-tree approach (Anthropic’s computer-use on web): read the ARIA tree rather than raw markup. More semantic, less verbose. BUA’s elements list is the a11y interactive set, and markdown is the semantic read.

CSS selectors survive as a fallback because every modern LLM has strong selector priors, they’re debuggable (valid document.querySelector input — reproduce in DevTools), and some targets are best expressed as an attribute-state predicate ([aria-hidden='false']) that no static uid can name.

The cost: a uid is only valid until the next navigation / re-extract. The element_stale / element_not_found error → re-extract pattern (see the multi-step example) is how we recover.

Implications of this design

• The backend works on any website — it has zero site-specific knowledge. All site shape lives in the extract payload and the uid / selector the LLM emits
• uids expire: a uid resolves against the map from the last extract; after a navigation (or if the DOM changed under an SPA re-render) the map is cleared and a stale uid returns element_stale — re-extract to get fresh handles
• extract before click isn’t optional — without it the model has no elements list to draw a uid from, and no Markdown to ground a selector on. Acting without grounding is brittle

Multi-step example — fetch candidate details

A single action is the atom; real workflows are chains. Here is a realistic trace for the prompt “fetch the top 3 candidates’ contact info from this hiring dashboard”.

Main-agent view — one tool call, one result:

main_agent → task({
  subagent_type: "browser",
  description: "Fetch 3 candidates' contact info",
  prompt: "Open the hiring dashboard (first action auto-creates the session). Extract the first 3 candidate rows …"
})
         ← { summary: "Extracted 3 candidates: [{name:'Jane',email:…}, …]", status: "completed", artifacts: [] }

Subagent view — 17 browser actions inside an isolated context. The subagent is the one iterating over elements by uid, waiting on aria-hidden transitions, and writing the final summary. The main agent’s context grows by exactly one task call + one summary return.

The trace below is the subagent’s internal loop. Every row is one browser tool call that the subagent emits; the agent loop blocks on each before deciding the next.

The agent loop is: observe the page → pick element uids → act → wait for the result to settle → observe again → act. Every physical click or type is preceded by an extract or wait_for so the uid (or selector) the model emits is grounded in something it actually saw.

Starting point: the user sent “fetch the top 3 candidates’ contact info from this hiring dashboard” and the agent is on the candidate list page. Every row below is one browser tool call (one WS round-trip); the agent loop blocks on each before deciding the next.

Total: 17 browser tool calls + 1 final text completion. Iterations 2 and 3 skip step 2’s wait_for because the list is already rendered — a small optimization the model learns by remembering that the list survived the close of the previous detail panel.

Patterns the trace exposes

• extract-before-act — never act on an element the model hasn’t seen. Without step 1 there is no uid for step 3’s button; with it, e4 comes straight from the extract’s elements list (and the Markdown grounds any selector fallback)
• wait_for bookends every async transition — steps 4 and 7 gate on aria-hidden flipping (a selector state predicate, the case where uid doesn’t apply). In SPAs this is the difference between “works on fast networks” and “works reliably”
• Loops happen in the agent loop — BUA has no for-each action. The model issues a fresh click/wait/extract triplet per iteration; the agent-loop iteration count is implicit from its plan
• One session, many actions — all calls hit the same (domain, tabId) session. Sessions don’t time out in the UX-first model; they only end on tab-close / user idle / Stop / blocklist-add. If a user clicks Stop all mid- loop, step 3’ returns session_not_found and the agent stops with a partial result

When something goes wrong

Swap step 4 for the unhappy path — detail panel takes longer than 5s:

The model treats timeout as a reason to re-extract and understand what’s blocking, not to retry the same wait — exactly what BROWSER_TOOL_INSTRUCTIONS says. Compare element_stale / element_not_found: the model re-extracts to get fresh uid handles instead.

This is why the tool’s prompt spells out error handling action-by-action rather than a single “retry on error” rule: each code means a different next step.

Adding a new action

Three files. No new ServerToolConfig, no new tool registration, no subagent touch:

1. Schema — add a variant to browserActionSchema in @zapvol/common/schemas/browser-bridge.ts
2. Extension dispatcher — add a case to src/action-dispatcher.ts’s executeAction switch; add a method on src/debugger-controller.ts if the action needs a new CDP command
3. Prompt — one line in the action table inside BROWSER_TOOL_INSTRUCTIONS (role-neutral reference) OR — if the new action changes the subagent’s expected behavior (e.g. terminal-error semantics) — also touch BROWSER_SUBAGENT_INSTRUCTIONS in browser-subagent.ts

Compare: adding a new filesystem tool touches TOOL_KEY_TO_NAMES, TOOL_CONFIG_METAS, clientToolConfigs, init-tools.ts, the tool file itself. The tool-group shape has more plumbing per operation.

Note: the subagent’s toolKeys (["browser", "complete"]) is not auto-expanded by adding actions — actions are variants of the single browser tool, not new tool registrations. You only change toolKeys if you want to add a different capability to the browser subagent (e.g. a filesystem tool for artifact delivery).

Related

• Runtime topology — where the extension and backend live relative to each other
• Protocol — message envelope and action schema details
• Session model — authorization rules enforced in the extension dispatcher