Intent commands

Semantic automation is available through the authenticated HTTP / TypeScript / MCP surfaces when the principal holds intent:execute. There are no dedicated intent HTTP routes or MCP tools — submit via POST /v1/commands / command_execute / BrowserRuntimeClient.submit with

{ "kind": "intent", "input": { "kind": "<intent>", "input": { … } } }

inside a CommandEnvelope (schemaVersion: 2). TypeScript helpers: locateEnvelope, fillEnvelope, submitAndVerifyEnvelope, waitForStateEnvelope, followEnvelope, dismissObstructionEnvelope, extractEnvelope.

Command classes

Class	Meaning
`Replayable`	Safe to retry under policy without a prior checkpoint
`Reconciliable`	May need inspection / reconcile before replay
`Boundary`	Mutating / side-effecting; checkpoint gate applies

Intent	Class
`locate`	Replayable
`waitForState`	Replayable
`extract`	Replayable
`fill`	Reconciliable
`dismissObstruction`	Reconciliable
`submitAndVerify`	Boundary
`follow`	Boundary if `boundary: true`, else Reconciliable

Envelope examples

Shared meta (ids are UUIDs; deadline RFC3339):

const meta = {
  commandId: crypto.randomUUID(),
  workflowId: crypto.randomUUID(),
  attemptId: crypto.randomUUID(),
  sessionId: session.id,
  pageId: page.id,
  deadline: new Date(Date.now() + 60_000).toISOString(),
};

Locate (Replayable)

import { locateEnvelope } from "@bobby-browser/sdk";
await client.submit(locateEnvelope(meta, "primary search box"), { idempotencyKey: crypto.randomUUID() });

Wire command: { kind: "intent", input: { kind: "locate", input: { purpose, hints } } }.

Fill (Reconciliable)

import { fillEnvelope } from "@bobby-browser/sdk";
await client.submit(
  fillEnvelope(meta, "email field", { kind: "text", text: "a@example.com", clearFirst: true }),
  { idempotencyKey: crypto.randomUUID() },
);

FillValue kinds: text, select, files (files need file:upload).

SubmitAndVerify (Boundary)

import { submitAndVerifyEnvelope } from "@bobby-browser/sdk";
await client.submit(
  submitAndVerifyEnvelope(meta, "submit login", { /* WaitForCommand expectedState */ }),
  { idempotencyKey: crypto.randomUUID() },
);

WaitForState (Replayable)

import { waitForStateEnvelope } from "@bobby-browser/sdk";
await client.submit(
  waitForStateEnvelope(meta, { /* WaitCondition */ }, 15_000),
  { idempotencyKey: crypto.randomUUID() },
);

Follow

import { followEnvelope } from "@bobby-browser/sdk";
await client.submit(
  followEnvelope(meta, "docs link", { /* expectedDestination WaitForCommand */ }, { boundary: false }),
  { idempotencyKey: crypto.randomUUID() },
);

Set boundary: true when activation may mutate (for example sign-out); requires a matching workflow checkpoint.

DismissObstruction (Reconciliable)

Clears a popup / overlay / cookie banner. No caller boundary flag — always reconciliable. Default timeoutMs is 5000.

import { dismissObstructionEnvelope } from "@bobby-browser/sdk";
await client.submit(
  dismissObstructionEnvelope(meta, "dismiss cookie banner"),
  { idempotencyKey: crypto.randomUUID() },
);

Extract (Replayable)

import { extractEnvelope } from "@bobby-browser/sdk";
await client.submit(
  extractEnvelope(meta, "product fields", [
    { name: "title", purpose: "product title", value: { kind: "text" } },
    { name: "link", purpose: "product link", value: { kind: "href" } },
  ]),
  { idempotencyKey: crypto.randomUUID() },
);

ExtractValueKind: text, attribute (+ attribute name), href.

Vision double-gate

Vision-assisted resolution is deny-by-default. Both must pass:

Bearer holds vision:assist
Session created with executionPolicy.visionAssist = true

Otherwise vision escalation is denied.

Purpose bounds

Intent purpose strings are non-empty and bounded (see MAX_INTENT_PURPOSE_BYTES in the TypeScript SDK). Helpers call assertIntentPurpose.