Reusable workflows with guarded action caching and self-healing replay

[alectimison-maker]

Problem

Repeated browser tasks currently pay the planning/tool-selection cost on every run, even after WebBrain has completed the same flow successfully. Custom skills provide Markdown instructions plus optional HTTP tools, but they are not an executable, parameterized workflow layer. Traces contain the successful tool chain, but replaying a trace verbatim would be unsafe and brittle:

• ref_id values only remain valid while their live DOM elements survive; navigation and SPA rerenders invalidate them.
• trace arguments may contain user-entered values, including credentials, so copying raw arguments into durable storage can persist secrets.
• a saved workflow must not inherit or bypass capability grants, submit confirmation, WebMCP confirmation, /allow-api, or completion verification.

This gap is also visible in docs/claude-chrome-comparison.md, which calls out missing shortcuts_list / shortcuts_execute primitives. Similar projects are converging on this layer:

• Stagehand caches observed actions/agent steps, reuses them without another LLM call, and falls back when the page no longer matches: https://docs.stagehand.dev/v3/best-practices/caching
• BrowserOS describes saved, repeatable visual workflows: https://docs.browseros.com/

Proposed direction

Add a local-only Saved Workflows layer that can compile a successful run into a sanitized, parameterized workflow, replay matching steps deterministically, and hand control back to the agent when the page changes.

This should reuse the existing trace recorder as an input source, but define a separate workflow schema rather than treating webbrain-trace/1 as executable.

Workflow schema (suggested v1)

• name, description, created/updated timestamps, source WebBrain version
• allowed start origin and URL-family pattern
• declared runtime parameters, with sensitive parameters stored as placeholders only
• ordered steps containing:
  • tool name and sanitized/parameterized arguments
  • a semantic target descriptor where applicable (role, accessible name, label, field metadata, href, stable id/selector evidence)
  • expected precondition and postcondition
  • retry/fallback policy
• no reusable raw ref_id, element index, screen coordinate, password, OTP, API key, cookie, hidden form token, or captured request body/header

Replay rules / safety invariants

1. Validate the current origin and URL family before starting.
2. Rebuild the current accessibility tree and resolve each target from its semantic descriptor. Never reuse a historical ref_id; ambiguity must fail closed.
3. Route replayed calls through the normal agent dispatch path so capability × origin permissions, submit confirmation, form verification, credential-field handling, WebMCP's mandatory per-call confirmation, and /allow-api still apply. Saving a workflow grants no authority.
4. Initially allow direct replay only for an explicit tool allowlist. Consequential actions require their normal fresh checks and an observable postcondition before advancing.
5. On locator mismatch, failed verification, unexpected navigation, or unknown action outcome, stop deterministic replay and resume the model from that step with current page evidence and the remaining workflow intent.
6. A self-healed locator/update is saved only after successful verification and explicit user opt-in; page content can never rewrite a workflow by itself.
7. Keep unattended/scheduled consequential execution out of the MVP. A future scheduler integration must preserve scheduledRequireConsequentialConfirmation semantics.

Suggested phases

Phase 1 — local saved workflow model and guided execution

• save/list/rename/delete/run UI
• local versioned schema + import/export validation
• parameter prompt that never pre-fills secret values from traces
• semantic target extraction and current-page re-resolution
• replay telemetry in traces (matched, miss, model_fallback, healed) without recording secret parameter values

Phase 2 — guarded deterministic replay

• direct replay for a reviewed allowlist
• per-step pre/postcondition validation
• model fallback from the first mismatch instead of restarting the task
• optional, user-approved locator healing

Phase 3 — product surface

• model-callable workflows_list and workflow_execute (or equivalent slash commands)
• optional scheduler integration
• visual graph editor only if the simpler list/editor proves useful

Out of scope for the first implementation

• cloud sync or sharing workflows
• a credential vault
• replaying captured network bodies/headers or opaque API replay IDs
• unattended consequential workflows
• a visual graph builder

Acceptance criteria

• Chrome and Firefox use the same workflow schema and pure normalization/validation tests.
• Saving a trace with form input does not persist secret literals or historical ref_id values.
• A workflow can rerun on an unchanged page with fewer LLM calls than the original run.
• A DOM rerender/navigation causes fresh target resolution rather than a stale action.
• Ambiguous targets, changed origins, permission denial, and unknown outcomes stop or fall back safely.
• Existing permission, submit-confirmation, verify_form, WebMCP, and completion invariants have regression coverage for replayed calls.
• Replay exposes hit/miss/fallback metrics so reliability and token savings can be measured.

Why this fits WebBrain

Most of the necessary safety substrate already exists: local trace events with structured tool args/results, stable-within-document AX references plus semantic element metadata, deterministic capability/origin permissions, submit confirmation, verified field operations, loop detection, and scheduled-task policies. The missing piece is a durable compiler/executor between traces and the agent loop—not another set of site-specific prompt instructions.

[esokullu]

love it, will look into your PR today!