Add an in-app live browser view for agent-browser CDP and headless sessions

[manishprakashiitg]

Summary

Add a workflow-scoped Live Browser view inside the AgentWorks Electron app so users can watch the exact tab/page currently controlled by the managed agent_browser tool.

The experience should be a low-latency live stream, not periodic screenshot polling. It must support both:

1. external/user-visible Chrome attached through configured CDP ports; and
2. isolated headless Chromium sessions owned by agent-browser.

Related browser consolidation/migration work: #128.

User problem

Agent-browser can automate, inspect, screenshot, and record a page, but the Electron app cannot currently show that page live. Users must switch to external Chrome in CDP mode, and headless workflow execution has no useful live visual surface. Periodic screenshots are too stale and visually poor for QA, debugging, login assistance, and understanding what the agent is doing.

Current architecture

CDP mode

• AgentWorks knows the configured Chrome CDP port.
• Workflow ownership is mapped to a stable labeled tab.
• Shared-tab actions are serialized per CDP port.
• A live viewer can attach to the exact owned page target and use the Chrome DevTools Protocol Page.startScreencast stream.

Headless mode

• Agent-browser owns a private daemon and Chromium process per session.
• AgentWorks currently tracks session identity and daemon/Chrome PIDs primarily for limits and cleanup.
• The underlying page CDP target/WebSocket is not exposed to AgentWorks.
• The current command/response integration supports screenshots and recordings but no persistent frame stream.

Chromium has the underlying screencast capability in both modes. The missing part for headless mode is a safe, managed transport from the agent-browser-owned target to AgentWorks.

Proposed design

Introduce a backend abstraction such as:

Electron Live Browser panel
        |
Authenticated browser-frame WebSocket
        |
Managed BrowserStreamService
    |-- external CDP tab source
    `-- agent-browser headless session source

Backend responsibilities

• Resolve a requested workflow/execution/browser session to its exact owned tab or headless page.
• Start and stop the stream through the managed browser layer; the renderer must not connect to raw CDP directly.
• For CDP, preserve existing port ownership, tab selection, and select-and-act locking rules.
• For headless, add a supported stream capability to the agent-browser daemon/integration or expose a scoped internal page endpoint. Do not expose an unauthenticated debugging port.
• Decode CDP base64 frames server-side when useful and send binary WebSocket frames to the renderer.
• Keep at most the latest frame; drop stale queued frames.
• Acknowledge CDP frames with Page.screencastFrameAck to provide backpressure.
• Stop capture when the viewer closes, the app is minimized, the execution/session ends, or the target disappears.
• Emit session/tab metadata and lifecycle events: URL, title, tab label, mode, CDP port where applicable, connected/disconnected, target changed, and stream stopped reason.

Frontend responsibilities

• Add a workflow/execution-scoped Live Browser panel, not a global ambiguous browser window.
• Render the stream into a responsive canvas while preserving aspect ratio.
• Show mode (CDP or Headless), workflow/execution owner, session/tab label, URL, and connection state.
• Allow selection only among tabs/sessions owned by the current workflow/execution.
• In CDP mode, offer Open in Chrome.
• Initial release may be read-only; interactive input can be a second phase.

Resource controls

The viewer must not become an always-on capture system.

Recommended defaults:

• stream only while the Live Browser panel is visible;
• stream only one selected tab/session per viewer;
• JPEG at approximately quality 60-70;
• maximum dimensions approximately 1280x720;
• target approximately 8-12 displayed FPS;
• latest-frame-only buffering;
• pause/stop when hidden or minimized;
• never persist frames unless explicit recording is requested;
• cap concurrent viewers globally and per workflow;
• expose lightweight diagnostics for current FPS, dropped frames, bandwidth, and stream age.

Multiple workflows may run concurrently, but the UI must not silently start a screencast for every active browser session.

Security and privacy

A CDP connection provides powerful access to page contents, cookies, storage, and browser control. Therefore:

• never expose raw CDP endpoints or WebSocket debugger URLs to the renderer;
• authenticate and authorize every stream against the AgentWorks session and workflow owner;
• bind headless stream endpoints to localhost/internal transport only;
• do not log or persist frame contents;
• stop streams and revoke viewer leases when sessions end;
• prevent one workflow/user from viewing another workflow's tab;
• treat the live view as sensitive UI and apply the same visibility rules as terminal/session output.

Phased implementation

Phase 1: CDP read-only live view

• Implement the managed stream service against an existing workflow-owned CDP tab.
• Add frame throttling, acknowledgement, latest-frame buffering, lifecycle cleanup, and Electron canvas rendering.
• Confirm normal agent-browser actions and shared-tab locking continue to work while viewing.

Phase 2: headless read-only live view

• Add a supported bridge to the agent-browser-owned Chromium target.
• Reuse the same frontend and stream protocol.
• Ensure session limits, cleanup, and one-daemon/one-Chrome ownership remain intact.

Phase 3: optional interaction

• Forward mouse, scroll, and keyboard events through a managed backend path.
• Handle coordinate scaling, page scale factor, resize, focus, dialogs, clipboard, downloads, and file choosers.
• Clearly arbitrate user input versus agent actions to avoid races.

Acceptance criteria

• A user can open a workflow/execution-level Live Browser panel for an active CDP session.
• The panel shows the exact labeled tab owned by that workflow, not whichever Chrome tab is globally active.
• Visual updates are streamed with low latency and without screenshot polling.
• Opening the viewer does not bypass or break existing CDP tab ownership/locking.
• A user can open the same panel for an active headless agent-browser session.
• CDP and headless use one frontend protocol and UI.
• Closing/hiding the panel stops capture and releases resources promptly.
• Frames are latest-only; a slow renderer cannot create an unbounded queue.
• Multiple active workflows do not automatically create multiple streams.
• Authorization prevents cross-workflow session/tab viewing.
• No raw debugger endpoint, cookie, frame, or page content is persisted or logged.
• Tests cover target ownership, disconnect/reconnect, workflow completion cleanup, hidden-panel cleanup, throttling/backpressure, and concurrent workflows.
• Runtime metrics demonstrate acceptable CPU, memory, and bandwidth at the chosen defaults.

Non-goals

• Do not embed or re-parent an external Google Chrome window inside Electron; Electron cannot directly host an external Chrome tab as a native WebContentsView.
• Do not replace agent-browser automation with an Electron webview.
• Do not use periodic screenshots as the primary live-view transport.
• Do not enable frame capture for all browser sessions by default.
• Do not expose raw CDP access to frontend code.

Open questions

1. Can the pinned agent-browser version expose a supported live frame stream or scoped page endpoint, or is an upstream contribution required?
2. Should screencast ownership be treated like existing exclusive recording/HAR features, or can one read-only viewer coexist safely with diagnostics?
3. What global/per-workflow viewer limits should ship initially?
4. Should the first release be strictly read-only, with interaction tracked in a follow-up ticket?
5. Should scheduled/background workflows surface a notification or badge when a live browser becomes available?