feat(server): let agents save browser screenshots - #6387
Draft
adinschmidt wants to merge 1 commit into
Draft
Conversation
Agents could save a recording of a collaborative browser tab but had no way to save a still image. preview_snapshot returns a PNG inline for the agent to look at, and the desktop camera button writes a file, but that path was never exposed over MCP. Add a preview_screenshot tool that reuses the existing desktop capture and returns the saved artifact's path, so an agent can hand the user a still the same way it hands over a recording.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Agents can save a recording of a collaborative browser tab, but they have no way to save a still.
preview_snapshotreturns a downscaled PNG inline for the agent to look at, and the desktop camera button writes a file, but that capture was never exposed over MCP — so an agent that just wants to hand the user a still image has to record a video instead.This adds a
preview_screenshottool that reuses the existing desktop capture and returns the saved artifact's path, mirroringpreview_recording_stop:Artifacts land in the same
browser-artifactsdirectory as recordings and the camera button, so reveal and copy-path work unchanged.The tool returns a path rather than image data on purpose:
preview_snapshotalready covers "let the agent see the page", and re-encoding a full-resolution PNG into the transcript on every capture would be a lot of tokens for an evidence artifact the human is meant to open."screenshot"joinsPREVIEW_AUTOMATION_OPERATIONSbut not the V1 list, so desktop hosts predating this change are filtered out by the broker and get a capability error rather than an operation they cannot serve.One inherited constraint worth knowing: capture needs a composited tab. This calls the same
webContents.capturePage()aspreview_snapshot, which fails withUnknownVizErroron a hidden guest — so a tab opened withopen: false, or one whose thread is not routed in the desktop window, will not produce an artifact.Verification
Against a dev desktop build, an agent called
preview_screenshotafterpreview_openand reportedsizeBytes: 57265, matching the artifact written to disk byte for byte. The image below is that artifact, unmodified — full resolution, not downscaled the way snapshot captures are.tsgo --noEmitis clean for contracts, server, and web; the preview toolkit and automation broker tests pass.Model: Claude Opus 5, via Claude Code.
Note
Add
preview_screenshottool to let agents capture browser tab screenshotspreview_screenshotMCP tool toPreviewToolkitandPreviewStandardToolkitthat captures a full-resolution PNG of a targeted browser tab and returns aPreviewAutomationScreenshotArtifactwith path, size, and timestamp.PreviewAutomationHostto handle thescreenshotautomation operation by callingbridge.captureScreenshotwith the runtime tab id.screenshotoperation toPREVIEW_AUTOMATION_OPERATIONSand defines thePreviewAutomationScreenshotArtifactschema in previewAutomation.ts.Macroscope summarized 63b662c.