core: view_image workspace tool + returns_images ambient gating; v0.9.46 - #192
Merged
Conversation
The agent can now look at images it produces (issue #172, PR 2 of 3): - view_image joins the workspace read tools: read_file's exact path pipeline (read_outside honored, same approval predicate), extension mime sniff via the shared parts.image_mime, and a 5MB decoded cap whose refusal tells the model to downscale via shell. - WorkspaceSession grows read_bytes (sized before reading, so the cap bounds memory; FileTooLargeError lets callers attach their own remedy). - Tool.returns_images: descriptive output-side metadata, dual of parameters. The runner filters *ambient* sources (workspace/plugin bundles) for providers without vision and logs the drop; agent.tools is never filtered — adapter degrade covers it, so the flag is a UX hint, never load-bearing for correctness. - Live-verified end-to-end: qwen reads a workspace PNG through view_image and names its color. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds first-party image viewing support to the workspace tool bundle, enabling vision-capable providers to receive ImagePart results from a new view_image tool while keeping text-only providers safe via ambient tool filtering (Tool.returns_images).
Changes:
- Introduces
view_imageworkspace tool that returns[TextPart(path), ImagePart(...)], with extension-based MIME gating and a 5MB byte cap enforced before read. - Adds
WorkspaceSession.read_bytes()(andFileBytes/FileTooLargeError) to support bounded binary reads used byview_image. - Implements
Tool.returns_imagesand runner-side filtering of ambient image-returning tools for providers without vision; adds unit + live coverage.
Reviewed changes
Copilot reviewed 16 out of 17 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| uv.lock | Bumps package version to v0.9.46 in lockfile. |
| pyproject.toml | Bumps project version to v0.9.46. |
| lovia/init.py | Updates __version__ to v0.9.46. |
| lovia/parts.py | Factors out shared image_mime() extension gate used by image producers. |
| lovia/tools/base.py | Adds Tool.returns_images metadata + decorator support. |
| lovia/runtime/loop.py | Filters ambient tools that return images when provider lacks vision capability. |
| lovia/workspace/workspace.py | Adds view_image to the workspace tool bundle (subject to runner filtering). |
| lovia/workspace/types.py | Adds FileBytes type for binary read results. |
| lovia/workspace/protocol.py | Extends WorkspaceSession protocol with read_bytes(max_bytes=...). |
| lovia/workspace/local.py | Implements LocalWorkspaceSession.read_bytes() with pre-read size cap + lock. |
| lovia/workspace/errors.py | Adds FileTooLargeError for actionable, caller-specific remedies. |
| lovia/workspace/init.py | Re-exports FileBytes and FileTooLargeError. |
| lovia/workspace/tools.py | Implements view_image tool with MIME gating, size cap, and image parts output. |
| tests/workspace/test_workspace_tools.py | Adds direct tool tests for view_image behavior and approvals. |
| tests/workspace/test_workspace_agent.py | Adds runner-level tests for ambient filtering vs explicit tool wiring. |
| tests/workspace/test_local_session.py | Updates expected tool bundle + tests read_bytes cap semantics. |
| tests/providers/test_live.py | Adds end-to-end live test verifying view_image reaches a vision model. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR 2 of 3 for #172 (design in this comment; PR 1 was #191): the tool and its gating. With this, a vision-capable model can look at any image in its workspace — including ones it just produced.
view_imageread_file's sibling): exact same path pipeline —read_outsidehonored, same_path_needs_approval("read")predicate — so/tmpscreenshots work undercodingwith approval, and the old hard outside-root refusal pattern is gone.parts.image_mime(also the future gate for the web layer's delegation tool in PR 3); jpg/jpeg/png/gif/webp.sips -Z 1568/magick -resize) and the agent loop self-serves. Sized before reading via the newWorkspaceSession.read_bytes(path, max_bytes=…), so the cap also bounds memory;FileTooLargeErroris distinct so callers attach use-case-specific remedies.[TextPart(display_path), ImagePart(...)]— projection readsshots/app.png\n[image: image/png, 210.3 KB].Gating:
Tool.returns_imagesResolves the issue's "not auto-added blind" question as settled in the design:
parameters), not a precondition flag.agent.tools— the user's own explicit wiring — is never filtered; PR 1's adapter degrade covers it. No flag is ever load-bearing for correctness.describe_image(PR 3) falls out naturally: for text-only mainsview_imagedisappears from the bundle on its own.Verified live
test_live_view_image_workspace_tool_end_to_end: qwen (vision, OpenAI flavor) calls the built-inview_imageon a workspace PNG and names its color — the full loop through path policy,read_bytes, parts, and the PR 1 synthetic-user-message wire.Web transition note: the SSE
tool_resultevent already sendsoutputonly, so until PR 3 lands the web UI shows the textual projection for image results — graceful, no payload bloat.🤖 Generated with Claude Code