Skip to content

core: view_image workspace tool + returns_images ambient gating; v0.9.46 - #192

Merged
cymoo merged 1 commit into
mainfrom
view-image-tool
Aug 8, 2026
Merged

core: view_image workspace tool + returns_images ambient gating; v0.9.46#192
cymoo merged 1 commit into
mainfrom
view-image-tool

Conversation

@cymoo

@cymoo cymoo commented Aug 8, 2026

Copy link
Copy Markdown
Owner

PR 2 of 3 for #172 (design in this comment; PR 1 was #191): the tool and its gating. With this, a vision-capable model can look at any image in its workspace — including ones it just produced.

view_image

  • Joins the workspace read tools (read_file's sibling): exact same path pipelineread_outside honored, same _path_needs_approval("read") predicate — so /tmp screenshots work under coding with approval, and the old hard outside-root refusal pattern is gone.
  • Extension mime sniff via the new shared parts.image_mime (also the future gate for the web layer's delegation tool in PR 3); jpg/jpeg/png/gif/webp.
  • 5 MB decoded cap (Anthropic's per-image limit, the strictest provider; deliberately a constant, not config). The refusal is actionable — it tells the model to downscale via shell (sips -Z 1568 / magick -resize) and the agent loop self-serves. Sized before reading via the new WorkspaceSession.read_bytes(path, max_bytes=…), so the cap also bounds memory; FileTooLargeError is distinct so callers attach use-case-specific remedies.
  • Returns [TextPart(display_path), ImagePart(...)] — projection reads shots/app.png\n[image: image/png, 210.3 KB].

Gating: Tool.returns_images

Resolves the issue's "not auto-added blind" question as settled in the design:

  • A descriptive output-side field (dual of parameters), not a precondition flag.
  • The runner filters ambient sources only (workspace + plugin bundles) when the provider lacks vision, with an info log. agent.tools — the user's own explicit wiring — is never filtered; PR 1's adapter degrade covers it. No flag is ever load-bearing for correctness.
  • Mutual exclusion with the web describe_image (PR 3) falls out naturally: for text-only mains view_image disappears from the bundle on its own.

Verified live

test_live_view_image_workspace_tool_end_to_end: qwen (vision, OpenAI flavor) calls the built-in view_image on a workspace PNG and names its color — the full loop through path policy, read_bytes, parts, and the PR 1 synthetic-user-message wire.

Web transition note: the SSE tool_result event already sends output only, so until PR 3 lands the web UI shows the textual projection for image results — graceful, no payload bloat.

🤖 Generated with Claude Code

The agent can now look at images it produces (issue #172, PR 2 of 3):

- view_image joins the workspace read tools: read_file's exact path
  pipeline (read_outside honored, same approval predicate), extension
  mime sniff via the shared parts.image_mime, and a 5MB decoded cap
  whose refusal tells the model to downscale via shell.
- WorkspaceSession grows read_bytes (sized before reading, so the cap
  bounds memory; FileTooLargeError lets callers attach their own remedy).
- Tool.returns_images: descriptive output-side metadata, dual of
  parameters. The runner filters *ambient* sources (workspace/plugin
  bundles) for providers without vision and logs the drop; agent.tools
  is never filtered — adapter degrade covers it, so the flag is a UX
  hint, never load-bearing for correctness.
- Live-verified end-to-end: qwen reads a workspace PNG through
  view_image and names its color.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 8, 2026 10:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds first-party image viewing support to the workspace tool bundle, enabling vision-capable providers to receive ImagePart results from a new view_image tool while keeping text-only providers safe via ambient tool filtering (Tool.returns_images).

Changes:

  • Introduces view_image workspace tool that returns [TextPart(path), ImagePart(...)], with extension-based MIME gating and a 5MB byte cap enforced before read.
  • Adds WorkspaceSession.read_bytes() (and FileBytes/FileTooLargeError) to support bounded binary reads used by view_image.
  • Implements Tool.returns_images and runner-side filtering of ambient image-returning tools for providers without vision; adds unit + live coverage.

Reviewed changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated no comments.

Show a summary per file
File Description
uv.lock Bumps package version to v0.9.46 in lockfile.
pyproject.toml Bumps project version to v0.9.46.
lovia/init.py Updates __version__ to v0.9.46.
lovia/parts.py Factors out shared image_mime() extension gate used by image producers.
lovia/tools/base.py Adds Tool.returns_images metadata + decorator support.
lovia/runtime/loop.py Filters ambient tools that return images when provider lacks vision capability.
lovia/workspace/workspace.py Adds view_image to the workspace tool bundle (subject to runner filtering).
lovia/workspace/types.py Adds FileBytes type for binary read results.
lovia/workspace/protocol.py Extends WorkspaceSession protocol with read_bytes(max_bytes=...).
lovia/workspace/local.py Implements LocalWorkspaceSession.read_bytes() with pre-read size cap + lock.
lovia/workspace/errors.py Adds FileTooLargeError for actionable, caller-specific remedies.
lovia/workspace/init.py Re-exports FileBytes and FileTooLargeError.
lovia/workspace/tools.py Implements view_image tool with MIME gating, size cap, and image parts output.
tests/workspace/test_workspace_tools.py Adds direct tool tests for view_image behavior and approvals.
tests/workspace/test_workspace_agent.py Adds runner-level tests for ambient filtering vs explicit tool wiring.
tests/workspace/test_local_session.py Updates expected tool bundle + tests read_bytes cap semantics.
tests/providers/test_live.py Adds end-to-end live test verifying view_image reaches a vision model.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@cymoo
cymoo merged commit 69dd67f into main Aug 8, 2026
10 checks passed
@cymoo
cymoo deleted the view-image-tool branch August 8, 2026 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants