web: render tool-result images — describe_image rename, transcript image route; v0.9.47 - #193
Merged
Conversation
…age route; v0.9.47 The web layer's half of issue #172 (PR 3 of 3): - see_image -> describe_image: the name states the contract (a text description, never pixels). Slimmed to a shell over the shared core pipeline: same read path policy as read_file/view_image (read_outside honored, ask via the approval predicate — fixes the old hard outside-root refusal bypassing policy), shared image_mime sniff and 5MB cap. make_describe_image_tool no longer needs workspace_root. - Tool-result images serve from the persisted transcript: GET /api/sessions/{sid}/tool-images/{call_id}/{index} decodes the stored entry (live-run mirror first, then session, then a resumable checkpoint), so the UI shows exactly the pixels the model saw — /tmp paths and since-changed files included. Immutable cache headers. - SSE tool_result and session-detail tool messages carry byte-free image stubs ({index, mime_type}); the supervisor mirror reconstructs parts from the tool's raw return value so a live run's images are servable before any checkpoint lands. - Frontend: thumbnail strip under the tool result (auto-opens the card — the image is the payload), lightbox on click, chip fallback when a fetch fails. media.py's model image table now aliases the core one. Verified end-to-end with the stub-model recipe: live SSE and reload replay both render the strip from the transcript route; lightbox works. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR completes the web-layer work for #172 by renaming the delegation vision tool to describe_image, adding transcript-backed serving of tool-result images via a new sessions API route, and wiring UI rendering so tool-result images show as thumbnails/lightbox without inlining base64.
Changes:
- Rename
see_image→describe_imageand align its read/approval/size-cap behavior with the core workspace read pipeline. - Add tool-result image “stubs” to SSE + session replay, plus a new route to fetch the exact stored image bytes from the transcript.
- Render tool-result images in the frontend (thumbnail strip + lightbox), and unify model-image mime handling via the core table.
Reviewed changes
Copilot reviewed 21 out of 22 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| uv.lock | Bump editable package version to 0.9.47. |
| pyproject.toml | Bump project version to 0.9.47. |
| lovia/init.py | Bump __version__ to 0.9.47. |
| lovia/workspace/tools.py | Export path_needs_approval and update built-in tools to use it. |
| lovia/web/vision.py | Implement describe_image as a workspace-session reader with shared limits/approval. |
| lovia/web/builder.py | Wire describe_image instead of see_image under the same gating rules. |
| lovia/web/config/schema.py | Update role comment to “no describe_image delegation”. |
| lovia/web/attachments.py | Update docs/comments to reference describe_image. |
| lovia/web/supervisor.py | Mirror tool-result parts for live tool-image serving before checkpoint persistence. |
| lovia/web/sse.py | Add byte-free images stubs to tool_result SSE events when parts include images. |
| lovia/web/schemas.py | Add ToolImageOut and MessageOut.images for session replay. |
| lovia/web/api/serialization.py | Stamp tool image stubs onto replayed tool messages from stored transcript entries. |
| lovia/web/api/sessions.py | Add GET /api/sessions/{sid}/tool-images/{call_id}/{index} to serve stored image bytes. |
| lovia/web/media.py | Alias web model-image mime table to the core parts table and delegate model_image_mime. |
| lovia/web/static/js/api.js | Add toolImageUrl() helper for the new tool-images route. |
| lovia/web/static/js/i18n.js | Add i18n string for tool-result image alt text. |
| lovia/web/static/js/chat.js | Render tool-result image thumbnails and lightbox, for live SSE and history replay. |
| lovia/web/static/styles.css | Add styling for tool-result image thumbnails/chips. |
| tests/web/test_web.py | Add end-to-end test for SSE stubs + session replay + tool-images route; add no-stubs test for plain tool results. |
| tests/web/test_web_cli.py | Update default-agent wiring test to assert describe_image. |
| tests/web/test_vision_tool.py | Update and expand delegation tool tests for new behavior/policy/limits. |
| tests/web/test_config_api.py | Update config wiring test to assert describe_image. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+829
to
+834
| img.addEventListener('error', () => { | ||
| const chip = document.createElement('span'); | ||
| chip.className = 'tool-image-chip'; | ||
| chip.innerHTML = `${icon('image', { size: 13 })}<span>${im.mime_type || 'image'}</span>`; | ||
| img.replaceWith(chip); | ||
| }); |
Comment on lines
+215
to
+217
| if 0 <= index < len(images): | ||
| return images[index] | ||
| return None |
Comment on lines
+28
to
+32
| from ..parts import _IMAGE_MIME_BY_SUFFIX, image_mime | ||
|
|
||
| # Suffix → mime for the raster formats a vision model API accepts — the core | ||
| # table, aliased for the web modules that iterate it. | ||
| MODEL_IMAGE_MIME_BY_EXT = _IMAGE_MIME_BY_SUFFIX |
…ds, public IMAGE_MIME_BY_SUFFIX Copilot review fixes: the image-chip mime label is built with textContent (tool-supplied string, was an innerHTML injection vector); the tool-images lookup keeps scanning older entries when a reused call_id's newest entry lacks the requested index; media.py imports the now-public IMAGE_MIME_BY_SUFFIX instead of a private symbol. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR 3 of 3 for #172 (design in this comment; PR 1 #191, PR 2 #192): the web layer. Closes #172.
see_image→describe_imageThe name states the contract: the model gets a description, never pixels — the counterpart of core
view_image(real pixels), with which it stays mutually exclusive by construction. Slimmed to a shell over the shared core pipeline:read_file/view_image—read_outsidehonored, ask resolved by the approval predicate. This fixes the old hard outside-root refusal that bypassed the policy's expressiveness.parts.image_mimesniff and the 5 MB cap with the same actionable downscale refusal.make_describe_image_tool(vision_model)— no moreworkspace_rootparam; the image reads through the run's workspace session.Tool-result images in the UI
GET /api/sessions/{sid}/tool-images/{call_id}/{index}serves bytes straight from the stored entry — live-run in-memory mirror first (a checkpoint may not have landed yet), then the persisted session, then a resumable checkpoint. The UI therefore shows exactly the pixels the model saw —/tmppaths and since-changed files included. Immutable cache headers (transcripts are append-only).tool_resultevent and session-detail tool messages carryimages: [{index, mime_type}]; the supervisor mirror reconstructs parts from the tool's raw return value so a live run's images are servable immediately.media.py's model-image table now aliases the corepartstable — one source of truth for what a vision model ingests.Verified
MessageOut.images; the lightbox opens. Screenshot of the card:the
view_imagecall header, the projection text (shot.png/[image: image/png, 168 B]), and the red thumbnail.tsc --checkJsall green.🤖 Generated with Claude Code