grok: degrade tool-result images instead of 400 (L1) + opt-in vision via CCP_GROK_TOOL_IMAGE (L2) - #69
Merged
Merged
Conversation
raine
force-pushed
the
release/grok-tool-image
branch
2 times, most recently
from
July 28, 2026 13:29
5b8eb32 to
d9eaadf
Compare
added 4 commits
July 28, 2026 15:33
Mirror the Codex translator: image children in tool_result become "[image omitted: <media_type>]" (base64) / "[image omitted: url]" fragments joined with text children via "\n", and top-level user image blocks degrade to the same placeholder text rather than erroring with "unsupported content block: image". Multi-part text tool_results are newline-joined to match Codex. Malformed image blocks (bad source, unknown fields) still fail strict validation.
Reattach mode keeps the image omission marker in each tool result and sends gate-passing images in a following user message as input_image data URLs. Minimum dimensions, decoded-size limits, and the request-wide last-four cap degrade individual failures to reasoned omission markers. URL sources remain omitted because the proxy cannot validate them without fetching. Traffic captures redact image data URLs and Anthropic source.data payloads. Token estimates charge a flat amount for each attached image. Omit remains the default, and reject retains strict image errors.
Inline mode sends image-bearing function call outputs as untagged arrays of input_text and input_image parts. String and text-only outputs retain their plain-string wire shape. The same dimension, decoded-size, URL, and request-wide image gates apply as reattach mode. Per-image failures remain visible as omission text within mixed arrays. Top-level user images use input_image parts in both vision modes. Token counting charges the image estimate for inline parts, and traffic capture redaction preserves array structure without persisting image payloads.
Keep startup mode validation consistent with request parsing and apply conservative image gates before opt-in vision payloads reach Grok. Redact Anthropic and provider image representations across nested traffic capture shapes while preserving useful JSON structure. Keep unsupported formats and unverifiable URLs on the per-image omission path instead of failing the request.
raine
force-pushed
the
release/grok-tool-image
branch
from
July 28, 2026 13:36
d9eaadf to
26e3cf1
Compare
Use format metadata when estimating decoded image size so ordinary 8-bit screenshots under the upstream limit retain their pixels. Keep conservative alpha accounting for formats that can carry transparency. Redact image source types and data URL markers case-insensitively so malformed pre-translation captures cannot persist image payloads. Pin omission tests to their intended mode so developer environment settings do not affect results. Place the feature notes in the active unreleased changelog section and document the image policy in the current Grok and configuration documentation.
raine
force-pushed
the
release/grok-tool-image
branch
from
July 28, 2026 13:39
26e3cf1 to
ed3bf32
Compare
Owner
|
Thank you |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Grok provider currently hard-fails with HTTP 400 (
tool result supports text children only) whenever atool_resultcontains an image block — which is what every Claude CodeRead(png)produces. Top-level user images fail similarly (unsupported content block: image). This makes any session that reads a screenshot/diagram unrecoverable on the Grok path, even though Codex (and Cursor) already degrade gracefully and the upstream CLI endpoint (cli-chat-proxy.grok.com/v1/responses) acceptsinput_imagefine.This PR fixes that in two layers, with default behaviour = the safe layer:
L1 — omit (default, mirrors Codex'
tool_result_to_string). Image children intool_resultdegrade to[image omitted: {media_type}]/[image omitted: url]string fragments joined into the tool output; top-level user images degrade the same way. Malformed blocks still error (strict field validation per house style).L2 — opt-in vision behind
CCP_GROK_TOOL_IMAGE. Setreattach(omit marker in the tool output + images sent asinput_imageparts in a following user message) orinline(tool output as an untagged array ofinput_text+input_imageparts; string-only outputs serialize byte-identical to before).rejectrestores the old bail for debugging. Live-verified againstcli-chat-proxy.grok.com(model correctly answered visual questions in both modes).Gates match upstream-verified limits, and any gate failure degrades that single image to an omit fragment with a reason — never a 400 for the whole turn:
reattach(can't gate without fetching)image_url+ anthropicsource.data), images counted in token estimatesImplementation notes: mode threaded explicitly through
translate_request_with_mode(pure, env-free tests); dimension checks are hand-rolled PNG/JPEG/GIF header parsers — no new dependencies.Test plan
cargo test— 624 passed / 0 failed (10 suites); 26 new grok tests covering all four modes, gates, caps, serialization regression, redactioncargo fmt --checkcleancli-chat-proxy.grok.com(via Claude CodeRead(png)+ follow-up tool call): default omit — upstream 200, omit marker in outbound request, no base64 leakage;reattach/inline— model correctly identified test image contents; 1×1 PNG degrades with reason, no 400🤖 Generated with Claude Code