fix: image handling (both directions) + never reply with reaction-only#17
Open
pdsullivan wants to merge 2 commits into
Open
fix: image handling (both directions) + never reply with reaction-only#17pdsullivan wants to merge 2 commits into
pdsullivan wants to merge 2 commits into
Conversation
Outbound: OpenAI retired dall-e-3, so every generate_image call was failing with a 400. Switch to gpt-image-1, which returns base64 instead of a hosted URL - store the bytes in an in-memory store and serve them at GET /images/:id so the Linq API has a URL to download from. The public base URL comes from PUBLIC_BASE_URL or is learned from incoming webhook Host headers. Inbound: images over Claude's 8000px dimension limit (long screenshots) 400'd the whole request and the user got no reply. Fetch incoming images and downscale oversized ones to 2048px JPEG via sharp before sending to Claude; normal images still pass through as URL sources. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Over the last 72h, 26% of Sonnet's replies were reaction-only with no text (e.g. thumbs-upping 'how can I be an expert in geoguessr?'), despite prompt rules against it. In group chats the Haiku classifier already routes reaction-worthy messages to a quick reaction before Sonnet is invoked, so a reaction-only result from chat() is a misfire in any chat type. Two changes: - Tighten the reaction rules in the system prompt (reactions are a bonus on top of text, never a replacement; drop the 'like = no text needed' guidance that invited this) - Deterministic guardrail: if Sonnet returns a reaction with no text and no other action, continue the tool loop with a tool_result asking for the actual text reply Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problems (all found in prod logs)
1. Outbound image generation broken: OpenAI retired the
dall-e-3model, so everygenerate_image/set_group_chat_iconcall was failing instantly with400 The model 'dall-e-3' does not exist.2. Inbound image analysis broken for large images: images exceeding Claude's 8000px dimension limit (typically long iPhone screenshots) caused the entire Claude API request to 400 — the user got no reply at all.
3. Reaction-only replies to real questions: over a 72h window, 26% of Sonnet's replies (221 of ~834) were a tapback with no text — e.g. thumbs-upping "How can I be an expert in geoguessr game?" — despite existing prompt rules against it.
Fixes
Outbound images
gpt-image-1(1024x1024, medium quality — roughly the same ~$0.04/image as before)gpt-image-1returns base64 only (no hosted URL like DALL-E did), but the Linq media API needs a fetchable URL. Added a small in-memory image store (src/state/imageStore.ts, 15 min TTL) served atGET /images/:id; Linq downloads the image immediately after send, so short-lived in-process storage is sufficient while the app runs single-instancePUBLIC_BASE_URLif set, otherwise it's learned from theHostheader of incoming webhook requests — works for both App Runner and ngrok with zero configInbound images
sharp(with EXIF rotation), then sent as base64sharppinned to^0.33.5(0.34+ requires Node ≥ 20.9)Reaction-only replies
chat(): if Sonnet returns a reaction with no text and no other action, continue the tool loop with atool_resulttelling it a reaction isn't a sufficient reply and to write the actual text response. In group chats the Haiku classifier already routes reaction-worthy messages to a quick reaction before Sonnet is invoked, so a reaction-only result fromchat()is a misfire in any chat typeVerification
dall-e-3is gone andgpt-image-1works on our OpenAI account via direct API callsgpt-image-1generation → stored → served bytes are a valid 1024x1024 PNG; a synthetic 9000x500 image resizes to 2048x114 JPEG through the exactprepareImageBlockpipelinechat()with the actual GeoGuessr question from the bug report🤖 Generated with Claude Code