Skip to content

fix: image handling (both directions) + never reply with reaction-only#17

Open
pdsullivan wants to merge 2 commits into
mainfrom
fix/image-handling
Open

fix: image handling (both directions) + never reply with reaction-only#17
pdsullivan wants to merge 2 commits into
mainfrom
fix/image-handling

Conversation

@pdsullivan

@pdsullivan pdsullivan commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Problems (all found in prod logs)

1. Outbound image generation broken: OpenAI retired the dall-e-3 model, so every generate_image / set_group_chat_icon call was failing instantly with 400 The model 'dall-e-3' does not exist.

2. Inbound image analysis broken for large images: images exceeding Claude's 8000px dimension limit (typically long iPhone screenshots) caused the entire Claude API request to 400 — the user got no reply at all.

3. Reaction-only replies to real questions: over a 72h window, 26% of Sonnet's replies (221 of ~834) were a tapback with no text — e.g. thumbs-upping "How can I be an expert in geoguessr game?" — despite existing prompt rules against it.

Fixes

Outbound images

  • Switch generation to gpt-image-1 (1024x1024, medium quality — roughly the same ~$0.04/image as before)
  • gpt-image-1 returns base64 only (no hosted URL like DALL-E did), but the Linq media API needs a fetchable URL. Added a small in-memory image store (src/state/imageStore.ts, 15 min TTL) served at GET /images/:id; Linq downloads the image immediately after send, so short-lived in-process storage is sufficient while the app runs single-instance
  • The public base URL for those links comes from PUBLIC_BASE_URL if set, otherwise it's learned from the Host header of incoming webhook requests — works for both App Runner and ngrok with zero config

Inbound images

  • Incoming images are now fetched and checked against Claude API limits (8000px max dimension, ~5MB). Oversized ones are downscaled to fit 2048px and re-encoded as JPEG via sharp (with EXIF rotation), then sent as base64
  • Normal-sized images pass through as URL sources exactly as before; any fetch/resize failure falls back to the previous URL-passthrough behavior
  • sharp pinned to ^0.33.5 (0.34+ requires Node ≥ 20.9)

Reaction-only replies

  • Tightened the reaction rules in the system prompt (reactions are a bonus on top of text, never a replacement; dropped the "like = simple acknowledgment when no text response needed" guidance that invited this)
  • Deterministic guardrail in chat(): if Sonnet returns a reaction with no text and no other action, continue the tool loop with a tool_result telling it a reaction isn't a sufficient reply and to write the actual text response. In group chats the Haiku classifier already routes reaction-worthy messages to a quick reaction before Sonnet is invoked, so a reaction-only result from chat() is a misfire in any chat type
  • Costs one extra Sonnet call only when the misfire happens; if the follow-up fails the reaction still goes out as before

Verification

  • Verified dall-e-3 is gone and gpt-image-1 works on our OpenAI account via direct API calls
  • Live end-to-end checks: real gpt-image-1 generation → stored → served bytes are a valid 1024x1024 PNG; a synthetic 9000x500 image resizes to 2048x114 JPEG through the exact prepareImageBlock pipeline
  • Reaction guardrail verified against the real API: forced a reaction-only first turn, confirmed the tool-loop continuation round-trips and produces a proper text answer; also smoke-tested the compiled chat() with the actual GeoGuessr question from the bug report

🤖 Generated with Claude Code

pdsullivan and others added 2 commits July 2, 2026 08:49
Outbound: OpenAI retired dall-e-3, so every generate_image call was
failing with a 400. Switch to gpt-image-1, which returns base64 instead
of a hosted URL - store the bytes in an in-memory store and serve them
at GET /images/:id so the Linq API has a URL to download from. The
public base URL comes from PUBLIC_BASE_URL or is learned from incoming
webhook Host headers.

Inbound: images over Claude's 8000px dimension limit (long screenshots)
400'd the whole request and the user got no reply. Fetch incoming
images and downscale oversized ones to 2048px JPEG via sharp before
sending to Claude; normal images still pass through as URL sources.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Over the last 72h, 26% of Sonnet's replies were reaction-only with no
text (e.g. thumbs-upping 'how can I be an expert in geoguessr?'),
despite prompt rules against it. In group chats the Haiku classifier
already routes reaction-worthy messages to a quick reaction before
Sonnet is invoked, so a reaction-only result from chat() is a misfire
in any chat type.

Two changes:
- Tighten the reaction rules in the system prompt (reactions are a
  bonus on top of text, never a replacement; drop the 'like = no text
  needed' guidance that invited this)
- Deterministic guardrail: if Sonnet returns a reaction with no text
  and no other action, continue the tool loop with a tool_result asking
  for the actual text reply

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pdsullivan pdsullivan changed the title fix: repair image handling in both directions (gpt-image-1 + inbound resize) fix: image handling (both directions) + never reply with reaction-only Jul 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant