Skip to content

feat(llm): feed tool screenshots back into vision models#190

Merged
alexk-dev merged 6 commits intomainfrom
feat/tool-image-llm-roundtrip
Mar 18, 2026
Merged

feat(llm): feed tool screenshots back into vision models#190
alexk-dev merged 6 commits intomainfrom
feat/tool-image-llm-roundtrip

Conversation

@alexk-dev
Copy link
Owner

Summary

  • preserve tool-produced image artifacts in history as lightweight metadata instead of inline base64
  • rehydrate tool screenshots back into multimodal LLM requests for vision-capable models
  • keep non-vision models and broken/missing artifacts on the existing text-only path

Why

Tools like pinchtab_screenshot already return image attachments to the user, but the next LLM call could not see those images.
That meant the agent could capture a screenshot and still be blind to it on the following reasoning step.

What changed

  • DefaultHistoryWriter now stores image artifact references on tool messages via toolAttachments metadata
  • Langchain4jAdapter now:
    • detects whether the target model supports vision
    • loads tool image artifacts through ToolArtifactService
    • appends a synthetic multimodal context message next to the matching ToolExecutionResultMessage
  • tool image rehydration degrades safely to text-only when:
    • the model is not vision-capable
    • the artifact metadata is absent or malformed
    • the stored file no longer exists

Tests

  • ./mvnw -Dmaven.gitcommitid.skip=true surefire:test -Dtest=Langchain4jAdapterTest,Langchain4jAdapterRetryTest,DefaultHistoryWriterTest
  • ./mvnw -Dmaven.gitcommitid.skip=true pmd:check spotbugs:check -DskipTests

Coverage added

  • tool image metadata is persisted on tool history messages
  • vision models receive multimodal context from tool screenshots
  • non-vision models stay text-only
  • missing/broken artifacts do not break the LLM loop

@sonarqubecloud
Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
71.1% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

@alexk-dev alexk-dev merged commit b98cb3b into main Mar 18, 2026
18 of 19 checks passed
@alexk-dev alexk-dev deleted the feat/tool-image-llm-roundtrip branch March 18, 2026 15:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant