feat(eval): add a vision= kwarg to the harbor adapter - #801
Merged
Conversation
The vision_analyze tool (#800) is configured through the GLOBAL config's top-level `vision` block, and the harbor adapter had no way to seed it — so a container run could not use the tool at all. `fusion=` and `advisor=` both already ride the seeded config; this is the third of the same shape. Two halves, and the second is the one that fails quietly: --ak vision=openai:gpt-5.6-luna seeds `{"vision": {"enabled": true, "provider": ..., "model": ...}}` at the TOP LEVEL, not under `settings` — vision_config reads the global tier only, deliberately, because the key names the provider that receives image bytes. And it forwards that provider's key. Without it the model is offered a tool that cannot authenticate, and because Read's stub starts naming the tool the moment vision is configured, the failure reads like a tool bug rather than a missing credential. This is exactly the moonshot advisor gap: the run looks clean, the task can still score 1.0, and only a trajectory grep reveals it. For a worker+vision+advisor run the container now carries all three vendors' keys. Half-configured values (`openai:`, `:gpt-5.6-luna`) raise at construction rather than seeding a silently-inert config. Tests drive the real constructor rather than mirroring its check — a mirrored assertion passes even when the adapter drops validation entirely. Mutation-tested: removing the validation fails 4 of them. Added to the Harbor adapter CI job's file list, without which the file would never run at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Test Results 1 files 1 suites 8m 14s ⏱️ Results for commit 61985d0. |
_build_env now consults self._vision, and the advisor tests build a Clawcodex
via __new__ with a hand-listed field set — so every one of them raised
AttributeError. The factory models a constructed agent and has to track the
constructor.
Caught by CI, not locally: `pytest -k harbor` in the main venv SKIPS these
files (importorskip("harbor")), so a local green tick said nothing about
them. The Harbor adapter job's exact command is the one to run:
uv run --isolated --python 3.13 --with harbor --with pytest python -m pytest \
tests/test_headless_usage_events.py tests/test_harbor_adapter_*.py
62 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
vision_analyze(#800) reads its config from the globalvisionblock, and the harbor adapter had no way to seed it — so a container run could not use the tool at all.fusion=andadvisor=already ride the seeded config; this is the third of the same shape.Two halves — the second fails quietly
Seeding: writes
{"vision": {"enabled": true, ...}}at the top level, not undersettings.vision_configreads the global tier only, deliberately, because the key names the provider that receives image bytes.Credential forwarding: without it the model is offered a tool that cannot authenticate — and since
Read's stub starts naming the tool the moment vision is configured, the failure reads like a tool bug rather than a missing key. This is precisely the moonshot advisor gap: the run looks clean, the task can still score 1.0, and only a trajectory grep reveals it. A worker+vision+advisor run now carries all three vendors' keys.Half-configured values (
openai:,:gpt-5.6-luna) raise at construction rather than seeding a silently-inert config.Tests
11 tests, driving the real constructor rather than mirroring its check — a mirrored assertion passes even when the adapter drops validation entirely. Mutation-tested: removing the validation fails 4 of them.
Added to the "Harbor adapter (3.13)" CI job's file list. That job is the only place these can run (
importorskip("harbor")skips them silently elsewhere), so a file omitted there never runs at all.🤖 Generated with Claude Code