Skip to content

fix: make adversarial reviewer failures visible [refs OSWE-239] - #117

Merged
ericlitman merged 1 commit into
mainfrom
claude/oswe-239-review-726b7b
Aug 3, 2026
Merged

fix: make adversarial reviewer failures visible [refs OSWE-239]#117
ericlitman merged 1 commit into
mainfrom
claude/oswe-239-review-726b7b

Conversation

@ericlitman

Copy link
Copy Markdown
Owner

Why

The first production adversarial review (tonal-api#3) published nothing and exited success in 4.6 seconds, and the control-plane log was completely silent. The incident read as unexplained for hours.

It wasn't. Both finders failed with RuntimeError: bounded reviewer stage returned no structured response, and the graph bailed at dedupesettle. The reason nothing surfaced is structural: every failure path in the adversarial spine — prepare, find, dedupe, adjudicate, prepublish, record_publish — captures its error into state["error"] and routes to settle, which never read it. The spine has no error logging at all, so "no error in the log" is the guaranteed outcome for every failure mode it has. The root cause was only recoverable from the LangSmith trace.

What changed

settle logs the spine error and the per-finder errors. Every failure route lands there and the graph still exits success, so it's the one place a swallowed error can reach the log. One funnel rather than six call sites.

_run_stage logs diagnostics before raising — the returned state's keys, its message count, and the resolved configurable's keys. message_count=0 means the stage graph ran no nodes at all, which is what happened in production and is a materially different failure from a model that answered without the tool. Config is logged as keys only; the configurable carries auth values.

A regression test for the finder sub-agent production actually builds. Every compiled-graph test in the suite patches out both _bounded_agent (replaced with bare object() sentinels) and _run_stage, so create_deep_agent(..., response_format=ToolStrategy(FinderOutput)) was never constructed or invoked under test. The new test drives all three finders through the real build_subagents_bounded_agent_run_stage path with a stub tool-calling model and asserts, per finder, that structured_response is a FinderOutput, that the model was actually invoked, and that FinderOutput was among the bound tools.

Two smaller tests cover the instrumentation itself: one asserts the _run_stage diagnostic fires with message_count and configurable_keys, and the existing finder-timeout test now asserts the settle-level log names both the spine error and the underlying finder error.

What this does not do

It does not fix the bug. The production failure needs the LangGraph platform runtime to reproduce — the identical construction, middleware, subagent specs, backend factory, parent-graph-with-checkpointer shape and production-shaped config all work locally. The open suspect is _run_stage passing the graph-construction config (carrying the parent's thread_id, run_id and checkpoint_ns) into every bounded sub-agent invoke. This PR makes the next adversarial run self-diagnosing so that question gets answered outright rather than re-investigated.

Analysis and evidence are on OSWE-239.

Verification

  • 2355 passed across the full unit suite; 35/35 in tests/reviewer/test_reviewer_adversarial.py
  • make lint clean, make typecheck 0 errors
  • Mutation check: removing response_format=ToolStrategy(...) from _bounded_agent fails the new regression test as intended

🤖 Generated with Claude Code

The first production adversarial review published nothing and exited
"success" in 4.6s. Both finders failed with "bounded reviewer stage
returned no structured response", but nothing reached the log: every
spine failure is captured into state["error"] and routed to settle,
which never read it. The root cause was only recoverable from the
LangSmith trace.

Log the spine error and the per-finder errors at settle, the single
funnel every failure route reaches. In _run_stage, log the returned
state's keys, its message count, and the resolved configurable's keys
before raising -- a message count of 0 means the stage graph ran no
nodes at all, which is what happened in production and is a different
failure from a model that answered without the tool. Keys only, since
the configurable carries auth values.

Cover the finder sub-agent that production actually builds. Every
compiled-graph test patches out both _bounded_agent and _run_stage, so
create_deep_agent(..., response_format=ToolStrategy(FinderOutput)) was
never constructed or invoked under test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ericlitman
ericlitman enabled auto-merge (squash) August 3, 2026 23:10

@mobilyze-open-swe-studio2 mobilyze-open-swe-studio2 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Open SWE Review: No issues found

Open SWE reviewed this PR and found no potential bugs to report.

Open in Web

@ericlitman
ericlitman merged commit 3a70985 into main Aug 3, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant