fix: make adversarial reviewer failures visible [refs OSWE-239] - #117
Merged
Conversation
The first production adversarial review published nothing and exited "success" in 4.6s. Both finders failed with "bounded reviewer stage returned no structured response", but nothing reached the log: every spine failure is captured into state["error"] and routed to settle, which never read it. The root cause was only recoverable from the LangSmith trace. Log the spine error and the per-finder errors at settle, the single funnel every failure route reaches. In _run_stage, log the returned state's keys, its message count, and the resolved configurable's keys before raising -- a message count of 0 means the stage graph ran no nodes at all, which is what happened in production and is a different failure from a model that answered without the tool. Keys only, since the configurable carries auth values. Cover the finder sub-agent that production actually builds. Every compiled-graph test patches out both _bounded_agent and _run_stage, so create_deep_agent(..., response_format=ToolStrategy(FinderOutput)) was never constructed or invoked under test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ericlitman
enabled auto-merge (squash)
August 3, 2026 23:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The first production adversarial review (tonal-api#3) published nothing and exited
successin 4.6 seconds, and the control-plane log was completely silent. The incident read as unexplained for hours.It wasn't. Both finders failed with
RuntimeError: bounded reviewer stage returned no structured response, and the graph bailed atdedupe→settle. The reason nothing surfaced is structural: every failure path in the adversarial spine —prepare,find,dedupe,adjudicate,prepublish,record_publish— captures its error intostate["error"]and routes tosettle, which never read it. The spine has no error logging at all, so "no error in the log" is the guaranteed outcome for every failure mode it has. The root cause was only recoverable from the LangSmith trace.What changed
settlelogs the spine error and the per-finder errors. Every failure route lands there and the graph still exitssuccess, so it's the one place a swallowed error can reach the log. One funnel rather than six call sites._run_stagelogs diagnostics before raising — the returned state's keys, its message count, and the resolved configurable's keys.message_count=0means the stage graph ran no nodes at all, which is what happened in production and is a materially different failure from a model that answered without the tool. Config is logged as keys only; the configurable carries auth values.A regression test for the finder sub-agent production actually builds. Every compiled-graph test in the suite patches out both
_bounded_agent(replaced with bareobject()sentinels) and_run_stage, socreate_deep_agent(..., response_format=ToolStrategy(FinderOutput))was never constructed or invoked under test. The new test drives all three finders through the realbuild_subagents→_bounded_agent→_run_stagepath with a stub tool-calling model and asserts, per finder, thatstructured_responseis aFinderOutput, that the model was actually invoked, and thatFinderOutputwas among the bound tools.Two smaller tests cover the instrumentation itself: one asserts the
_run_stagediagnostic fires withmessage_countandconfigurable_keys, and the existing finder-timeout test now asserts the settle-level log names both the spine error and the underlying finder error.What this does not do
It does not fix the bug. The production failure needs the LangGraph platform runtime to reproduce — the identical construction, middleware, subagent specs, backend factory, parent-graph-with-checkpointer shape and production-shaped config all work locally. The open suspect is
_run_stagepassing the graph-constructionconfig(carrying the parent'sthread_id,run_idandcheckpoint_ns) into every bounded sub-agent invoke. This PR makes the next adversarial run self-diagnosing so that question gets answered outright rather than re-investigated.Analysis and evidence are on OSWE-239.
Verification
tests/reviewer/test_reviewer_adversarial.pymake lintclean,make typecheck0 errorsresponse_format=ToolStrategy(...)from_bounded_agentfails the new regression test as intended🤖 Generated with Claude Code