fix(evaluator): PTY driver deadlock + multi-select gate handling - #691
Open
KBB99 wants to merge 1 commit into
Open
fix(evaluator): PTY driver deadlock + multi-select gate handling#691KBB99 wants to merge 1 commit into
KBB99 wants to merge 1 commit into
Conversation
Two fixes to the cli-harness PTY driver, both found running the first scored v2 evaluation (awslabs#684): 1. drive_until() pipe deadlock: when a menu stayed painted on screen, the loop re-sent Enter every iteration without ever draining PTY output. Both pipe buffers filled and driver + TUI froze against each other (blocked on write(2)). Now a given menu screen is answered once — re-answer only when the screen changes — and the loop always drains after answering. 2. answer_gate_default() only knew single-select menus. On a multi-select widget (e.g. the v2 Learnings ritual) Enter TOGGLES the checkbox instead of selecting, so the driver toggled forever — the same failure tui-drive.ts documents as the t73 hang. Port its key model: checkbox option lines -> Space to toggle the Recommended option, Right to advance a multi-tab form (Enter for a lone question); "Submit answers" review screen -> plain Enter. These became load-bearing after v2 removed --test-run (2.1.4, awslabs#448): every gate now surfaces to the driver, including multi-select ones. 51/51 cli-harness tests pass; ruff check + format clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two fixes to the cli-harness PTY driver (
_pty_terminal.py), both hit while producing the first scored v2 evaluation run (#684). Root cause for both: the evaluator was written against v2 ~2.1.x when--test-runauto-approved gates; v2 removed that bypass in 2.1.4 (#448), so every gate — including multi-select widgets — now surfaces to the driver.1.
drive_until()pipe deadlockWhen a menu stayed painted (any widget Enter doesn't dismiss), the loop re-sent Enter every iteration without draining PTY output. The TUI's output filled the kernel pipe buffer, the TUI blocked writing, the driver blocked on
write(2)sending the next keystroke, and the pair froze permanently — confirmed twice via process sampling (sampleshowed both sides parked inwrite), costing multi-hour runs each time.Fix: answer a given menu screen once — re-answer only when the rendered screen changes — and always
_drain()after answering.2. Multi-select gates spun forever
answer_gate_default()only knew single-select menus (bare Enter accepts the highlighted option). On a multi-select widget — e.g. the v2 Learnings ritual — Enter toggles the checkbox instead of selecting, so the driver toggled[ ]↔[✔]indefinitely (observed for 19h). This is exactly the failure the repo's own e2e driver documents as the t73 hang ("the loop toggled[ ]↔[✔]forever",tests/harness/tui-drive.ts).Fix: port tui-drive.ts's per-widget key model to the Python driver —
1. [ ]) → Space toggles the Recommended option, then Right advances a multi-tab form / Enter commits a lone questionSubmit answersreview screen → plain EnterValidation
ruff check+ruff format --checkclean./aidlc --resumesessions), producing the scored results on [Feature]: Benchmarks evaluating AI-DLC performance #684 — 88/88 contract tests, 293 generated tests at 100% coverage.Known gaps NOT addressed here (follow-ups)
Found in the same exercise, needing owner decisions rather than mechanical fixes:
timeout_seconds: 7200,adapter.py) is far too small for v2/mvp (~7.5h observed). Suggest raising it and/or teaching the adapter to chain/aidlc --resumesessions (which worked flawlessly when driven manually).find_aidlc_docs()doesn't know v2's layout (aidlc/spaces/<space>/intents/<intent>/), so a completed run still reports "no aidlc-docs" — we extracted docs manually for [Feature]: Benchmarks evaluating AI-DLC performance #684.--region(NoRegionError otherwise); should default from config or the dist's pinned region.run-metrics.yamlreports zero. Stopgap documented on [Feature]: Benchmarks evaluating AI-DLC performance #684: sumAWS/BedrockCloudWatch metrics over the run window fromrun-meta.yaml.Refs #684.
🤖 Generated with Claude Code