Skip to content

fix(evaluator): PTY driver deadlock + multi-select gate handling - #691

Open
KBB99 wants to merge 1 commit into
awslabs:evaluator-keepfrom
KBB99:fix/evaluator-pty-driver
Open

fix(evaluator): PTY driver deadlock + multi-select gate handling#691
KBB99 wants to merge 1 commit into
awslabs:evaluator-keepfrom
KBB99:fix/evaluator-pty-driver

Conversation

@KBB99

@KBB99 KBB99 commented Jul 31, 2026

Copy link
Copy Markdown

Summary

Two fixes to the cli-harness PTY driver (_pty_terminal.py), both hit while producing the first scored v2 evaluation run (#684). Root cause for both: the evaluator was written against v2 ~2.1.x when --test-run auto-approved gates; v2 removed that bypass in 2.1.4 (#448), so every gate — including multi-select widgets — now surfaces to the driver.

1. drive_until() pipe deadlock

When a menu stayed painted (any widget Enter doesn't dismiss), the loop re-sent Enter every iteration without draining PTY output. The TUI's output filled the kernel pipe buffer, the TUI blocked writing, the driver blocked on write(2) sending the next keystroke, and the pair froze permanently — confirmed twice via process sampling (sample showed both sides parked in write), costing multi-hour runs each time.

Fix: answer a given menu screen once — re-answer only when the rendered screen changes — and always _drain() after answering.

2. Multi-select gates spun forever

answer_gate_default() only knew single-select menus (bare Enter accepts the highlighted option). On a multi-select widget — e.g. the v2 Learnings ritual — Enter toggles the checkbox instead of selecting, so the driver toggled [ ][✔] indefinitely (observed for 19h). This is exactly the failure the repo's own e2e driver documents as the t73 hang ("the loop toggled [ ][✔] forever", tests/harness/tui-drive.ts).

Fix: port tui-drive.ts's per-widget key model to the Python driver —

  • checkbox option lines (1. [ ]) → Space toggles the Recommended option, then Right advances a multi-tab form / Enter commits a lone question
  • Submit answers review screen → plain Enter
  • anything else → single-select, bare Enter (unchanged)

Validation

  • 51/51 cli-harness tests pass; ruff check + ruff format --check clean.
  • Proven in anger: with these two fixes the claude-cli adapter drove a complete 21-stage v2/mvp workflow end-to-end (three /aidlc --resume sessions), producing the scored results on [Feature]: Benchmarks evaluating AI-DLC performance #684 — 88/88 contract tests, 293 generated tests at 100% coverage.

Known gaps NOT addressed here (follow-ups)

Found in the same exercise, needing owner decisions rather than mechanical fixes:

  1. 2h adapter timeout (timeout_seconds: 7200, adapter.py) is far too small for v2/mvp (~7.5h observed). Suggest raising it and/or teaching the adapter to chain /aidlc --resume sessions (which worked flawlessly when driven manually).
  2. find_aidlc_docs() doesn't know v2's layout (aidlc/spaces/<space>/intents/<intent>/), so a completed run still reports "no aidlc-docs" — we extracted docs manually for [Feature]: Benchmarks evaluating AI-DLC performance #684.
  3. Stage-5 scorer needs an explicit --region (NoRegionError otherwise); should default from config or the dist's pinned region.
  4. No token-usage capture in the CLI adapters — run-metrics.yaml reports zero. Stopgap documented on [Feature]: Benchmarks evaluating AI-DLC performance #684: sum AWS/Bedrock CloudWatch metrics over the run window from run-meta.yaml.

Refs #684.

🤖 Generated with Claude Code

Two fixes to the cli-harness PTY driver, both found running the first
scored v2 evaluation (awslabs#684):

1. drive_until() pipe deadlock: when a menu stayed painted on screen,
   the loop re-sent Enter every iteration without ever draining PTY
   output. Both pipe buffers filled and driver + TUI froze against
   each other (blocked on write(2)). Now a given menu screen is
   answered once — re-answer only when the screen changes — and the
   loop always drains after answering.

2. answer_gate_default() only knew single-select menus. On a
   multi-select widget (e.g. the v2 Learnings ritual) Enter TOGGLES
   the checkbox instead of selecting, so the driver toggled forever —
   the same failure tui-drive.ts documents as the t73 hang. Port its
   key model: checkbox option lines -> Space to toggle the Recommended
   option, Right to advance a multi-tab form (Enter for a lone
   question); "Submit answers" review screen -> plain Enter.

These became load-bearing after v2 removed --test-run (2.1.4, awslabs#448):
every gate now surfaces to the driver, including multi-select ones.

51/51 cli-harness tests pass; ruff check + format clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant