fix(wrappers): v1.0.4 beat-reliability set — deadlines, timeout shim, cold-open guards, party heartbeat + soft-tick (audit F12-1/3/4/5/8) - #828
Merged
Conversation
… timeout(1) shim, cold-open guards, party heartbeat + soft-tick (audit F12-1, F12-3, F12-4, F12-5, F12-8) The five wrapper-reliability findings from docs/audits/ENGINE-AUDIT-2026-06-11.md unit 12 (skeptic-verified), fixed together because they share files: * F12-1 (enriches #753): routine beat deadline 200s -> 360s in clawdnd_dm_timeout (measured: routine p90=224s max=360s — the flat 200s killed ~18% of HEALTHY beats), and BOTH dm_turn paths (play.sh + play_party.sh) now recompute the retry deadline via the new clawdnd_dm_retry_timeout (attempt 2 escalates to the model-aware cold-open tier, never de-escalates, never reuses verbatim). * F12-8 (closes #787): worldos_timeout shim in qa/lib_beat_driver.sh — timeout(1) when present, else a python3 subprocess fallback preserving rc=124/127/126 semantics — swapped into both play-lane _dm_invoke call sites; launch_common.sh preflight now warns (non-fatal, the shim covers absence) with the brew-install-coreutils hint. * F12-3 (closes #777): play.sh cold open now ABORTS NON-ZERO on an empty opening DMSG and on a no-campaign mint (was: unflagged empty chat row + an indefinitely-"running" unplayable session), and runs the seating guard with one reseat retry then a loud abort. pc_seated() factored into the lib as clawdnd_pc_seated (snapshot-read-only, viewer _action_actor contract); play_party.sh now REUSES the shared helper instead of its local copy. * F12-4 (closes #790, completes #623): play_party.sh emits the model-independent progress heartbeat — before the cold-open turn (campaign id is pre-seeded in this lane) and after the human move BEFORE companion_moves — via the same shared helper play.sh calls (post-#763 contamination-safe). * F12-5 (closes #791): play_party.sh beat loop captures PREV_DAY/PREV_TOD pre-beat and runs clawdnd_soft_tick after record_dm_reply (mirror of play.sh:475-478/504) — it was the only beat loop without the backstop. Tests: servers/engine/tests/test_wrapper_reliability.py (22 new, red-first) — worldos_timeout behavioral matrix (rc pass-through, rc=124 deadline, PATH- stripped fallback, native-binary preference), retry-deadline escalation, clawdnd_pc_seated behavioral matrix, static anti-drift asserts on all call sites, bash -n on every touched script. test_adversarial_release.py seating assert updated to track the factored helper (intent unchanged). Full engine suite 1805 passed; qa/fast_gate.sh PASS. Engine stays the sole writer (guards are snapshot-read-only; heartbeat/tick route through log_engine_narration/advance_time). Wire contracts frozen (CLAWDND_* names kept; no new env names). Bash-3.2-clean.
|
Warning Review limit reached
More reviews will be available in 18 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (6)
Comment |
100yenadmin
added a commit
that referenced
this pull request
Jun 14, 2026
…results, visible failure beats, single chatlog impl (Refs #757 #745) (#841) Root cause (audit SYN-01 = F12-7 + F12-14 + F13-5, docs/audits/ENGINE-AUDIT-2026-06-11.md): a 401/auth-failed `claude -p` result carries NON-empty error text (subtype:"success", is_error:true, api_error_status:401), bypassing the empty-only retry (run_duo turn_retry) AND the #357 narration-fallback gate — so the error string was chatlogged AS DM PROSE; a fully-dead beat either recycled the previous beat's prose into a client-hidden row or wrote an unflagged EMPTY dm row; and 3 QA runners' local chatlog() overrides (run_duo:135, ui_playtest:138, run_party:169) shadowed the shared lib, killing the fallback_recovered stamp (which also had zero consumers). Fix (qa/lib_beat_driver.sh, shared by all 5 DM wrappers): - clawdnd_dm_final_text — the ONE extraction front door: notes the final attempt's stream-json in $STATE_DIR/.dm_last_result and echoes NOTHING on an error-class result (is_error / api_error_status), surfacing the real failure + 401/403 re-auth hint via the existing clawdnd_report_attempt_failure pattern. Empty echo makes the empty-only retries fire on error results too. - clawdnd_resolve_dm_reply — parses the FINAL result event FIRST: error-class => beat FAILED (never chat the error text, never fallback-recycle). A fallback recovery is GENUINE only if the DM logged NEW prose past the pre-beat log-tail mark (clawdnd_dm_prebeat_mark + qa/dm_beat_mark.py — same prose filters as the #357 fallback; fail-open without a mark); recycled-only prose => beat FAILED. - record_dm_reply blank guard + clawdnd_chatlog_dm_failed — a failed/blank beat records ONE wrapper-authored VISIBLE failure row stamped {"beat_failed":true} (chat-only by design: engine-logging it would dedup-hide repeats and pollute recap/FTS/lean-tail/fallback memory). - Deleted the 3 runner chatlog overrides — the lib chatlog (drop-in superset) is the single implementation, so the fallback_recovered/beat_failed stamps land in every runner. - qa/assert_behavioral.py dm_beat_honesty — counts + reports beats_failed / fallback_recovered per run (WARN only; the gate-discount policy stays #757's call). #828's worldos_timeout / retry-escalation / re-mint behavior is preserved exactly (only the final jq extraction is swapped for the front door; the pre-beat mark is taken once per beat, before attempt 1). Tests: servers/engine/tests/test_dead_beat_classification.py (25 new, red-first) — 401 fixture never reaches chat; recycled recovery fails the beat; genuine #357 win preserved; heartbeat rows never count as prose; blank record => exactly one visible, never-hidden failure row; override deletion + wiring shape locks; assert_behavioral counts without gating; bash -n on every touched script. Full engine suite 1830 passed; qa/fast_gate.sh PASS. Finding-ids: SYN-01 (F12-7, F12-14, F13-5). Refs #757, Refs #745. Co-authored-by: Eva <arncalso@gmail.com>
100yenadmin
added a commit
that referenced
this pull request
Jun 14, 2026
…ta-aborted); fix sweep header parallel→sequential (#846) - OPERATING-GOAL state block → current truth: main ~347b6c2, the audit fix-wave + the sibling full-engine-audit fork's fixes (mech #832/#834, image_render #829, dead-beat #841, seat #833, spells #830, economy #831, beat-reliability #828) all landed; rc3 QUOTA-ABORTED + invalid (#845, newbie-canary clean); NEXT ACTION = re-measure with the hardened sequential sweep at current main, blocked on claude quota. - sweep_v2.sh header: the top comment + start-line still said PARALLEL after #844 made the persona batch sequential — corrected so the file is internally consistent. Co-authored-by: Eva <arncalso@gmail.com>
100yenadmin
added a commit
that referenced
this pull request
Jun 14, 2026
…meout/stale-status/budget, claude provider_status, run_duo deadline, party actor timeout, play.sh idle+lock, sonnet cold-open margin (Refs #811) (#863) Brings the codex/openclaw/claude lanes up to the play.sh protections per the audited P2 cluster. All changes additive + bash 3.2-clean; preserves #828's worldos_timeout shim + #841's failure-classification (both confirmed intact). F12-2 sonnet cold-open deadline 400 had thin (band-top) margin → non-opus default 400→550 in clawdnd_dm_timeout (opus unchanged 500). Updated the opus-tuning + wrapper-reliability guards; rewrote the (already-stale, routine=200) cold-open timeout proof to 360 + added opus/sonnet margin scenarios. F12-9 codex DM wrapper (play_codex_dm.sh): EXIT/INT/TERM trap now stamps provider_status "failed" on abnormal exit; codex exec wrapped in an inline worldos_timeout (WORLDOS_CODEX_TURN_TIMEOUT) + ONE session-safe retry; session budget ENFORCED via token-count spend accounting (WORLDOS_CODEX_USD_PER_MTOK) → "exhausted" stop (the budget envs were validated then never used). F12-10 claude lanes never wrote provider_status.json → factored clawdnd_write_provider_status (atomic, v1 schema) into the lib; play.sh + play_party.sh write running/stopped(turn_cap|budget|idle)/failed so the viewer buckets a dead/stopped session as no_provider instead of "unknown". F12-11 run_duo DM turn was unbounded + swallowed the real cause → wrap in worldos_timeout(clawdnd_dm_timeout); report on rc!=0 (dedup-guarded); replace the inline cold-open remint with the shared clawdnd_dm_remint_session_on_retry; keep empty-output retry as 2nd trigger. F12-12 play_party companion (actor) turn was unbounded → worldos_timeout ${WORLDOS_ACTOR_TIMEOUT:-120}; empty on failure (companion_moves skip-safe). F12-13 play.sh had no idle ceiling and no launch lock → port play_party's MAX_IDLE block + acquire/release the single-flight launch lock. Skipped as already-done (confirmed on main): F12-1/3/4/5/8 (#828), SYN-01 (#841). Tests: fast_gate Tier-0 191 passed; codex wrapper 34 (incl 6 new F12-9, 3 runtime fake-codex); wrapper-reliability/opus/adversarial 43; new shell tests test_play_party_actor_timeout.sh, test_run_duo_dm_timeout.sh, test_provider_status_claude_lanes.sh + extended single-flight + cold-open proof. Source: docs/audits/ENGINE-AUDIT-2026-06-11.md Co-authored-by: Eva <arncalso@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the five-finding wrapper-reliability cluster from the 2026-06-11 engine audit (
docs/audits/ENGINE-AUDIT-2026-06-11.md, unit 12 — skeptic-verified at a245a2c, re-validated against current main c1061e7). One branch because they shareqa/lib_beat_driver.sh+ both play lanes.Closes #777 (F12-3) · Closes #787 (F12-8) · Closes #790 (F12-4) · Closes #791 (F12-5) · enriches #753 (F12-1 wrapper half — no dedicated issue).
Root causes
clawdnd_dm_timeoutreturned a flat 200s for routine beats while measured healthy routine beats run p90=224s / max=360s (206-beat run_duo counterfactual, same opus/medium defaults) → ~18% of healthy beats were killed; and bothdm_turnpaths capturedbeat_timeoutonce and re-invoked the retry with the SAME deadline verbatim → the long beat was killed twice.timeout(1)is a coreutils binary absent on stock Darwin;play.sh/play_party.shinvoked it bare → on a non-coreutils Mac every beat died rc=127 in <1s, masked. Preflight (clawdnd_missing_commands) never checked it.play.shrecorded the cold-open DMSG UNCONDITIONALLY and entered the move loop: a double-failed cold open (401-class, proven 2026-06-02) left an unflagged EMPTY chat row +CAMPAIGN_ID=""+ every beat resuming a nonexistent session — an indefinitely-"running" unplayable session under a live viewer.play_party.shalready carried both guards; play.sh had neither.clawdnd_emit_progress_heartbeat(factored in [reliability] DM beats silently drop / hang >10min with no recovery — sabotages story personas (crit) #623 "so every harness shares it") had zeroplay_party.shcallers; post-fix(qa+engine): make the beat heartbeat real — flip progress at ingest, decontaminate recap/FTS/lean-tail/fallback (#749) #763 the viewer flips progress at heartbeat INGEST, so the party lane never flipped.play_party.shwas the ONLY beat loop withoutclawdnd_soft_tick(noPREV_DAY/PREV_TODcapture) → party sessions could freeze at day-1 morning indefinitely.Fix
qa/lib_beat_driver.sh: routine default 200→360 (env overrides win unchanged); NEWclawdnd_dm_retry_timeout(attempt 2 escalates to the model-aware cold-open tier — opus 500 / default 400 — never de-escalates below attempt 1); NEWworldos_timeoutshim (timeout(1) when present, else python3 subprocess preserving rc=124/127/126; stdin/stdout pass-through); NEWclawdnd_pc_seated(snapshot-read-only, viewer_action_actorcontract).scripts/launch_common.sh:clawdnd_warn_if_no_timeout— WARNS with thebrew install coreutilshint (non-fatal by design: the shim fully serves a timeout-less host, so failing closed would be a regression; flagged as a deliberate deviation from the "hard preflight" reading of the spec).scripts/play.sh: empty-opening abort (non-zero, BEFORE the move loop), no-campaign abort, seating guard with ONE reseat retry (fresh session id, canon-pickup directive) then loud abort; shim swap; retry-deadline recompute;CLAWDND_BEAT_TIMEOUTseed line 200→360.scripts/play_party.sh: localpc_seated()deleted in favor of the shared lib helper (unify, not duplicate); cold-open heartbeat (pre-seeded campaign id) + per-beat heartbeat after the human move BEFOREcompanion_moves(same shared helper play.sh calls — no local text bank, fix(qa+engine): make the beat heartbeat real — flip progress at ingest, decontaminate recap/FTS/lean-tail/fallback (#749) #763 contamination-safe);PREV_DAY/PREV_TODcapture +clawdnd_soft_tickafterrecord_dm_reply; shim swap; retry-deadline recompute.Tests (red-first)
servers/engine/tests/test_wrapper_reliability.py— 22 new tests, watched fail before the fix (19 red / 3 pre-existing-pass), in thetest_dm_session_remint.pypytest-over-/bin/bash pattern:worldos_timeout: rc/stdout pass-through, rc=124 at the deadline, PATH-stripped no-binary fallback (124/7/127), native-binary preference, both-lane call-site swap, preflight brew hint (static + behavioral).clawdnd_pc_seatedbehavioral matrix (seated / npc-only / pc-not-in-party / no-snapshot / blank-id), play.sh abort-before-loop + reseat-then-loud-abort statics, play_party lib-reuse statics.companion_moves; soft-tick afterrecord_dm_replywith pre-beat capture).bash -non all four touched scripts.test_adversarial_release.py::test_coldopen_play_party_guards_player_pc_seatingupdated to track the factored helper (assert intent unchanged — the_action_actor-matching python now lives in the lib).Verification
-p no:xdist).qa/fast_gate.sh: PASS (188 deterministic tier tests)./bin/bash -nclean onscripts/play.sh,scripts/play_party.sh,scripts/launch_common.sh,qa/lib_beat_driver.sh; new bash is 3.2-clean (no new arrays, no globstar, heredocs never inside$()).log_engine_narration/advance_time); wire contracts frozen (CLAWDND_* names kept, no new env names); additive (explicit env overrides keep today's behavior).Notes / residual
qa/ui_playtest.sh(+run_duo.sh, codex wrapper, companion turns) still use bare/notimeout— deliberately out of scope here; F12-9/F12-11/F12-12 land on this sameworldos_timeoutshim.timeout=Ns→deadline escalated to Ns); repo grep found no consumer of the old wording.🤖 Generated with Claude Code