Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: CI

on:
pull_request:
branches: [main]
push:
branches: [main]

permissions:
contents: read

concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
tests:
# The hooks target macOS /bin/bash 3.2; macos-15 ships it as /bin/bash.
runs-on: macos-15
timeout-minutes: 30
steps:
- uses: actions/checkout@v5
- name: Run test suites
# /bin first so run.sh's unqualified `bash` resolves to 3.2, not brew bash 5.
run: |
export PATH="/bin:$PATH"
/bin/bash tests/run.sh
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
[![Bash](https://img.shields.io/badge/shell-bash-4EAA25?style=for-the-badge&logo=gnubash&logoColor=white)](hooks/)
[![Node](https://img.shields.io/badge/runtime-Node-339933?style=for-the-badge&logo=nodedotjs&logoColor=white)](install.mjs)
[![Platform](https://img.shields.io/badge/platform-macOS%20·%20Linux-0A84FF?style=for-the-badge&logo=apple&logoColor=white)](#requirements)
[![Tests](https://img.shields.io/badge/tests-6%20suites-00B34A?style=for-the-badge)](tests/)
[![Tests](https://img.shields.io/badge/tests-11%20suites-00B34A?style=for-the-badge)](tests/)
[![No API key](https://img.shields.io/badge/no%20API%20key-ChatGPT%20login-FF6B35?style=for-the-badge)](#requirements)

</div>
Expand Down Expand Up @@ -81,7 +81,7 @@ Write contention waits without arrival ordering only while the current lease has

Foreground write supervisors update a separate lease heartbeat every `MAESTRO_LOCK_HEARTBEAT_INTERVAL_SEC` (default 20 seconds, minimum 1; invalid values use 20). `MAESTRO_LOCK_HEARTBEAT_STALE_SEC` controls when a missed heartbeat is reported and makes `--clear-lease` eligible after its independent writer check (default 90 seconds; `0` disables staleness reporting; invalid values use 90). A stale heartbeat is diagnostic only and never transfers ownership automatically.

Every dispatch has an absolute deadline (`MAESTRO_MAX_DISPATCH_SEC`, default 1200 seconds), while the local verifier has its own process-group deadline (`MAESTRO_VERIFY_TIMEOUT_SEC`, default 900 seconds), and `MAESTRO_COMPANION_TIMEOUT_SEC` bounds each companion call (default 120 seconds) so a wedged companion or broker cannot block the poll loop indefinitely, where neither the idle guard nor dispatch deadline can fire. Cancellation is detected after the poll interval plus the bounded status call, so worst-case lag is roughly `POLL + MAESTRO_COMPANION_TIMEOUT_SEC`, not an exact deadline. Cancelling a write job cannot prove its brokered turn stopped, so Maestro retains and poisons the lease, ends the loop as `BLOCKED`, and does not re-dispatch. Once no Codex job is writing, recover with `bash hooks/implementer-loop.sh --clear-lease` (installed: `bash ~/.claude/hooks/implementer-loop.sh --clear-lease`). Read-only discussions hold no write lease and are never poisoned.
Every dispatch has an absolute deadline (`MAESTRO_MAX_DISPATCH_SEC`, default 1200 seconds), while the local verifier has its own process-group deadline (`MAESTRO_VERIFY_TIMEOUT_SEC`, default 900 seconds), and `MAESTRO_COMPANION_TIMEOUT_SEC` bounds each companion call (default 120 seconds) so a wedged companion or broker cannot block the poll loop indefinitely, where neither the idle guard nor dispatch deadline can fire. At the poll site, a timed-out companion call yields an empty status; four consecutive empty statuses, or the dispatch deadline being crossed during status loss, cancel the job and fail closed. The worst case from a wedged broker to that exit is roughly 4 × (`POLL` + `MAESTRO_COMPANION_TIMEOUT_SEC`) — about 9 minutes at the defaults. For a write job, that fail-closed exit retains and poisons the lease, ends the loop as `BLOCKED`, and does not re-dispatch because nothing observable proves the brokered turn stopped. Once no Codex job is writing, recover with `bash hooks/implementer-loop.sh --clear-lease` (installed: `bash ~/.claude/hooks/implementer-loop.sh --clear-lease`). Read-only discussions hold no write lease and are never poisoned.

On every SessionStart source, the hook appends a validated `MAESTRO_SESSION_ID` export to `$CLAUDE_ENV_FILE`. The value is attribution only: the token, PID/process-start identity, and companion job liveness remain the ownership checks. A missing or invalid value is recorded as `unknown`; the session appears in lease metadata, contention/poison messages, and provenance records.

Expand Down
3 changes: 2 additions & 1 deletion hooks/implementer-loop.sh
Original file line number Diff line number Diff line change
Expand Up @@ -313,7 +313,8 @@ while [ "$i" -lt "$MAX_ITERS" ]; do
# Close FD 3 so verifier progress re-points to stdout and lands in VOUT instead of the operator channel.
: > "$VOUTF"
set -m
bash -c "$VERIFY" > "$VOUTF" 2>&1 3>&- &
env -u MAESTRO_LOCK_ACQUIRED -u MAESTRO_LOCK_TOKEN -u MAESTRO_LOCK_DIR \
bash -c "$VERIFY" > "$VOUTF" 2>&1 3>&- &
vpid=$!
vstarted=$(date +%s)
vtimed_out=0
Expand Down
5 changes: 1 addition & 4 deletions hooks/implementer-watchdog.sh
Original file line number Diff line number Diff line change
Expand Up @@ -191,16 +191,13 @@ case "$rc" in
125)
reason=${MAESTRO_CANCEL_REASON:-unknown}
if [ "${MAESTRO_CANCEL_REQUESTED:-1}" -eq 0 ]; then
progress "WATCHDOG_POISONED: job $JOB was not cancelled because poison metadata could not be staged and may still be running; the write lease is retained and this run is over."
progress "WATCHDOG_POISONED: job $JOB was not confirmed cancelled and may still be running; the write lease is retained and this run is over."
else
progress "WATCHDOG_POISONED: job $JOB was cancelled ($reason) and turn quiescence could not be confirmed; the write lease is retained and this run is over."
fi
progress "WATCHDOG_POISONED: recover only after no Codex job is writing: bash hooks/implementer-loop.sh --clear-lease (installed path: bash ~/.claude/hooks/implementer-loop.sh --clear-lease)"
printf 'RESULT: BLOCKED\n'
maestro_finish "POISONED" 125 ;;
6)
echo "WATCHDOG_FAILED: companion status unreachable; job $JOB state unknown. Check the tree before re-dispatching." >&2
maestro_finish "FAILED" 4 ;;
*)
echo "WATCHDOG_FAILED: job $JOB ended failed. Re-dispatch with the failure evidence, or ask the user." >&2
maestro_finish "FAILED" 4 ;;
Expand Down
51 changes: 25 additions & 26 deletions hooks/lib-companion.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,12 @@
# companion_verify_pin <C> <job> <model> <effort> → returns 0 match | 4 mismatch
# companion_workspace_writers <C> → prints job<TAB>write, or returns 4
# companion_poll <C> <job> <idle> <sec> → returns 0 done | 4 failed | 124 read-only timeout
# | 125 write timeout | 6 status-lost
# | 125 write timeout
# companion_result <C> <job> → prints result (retried), or returns 4
#
# Error-handling contract: a companion that cannot answer `status` 4 times in a row
# is declared lost (6) instead of being polled forever; an empty `result` is fetched
# once more before being declared a failure. Hangs are cancelled by the poll itself.
# is cancelled instead of being polled forever; an empty `result` is fetched once
# more before being declared a failure. Hangs are cancelled by the poll itself.

progress_init() {
if ! { true >&3; } 2>/dev/null; then exec 3>&1; fi
Expand Down Expand Up @@ -175,7 +175,7 @@ repo_digest() {

run_bounded() {
local timeout="$1" label="$2" default="${run_bounded_default-120}"
local invalid output pid elapsed timed_out grace rc
local invalid output pid elapsed timed_out grace rc start hb
shift 2
invalid=0
case "$timeout" in
Expand All @@ -195,16 +195,22 @@ run_bounded() {
pid=$!
set +m

start=$SECONDS
hb=0
elapsed=0
timed_out=0
while kill -0 "$pid" 2>/dev/null; do
elapsed=$((SECONDS - start))
if [ "$elapsed" -ge "$timeout" ]; then
timed_out=1
break
fi
sleep 1
write_lock_heartbeat_write
elapsed=$((elapsed + 1))
sleep 0.1
hb=$((hb + 1))
if [ "$hb" -ge 10 ]; then
write_lock_heartbeat_write
hb=0
fi
done
if [ "$timed_out" -eq 1 ]; then
kill -TERM -"$pid" 2>/dev/null || :
Expand Down Expand Up @@ -1092,17 +1098,8 @@ companion_verify_pin() {
return 4
}

companion_cancel_job() { # C JOB REASON → 124 read-only, 125 write-mode
local C="$1" JOB="$2" REASON="$3" ST pid log write_mode=0
ST=$(companion_call "$C" status "$JOB" --json 2>/dev/null) || ST=""
pid=$(printf '%s\n' "$ST" |
grep -oE '"pid"[[:space:]]*:[[:space:]]*(null|[0-9]+)' |
head -1 |
sed -E 's/^"pid"[[:space:]]*:[[:space:]]*//')
log=$(printf '%s' "$ST" |
sed -n 's/.*"logFile"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' |
head -1)
[ "$pid" = "null" ] && pid=""
companion_cancel_job() { # C JOB REASON LOG → 124 read-only, 125 write-mode
local C="$1" JOB="$2" REASON="$3" log="$4" crc write_mode=0
MAESTRO_CANCEL_REASON="$REASON"
MAESTRO_CANCEL_REQUESTED=0
if write_lock_is_owner; then
Expand All @@ -1114,8 +1111,9 @@ companion_cancel_job() { # C JOB REASON → 124 read-only, 125 write-mode
fi
fi
companion_call "$C" cancel "$JOB" >/dev/null 2>&1
MAESTRO_CANCEL_REQUESTED=1
progress "MAESTRO_POLL: cancelled reason=$REASON job=$JOB pid=${pid:-unknown} log=${log:-unknown}"
crc=$?
if [ "$crc" -eq 125 ]; then MAESTRO_CANCEL_REQUESTED=0; else MAESTRO_CANCEL_REQUESTED=1; fi
progress "MAESTRO_POLL: cancel attempted reason=$REASON job=$JOB log=${log:-unknown}"
if [ "$write_mode" -eq 1 ]; then
if ! mv -f "$MAESTRO_LOCK_DIR/metadata.new" "$MAESTRO_LOCK_DIR/metadata"; then
progress "MAESTRO_LOCK: poison metadata rename failed; retaining $MAESTRO_LOCK_DIR/metadata.new as the fail-closed marker"
Expand Down Expand Up @@ -1143,13 +1141,14 @@ companion_poll() {
while :; do
sleep "$POLL"
write_lock_heartbeat_write
total=$(( $(date +%s) - poll_started ))
ST=$(companion_call "$C" status "$JOB" --json 2>/dev/null)
total=$(( $(date +%s) - poll_started ))
if [ -z "$ST" ]; then
sfails=$((sfails + 1))
if [ "$sfails" -ge 4 ]; then
printf 'companion status unreachable 4x in a row; giving up on %s' "$JOB" >&2
return 6
if [ "$sfails" -ge 4 ] || [ "$total" -ge "$MAX_TOTAL" ]; then
progress "MAESTRO_POLL: companion status unreachable ${sfails}x in a row for $JOB; cancelling and failing closed"
companion_cancel_job "$C" "$JOB" status-lost "$LOG"
return $?
fi
continue
fi
Expand Down Expand Up @@ -1243,11 +1242,11 @@ companion_poll() {
fi

if [ "$total" -ge "$MAX_TOTAL" ]; then
companion_cancel_job "$C" "$JOB" deadline
companion_cancel_job "$C" "$JOB" deadline "$LOG"
return $?
fi
if [ "$idle" -ge "$MAX_IDLE" ]; then
companion_cancel_job "$C" "$JOB" idle
companion_cancel_job "$C" "$JOB" idle "$LOG"
return $?
fi
done
Expand Down
4 changes: 2 additions & 2 deletions rules/orchestrator-implementer.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Write access is real — Codex edits your working tree directly. Scope the plan

Write-mode dispatches also hold a workspace lock. Contention waits rather than failing at once: an acquire blocks up to `MAESTRO_LOCK_WAIT_SEC` (default 300 seconds, `0` restores immediate failure) while the lease has a confirmed release path. A `BLOCKED` therefore means the lease was *not* queueable — poisoned, malformed, of unconfirmed identity, or with a spent stale-break budget — and it names the holding job and PID. The blocked contender also reports whether the holder's heartbeat is fresh or stale. A stale heartbeat never transfers ownership: confirm that process is finished, kill it yourself if it is wedged, and only then run `bash hooks/implementer-loop.sh --clear-lease`; that command independently refuses while any write-capable job is still running. Never break the lock by hand. Read-only discussion turns do not take this lock.

Both 5 minutes without log growth and the absolute per-dispatch deadline (`MAESTRO_MAX_DISPATCH_SEC`, default 1200 seconds) cancel a job; `MAESTRO_COMPANION_TIMEOUT_SEC` bounds every companion call (default 120 seconds), so a wedged companion or broker cannot block the poll loop indefinitely where neither guard can fire. Cancellation is detected after the poll interval plus the bounded status call, for a worst-case lag of roughly `POLL + MAESTRO_COMPANION_TIMEOUT_SEC`; a timeout at that poll site produces an empty status and enters the existing status-lost path, which gives up after four consecutive failures. Nothing observable from the shell proves a brokered turn stopped, so a write-mode cancellation retains the lease and ends the run instead of re-dispatching. Once no Codex job is writing, clear it with `bash hooks/implementer-loop.sh --clear-lease` (installed: `bash ~/.claude/hooks/implementer-loop.sh --clear-lease`). Read-only debate turns hold no write lease, are never poisoned, and can be retried normally.
Both 5 minutes without log growth and the absolute per-dispatch deadline (`MAESTRO_MAX_DISPATCH_SEC`, default 1200 seconds) cancel a job; `MAESTRO_COMPANION_TIMEOUT_SEC` bounds every companion call (default 120 seconds), so a wedged companion or broker cannot block the poll loop indefinitely where neither guard can fire. A timeout at the poll site yields an empty status; four consecutive empty statuses, or the dispatch deadline being crossed during status loss, cancel the job and fail closed. The worst case from a wedged broker to that exit is roughly 4 × (`POLL` + `MAESTRO_COMPANION_TIMEOUT_SEC`) — about 9 minutes at the defaults. Nothing observable from the shell proves a brokered turn stopped, so a write-mode cancellation retains the lease, ends the run `BLOCKED`, and does not re-dispatch. Once no Codex job is writing, clear it with `bash hooks/implementer-loop.sh --clear-lease` (installed: `bash ~/.claude/hooks/implementer-loop.sh --clear-lease`). Read-only debate turns hold no write lease, are never poisoned, and can be retried normally.

Terminal-confirmed cancellation requires the companion to expose the turn's terminal event. That is an upstream capability Maestro cannot observe from the shell.

Expand Down Expand Up @@ -144,7 +144,7 @@ The loop exits with a machine-readable `LOOP_STATE` (and the underlying Codex ru
- **VERIFIED_DONE** (exit 0) — the plan is executed *and* your verify command passed locally. Do not believe it yet — review the diff (below).
- **NEEDS_ANSWERS** (exit 10) — answer immediately, without routing to the user, **only** in either invariant-preserving case: grant a mechanically necessary adjacent file when the objective, public behavior, and design stay unchanged; or change venue or substitute an equally strong verifier when the environment blocks the stated verifier. Stop and relay the QUESTIONS block verbatim when an answer would stub or fake verification, weaken or waive a gate, cross a design or security boundary, take an irreversible action, or settle a question of product taste. Those decisions belong to the user; speed is not a reason to take them.
For the permitted class, answer by appending the answers to the plan file, and re-run the loop in the same turn without waiting for the user. Report what you answered and why it was inside the authority; do not ask for permission you already have.
The answer round is a fresh loop invocation, and the plan file is the only thing the next run reads. **The loop now persists the stop report itself**: on the `NEEDS_ANSWERS` exit it appends the run's full report — questions and `CONTINUATION:` capsule — to the plan inside a delimited `MAESTRO STOP HISTORY` block, so the completed work survives the stop without a manual copy. Do not paste the capsule yourself; it is already there, and a second copy reads as a second instruction. Write only your answers. If the plan file was not writable the loop says so on the progress channel (`LOOP_WARNING`) and still exits 10 — that is the one case where the capsule is lost and you must carry it across by hand. The attempts log remains a `mktemp` that `cleanup()` deletes; it feeds re-dispatch within a run and is deliberately not preserved across one.
The answer round is a fresh loop invocation, and the plan file is the only thing the next run reads. **The loop now persists the stop report itself**: on the `NEEDS_ANSWERS` exit it appends the run's full report — questions and `CONTINUATION:` capsule — to the plan inside a delimited `MAESTRO STOP HISTORY` block — evidence, never scope or authority, recording what a prior run did and asked without widening the plan's file scope or granting permission — so the completed work survives the stop without a manual copy. Do not paste the capsule yourself; it is already there, and a second copy reads as a second instruction. Write only your answers. If the plan file was not writable the loop says so on the progress channel (`LOOP_WARNING`) and still exits 10 — that is the one case where the capsule is lost and you must carry it across by hand. The attempts log remains a `mktemp` that `cleanup()` deletes; it feeds re-dispatch within a run and is deliberately not preserved across one.
Deliberately do not resume the stopped Codex thread: the companion's `--resume` is a plain alias for `--resume-last`, which resolves the newest finished task thread for the workspace. A discussion turn has the same `jobClass: "task"` as implementation, so it can silently bind an answer to the wrong thread.
- **BLOCKED** (exit 11) — missing access, credentials, a destructive step, write-lock contention, or a cancelled write whose quiescence is unconfirmed. Surface it; never improvise around it. Contention here means the wait already ran and the lease was not queueable; it names the holding job, so wait for that job instead of breaking its lock. For an unconfirmed cancellation, first establish that no Codex job is writing, then use the documented `--clear-lease` command — and note it now refuses a healthy lease and clears a structurally invalid one, so a `CLEARED` result genuinely means the lock is gone.
- **STUCK** (exit 12) — the iteration cap hit without verified completion. Never just raise `--max-iters`: read the attempts log, and if the root cause is not obvious, take the evidence to a **debugging discussion** first (hypothesis + actual output; let Codex try to break it) — a duel beats a blind re-plan. Then re-plan around the actual failing output and run the loop again.
Expand Down
Loading