diff --git a/docs/known-issues/2026-07-16-codex-app-false-busy.md b/docs/known-issues/2026-07-16-codex-app-false-busy.md new file mode 100644 index 000000000..52a187ea5 --- /dev/null +++ b/docs/known-issues/2026-07-16-codex-app-false-busy.md @@ -0,0 +1,232 @@ +# Codex App 会话长时间无进展却持续显示“工作中” + +> 状态（2026-07-18 21:30）：**旧工作稿，不是最终 PR / 飞书 Doc，不可直接送审**。ORIGINAL WIP 的 `HEAD=44e4113937080303afca9037aae1bf483dd2982c`，本地 tracking ref `origin/master=6c75d44c4fd942b490dc8648038d89a1541eb7fc`，两者并不相等；另有独立 clean migration target 固定在 `c14b0180cf39cd8c219d5a6a1389b2cd794faf94`，不能与本树混称。当前 `git status --porcelain` 为 137 个 tracked status row、40 个 untracked row，树不 clean。下表除明确标为“2026-07-18 fresh”的 supervisor 增量外，均是旧快照历史记录，不能作为当前整树 PR 证据。当前仍未 commit、push、建 PR、切 live 或做真实飞书 E2E。 +> 本次迁移前 include-untracked 快照为 stash `23db3c031d9d83b96bb1decd94721d8673f0e627`；此前 `f46a49c1edae00981e29e0893d0d7d700544066c`、`768f2ec77c4886100761a0f299056feafa0c120f`、`baa211dc93d94e6136e519629d08f6fa9ba6315b` 也仍保留，均 apply、未 pop/drop。 +> 记录日期：2026-07-16。 + +## 问题与原始反馈 + +Codex App turn 提交后，如果 app-server 或工具链不再产生可观察进展，Botmux 过去仍会因 CLI 未回到 prompt 而无限投影 `working`。飞书卡片和 Dashboard 因而把“没有可观察进展”写成“正常工作中”，用户无法判断应继续等待还是检查当前 Session。 + +| 项目 | 原始值 / 事实 | 证据层级 | +| --- | --- | --- | +| 本轮缺陷入口 | `om_x100b6ab0f5be6508b1db261a4562aee` | 飞书话题中回读的四问题汇总入口 | +| 用户原始反馈 | `om_x100b6a4d3949dca4deaf50cc4452817`：“一个小时了 什么进度了 什么什么消息都没有” | 飞书 chat history 回读 | +| 关联状态卡 | `om_x100b6a4d36e9a934c4474c3ade214d9`，卡片转录仍为“工作中” | 飞书 chat history；未取得原始 card payload | + +历史 Session 缺少可对应的 `sessionId`、Codex App `threadId`、app-server 通知、PTY 时间线和 daemon 日志。因此本文不把原始卡点断言为 MCP、网络、模型服务或工具初始化故障；修复只检测“连续没有可观察进展”。 + +## 根因 + +1. `codex-app-runner` 已直接消费 `turn/started`、item 和 `turn/completed`，worker 的状态却主要来自 prompt / screen heuristic；未回到 prompt 就一直是 `working`。 +2. 一次 flush 可以向 serial runner 排入多个 turn。turn 1 和 turn 2 之间可能出现短暂真实 prompt；若 turn 2 的 chunked submit 随后失败，旧逻辑不会重放刚才被拒绝的 ready 边界。 +3. daemon 重启后，tmux、Herdr 或 Zellij 里的 runner 可以继续存活，但新 worker 的内存队列已经丢失。仅凭 backend 的 `isReattach` 或终端内容，既不能证明它是旧 runner，也不能安全接受 completion/final。 +4. OSC 处于用户、模型和工具都能写入的终端数据面。任何仅靠前缀、JSON 字段或共享 nonce 的方案都可被伪造；Herdr/Zellij 的渲染接口也没有给 Codex App 控制帧提供可靠的 raw-byte 传输保证。 +5. Mira/Mir 的 terminal marker 路径与本问题无关，继续完整保留上游 `RunnerControlDecoder` 行为，不在本修复中改动其 parser。 +6. terminal prompt 仍可能在 PTY、Tmux、Herdr 或 Zellij 的观察链路中延迟或丢失；反过来，它也可能先于 socket final 抵达 worker。若把 terminal prompt 作为 Codex App 的最终 idle 权威，就无法同时保证“prompt 丢失仍恢复 ready”和“final_output 先于 prompt_ready”。 +7. worker、CLI 或 daemon 在 turn 的不同阶段退出时，原实现没有一份由 daemon 持久化的完整输入与归属账本。仅恢复 Codex App thread、只看 runner idle，或把下一条输入重新塞入同一 runner，都无法证明上一条输入尚未写入、已经提交还是 final 已送达，可能造成错序、重复执行或永久卡在旧归属上。 + +## 方案 + +### 1. 明确的 turn liveness + +- runner 在 turn 出队时发 `submitted`，app-server 通知最多每 5 秒发一次 `progress`，完成时发 `completed`；thread 与 final 也走同一控制面。 +- worker 为每条已开始提交的消息建立队列槽位。完成只推进队首；后续 turn 从真正成为 runner 当前任务时重新起算超时。 +- 当前 turn 90 秒无活动时投影 `stalled`，飞书与 Dashboard 显示“长时间无进展”，同一 turn 只通知一次；后续真实活动恢复 `working`。 +- 不自动重启或取消正在执行的 turn。只有仍处于 daemon `accepted`、尚未进入 runner 写入阶段的完整输入可以在 replacement worker 恢复；任何 `prepared` 或写入结果不确定的 turn 都 fail closed，不自动重放，避免重复模型或工具副作用。 +- turn 间 prompt 在队列非空时不投影 `idle`。若后续 submit 失败并精确取消、队列已空，则在 flush mutex 释放后重放 deferred ready。 +- runner 初始化完成时，以及 serial queue 真正清空时，发送签名 `state { busy:false }`。queue drain 的 idle state 只在所有 queued turns 的 `completed` 与全部 final fragments 已进入可靠控制队列后追加；两条已排队 turn 之间不发送瞬时 idle。 +- worker 把签名 idle state 作为 Codex App ready 权威。terminal prompt 在 signed idle 到达前只会被 defer；因此 terminal 渲染完全丢失仍可进入 ready，而正常完成顺序固定为 `final_output → prompt_ready`。 + +### 2. backend-independent 的非对称控制通道 + +Codex App 不再通过 OSC 发控制消息。POSIX 使用随机 AF_UNIX endpoint，Windows 使用随机 named-pipe endpoint；runner 通过独立 OOB channel 上报。该路径与 PTY、Tmux、Herdr、Zellij 的终端渲染无关。 + +- 每次 spawn 尝试生成新的 Ed25519 keypair 与随机 generation。 +- worker 只持有、持久化公钥；私钥仅写入一个 `O_EXCL`、精确 `0600` 的 one-shot bootstrap。 +- 启动环境只携带 bootstrap **路径**。私钥不进入 env、argv、tmux command、Zellij layout、持久状态或 socket wire。 +- runner 在 app-server 启动前删除环境变量，并用单一 `O_NOFOLLOW` fd 校验 owner/mode/link/size；随后先 unlink，再从同一 fd 读取并导入私钥。文件缺失、symlink、hardlink、宽权限、session 不匹配或 unlink 失败均 fail closed。 +- app-server 只有在 worker 完成验证并返回 `accepted` 后才启动，所以模型及其工具不会继承 bootstrap path。 +- bootstrap 位于 bot 私有目录。POSIX 控制根固定为 owner-only `/tmp/botmux-codex-app-`：每次 endpoint 使用随机叶子，runner 只从固定、受保护、按 Session 哈希命名的 locator 发现它；locator 校验必须由调用方显式提供 trusted control root，不能从 locator path 反推信任。 +- POSIX worker 在触碰 locator 前持有跨进程、进程生命周期级 owner-directory lease。v2 owner / reaper actor record 都绑定创建时目录的精确 bigint `dev + ino`；owner 另带随机 nonce、PID 与 process-start token，reaper 还绑定目标 owner nonce、owner 文件 inode 与进程身份。写后回读且 exact actor CAS 成功才算持有或有权清理。dead actor、过 grace 的 secure partial actor与同目录 losing contender 残留可以恢复；live、unknown、EPERM、无法验证 start token 或多个不一致 candidate 一律 fail closed。replacement directory 中迟到的旧 creator / cleaner 即使停在 publish→CAS 窗口，也不能撤销或毒化 successor。释放只删除本进程 exact actor 并尝试 `rmdir`，`killCli` 和 in-worker restart 都不释放。非 Linux `ps -o lstart` 固定 `LC_ALL=C / LANG=C / TZ=UTC`；同秒 PID 复用只会保守阻塞，不会授权删除 live owner。 +- 短控制根使 macOS endpoint 低于约 104-byte 的 AF_UNIX 上限。Linux bwrap 显式绑定 bootstrap、endpoint 与 locator 目录；macOS Seatbelt 的 own BOT_HOME 与 `/tmp` write carve-out覆盖它们。 +- bootstrap cleanup、worker challenge proof 和 runner authentication 共用现有 first-prompt hard cap：90 秒。超过 30 秒但未到 90 秒的慢启动不再被提前清理或 fail closed。 + +Windows 不把 named pipe 当成文件系统 socket，也不假定 named pipe 自身是 owner-only。方案 A 使用两层独立对象： + +- 控制根固定在当前用户的 drive-qualified `%LOCALAPPDATA%\\Botmux\\codex-app-control`，缺失时回退 Windows home；UNC 路径直接 fail closed，不采信 `SESSION_DATA_DIR`。根目录及子目录通过绝对 `System32\\whoami.exe` / `icacls.exe`、参数数组和 `shell:false` 去继承、设 owner、仅授予当前 SID 与 SYSTEM full control，并回读 `/save` 的 protected SDDL 做 exact verification。命令失败、未知/继承 ACE、非 canonical ACL 或不支持所需 ACL 的文件系统都 fail closed。 +- bootstrap 仍是 hardened root 下 `O_EXCL` 随机叶子；Windows 保留 lstat/open/fstat 的 regular-file、single-link、size 和同一文件 identity 校验，先 unlink 再从同一 fd 读取。只跳过 Windows/Node 不可靠的 `O_NOFOLLOW`、uid、精确 mode、`fchmod`、unlink 后 nlink 与目录 fsync；POSIX 原约束不变。 +- public state 与 locator 使用同目录随机 `O_EXCL` temp、file fsync、atomic rename、rename 后回读校验；Windows 不做 directory fsync。v3 state 只持有 public identities，不保存 bootstrap path 或私钥；不兼容或损坏的旧 state 读取失败后 cold start。既有固定 state 文件在信任前再经 exact ACL hardening；新 temp/locator/bootstrap 继承已验证的 restricted parent DACL。 +- 每个 worker 的 runner-facing endpoint 是独立 256-bit 随机 `\\\\?\\pipe\\botmux-codex-app-<64 hex>`；另生成独立 256-bit epoch。worker 必须先 bind，再原子发布 `{version, sessionId, epoch, endpoint}` locator，最后才允许 `backend.spawn`。 +- daemon 的 kill-then-fork 会短暂重叠进程。为避免旧 worker 在新 worker 之后覆盖固定 locator，每个 Session 另持有 deterministic owner-lease named pipe，直到进程退出才由内核释放；新 worker 对 `EADDRINUSE` 进行 50ms 有界等待，10 秒仍拿不到即 fail closed，其他 bind 错误立即失败。lease 不承载 secret、challenge、accepted 或 runner 流量。 +- stop、prepare 与 publish-failure 不对固定 Windows locator 做 read→unlink；那不是 CAS，可能误删 replacement epoch。关闭随机 pipe 后留下的 locator只会指向已失效 endpoint，下一 owner 在新 endpoint bind 成功后原子覆盖。 +- runner 从 bootstrap 取得 `locatorPath`，对 missing/corrupt locator 以 250ms 轮询。未 accepted 的 endpoint 可在每次独立 5 秒**绝对** handshake deadline 下重试到共享 90 秒 proof deadline；transport slow-drip 不会续命。accepted 还必须回显 protected locator 的 epoch；一旦 accepted，该 pipe name 永久 burn，关闭后只接受新 locator endpoint。 +- authenticated socket 关闭会在 POSIX/Windows 都重新 arm 90 秒 proof deadline，并先 bind+publish 新随机 endpoint，再退休旧 server；若 runner 未在 deadline 内重新完成 challenge proof，generation fail closed，而不是永久停在 proof-gated `working`。 +- `accepted` 仍有意不签名：POSIX 权威来自 owner-only fixed root、严格 locator、已 bind 的随机 AF_UNIX endpoint 与独立随机 epoch；Windows 权威来自 restricted locator、已 bind 的随机 named pipe 与独立随机 epoch。runner auth 与所有 marker/ACK 的 Ed25519 challenge、generation、seq 语义不变。 + +### 3. old reuse 与 fresh start 由密码学证明，不依赖预测 + +spawn 前，pending 状态同时保留旧 public identity 和 fresh candidate public identity。backend 如果实际复用旧 runner，会忽略新命令/bootstrap；若实际 fresh，则新 runner 消费 candidate bootstrap。 + +每条连接都由 worker 发送新的 256-bit challenge： + +1. runner 对 `session + generation + challenge` 的独立 auth domain 作 Ed25519 签名； +2. worker 只在 pending identities 中找 generation，并用对应公钥验签； +3. old key 验过即证明实际 warm reuse，candidate key 验过即证明实际 fresh； +4. 选中的 identity 被原子折叠为唯一 `active` 状态，之后才接受 marker 和 prompt。 + +这同时覆盖“存在性 probe 与 backend.spawn 之间发生变化”的竞态。Herdr 的 `isReattach` getter 也改为报告 `agent get` 是否真的复用了 pane，而不是构造参数；但控制面安全不依赖该 getter。 + +### 4. marker anti-replay 与可靠重连 + +- 每个 marker 的签名 domain 都包含 `session + generation + connection challenge + seq + event kind + canonical payload`；auth 与 marker 使用不同 domain separator。 +- worker 只在已经通过 challenge 的同一 socket 接受 marker；每条连接的第一条 seq 可在 worker replacement 后从任意正整数开始，但随后必须严格连续。连续性 fence 有意先于 generation replay window：这样 ACK 丢失后的完整连续重放可以逐条补 ACK，任何连接内 gap 又不会被后面的累计 ACK 掩盖。旧连接抓到的 auth/marker 因 challenge 不同，不能在新 worker 上重放。 +- 普通 marker 独立 ACK；chunked final 由每个认证连接自己的 assembler 接收，并以 `final-end` 作累计 ACK。start/chunk 在完整 assembly 前都留在 runner 队列；id/total/index、严格 base64、chunk 顺序、总大小或 final-end 完整性任一不满足，worker 都清掉该连接的 partial assembly、销毁 socket，且不 commit / ACK 该记录。runner 随后用新 challenge 重签并从 `final-start` 重放完整 transaction，而不是只发送缺少前缀的残片。 +- 若完整 `final-end` 已写入 daemon 的 generation highwater，但 ACK 在连接断开前丢失，重连后的 start/chunks/end 都落入 replay window：worker 只补累计 ACK，不重新 assembly，也不再次请求 daemon 投递。外部 provider 已接受、daemon 尚未持久化 settlement 时仍是 at-least-once 窗口；普通飞书消息使用稳定 UUID 在 provider 的 1 小时幂等窗口内去重，不能外推为无限期或全 sink exactly-once。 +- 未认证连接有独立 decoder、challenge 和 5 秒 timeout；worker 继续 accept，并在同时未认证连接达到 16 条时直接拒绝新连接，单个冒充客户端不能永久占住通道。 +- 控制 line、final 总大小、chunk 数和每连接单一 final transaction 都有硬上限；畸形、重叠或交错输入 fail closed。 + +### 5. daemon 持久归属、恢复与 final settlement + +- daemon 在把 Codex App 输入交给 worker 前生成稳定 `dispatchId`，冻结完整输入、effective turn id、原始 reply root、caller 与 delivery sink，并将 `accepted` ledger 原子写入 Session；持久化失败会回滚本次 acceptance。 +- worker 在写 runner 前先占用本地 FIFO，再要求 daemon 将 exact ledger head 从 `accepted` 持久化为 `prepared`。tmux/backend 的 `false`、throw、chunk 或 Enter 失败都不能证明“零字节写入”，因此统一视为 `dirty_unknown`：保留归属、隔离 generation，并让 daemon replacement/fence 接管，而不是取消后重投。 +- final 先在 worker 中只做 exact head 校验而不 pop；daemon 再校验 worker、Session、ledger 与 generation，完成可见投递和 durable terminal settlement，原子 pop exact ledger head 并写 cumulative highwater，最后 ACK worker。worker 收到 ACK 后才 pop 本地 FIFO 并 ACK runner。空 final 也走同一事务，不会被当作缺失消息。 +- replacement worker 会恢复完整 ledger FIFO。`accepted` 输入可以在新的安全写入前恢复；`prepared` 输入即使来自 warm runner 且收到签名 idle，也不能证明 raw input buffer 未写，因此不自动 reset / replay。只有 runner 重放 final 或 daemon highwater 能关闭该归属；fresh runner 遇到 recovered `prepared` 会立即 fail closed。 +- 自然 runner exit、worker SIGKILL、daemon restart、VC receiver expiry/reset、boot recovery 都保留 exact turn/attempt ownership。只有 backing 被权威证明 missing 且旧 ledger exact-retire 已持久化后，runtime fence 才允许下一 attempt；持久化失败继续 gate/reprobe。 +- `/restart`、`/cd`、suspend、transfer、native thread/adopt 与 double-fork 在 ledger 非空时拒绝或安全排入既有 FIFO；显式 `/close` 是唯一放弃边界，会原子清 ledger/highwater。read-isolation 切换同样先关闭该 bot 的 admission，并只在 runtime 与持久态均无 active Session、closed Session 的 PID 已消失且所有 owned persistent backing 都证明 missing 后写配置；否则返回冲突，要求关闭会话并稍后重试。 +- 普通飞书投递使用稳定 provider UUID `ca_`，其去重能力受飞书 1 小时窗口限制。doc-comment、HTTP wait/async、silent/suppressed 等 sink 会随 ledger 持久化；若 daemon 重启后原 sink 已不可恢复，final fail closed，不会降级泄漏到普通飞书消息。`botmux send` 的 sink 权限绑定可信 origin Session + turn/attempt，`--session-id`、voice、附件或 relay 不能绕过。 + +### 6. 状态与 UI + +- `ScreenStatus / StreamStatus` 增加 `stalled`；飞书卡片使用 danger header，Dashboard 可筛选并进入需关注列。 +- Dashboard 总览仍把 `stalled` 计为 active；Resource Monitor 将其计作 runtime `working`，因为含义是“可能仍执行，但暂无可观察进展”。 +- maintenance heartbeat 将 `working / stalled` 都视为 busy，不因展示状态变化而新增全局重启机会。 + +### 7. PM2 core fleet 的安全停启与首次升级边界 + +这部分不是 false-busy 状态机本身，而是本修复能否安全合入和升级的关键前置。Codex App / Riff 已把输入归属与 shutdown preparation 持久化后，原来的 PM2 控制方式仍可能在重启途中造成“只停了一半”“停错 PID 的下一代进程”或“旧任务已 prepare、却被下一次重启报告误消费”。因此本轮把 PM2 stop / restart / start / start-bot 一并收敛为可验证的 fleet transaction。 + +**要解决的问题与根因：** + +- PM2 的 `jlist`、God socket 和 `process.kill(pid)` 都不是 PID + process-birth 绑定的授权；检查之后若 PID 被复用，普通 OS signal 可能落到 successor。 +- PM2 把进程标成 `online` 早于 daemon 安装 signal handler、注册 shutdown endpoint 并发布 capability；只验 `online` 会把半启动进程当成可安全接管的新 fleet。 +- bundled PM2 在 signal-only child exit 上把 `code` 归一成 `code || 0`，因此 SIGKILL / OOM 也可能投影为 `exit_code=0`。若 daemon 的 `stop_exit_codes` 同样使用 0，prepare 中途被硬杀会被误判为 graceful no-restart。 +- PM2 达到 `unstable_restarts >= max_restarts` 时会抑制后续 restart，并可能把硬杀后的 row 留在 `errored`。因此“没有 restart timer / status=errored”只能证明 row 暂时静止，不能证明刚才接受 `202` 的 shutdown transaction 已完成。 +- 多 bot 启停期间，`bots.json`、PM2 rows、daemon descriptors、restart timers 和 God generation 可能分别变化。一次旧快照或批量 `delete` 无法证明每个实际 mutation 仍指向原对象。 +- 首次升级时，磁盘上的新 CLI 已经包含协议，但内存中的旧 daemon 没有 `riff-fleet-prepare-persist-commit-exit42-v2` shutdown capability。把“包已安装”误当成“daemon 已升级”会在 Riff 仍工作时强制停机。 +- restart breadcrumb 原先没有完整的 attempt 状态与原子 claim；部分重启或 report 并发可能提前消费一个尚未 verified / committed 的 intent。 + +**解决方法：** + +- 每次 mutation 前重新读取严格语义 `pm2 jlist`，拒绝 malformed row、重复 canonical `pm_id`、重复正 PID、重复 God、未注册的 live descriptor，并以 exact PM2 id + name + PID + process-birth 重新证明对象仍静止后才逐个 stop / delete。 +- daemon 只有在 SIGTERM / SIGINT handler 与 exact shutdown handler 都已安装后，才把 shutdown protocol 写入 fresh descriptor。CLI 对全部 live daemon 做 exact-set attestation，再用 route/port-bound HMAC 向 exact app + boot + process-birth 发并发 loopback POST；只有 exact `202` ACK 才算该 generation 接受 shutdown。缺 ACK 即使随后退出也按失败处理，并补偿恢复已退出 peer。 +- daemon-only PM2 policy 使用共享的非零 `DAEMON_GRACEFUL_EXIT_CODE=42`；只有 prepare / persist / commit 与 worker teardown 全部完成后才以该 sentinel 退出。signal death 归一出的 0 不在 daemon `stop_exit_codes` 中，必须继续 autorestart；无 Riff lineage 的 Dashboard 保留独立 exit-0 policy。 +- PM2 registry 投影保留原始 `stop_exit_codes` 元素，不做 parse / filter；daemon policy 只接受唯一 numeric `42` 或 canonical string `"42"`。因此 `[42,"0foo"]`、`[42,"0x0"]`、`[42,null]` 等会被拒绝，避免 PM2 自身 `parseInt` 语义把隐藏的额外值解释成 0 并抑制 signal-death restart。 +- timer-free 判定只用于初始 dormant admission 与失败补偿；post-signal quiet-success 另要求每个 name 的 latest 已 signal generation 在 quiet window **每一轮** fresh projection 中仍有 exact `name + pm_id` terminal row，并由 raw `stop_exit_codes` 接受其 `exit_code`。daemon 的 `errored + exit_code=0 + [42]` 必须进入恢复或 partial-failure，只有 `exit_code=42` 才能成为 graceful terminal proof；一次看到 42 后 row 又缺失也不能成功。只有 fresh 且 OS-live 的 successor row 携带上一代已接受 exit 时才缓存 predecessor；dead different-PID row 的 exit 可能已属于 successor 自身，不能反证上一代。successor 被 signal 后随即成为必须逐轮重证的 latest generation。Dashboard 仍按其独立 `[0]` policy 验证。 +- start / restart / start-bot 使用同一 fleet lock 与锁内 `bots.json` snapshot，PM2 start 有总超时、late-publication settle、fresh online + handler-ready exact-set verification；部分启动或验证失败只回滚本 attempt 可证明拥有的 rows。 +- restart intent 使用 `prepared → committed` / `aborted` attempt fence，并以原子 claim 取代 `consume + hasPrepared` 竞态；新 fleet 完整 verified 前不会 commit，也不会产生成功重启报告。 +- `restart --include-pm2` **不再宣称会重启一个正在运行的 PM2 God**。Node / PM2 没有 generation-bound God signal，因而该选项只允许“命令入场时零 live God”的干净场景；只要发现一个或多个 live God，命令就在 breadcrumb、PM2 RPC、daemon signal 和 fleet mutation 之前零改动拒绝。 + +**改动规模与兼容性：** 这是一次较大的 supervisor control-plane 改动，涉及 CLI fleet admission、daemon descriptor、loopback shutdown IPC、start rollback 和 restart report；它不改变 Codex App turn / final 的业务协议，也不把不安全的自动 kill 当成兼容降级。外部脚本若绕过 Botmux 直接改 `bots.json` 或直接调用 PM2，不受 advisory lock 保护，仍属于明确的外部边界。 + +**首次升级操作边界：** 已运行的旧 daemon 没有新 capability 时，普通 `botmux stop` / `botmux restart` 会在任何 daemon signal 前 fail closed；这是预期安全行为，不是可自动忽略的错误。操作者必须先独立确认所有 Session 与 Riff workload 均 idle，在获批维护窗口内一次性运行 `botmux restart --bootstrap-shutdown-protocol --yes`。该显式双确认入口逐个绑定并复核 PM2 `name + pm_id + PID + process birth` 后退役旧 core，再按新协议启动并验证全部配置 row 都 `online` 且发布 handler-ready descriptor；它不会由 upgrade/update 自动调用。自动更新只能报告“新包已安装、restart 已请求或被阻断”；在新 fleet 完整验证并提交 restart attempt 之前，不得宣称更新已应用。 + +## 修复前后效果 + +| 场景 | 修复前 | 修复后 | +| --- | --- | --- | +| turn 正常产生 app-server 活动 | `working` | 保持 `working`，活动刷新 90 秒窗口 | +| turn 连续 90 秒无活动 | 永久 `working` | `stalled`，提示一次并允许后续活动恢复 | +| 一次 flush 排入多个 turn | 无独立生命周期 | 队列化跟踪；等待前序不计作后序无进展 | +| turn 间 prompt 后下一条 submit 失败 | 可能不再 ready | 精确取消槽位并重放 deferred ready | +| terminal prompt 在 backend 观察链路中丢失 | 永远无法进入 idle | 签名 `busy:false` 独立驱动 ready / idle | +| terminal prompt 先于 socket final 抵达 | 可能先 idle、后 final | prompt 先 defer；signed idle 排在 final transaction 后，保证 final-before-ready | +| 用户/agent/tool 输出伪造 OSC | 可能推进状态或伪造 final | Codex App 完全不从终端接受控制消息 | +| worker 重启、旧 runner 存活 | 依赖 backend/屏幕推断 | fresh challenge + old public key 验签后才 warm reattach | +| Herdr 预测 reattach 但实际启动新 agent | `isReattach` 语义错误 | getter 报 actual；candidate key 证明实际 fresh | +| worker 在 final chunk 中途替换 | 已 ACK 前缀丢失后无法重组 final | final 事务到 `final-end` 才累计 ACK；新 worker 收到完整重放 | +| final-end 缺 chunk / total 不一致 | 仍可能累计 ACK 并永久丢 final | 不 commit、不 ACK、销毁连接；新 challenge 下重放完整 transaction | +| final-end settlement 已持久化但 ACK 丢失 | 重连可能重复发布 final | generation highwater 识别连续 duplicate replay，只补 ACK，不再次请求投递 | +| worker 在输入写入边界退出 | 无法区分未写、已写、已提交 | daemon ledger 区分 `accepted / prepared`；只恢复 safe accepted，prepared fail closed | +| daemon 在外部 provider 接受后、ledger commit 前退出 | 可能重复投递或改投其它 sink | 原 sink 与稳定 UUID 持久化；普通飞书在 1 小时窗口内幂等，其他不可恢复 sink fail closed；不宣称无限期 exactly-once | +| A/B 两条 chat-scope turn 排队后回复根变化 | 晚到的 A 可能落群顶层或串到 B | 每条 ledger 冻结 exact plain/thread reply target、caller 与 sink | +| pre-upgrade pane 无 public identity | 无可信 lifecycle | attach 前 fail-closed cold start，不接收 legacy completion/final | + +## 安全边界 + +本设计防止终端正文、agent message、工具 stdout、另一个 socket client或旧连接 transcript 伪造 Codex App lifecycle/final；也避免把可复用私钥放入 env、argv、持久态或控制 wire。 + +它不宣称能隔离“已经取得同 UID 任意进程读取/调试能力”的攻击者：能够在 trusted runner consume 前抢读 bootstrap、读取 runner 内存、使用 `ptrace`，或持续改写 locator / 干扰 socket 的同 UID 恶意进程，仍可窃取 key 或造成 DoS。该级别需要 OS 级进程隔离或不同 UID，超出本 PR。模型及其新启动工具不会继承 bootstrap path，但预先存在的同 UID 进程不在此保证内。warm reattach 仍会准备一个 fresh candidate bootstrap 以覆盖“预测复用但实际 fresh”的竞态；若旧 runner 真正复用，该文件会在旧 key proof 后清理，期间同样受上述同 UID 边界约束。 + +Windows named pipe 本身没有在本实现中设置或验证 owner-only DACL，因此不作该声明。跨用户本地进程可以尝试抢占 deterministic owner-lease name、枚举/连接 pipe 或制造连接耗尽，形成可用性 DoS；它拿不到 restricted locator 中的 256-bit epoch，不能据此伪造 accepted。owner lease 与 control pipe 都不承载可复用 secret。UNC root 已拒绝；drive-letter 映射是否为真实本地卷仍需真实 Windows 验证，ACL/rename/readback 任一步不满足都会 fail closed，不能把本机 mock 外推为 SMB/FAT 可用。 + +Windows 文件删除语义的实现依据是 Node 所用 libuv v1.x `src/win/fs.c::fs__open` 默认包含 `FILE_SHARE_DELETE`，因此 one-shot bootstrap 可以保持同一 handle、unlink 后同 fd 读取；这是源码依据，不是目标 Windows E2E 证明。ACL 命令与 SDDL 语义以 Microsoft `icacls` 文档为准：；libuv 源码：。 + +## Windows 原生验证缺口（实现已落地，不得标 Windows-ready） + +方案 A 的代码、纯状态机、文件策略和平台无关 locator orchestration 已落地。本机 integration 使用真实 runner + POSIX 随机 AF_UNIX endpoint 验证 locator polling、重复 challenge、wrong epoch、accepted endpoint burn 与 replacement endpoint re-auth；Windows-shaped endpoint/ACL 仅由纯逻辑和 source contract 覆盖。这不能验证 Windows kernel、NTFS ACL、`icacls /save` 的真实字节、named-pipe libuv 行为或进程退出释放 lease。 + +仍缺以下 Windows 原生证据： + +- 在受支持 Windows/Node 版本上真实执行 `whoami` / `icacls`，覆盖 exact current-SID + SYSTEM DACL、未知/继承 ACE 拒绝、UTF-16 `/save` 输出和 ACL-incapable volume fail closed； +- 真实 bootstrap create→lstat/open/fstat→unlink→same-fd read，确认当前 Node/libuv 的 delete-sharing 与 identity/size/nlink 分支； +- 两个 worker 进程竞争 deterministic owner lease，覆盖旧进程退出释放、新进程 acquire、10 秒超时和非 `EADDRINUSE` fail-fast； +- 真 named pipe 的 bind-before-publish、bind failure不发布、active close A→B、pre-auth retry、wrong epoch、accepted A 不重连，以及 locator atomic replacement； +- Windows CI 或目标开发机至少一层端到端 runner/app-server 验证。 + +在这些原生证据完成前： + +- 不得把本文或实现称为 Windows-ready、已上线或已通过目标机验证； +- 不得把 macOS 上的 mock/integration/full unit/build 外推成真实 Windows 可用； +- 后续若经用户批准创建 Issue1 PR，必须在 PR 中原样保留该证据缺口，并把 Windows CI / 目标开发机验证列为合入或发布前检查，不能宣称 Windows 已验证完成。 + +## 影响面 + +| 维度 | 结论 | +| --- | --- | +| CLI | liveness 与非对称 socket 只用于 `cliId=codex-app`。Mira/Mir 继续完整使用上游 canonical `RunnerControlDecoder`，本修复不改动其 terminal marker parser。 | +| backend | socket 是独立 OOB；Pty、Tmux、Herdr、Zellij 不需要传播 OSC。持久 backend 通过公钥 proof 区分 reuse/fresh。 | +| platform | POSIX 使用 fixed protected locator + random AF_UNIX endpoint + process-lifetime owner-directory lease；Windows 使用 hardened locator + random named pipe + kernel-lifetime owner pipe。Windows native E2E 仍是 blocker。 | +| Session | 现代 runner 可在 worker restart 后 challenge re-auth；pre-upgrade/无公钥 pane 会 cold start，可能失去仅在内存中的上下文。 | +| 消息 | `stalled` 不丢弃、不出队、不重投 pending message。final 通过 ACK 队列重连重发。 | +| workflow | 本 PR 不扩展 workflow one-shot event 的结束/通知策略。 | + +## 历史验证记录与本轮增量证据 + +本节保留旧工作稿的历史结果，目的是说明曾覆盖过什么，不是宣称它们已在当前 `HEAD` / dirty WIP 上复跑。只有明确标为“2026-07-18 fresh”的 supervisor focused、TypeScript 和 reconstructed-delta diff check 是本轮增量证据；当前整树的 full unit、build、clean-tree 与最终独立 PASS 仍待 clean target 迁移后重做。Windows 行为测试只证明平台无关 JS 逻辑，原生证据仍按上节保留 blocker。 + +| 层级 | 内容 | 当前结果 | +| --- | --- | --- | +| 历史：asymmetric protocol / POSIX lease / Windows policy unit | bootstrap single-fd consume、SID/SDDL、strict CSV second-column SID、fixed trusted root、random POSIX/Windows endpoint、v2 directory-bound owner/reaper actors、partial/dead actor recovery、双 contender、delayed creator/cleaner publish→CAS、auth/replay/line bounds、per-connection seq fence、final assembler reject/complete | 旧快照记录：`test/codex-app-control.test.ts` 48/48 passed；另连续复跑 5 次均 48/48；Windows 部分非 native；未在当前整树复跑 | +| 历史：liveness / ready authority | queue timeout/recovery、exact cancellation one-shot late prompt、absent/stale handle不授权、signed idle final-before-ready、new work/state/reset 清 arm | 旧快照记录：`test/codex-app-turn-liveness.test.ts` 23/23 passed；未在当前整树复跑 | +| 历史：runner local integration | accepted 前不启动 app-server、locator retry/burn、slow-drip deadline、signed lifecycle、queue drain idle、endpoint re-auth、chunk 中断完整重放、incomplete final-end 无 ACK 后完整重放、committed final ACK 丢失后只补 ACK、empty final transaction、OSC injection escaping、stable Botmux turn identity | 旧快照记录：`test/codex-app-runner.integration.test.ts` 16/16 passed；POSIX local integration；未在当前整树复跑 | +| 历史：worker + daemon durable recovery focused | N/N+1 FIFO、自然 runner exit、worker SIGKILL、surviving runner replacement、empty final ACK barrier、accepted/prepared ledger、generation highwater、VC runtime/boot exact retirement、frozen reply/sink/origin、mutation admission、CLI sink gate与生命周期冲突 | 旧快照记录：40 files / 1067 tests passed，其中真实 worker + tmux replacement integration 4/4；未在当前整树复跑 | +| PM2 / Riff supervisor safety focused（2026-07-18 fresh rerun） | strict jlist、duplicate God/PID、stale descriptor live-birth authority、raw exact `stop_exit_codes` policy、fresh-per-round overlimit terminal proof、live-successor predecessor proof、nonzero graceful-exit sentinel / signal-death restart、exact per-mutation identity、fleet rollback/late publication、idempotent handler-ready exact-set、restart intent CAS、Riff prepare/abort/restore、signed exact loopback shutdown、bots.json lock 与 maintenance compatibility | 20 files / 233 tests passed；其中真实 `startIpcServer({ authRequired: true, ready })` loopback 覆盖 unsigned 401、未注册 503、wrong boot/birth 409、exact signed tuple 202 | +| 历史：independent code review | 对旧冻结树复核 fatal exit、workerless/boot retirement、reply root/caller/sink、fresh origin、revocable admission lease、terminal/worktree re-admission与 read-isolation close-first proof | 旧快照记录：PASS、12 files / 253 tests passed；不是当前 supervisor frozen review 结论 | +| TypeScript | `./node_modules/.bin/tsc --noEmit` | 2026-07-18 在本轮 supervisor 增量上 fresh rerun，exit 0 | +| 历史：full unit suite | `./node_modules/.bin/vitest run --project unit --reporter=json --outputFile=/private/tmp/botmux-issue1-final-full.json` | 旧快照记录：2267/2269 suites、9207/9296 tests passed，82 failed、7 pending，并曾与当时 clean base 的失败 title 集一致；这些 JSON/hash 不代表当前整树 | +| 历史：build | `./node_modules/.bin/tsc && cp src/setup/lark-scopes.json dist/setup/ && node scripts/build-dashboard.mjs && chmod +x dist/cli.js` | 旧快照记录：exit 0；当前整树未 build，本轮也明确禁止 build | +| 2026-07-18 fresh：supervisor reconstructed-delta diff hygiene | `git diff --check` against the saved pre-supervisor frozen baseline | exit 0；仅证明 supervisor delta 无 whitespace error；ORIGINAL 仍有 137 个 tracked status row 与 40 个 untracked row，不是 clean tree | +| Windows native | 实现已落地，真实 `icacls` / bootstrap unlink / named pipe / process lease / app-server E2E 尚未执行 | 证据缺口；不得标 Windows-ready，后续 PR 必须显式携带并在 CI / 目标机补证 | +| target machine / live daemon / 真实飞书 E2E | 未执行 | 不得报告为已上线或原 Session 已修复 | + +## 当前工作树、迁移目标与远端风险 + +- ORIGINAL WIP 当前 `HEAD=44e4113937080303afca9037aae1bf483dd2982c`，本地 tracking ref `origin/master=6c75d44c4fd942b490dc8648038d89a1541eb7fc`；两者不相等。`git status --porcelain` 当前是 137 个 tracked status row + 40 个 untracked row，改动仍未 commit。clean migration target 另行固定在 `c14b0180cf39cd8c219d5a6a1389b2cd794faf94`，其 PR 文档与验证必须独立生成，不能复用本段旧稿状态。 +- 历史恢复点仍包括 include-untracked stash `23db3c031d9d83b96bb1decd94721d8673f0e627`；此前 `f46a49c1edae00981e29e0893d0d7d700544066c`、`768f2ec77c4886100761a0f299056feafa0c120f`、`baa211dc93d94e6136e519629d08f6fa9ba6315b` 也仍保留，均为历史恢复证据，不代表当前 base/clean 状态。 +- 本次 upstream 是 PR #470 的会议多 Agent 可靠投递，和 Issue1 有 12 个重叠文件：`src/codex-app-runner.ts`、`src/daemon.ts`、Dashboard i18n/style、CLI i18n、`src/types.ts`、`src/utils/child-env.ts`、`src/worker.ts`、runner integration fixture/test 与 worker source-contract test。 +- 三个文本冲突位于 `src/codex-app-runner.ts`、`src/worker.ts`、`test/codex-app-runner.integration.test.ts`。runner 保留上游 `RunnerControlWriter` 只做可见正文/错误转义，Codex lifecycle/final 不再写 terminal marker；worker 保留上游 `RunnerControlDecoder` 供 Mira/Mir 使用，`codex-app` 明确不在 OSC allowlist。 +- 上游 #470 的 stable Botmux/native turn identity、worker-owned dispatch attempt、durable terminal drain、backend-exit ambiguous fencing、durable expiry/restart语义均保留。审计还发现并修复一个迁移缺口：`spawnCli` 异步化后，in-worker restart 的 500ms respawn 曾未 `await`；现在 init、crash retry 与 in-worker restart 三条路径都等待 endpoint bind + locator publish，并识别 `CliSpawnSupersededError`。 +- 其余 9 个 auto-merge overlap 已逐项对照：daemon 只叠加 `stalled` maintenance busy；types 同时保留 #470 dispatch/receiver schema 与 Issue1 status；child-env 同时保留 dispatch attempt 和 one-shot bootstrap path；fixture 同时保留 #470 OSC injection 与 bootstrap/argv泄漏检查。 +- 2026-07-16 本轮远端复核：PR #484 仍为 `OPEN / BLOCKED / REVIEW_REQUIRED`，head `02b58abdd9126b0610c9e124794690cb2cca5169`，无 CI checks，`updatedAt=2026-07-16T08:31:44Z`。它仍与 Herdr/worker 文件级重叠；无论后续状态如何，本修复都不能退回“按 backend.isReattach 认证”，actual getter 只用于 screen seeding，控制面继续依赖公钥 challenge proof。 + +## 未覆盖项与回滚 + +- Windows native ACL、named-pipe、owner-lease 和 bootstrap delete-sharing 仍无目标机 E2E；在补证据前仅可报告“实现 + 平台无关行为测试通过”。 +- 90 秒是“没有可观察进展”而非故障判定；安静的长工具可能暂时进入 `stalled`。 +- 同一 turn 的通知去重在 worker 内存中；worker 重启后同一长任务可能再提醒一次。 +- worker 事件循环完全冻结时无法运行状态 tick；这需要 daemon watchdog，超出本 PR。 +- 本 PR 不提供自动取消、自动重启或 exactly-once 模型/工具执行。daemon ledger 与 ACK 保证归属、顺序、fail-closed 恢复和已持久化 settlement 的 replay ACK；普通飞书稳定 UUID 仅在 provider 的 1 小时窗口内提供幂等，超出窗口的 crash replay 仍是 at-least-once 边界。 +- 显式 doc-comment 成功发送后，`docCommentTargets[turn]` 的进程内条目尚未主动清理；它不会跨 sink 泄漏或覆盖 durable ledger，但会保留到进程结束，后续可单独补清理。 +- read-isolation 现在是 close-first / retry 语义：必须关闭该 bot 的全部 Session，并等待 PID 消失及 owned persistent backing 证明 missing 后才能切换。Dashboard 当前会显示后端 409 文本；更友好的专用 UI/i18n 提示可后续补充，不影响安全边界。 +- 回滚会移除 liveness、`stalled` UI 与非对称控制通道；回滚版本无法可信接管本版本的现代 runner，应 cold-start persistent Codex App pane。任何 live 切换/重启仍需单独维护窗口与用户确认。 diff --git a/docs/known-issues/2026-07-16-codex-running-reported-idle.md b/docs/known-issues/2026-07-16-codex-running-reported-idle.md new file mode 100644 index 000000000..42b7be8cf --- /dev/null +++ b/docs/known-issues/2026-07-16-codex-running-reported-idle.md @@ -0,0 +1,256 @@ +# Codex 任务运行中误报“等待输入 / idle” + +> 状态：已 rebase 到 origin/master（commit 12678852 基底 19893f1e）；冲突已消解；focused unit 135/135 PASS；tsc 仅见上游 desktop/electron 既有噪声；尚未 push / 开 PR / live E2E +> 日期：2026-07-16 ｜ 严重度：高 ｜ 建议 PR：`fix(codex): pending turn 不再误报等待输入` + +## 用户反馈与证据边界 + +问题入口是外层转发的飞书消息 `om_x100b6a4c39de2cb4c37ceab580a1735`，其中用户原话为： + +> “但是我看他已经是等待我输入状态了？真的在工作吗。。” + +已确认的观察是：转发上下文中 Codex rollout 仍继续产生进展，而 Dashboard API 已把会话投影为 `idle`，飞书流式卡片同时显示“等待输入”。 + +证据存在以下边界： + +- 上述 ID 是外层转发消息 ID，不是原会话 source message ID；原消息 ID 与精确时间未保留。 +- 当时的 worker 日志、完整 rollout、Dashboard 原始响应和卡片 payload 没有形成统一时间线。 +- 因此这是有真实用户反馈与运行观察支撑的缺陷，但不是已经补齐的线上可复现包；本文不会把本地测试称为线上复现或 E2E 证据。 + +## 问题现象 + +当结构化 transcript 中仍有已提交或已开始、但尚未出现 `assistant_final` / `turn_aborted` 终态的 turn 时，CLI 屏幕可能短暂出现 prompt，或 PTY 进入静默。旧逻辑随后会： + +1. `IdleDetector` 把屏幕 prompt / 静默判断为 ready； +2. `markPromptReady()` 立即把 `isPromptReady` 设为 `true`、发送 `prompt_ready` 并清空 in-flight 输入； +3. 即时 `screen_update`、后续周期 tick 与截图上传把状态投影为 `idle`； +4. Dashboard 与飞书卡片共同消费该投影，因此同时误报“等待输入”。 + +实际 runner 并没有完成。这是 **false idle（假空闲）**，可能诱发重复提问、不必要的重启，或把仍在运行的会话当作可回收空闲 worker。 + +## 根因 + +旧实现混淆了两类信号： + +| 信号 | 实际含义 | 可靠性边界 | +| --- | --- | --- | +| screen prompt / PTY quiescence | 当前 UI 看起来可输入或暂时静默 | 启发式；不能证明 turn 已结束 | +| adapter/history submit confirmation | 输入已被 CLI 的提交历史明确记录，但 type-ahead 时 transcript user event 可能尚未写入 | 可作为短时 hand-off 证据；必须有界，不能永久压制 ready | +| structured transcript lifecycle | transcript 已出现 user/start，直到 `assistant_final` 或 `turn_aborted` 关闭 | 对任务完成更权威；worker 侧 mark 本身不能证明输入已进入 CLI；中断终态不应伪造回复 | + +`CodexBridgeQueue` 已经维护了 pending、started 和 terminal 的 turn 生命周期，但状态投影没有使用它作为门控。即使只在 `markPromptReady()` 的即时 idle 更新上补条件，`startScreenUpdates()` 的周期 tick 和截图路径仍能重新写入 `idle`，所以零散条件不是完整修复。 + +## 修复方案 + +本修复让结构化 turn 生命周期优先于屏幕启发式，并把所有展示入口收敛到同一个状态投影： + +1. `CodexBridgeQueue.hasBlockingTurn()` 把 transcript 已 started、且尚无 `assistant_final` / `turn_aborted` 的 Codex turn 视为强运行证据。单纯 worker mark 不参与状态门控。强门控通过显式 allowlist **只对 Codex 开启**；其他 structured driver 继续使用共享队列做归属与输出，但在补齐全部正常、异常和中断终态契约前，不用 started turn 永久压制 screen-ready。 +2. worker 在每一条 fresh/type-ahead 输入和每一次 adopt 输入写入前，都先同步把 `isPromptReady` 置为 false 并 re-arm `IdleDetector`，再记录 `submitVerificationStartedAtMs`、进入 `await writeInput()`。在 Codex strong-gate 路径，history 轮询的数秒内若 screen-ready 先到，会被 30 秒的有界 verification lease 拦住；若极快 turn 的 `assistant_final` 先于 history 轮询返回，final edge 也已经有可用的 detector 接收，不会在 await 返回后被一次迟到 reset 擦掉。若 adapter 提供 20 秒 deferred recheck，则正常复核路径内 verification lease 保留到复核结束；状态门控硬上限仍为 30 秒，adapter promise / recheck 卡死时不会永久压制 ready。 +3. 对 adapter/history 已明确验证成功的提交，verification lease 原子切换为 `submitConfirmedAtMs`，在 transcript start 之前提供 20 秒的 hand-off lease。Codex 与 CoCo 只有实际 history 命中才返回 `submitted: true`；CoCo 新安装、history 尚不存在的“信任 Enter”路径仍返回 `undefined`，不会被误当成强证据。只有 Codex allowlist 会把该 lease 投影为 `working`。 +4. CoCo 等 type-ahead 时，下一条提交可能在上一条运行期间一直没有 transcript user event。上一条 `assistant_final` 到达时，会以**本机观察到该事件的时间**刷新下一条 confirmed lease 或 attribution-only lease，从真实 dequeue 边界重新计时；不使用 transcript 自带时间戳，避免外部时钟偏移把 lease 拉长到分钟/小时或让它立即过期。这是共享队列归属与清理语义；非 Codex driver 不因此获得 screen-status strong gate。 +5. `markPromptReady()` 遇到 blocking turn 时拒绝本次 screen-ready 信号，不发送 `prompt_ready`、不清空 in-flight 输入，并保存这次 ready evidence、重置 `IdleDetector`。started turn 由后续 `assistant_final` 或 `turn_aborted` 重驱；verification/confirmed lease 在状态切换或到期时主动重驱。如果期间出现新 PTY 输出，则重新检查当前 screen，而不是盲信旧 prompt。新输出证据使用单调递增的 PTY generation，而不是毫秒时间戳，因此同一毫秒内先后到达的两个 chunk 也不会被误判成“没有新输出”。grace timer 同时绑定创建时的 backend 对象和 `cliSpawnGeneration`，并拒绝 `cliRestartInProgress`；即使 restart 的 async teardown 尚未替换 backend 对象，旧 timer 也不会 prune/replay 或重驱新 generation。 +6. 即时更新、周期 `screen_update` 和截图上传统一调用 `projectRuntimeScreenStatus()`：在 Codex strong-gate 路径，即使 prompt 可见，只要存在 started turn、仍在验证中或仍在 confirmed lease 内，就保持 `working`；`analyzing` 与 usage-limit 的既有优先级保留。周期 tick 还把异步 tmux / observe snapshot 和状态投影绑定成“先完成 snapshot、再读取最新 lifecycle”，避免 tick 以 idle 开始、snapshot 期间新 turn 已启动、最后却晚发旧 idle。 +7. submit 的延迟复核确认失败时，只删除仍未 started 的结构化 mark；不会凭空触发 `fireIdle()`。延迟复核若确认成功，则原子切换到 confirmed lease；若 history 仍未命中、但 PTY / structured transcript / `botmux send` 已证明 CLI 活跃，也转成同样有界的 confirmed lease，而不是留下无法过期的 bare head。其他未确认结果结束 verification、刷新 attribution-only lease，并重驱此前被拒绝的 ready。已经 started 的 turn 不允许被失败清理路径删除。所有 begin / confirm / finish / drop 都匹配 `(turnId, dispatchAttempt)`；延迟复核同时绑定创建时的 CLI generation 与 backend，并在 adapter recheck 的 await 前后重复核对。Claude fallback 的 turn ID 属于 `BridgeTurnQueue`，只有 Codex structured mark 才启用 `CodexBridgeQueue` exact-target 检查，避免把所有 Claude deferred recheck 误判为 stale。 +8. 删除失败的队首 mark 后，会把因 head-of-line fingerprint 不匹配而缓存的近期 user / assistant / abort events 继续保留并重新匹配到下一 turn；即使连续两个队首都失败，第三个成功 turn 也不会在第一次重放时丢失。对 confirmed 或 attribution-only lease 到期、但仍未 started 的**队首**也执行同样清理：若有 started predecessor 则绝不提前清理，等待其 terminal event 刷新下一条 lease；若到期边界恰好到达的是该队首自己的 matching user event，则先允许它转为 started；只有不匹配的 stale head 才会被移除并重放 buffered successor。状态查询保持纯读取；所有 prune 都只发生在 worker 的显式 ingest / timer 边界，并在同一调用栈立刻 drain 可发送 completion，避免重放出 user+terminal 后反而把 completion 留在队列里等“下一条事件”。CoCo adopt 改为走同一 adapter/history 确认路径，但其 screen-status strong gate 仍未开启。 +9. Codex transcript 新增 `event_msg.payload.type=turn_aborted` 解析。中断会关闭 started lifecycle、释放 ready gate，但不会生成假的 assistant 输出；非 adopt 路径中暂时无法归属的 abort 会与其他近期 event 一起缓存，stale head 清理后再重放给正确 successor。 +10. 最新上游 native `/rename` 与 adopt 输入共用 TUI，需要避免管理员命令和用户消息交错。rename 使用 `idle → reserved → writing → sent` 四阶段：排队等待期间进入 `reserved`，从开始写文字到 Enter 的 await 完成前保持 `writing`，只有 Enter 真正落地后才进入 `sent`，且只有 `sent` 状态观察到的新 prompt 才能释放。异常写入也只通过同一 bounded fail-open window 释放，避免把半条命令与后续输入拼接。raw input、bundled follow-up、adopt message 与 rename 共用串行顺序；旧 turn 的快速 final 无法误清尚未发送或尚未完成 Enter 的 rename。 +11. adopt serial queue 是 process-lifetime 对象，因此每个排队任务都捕获 `{ cliSpawnGeneration, backend }`，并在真正 dequeue、transcript mark、raw Enter、bundled follow-up 与 adapter await 返回后核对；`cliRestartInProgress` 和 `rawInputRestartGate` 也属于 replacement-ready fence。尚未开始任何写入的 stale bundle 才能整包回队；若 raw command 写入期间检测到 generation/backend 已变化，则明确提示 ambiguous，并 withholding 依赖它的 follow-up，要求用户核对后整包重试。绝不把 `/cd ...` 等前置命令的 follow-up 单独投进不确定 repo/session 的 replacement CLI。既有 write exception 路径即使可能已部分写入，也仍只记日志并 withholding follow-up；本文不把它误写成已经具备用户可见 reconciliation。 +12. normal non-adopt flush 同样在 `await writeInput()` 前捕获 generation、backend 和 adapter，并把 await 成功/异常后的 generation check 放在 busy probe、session ID 持久化、bridge notify、ready redrive、deferred timer 与用户提示之前。旧 ordinary continuation 不修改全局 bridge queue、也不额外告警，因为 crash carryover 可能已用同一 `turnId + dispatchAttempt=undefined` 在新代重建 mark；durable input 不走 ordinary carryover，只发送对应 exact-attempt ambiguous terminal。 +13. `assistant phase=final_answer` 即使 `output_text` 为空也保留为正常 terminal event。queue 用空 `finalText` 表示“完成但无可见输出”，worker 跳过 `final_output`，但仍发送 exact-attempt `completed turn_terminal`，避免把合法空答复误留成永久 running。 +14. confirmed / attribution-only 的 pre-start lease 到期并删除 stale head 时，该 prune boundary 同时是 durable delivery 的终态边界：每个带 `dispatchAttempt` 的 dropped turn 先发送 exact `ambiguous / structured_start_timeout` terminal，再 drain 因 replay 暴露的 successor completion。这样 attempt N 在约 20 秒有界自愈时就交还 receiver，而不是让 `durableTurnInFlight` 继续阻塞到 daemon 的 15 分钟 watchdog；无 `dispatchAttempt` 的普通 turn 仍只记录日志。 + +### 后续审计发现并关闭的十八个 blocker + +1. **Deferred activity 只 suppress 告警、没有收口 lifecycle**：延迟复核未命中 history、但出现 CLI activity 时，旧补丁会清掉 verification，却把 bare fingerprint 永久留在队首。现在由 `settleDeferredSubmitConfirmation()` 统一完成“决策 + lifecycle 结算”；activity evidence 会建立有界 confirmed lease，其他未确认结果进入有界 attribution-only lease。 +2. **Adopt 并发写会交错 composer 与 Enter**：两个 IPC listener 可同时进入 CoCo/Codex 的 paste → wait → Enter/history 流程，造成后一条内容覆盖前一条 composer、确认结果串 turn。现在普通 adopt message 与 adopt `raw_input` 共用同一个异步串行队列，队列覆盖 transcript mark、完整 adapter write、确认/失败处理；`raw_input` 的 bundled follow-up 不再走 `sendToPty()` 的 fire-and-forget flush，而是在同一个 queue task 内直接 `await writeAdoptMessage()`，直到 adapter write、history verification 与 structured lifecycle 结算全部完成，才允许下一条 adopt message 使用 composer。非 adopt 仍保留原 pending/type-ahead 路径。 +3. **同毫秒 PTY 输出绕过 rejected-ready 失效判断**：只比较 `Date.now()` 时，两个 chunk 可能共享毫秒时间戳，timer 会误信旧 prompt。现在每个真正送入 `IdleDetector` 的 chunk 都递增 generation；重驱只接受仍为当前 generation 的 evidence，spawn/kill 时一并重置。 +4. **Hermes / Pi / MTR 的 `writeInput()` 返回 `undefined` 会留下永久 bare head**：每个 mark 现在从创建起都有 20 秒 attribution-only lease；它不参与 `hasBlockingTurn()`、不制造 false busy，verification 活跃的 30 秒内又不会被提前清理。verification 无确认结束时从本机完成时间刷新该 lease，started predecessor 完成时再刷新下一合法 head。1 秒 bridge ticker 的 `finally` 无条件执行显式 prune → replay → 同栈 emit，因此即使 transcript 尚未 attach、offset 没推进或本 tick 没有新 event，也能自动清掉 silent-write head；同一边界到达的 matching event 始终先 ingest、后 prune。 +5. **迁到最新上游后 native `/rename` 会与 adopt composer 竞态**：最初实现只用布尔 in-flight，排队等待期间的旧 final 可以过早清锁。现在先用 `reserved / sent` 区分“已占位但未写入”和“Enter 已写入”，并让 rename、raw follow-up 与 adopt message 走同一串行写队列；第 11 项继续封住实际 text→Enter await 窗口。 +6. **Codex interrupt 只写 `turn_aborted`，没有 `assistant_final`**：若只认 final，started turn 会永久阻塞。现在把 abort 作为无输出 terminal event，既释放 lifecycle，又不伪造回答。 +7. **把强 started gate 扩到所有 structured CLI 会制造新的永久假忙**：TRAE、CoCo、Hermes、MTR、Pi、Grok、Cursor 的正常/异常/中断终态契约并不都完整。现在强 gate 是 Codex-only allowlist；共享队列的归属、清理和输出能力不等于这些 CLI 已获得完成态保证。 +8. **split-live late attach 一次读到 terminal 时 ready 重驱会延迟**：late attach 直接 ingest 已完成 live turn 时，原实现可能等 20–30 秒 lease timer。现在同一批次观察到 final/abort 后立即 `fireIdle()`，与正常增量 ingest 的终态路径一致。 +9. **最新上游把 Codex 标成 reliable terminal 后，abort 只释放 idle gate 却不结算 durable receipt**：早期 Issue2 语义为 `turn_aborted` 直接删除 started lifecycle，确实不会伪造 assistant 回复，但 `fa9914f2` 的 Codex adapter 已声明 `reliableTurnTerminal=true`，这样会让会议投递一直等不到 exact-attempt `turn_terminal`。现在 abort 以空 `finalText`、`ambiguous` terminal 和安全错误码关闭 queue：worker 的空文本 guard 不发送 `final_output`，随后 terminal loop 仍发送带原 `dispatchAttempt` 的 `turn_terminal`；后继 lease 刷新语义保持不变。 +10. **Attempt N 的延迟 submit callback 会落到 replay N+1**：只按 `turnId` begin / confirm / finish / drop 时，N 到期重启后复用同一 delivery key 的 N+1 会被旧 timer 修改。现在所有 lifecycle 方法与 deferred target check 都匹配 exact `(turnId, dispatchAttempt)`；generation/backend fence 在 recheck await 前后核对，stale callback 不再持久化旧 session ID、redrive、递归设 timer、发 terminal 或告警。 +11. **Rename 的 `sent` 状态在 Enter await 前过早建立**：旧 final 若恰好落在 text→200ms→Enter 窗口，就能释放 gate，让用户输入插进半条命令。现在该窗口显式为 `writing`，仅 Enter promise resolve 后进入 `sent`；异常也通过 bounded fail-open 收口。 +12. **Process-lifetime adopt queue 可能跨 CLI generation 写入**：旧排队任务在 restart 后可能对 `backend=null` 做 mark，或直接写到 replacement CLI。现在 ordinary adopt、raw command 和 bundled follow-up 都使用 generation/backend/restart-ready fence；dequeue 前 stale 可安全重排，检测到 generation gap 的 partial write 会明确 ambiguous。与 generation 无关的既有 raw write exception 仍仅写日志并 withholding follow-up，不在本 blocker 中宣称已补通知。 +13. **空的正常 `final_answer` 被 parser 当成“没有终态”**：`if (!text) continue` 会让无输出但正常完成的 turn 永久阻塞 reliable terminal。现在 parser 保留空 normal final，queue 关闭 completed lifecycle，worker 只抑制空 `final_output` 而不抑制 exact terminal。 +14. **Normal `writeInput()` 的 await continuation 会跨代产生副作用**：旧 backend await 返回时可能已经完成 restart，随后却用新全局 backend 做 busy probe、持久化旧 session ID、notify/redrive 或再设 timer。现在 resolve 与 reject 两条 continuation 都先检查捕获的 generation/backend/adapter。ordinary stale continuation完全交给已有 crash carryover，不删除新代同 ID mark、不诱导手动重复重试；durable stale continuation只发 exact ambiguous terminal。 +15. **Claude deferred recheck 被 Codex exact-target 检查误杀**：Claude 与 structured path 共用 schedule，但 Claude `bridgeTurnId` 只存在 `BridgeTurnQueue`。现在 exact structured target 是显式 opt-in；Claude 仍受通用 generation fence 保护，同时会正常执行 `submitted:false` recheck、告警与 mark cleanup。 +16. **Adopt raw bundle 在前置命令不确定后单独重放 follow-up**：例如 `/cd ` 已向旧 backend 写入后 generation 改变，旧实现会把 dependent prompt 单独 push 到 replacement，可能在错误 repo/session 自动执行。现在只有 raw 写入前 stale 才整包 requeue；raw 已写或 Enter 后的任何 generation gap 都 withholding follow-up、提示核对并要求显式整包重试。 +17. **Structured ready-grace timer 只比较 backend 对象**：`restartCliProcess()` 在 jitter/async teardown 前已经递增 generation、拉起 restart fence，但 backend 仍可能是同一个对象；旧 timer 会在此窗口 prune/replay 旧 queue。现在 timer 捕获 generation + backend，并要求 `!cliRestartInProgress`；stale callback 无 queue/status/redrive 副作用。 +18. **Durable pre-start lease 到期只 prune、不发 terminal**：旧 common helper 会删除 attempt N 并先 emit replay 暴露的 successor，却不发送 N 的 exact terminal；`durableTurnInFlight` 因而可能继续阻塞相邻输入，直到 15 分钟 daemon lease 强制重启。现在七个 prune callsite 仍统一进入同一 helper，但 helper 在 successor emit 前先回调 dropped turns；worker 只对带 `dispatchAttempt` 的项发送 `ambiguous / structured_start_timeout`，并用 exact attempt 与顺序测试锁定 N terminal → N+1 completion。 + +Codex 状态顺序变为： + +```text +message written + → bridge mark（本身不覆盖 screen 状态） + → submit verification begins（bounded；先于 await writeInput） + → adapter/history 明确确认提交 + → verification 原子切换为 bounded confirmed-start lease = working + ↘ 若 adapter 无权威返回：bounded attribution-only lease（不改变 screen 状态） + → matching transcript user，或到期后显式 prune/replay + → transcript user event started + → started + unfinished = working（不再受 lease 到期影响） + → screen prompt / quiet (只作为启发式，状态仍 working) + → assistant_final（正常完成并可输出） + 或 turn_aborted（中断完成，不伪造输出） + → 若有下一条 type-ahead：刷新其 confirmed / attribution-only bounded lease + → 否则 blocking turn = false + → transcript terminal fireIdle + → prompt_ready + idle +``` + +## 前后效果 + +### 修复前 + +- prompt glyph 或静默可以先于 rollout 完成把状态切到 `idle`； +- 即时更新即使被局部抑制，下一次周期 tick 或截图仍可覆盖为 `idle`； +- false idle 会提前清空 in-flight tracker，并可能释放仅应在真实 ready 后投递的输入； +- Dashboard 和飞书卡片使用同一错误状态，无法互相校正。 + +### 修复后 + +- Codex transcript-started unfinished turn，或处于 bounded start lease 的已确认提交存在期间，即时、周期和截图三条路径都不会报告 `idle`； +- screen-ready 不再提前发送 `prompt_ready` 或清空 in-flight 输入； +- `assistant_final` 或 `turn_aborted` 到达后都能正常收敛为 `idle`，不会因为拒绝过一次 false-ready 而丢失真正的 ready edge；abort 只关闭 lifecycle，不生成不存在的 assistant 回复； +- history 轮询 / deferred recheck 尚未返回期间，不会先发布 ready 再迟到确认； +- 极快 turn 即使在 `writeInput` 的 submit verification 返回前已经产生 `assistant_final`，完成 edge 也不会被迟到 reset 吞掉；同一轮 flush 中的每条 type-ahead 输入都会单独 re-arm，不复用上一条已 idle 的 detector 状态； +- type-ahead 的下一条已确认提交不会在上一条刚结束、transcript 尚未 dequeue 的间隙误报 idle； +- 已确认但始终未 started 的 stale 队首在 bounded lease 到期后会被安全清理并重放 buffered successor，不会继续挡住后续真实 turn 的 started lifecycle；started predecessor 和边界上刚到达的 matching user event 都受保护； +- durable stale 队首被 bounded prune 时会先发送 exact `structured_start_timeout` ambiguous terminal、释放 receiver ownership，再发送 replay 后已完成的 successor；不会从约 20 秒自愈退化为 15 分钟 watchdog 等待； +- adapter 返回 `undefined` 或 silent write 时，unconfirmed mark 只在 attribution-only lease 内保留且从不压制真实 prompt；1 秒 bridge tick 即使没有新 event 也会显式清理到期 head，并重放/发送完整 buffered successor； +- deferred activity evidence 不再只 suppress 告警而遗留永久 head；它被结算成有界 lease，确认失败时仍只清理对应 unstarted mark，不伪造 ready； +- adopt 的普通消息、slash command 和 bundled follow-up 不会再跨 IPC 交错 composer / Enter / history confirmation；raw Enter → follow-up 完整 settle → 下一 adopt message 的顺序由同一个 serial queue transaction 保证； +- native `/rename` 在 idle、排队 reserved、text→Enter writing、等待新 prompt 的 sent 四个阶段间保持明确状态边界，不会与用户输入交错；旧 turn 的快速 final 不能提前释放尚未发送或尚未完成 Enter 的 rename； +- CLI restart / replacement 期间，旧 deferred timer、structured grace timer、normal write continuation、adopt/raw queued task 都不会把旧 generation 的 session、terminal、queue prune 或输入副作用落到新 generation；ordinary crash carryover 重建的同 ID mark也不会被旧 continuation 删除； +- raw command 与 dependent follow-up 保持语义原子：只有完全没写入时才整包重排；前置命令可能已落入旧 backend 后，follow-up 会被 withholding，不会在 replacement 的未知 repo/session 自动执行； +- 正常空 `final_answer` 会完成 lifecycle 和 durable receipt，但不会生成假的空回复；Claude 的 deferred submit recheck 仍沿用其自己的 BridgeTurnQueue 归属，不被 Codex queue 的 exact-attempt 检查误杀； +- split-live late attach 若一次读到已完成 turn，会在同一批次立即重驱 ready，不再无谓等待 lease timer； +- rejected-ready 在同一毫秒出现后续 PTY chunk 时也会失效并重新检查当前 screen； +- 周期 snapshot 期间若新 turn 启动，tick 会按 snapshot 完成后的最新 lifecycle 发 `working`，不会晚发 snapshot 前缓存的 `idle`。 + +## 影响面与兼容性 + +- **CLI**：强 started-turn 状态门控目前只对 **Codex** 开启，因为 Codex parser 已覆盖正常 `assistant_final` 与中断 `turn_aborted`。TRAE、CoCo、Hermes、MTR、Pi、Grok 和 Cursor adopt 仍可使用共享 `CodexBridgeQueue` 做 turn 归属、输出与 stale-head 清理，但不声称它们已具备完整终态契约，也不会用 started turn 永久压制 screen-ready。bounded verification/confirmation 只有在 Codex strong-gate 路径才影响状态；本修复对 CoCo adopt 的 adapter/history 串行化属于共享输入安全，不等于为 CoCo 开启 strong gate。 +- **Backend**：不依赖 PTY、tmux、pipe-pane 或 observe backend 的截图节奏；这些 backend 仍产生原有 screen 信号，在 Codex 路径不能越过更权威的 lifecycle / bounded-confirmation 门控。没有可靠 submit 确认或 bridge 尚未 attach 时，未 started mark 不会造成永久假忙或永久 head-of-line 阻塞；1 秒 tick 会清理 attribution lease，ready 重驱会用 PTY generation 检查期间是否出现新输出。 +- **Session**：Codex 覆盖新会话、resume/restore、late attach、type-ahead，以及可验证 adapter 的 adopt 结构化 turn；同时与最新上游 native `/rename` 生命周期协调。worker teardown 仍由既有 `clearPending()` 清理队列和 timer；deferred submit、normal flush continuation 与 process-lifetime adopt queue 都绑定 CLI/backend generation，replacement-ready 还要求 restart gate 已释放。 +- **展示**：不增加状态枚举，也不修改卡片 schema、Dashboard API 或持久化格式；仍使用既有 `working / idle / analyzing / limited`。 +- **输入**：新会话不改变 adapter 的按键、重试或 type-ahead 策略；Codex/CoCo 的“history 已命中但没有 sessionId”返回值从 `undefined` 明确化为 `{ submitted: true }`，只用于 bounded lifecycle hand-off。CoCo adopt 从通用 raw sendText 路径切到其已有的 bracketed-paste + history verification adapter，解决 adopt type-ahead 的同一空窗；所有 adopt 写通过共享 serial queue 保持原子顺序。无法验证的 fresh-install 以及 Hermes/Pi/MTR 返回 `undefined` 的路径使用 attribution-only lease，不冒充权威提交。 +- **Adopt replacement 边界**：当前 worker 不支持 adopt backend 的 in-place replacement；restart/crash 路径不会把退出的 adopted CLI 换成新 generation 后继续 drain。因此“写入前 stale 可整包 park”只描述当前安全兜底，不声称未来 replacement 场景已经证明跨多个 stale raw bundle 的 FIFO。若后续引入 in-place replacement，需要用 sequence ID 或专用 stale-front FIFO 保序，不能用逐项 `unshift()`。 + +## 与 PR #443 的关系 + +2026-07-16 在迁移到 `fa9914f2` 后重新通过 GitHub API 核对：PR #443（`fix(codex): 修复提交重试和告警锚点`）仍为 `OPEN / Draft`，head `f6b4e0fb`，`mergeable=CONFLICTING / mergeStateStatus=DIRTY`、`reviewDecision=REVIEW_REQUIRED`，已有 `build=SUCCESS`。它处理 Codex composer 重驱、提交确认与告警 turnId，未建立 pending-turn 到展示状态的 lifecycle gate，也未覆盖周期 screen tick / screenshot 投影。 + +两者功能正交，但都触及 `worker.ts` 的 submit-failure 附近，后合并者可能需要做文本冲突处理。本修复没有复制或改变 #443 的重试节奏；如果 #443 后续合入，应保留“失败提交清理未 started structured mark”的语义并重新运行双方测试。 + +## 验证结果 + +### 已完成：最新上游迁移、重叠语义审计、focused / TypeScript / build 与 full A/B + +最终工作树已迁到并重新验证于： + +```text +origin/master = fa9914f2023e26833ceba6058e7364e4454c5c8e +branch HEAD = fa9914f2023e26833ceba6058e7364e4454c5c8e +``` + +2026-07-16 收尾时重新执行 `git ls-remote origin refs/heads/master`，远端仍为上述 commit。全程没有 commit、push、开 PR、切换 live runtime 或 drop/pop stash。迁移前新的 include-untracked 快照 `8a14edf5bcdc85079365e36cde9a0241951a67b1`、上一轮快照 `110c00b1353c42da3aaa179b153b144e71de7947`，以及更早的 `8dd44143…`、`1e92e03f…`、`9d85b891…` 均保留。`stash apply --index` 没有修改工作树，随后使用不带 `--index` 的 `apply` 恢复并手工解决冲突，始终未 pop/drop。 + +`1b397073..fa9914f2` 是 1 个大型 merge commit，共 162 个文件（`+48193/-1961`）。它与本修复直接交叉 10 个路径： + +```text +src/adapters/cli/codex.ts +src/services/codex-bridge-queue.ts +src/services/codex-transcript.ts +src/worker.ts +test/claude-turn-terminal-contract.test.ts +test/codex-bridge-queue.test.ts +test/raw-input-followup-atomicity.test.ts +test/session-rename-worker.test.ts +test/worker-durable-expiry-order.test.ts +test/write-input.test.ts +``` + +实际冲突位于 `codex-bridge-queue.ts`、`worker.ts`、`raw-input-followup-atomicity.test.ts` 与 `session-rename-worker.test.ts`。最终合并同时保留： + +- 本修复的 Codex-only lifecycle gate、有界 verification/confirmed/attribution lease、terminal ready 重驱、PTY/backend generation、统一状态投影、串行 adopt/rename 写入； +- 上游的 durable `dispatchAttempt`、terminal status/error、restart fence、VC meeting IM turn origin、独立 TRAE-X rollout parser，以及 adopt/follow-up 的最新 worker 契约； +- submit failure 默认只能删除 exact-attempt 且尚未 started 的 mark；只有权威 failed/ambiguous terminal 才允许删除对应 started attempt，并刷新下一条 pre-start lease； +- `assistant_final` 继续保留上游 terminal metadata，`turn_aborted` 继续作为无输出终态释放 lifecycle； +- 两个上游 source-contract tests（`claude-turn-terminal-contract`、`worker-durable-expiry-order`）按合并后的等价语义更新，而非放宽为无约束匹配。 + +当前 staged code/test patch 共 23 个文件（`+3009/-296`），SHA-256=`6ffb1cd9da6d0850d0cf064348ad656382cd19c3af552985459332c987d250b5`。Focused 覆盖 Issue2 原测试及本次重叠的 queue、worker、durable terminal、TRAE-X、Grok、Codex App、backend crash/suspend 等 31 个文件： + +```text +31 test files: 31 passed / 0 failed +754 tests: 753 passed / 0 failed / 1 skipped +success: true +``` + +Focused JSON 为 `/private/tmp/botmux-false-idle-fa991-focused-prune-fix.json`，SHA-256=`47523d8cecf97a45251f2469688ca48a508fca856812f9c6efadedbe1ab3c909`。关键覆盖包括 Codex-only strong gate、正常/空 final 与 abort、stale-head prune/replay、同栈 completion drain、周期 snapshot 竞态、deferred submit exact-attempt/generation settlement、Claude non-structured recheck、normal await continuation、raw bundle withholding、adopt generation/restart gate、native rename 四阶段、same-backend restart grace timer、split-live late attach、exact `dispatchAttempt` terminal 清理与 successor lease 刷新，以及 abort 无 `final_output` 但仍结算 `ambiguous turn_terminal` 的 durable 契约。durable prune 的 4-file 定向复核为 93/93 PASS，JSON `/private/tmp/botmux-false-idle-prune-terminal-fix.json`，SHA-256=`e135db1f7a6c7244509831498446842bcf513c4a3a99f8e247721dd2ac50e63f`，覆盖 exact attempt 与 N terminal 先于 N+1 completion。 + +Standalone `tsc --noEmit` 通过；完整 build pipeline（TypeScript、scope copy、Dashboard bundle、CLI executable）通过；`git diff --cached --check` 与 `git diff --check` 通过；冲突标记和 unmerged path 均为 0。 + +### Full unit 结果与上游基线 blocker + +最新补丁树已完整执行 full JSON suite： + +```text +560 test files: 559 passed / 1 failed +9202 tests: 9113 passed / 82 failed / 7 skipped +success: false +``` + +唯一失败文件是本次上游新加入、与 Issue2 patch 路径不相交的 `test/vc-meeting-daemon-session.test.ts`；其他 559 个文件全部通过。Full JSON 为 `/private/tmp/botmux-false-idle-fa991-full-prune-fix.json`，SHA-256=`deae9155cb2a50418bc9dd84acf59af78e762d8987092588dbe2fa9ef96633f8`。 + +为避免把真实回归误归因于上游，把最终补丁 full 中该文件的失败集合与纯净上游 standalone 做了 A/B： + +| 工作树 | tests | 失败 title 集合 SHA-256 | 证据 | +| --- | ---: | --- | --- | +| 最终 Issue2 patch full 中的 VC 文件 | 35 pass / 82 fail | `3e85f5af02abaee3e450c07779f9cc5e48be4f54cfd70cd21fc57e6f5b9b4fa5` | `/private/tmp/botmux-false-idle-fa991-full-prune-fix.json`, SHA-256=`deae9155cb2a50418bc9dd84acf59af78e762d8987092588dbe2fa9ef96633f8` | +| 纯净 detached `fa9914f2`，无 Issue2 patch | 35 pass / 82 fail | `3e85f5af02abaee3e450c07779f9cc5e48be4f54cfd70cd21fc57e6f5b9b4fa5` | `/private/tmp/botmux-fa991-clean-vc.json`, SHA-256=`705492eee44bf17faee4378c6cda33471019d69c31136ba320de17cc20ea233a` | + +两边失败数量和完整失败 title 集合完全一致，首批失败都是 VC consumer profile activation 未从“会议多 agent 设置中”收敛为“会议 agents 已启用”，随后产生级联断言失败。因此可以确认：**这 82 个失败在纯净最新上游即可独立复现，不是 Issue2 patch 引入**。但它仍然阻止本分支声称“最新 full suite 全绿”；在上游基线修复或明确调整运行前提之前，验证状态只能是“focused/tsc/build PASS，full 已执行且受可复现上游 blocker 阻断”。 + +### 尚未完成 + +- 目标开发机上的真实 Codex CLI / rollout local integration； +- live daemon 切换或重启； +- 真实飞书卡片与 Dashboard API 的统一时间线 E2E； +- generation-change ambiguity 的新增 `user_notify` 目前沿用 worker 既有硬编码通知风格，尚未补齐多语言 key；与 generation 无关的既有 raw write exception 即使可能发生在部分文本/Enter 已写入后，也仍只写 worker 日志，但 bundled follow-up 会被 withholding； +- PR CI、review、合入、发布和线上效果验证。 + +这些缺口必须在后续证据中单独标注，不能由本地 unit/build 推导为“已上线修复”。 + +## 回滚 + +该变更不改数据格式，可按独立 commit 回滚。回滚后 runner 本身仍可继续工作，但 screen heuristic 会重新获得覆盖 lifecycle 的优先级，本文所述 false idle 风险随之恢复。 + +## 关联文件 + +- `src/services/codex-bridge-queue.ts` +- `src/services/codex-transcript.ts` +- `src/services/structured-bridge-clis.ts` +- `src/services/submit-confirmation.ts` +- `src/services/adopt-input-sequence.ts` +- `src/utils/async-serial-queue.ts` +- `src/utils/runtime-screen-status.ts` +- `src/adapters/cli/codex.ts` +- `src/adapters/cli/coco.ts` +- `src/worker.ts` +- `test/codex-bridge-queue.test.ts` +- `test/codex-transcript.test.ts` +- `test/structured-bridge-clis.test.ts` +- `test/submit-confirmation.test.ts` +- `test/async-serial-queue.test.ts` +- `test/runtime-screen-status.test.ts` +- `test/worker-structured-lifecycle-status.test.ts` +- `test/raw-input-followup-atomicity.test.ts` +- `test/session-rename-worker.test.ts` +- `test/claude-turn-terminal-contract.test.ts` +- `test/worker-durable-expiry-order.test.ts` +- `test/worker-pipe-initial-screen-order.test.ts` +- `test/write-input.test.ts` diff --git a/src/adapters/backend/herdr-backend.ts b/src/adapters/backend/herdr-backend.ts index c976bcb2b..6c7f04bf7 100644 --- a/src/adapters/backend/herdr-backend.ts +++ b/src/adapters/backend/herdr-backend.ts @@ -136,6 +136,7 @@ export class HerdrBackend implements SessionBackend { private lastText = ''; private exited = false; private started = false; + private actuallyReattached = false; private cols = 200; private rows = 50; private agentProbeFailures = 0; @@ -212,7 +213,7 @@ export class HerdrBackend implements SessionBackend { } get isReattach(): boolean { - return this.opts.isReattach ?? false; + return this.actuallyReattached; } spawn(bin: string, args: string[], opts: SpawnOpts): void { @@ -245,6 +246,7 @@ export class HerdrBackend implements SessionBackend { const external = this.opts.externalTarget; if (external) { + this.actuallyReattached = false; this.paneId = external.paneId ?? external.target; } else { // Reuse an existing `botmux` agent ONLY when we're genuinely re-attaching @@ -255,10 +257,12 @@ export class HerdrBackend implements SessionBackend { // row from persisted metadata, and reuse would skip `agent start` so the // new command never ran. killSession() now deletes that metadata, but we // also gate reuse on isReattach so a stale row can never be adopted. - const existing = this.isReattach ? this.getAgent() : undefined; + const existing = this.opts.isReattach ? this.getAgent() : undefined; if (existing) { + this.actuallyReattached = true; this.paneId = existing.pane_id; } else { + this.actuallyReattached = false; const started = jsonCommand(herdrSessionArgs(this.sessionName, [ 'agent', 'start', this.agentName, '--cwd', opts.cwd, @@ -278,25 +282,31 @@ export class HerdrBackend implements SessionBackend { // - Re-attach / external adopt: snapshot the current screen so we only // stream new deltas. Worker.ts explicitly seeds the initial screen // via captureCurrentScreen() in those paths. - this.lastText = (this.isReattach || this.opts.externalTarget) ? this.readRecentAnsi() : ''; + this.lastText = (this.actuallyReattached || this.opts.externalTarget) ? this.readRecentAnsi() : ''; this.startPolling(); this.startStatusWatcher(); } - write(data: string): void { - if (this.exited) return; + write(data: string): boolean { + if (this.exited) return false; const target = this.paneId ?? this.agentName; - runHerdr(herdrSessionArgs(this.sessionName, ['pane', 'send-text', target, data]), { timeout: 5000 }); + return runHerdr( + herdrSessionArgs(this.sessionName, ['pane', 'send-text', target, data]), + { timeout: 5000 }, + ); } - sendText(text: string): void { - this.write(text); + sendText(text: string): boolean { + return this.write(text); } - sendSpecialKeys(...keys: string[]): void { - if (this.exited) return; + sendSpecialKeys(...keys: string[]): boolean { + if (this.exited) return false; const target = this.paneId ?? this.agentName; - runHerdr(herdrSessionArgs(this.sessionName, ['pane', 'send-keys', target, ...keys]), { timeout: 5000 }); + return runHerdr( + herdrSessionArgs(this.sessionName, ['pane', 'send-keys', target, ...keys]), + { timeout: 5000 }, + ); } pasteText(text: string): void { diff --git a/src/adapters/backend/pty-backend.ts b/src/adapters/backend/pty-backend.ts index be38c7569..a5ee86b35 100644 --- a/src/adapters/backend/pty-backend.ts +++ b/src/adapters/backend/pty-backend.ts @@ -45,8 +45,10 @@ export class PtyBackend implements SessionBackend { logger.debug(`[pty] spawned pid=${this.process.pid}`); } - write(data: string): void { - this.process?.write(data); + write(data: string): boolean { + if (!this.process) return false; + this.process.write(data); + return true; } resize(cols: number, rows: number): void { diff --git a/src/adapters/backend/riff-backend.ts b/src/adapters/backend/riff-backend.ts index 4eb793a65..56bd85b5e 100644 --- a/src/adapters/backend/riff-backend.ts +++ b/src/adapters/backend/riff-backend.ts @@ -2,7 +2,12 @@ import { readFileSync } from 'node:fs'; import { join } from 'node:path'; import { execFileSync } from 'node:child_process'; import { createHash } from 'node:crypto'; -import type { SessionBackend, SpawnOpts } from './types.js'; +import type { + SessionBackend, + SessionDestroyResult, + SessionShutdownDetachResult, + SpawnOpts, +} from './types.js'; import { logger } from '../../utils/logger.js'; /** @@ -339,6 +344,24 @@ export class RiffBackend implements SessionBackend { private cancelTimeoutMs = 4_000; private createTimeoutMs = 10_000; private destroyDeadlineMs = 20_000; + /** Exact late/current task whose close cancellation failed. Retained across + * the prepare-close handshake so the daemon can persist a retry handle. */ + private closeFailureTaskId: string | null = null; + private closeFailureError: string | null = null; + private closeLateTaskHandled = false; + private closePrepared = false; + private closeAttempt: symbol | null = null; + private destroyInFlight: Promise | null = null; + private cancelInFlight: Promise | null = null; + private abortInFlight: Promise | null = null; + /** Graceful daemon shutdown is a non-cancelling two-phase detach. It fences + * only writes arriving after prepare; writes already appended to writeChain + * still drain so a late child id can be durably handed to the daemon. */ + private shutdownDetaching = false; + private shutdownDetachPrepared = false; + private shutdownDetachAttempt: symbol | null = null; + private shutdownDetachInFlight: Promise | null = null; + private shutdownDetachAbortInFlight: Promise | null = null; /** Serializes write() → createTask/followUp. Without this, a second message * arriving before the first task-execute HTTP returns would see * currentTaskId === null and create a duplicate task. */ @@ -417,8 +440,16 @@ export class RiffBackend implements SessionBackend { // No actual process to spawn. Task creation happens on first write(). } - write(data: string): void { - if (this.killed || this.closing) return; + write(data: string): boolean { + if (this.killed) return false; + if (this.shutdownDetaching) { + logger.warn('[riff] write rejected while graceful shutdown detach is preparing/prepared'); + return false; + } + if (this.closing) { + logger.warn('[riff] write rejected while explicit close is preparing/prepared'); + return false; + } const { text, attachments } = this.extractAttachments(data); @@ -440,6 +471,7 @@ export class RiffBackend implements SessionBackend { .catch((err) => { logger.warn(`[riff] queued write failed: ${err}`); }); + return true; } resize(_cols: number, _rows: number): void { @@ -467,8 +499,8 @@ export class RiffBackend implements SessionBackend { this.exitCb?.(0, null); } - async destroySession(): Promise { - // /close（及 /restart 的替换路径）必须把远端任务真正取消掉——fire-and-forget + async destroySession(): Promise { + // /close 必须把远端任务真正取消掉——fire-and-forget // 在 worker 紧接 process.exit 时大概率发不出去，已关闭话题的远端 agent 会 // 继续拿着注入的凭证发消息。有界 await + 一次重试，失败也明确留痕。 // @@ -476,36 +508,249 @@ export class RiffBackend implements SessionBackend { // currentTaskId 还是 null/旧值，直接 cancel 会漏掉 late task。先立 closing // 门（拒新写 + 令 in-flight 完成后自取消），再有界等 writeChain 沉降，最后 // cancel 沉降后的 current task。 + if (this.shutdownDetaching) { + return { + ok: false, + ...(this.currentTaskId ? { taskId: this.currentTaskId } : {}), + error: 'shutdown_detach_in_progress', + }; + } + if (this.destroyInFlight) return this.destroyInFlight; + if (this.closePrepared) { + return { + ok: true, + ...(this.currentTaskId ? { taskId: this.currentTaskId } : {}), + }; + } + const attempt = Symbol('riff-close-prepare'); + this.closeAttempt = attempt; this.closing = true; + this.closeFailureTaskId = null; + this.closeFailureError = null; + this.closeLateTaskHandled = false; // 预算层级（单调覆盖，无内层 race——writeChain 本身有界）： // create/follow-up fetch 10s + late cancel 4s×2 = chain 最坏 18s // own cancel 4s×2 = 8s（与 late 情形互斥：closing 分支不登记 current） - // → destroySession 总 deadline 20s → worker close/restart race 22s + // → destroySession 总 deadline 20s → worker close handshake 22s // → daemon SIGTERM backstop 24s / SIGKILL 29s。 // 对 writeChain 只整体 await：单独给它小窗口会在窗口边缘掐掉链内的 // late cancel（create 于 t≈窗口末返回 → cancel 尚 pending → teardown 提前 // resolve → process.exit 掐断取消）。 - const teardown = (async () => { + const teardown = (async (): Promise => { try { await this.writeChain; } catch { /* writeChain never rejects (caught internally) */ } - if (this.currentTaskId && !this.taskDone) { + if (this.closeAttempt !== attempt) { + return { + ok: false, + ...(this.currentTaskId ? { taskId: this.currentTaskId } : {}), + error: 'close_aborted', + }; + } + if (this.closeFailureTaskId) { + return { + ok: false, + taskId: this.closeFailureTaskId, + error: this.closeFailureError ?? 'late_task_cancel_failed', + }; + } + // A task materialized while closing was already cancelled inside the + // writeChain. Do not then cancel its stale parent lineage as if it were + // still the active execution. + if (!this.closeLateTaskHandled && this.currentTaskId && !this.taskDone) { const id = this.currentTaskId; - try { - await this.cancelTask(id); + const cancelled = await this.cancelTaskWithRetry(id, 'close'); + // abortDestroySession invalidates the exact attempt before waiting for + // an already-issued cancellation. A late successful HTTP response must + // not resurrect that aborted generation as a prepared close. + if (this.closeAttempt !== attempt || !this.closing) { + return { + ok: false, + ...(this.currentTaskId ? { taskId: this.currentTaskId } : {}), + error: 'close_aborted', + }; + } + if (cancelled) { logger.info(`[riff] task ${id} cancelled on close`); - } catch (err) { - try { - await this.cancelTask(id); - logger.info(`[riff] task ${id} cancelled on close (retry)`); - } catch (err2) { - logger.warn(`[riff] task-cancel failed on close (task ${id} may keep running remotely): ${err2}`); - } + } else { + return { + ok: false, + taskId: id, + error: this.closeFailureError ?? 'task_cancel_failed', + }; } } + if (this.closeAttempt !== attempt || !this.closing) { + return { + ok: false, + ...(this.currentTaskId ? { taskId: this.currentTaskId } : {}), + error: 'close_aborted', + }; + } + return { + ok: true, + ...(this.currentTaskId ? { taskId: this.currentTaskId } : {}), + }; })(); - await Promise.race([teardown, new Promise((r) => setTimeout(r, this.destroyDeadlineMs))]); - this.kill(); + this.destroyInFlight = Promise.race([ + teardown, + new Promise(resolve => setTimeout(() => resolve({ + ok: false, + ...(this.closeFailureTaskId || this.currentTaskId + ? { taskId: this.closeFailureTaskId ?? this.currentTaskId! } + : {}), + error: 'close_timeout', + }), this.destroyDeadlineMs)), + ]).then(async (result) => { + // Promise.race and the teardown continuation each add a microtask + // boundary. Revalidate the generation immediately before publishing the + // prepared bit so a concurrent abort can never be overwritten. + if (result.ok && (this.closeAttempt !== attempt || !this.closing)) { + result = { + ok: false, + ...(this.currentTaskId ? { taskId: this.currentTaskId } : {}), + error: 'close_aborted', + }; + } + if (result.ok) { + this.closePrepared = true; + } else { + // A failed prepare is not a terminal close. Restore admission so the + // still-active durable owner can accept a follow-up or a close retry. + await this.abortDestroySession(); + } + return result; + }).finally(() => { + this.destroyInFlight = null; + }); + return this.destroyInFlight; + } + + async abortDestroySession(): Promise { + if (this.killed) return; + if (this.abortInFlight) return this.abortInFlight; + this.closeAttempt = null; + this.closePrepared = false; + const pendingCancel = this.cancelInFlight; + this.abortInFlight = (async () => { + // A close timeout can win Promise.race after task-cancel was already + // issued. Reopening admission before that request settles lets a new + // follow-up race a late successful cancellation of its parent. Keep the + // backend fenced until the exact cancellation attempt reaches terminal. + if (pendingCancel) { + try { await pendingCancel; } catch { /* cancel helper returns boolean */ } + } + if (this.killed || this.closeAttempt !== null || this.closePrepared) return; + this.closing = false; + this.closeFailureTaskId = null; + this.closeFailureError = null; + this.closeLateTaskHandled = false; + logger.info('[riff] explicit close aborted; write admission restored'); + })().finally(() => { + this.abortInFlight = null; + }); + return this.abortInFlight; + } + + commitDestroySession(): void { + // The daemon has durably published the closed row. Keep admission fenced + // until the worker immediately detaches/exits. + this.closePrepared = false; + this.closeAttempt = null; + this.closing = true; + } + + async prepareShutdownDetach(): Promise { + if (this.shutdownDetachInFlight) return this.shutdownDetachInFlight; + if (this.shutdownDetachPrepared) { + return { ok: true, taskId: this.currentTaskId }; + } + if (this.killed) { + return { ok: false, taskId: this.currentTaskId, error: 'backend_killed' }; + } + if (this.closing || this.destroyInFlight || this.closePrepared) { + return { ok: false, taskId: this.currentTaskId, error: 'explicit_close_in_progress' }; + } + + const attempt = Symbol('riff-shutdown-detach'); + this.shutdownDetachAttempt = attempt; + this.shutdownDetaching = true; + // Existing SSE delivery is presentation-only. Stop it now, but do not + // cancel the remote task. Any create/follow-up already accepted before the + // fence remains in writeChain and is allowed to materialize below. + this.abortController?.abort(); + + const drain = (async (): Promise => { + try { await this.writeChain; } + catch { /* writeChain catches its own failures */ } + if (this.killed || this.shutdownDetachAttempt !== attempt || !this.shutdownDetaching) { + return { ok: false, taskId: this.currentTaskId, error: 'shutdown_detach_aborted' }; + } + if (this.closing || this.closePrepared) { + return { ok: false, taskId: this.currentTaskId, error: 'explicit_close_in_progress' }; + } + this.shutdownDetachPrepared = true; + logger.info( + `[riff] graceful shutdown detach prepared` + + `${this.currentTaskId ? ` (task ${this.currentTaskId})` : ' (no task lineage)'}`, + ); + return { ok: true, taskId: this.currentTaskId }; + })(); + this.shutdownDetachInFlight = drain.finally(() => { + this.shutdownDetachInFlight = null; + }); + return this.shutdownDetachInFlight; + } + + async abortShutdownDetach(): Promise { + if (this.killed) { + return { ok: false, taskId: this.currentTaskId, error: 'backend_killed' }; + } + if (this.shutdownDetachAbortInFlight) return this.shutdownDetachAbortInFlight; + const pending = this.shutdownDetachInFlight; + const pendingCancel = this.cancelInFlight; + this.shutdownDetachAttempt = null; + this.shutdownDetachPrepared = false; + this.shutdownDetachAbortInFlight = (async (): Promise => { + // Normally shutdown detach never cancels a remote task. Still wait for + // any exact cancellation already issued by an overlapping explicit close + // before reopening admission, otherwise its late result could invalidate + // a newly accepted follow-up. + await Promise.all([ + pending ? pending.catch(() => undefined) : Promise.resolve(), + pendingCancel ? pendingCancel.catch(() => false) : Promise.resolve(), + ]); + if (this.killed) { + return { ok: false, taskId: this.currentTaskId, error: 'backend_killed' }; + } + if (this.closing || this.shutdownDetachAttempt !== null) { + return { + ok: false, + taskId: this.currentTaskId, + error: this.closing ? 'explicit_close_in_progress' : 'new_shutdown_detach_in_progress', + }; + } + this.shutdownDetaching = false; + // prepare stopped SSE before the persistence ACK. If shutdown is + // aborted, reconnect the exact current task so the still-live owner + // resumes normal output and completion tracking. + if (this.currentTaskId && !this.taskDone) { + this.reconnectAttempts = 0; + void this.streamTask(this.currentTaskId); + } + logger.info('[riff] graceful shutdown detach aborted; write admission restored'); + return { ok: true, taskId: this.currentTaskId }; + })().finally(() => { + this.shutdownDetachAbortInFlight = null; + }); + return this.shutdownDetachAbortInFlight; + } + + commitShutdownDetach(): void { + this.shutdownDetachPrepared = false; + this.shutdownDetachAttempt = null; + // Keep admission fenced until the worker exits immediately after commit. + this.shutdownDetaching = true; } getChildPid(): number | null { @@ -760,8 +1005,8 @@ export class RiffBackend implements SessionBackend { * Post-await adoption gate for a freshly created/followed-up task id. * - closing（/close 竞态窗口）：这个 late task 已经没有会话可服务——立即取消 * （有界+一次重试），绝不 stream/登记，防远端 orphan； - * - killed（detach：重启/休眠）：登记 id 让 daemon 持久化血缘，但不 stream - * （任务合法续跑，重启后 follow-up 接上）； + * - killed / shutdownDetaching（detach）：登记 id 让 daemon 持久化血缘， + * 但不 stream（任务合法续跑，重启后 follow-up 接上）； * - 正常：登记 + 由调用方启动 stream。 */ private async adoptLateTask(taskId: string): Promise { @@ -769,20 +1014,21 @@ export class RiffBackend implements SessionBackend { // 在 writeChain 内 await——destroySession 等 writeChain 沉降时就能把这次 // 取消一起等到（void 触发会在 worker exit 时被掐断）。 logger.info(`[riff] task ${taskId} created during close — cancelling late task`); - try { - await this.cancelTask(taskId); - } catch { - try { - await this.cancelTask(taskId); - } catch (err) { - logger.warn(`[riff] late-task cancel failed (task ${taskId} may keep running remotely): ${err}`); - } - } + this.closeLateTaskHandled = true; + // Preserve the exact newest lineage even when cancellation succeeds. + // If the daemon cannot durably commit the close and sends abort, the + // next follow-up must continue from this child rather than its stale + // parent. Publishing before the cancel also makes a failed cancel + // retryable by the daemon. + this.currentTaskId = taskId; + this.taskIdCb?.(taskId); + const cancelled = await this.cancelTaskWithRetry(taskId, 'late-task close'); + if (!cancelled) this.taskDone = false; return false; } this.currentTaskId = taskId; this.taskIdCb?.(taskId); - if (this.killed) return false; + if (this.killed || this.shutdownDetaching) return false; return true; } @@ -809,6 +1055,32 @@ export class RiffBackend implements SessionBackend { if (!resp.ok) throw new Error(`task-cancel HTTP ${resp.status}`); } + private async cancelTaskWithRetry(taskId: string, context: string): Promise { + const operation = (async (): Promise => { + try { + await this.cancelTask(taskId); + return true; + } catch { + try { + await this.cancelTask(taskId); + logger.info(`[riff] task ${taskId} cancelled on ${context} (retry)`); + return true; + } catch (err) { + this.closeFailureTaskId = taskId; + this.closeFailureError = err instanceof Error ? err.message : String(err); + logger.warn(`[riff] ${context} cancel failed (task ${taskId} may keep running remotely): ${err}`); + return false; + } + } + })(); + this.cancelInFlight = operation; + try { + return await operation; + } finally { + if (this.cancelInFlight === operation) this.cancelInFlight = null; + } + } + private async streamTask(taskId: string): Promise { const url = `${this.config.baseUrl}/api2/task-stream?id=${encodeURIComponent(taskId)}`; const headers: Record = {}; diff --git a/src/adapters/backend/sandbox.ts b/src/adapters/backend/sandbox.ts index 23fe03e08..3e7948c6a 100644 --- a/src/adapters/backend/sandbox.ts +++ b/src/adapters/backend/sandbox.ts @@ -1395,6 +1395,7 @@ export function buildRelayHostEnv( const env: NodeJS.ProcessEnv = { ...baseEnv }; delete env.BOTMUX_SEND_RELAY; delete env.BOTMUX_CARD_PREPARED_CONTENT_FILE; + delete env.BOTMUX_HOST_RELAY_REQUIRES_CODEX_APP_LEDGER; if (preparedContentFile) { env.BOTMUX_CARD_LOCAL_LINK_MODE = 'disabled'; env.BOTMUX_CARD_PREPARED_CONTENT_FILE = preparedContentFile; @@ -1416,7 +1417,19 @@ export function startOutboxWatcher( * absent the relay still runs, but carries NO durable origin — a missing * hook must never let the sandbox promote its own origin fields. */ authorize?: (claim: { capability?: string }) => - | { ok: true; origin: { turnId?: string; dispatchAttempt?: number } } + | { + ok: true; + origin: { + turnId?: string; + dispatchAttempt?: number; + /** The worker matched an unsettled Codex App ledger entry. The + * host child must still find that exact entry before any provider + * side effect; terminal settlement/revocation between authorize + * and re-exec therefore fails closed instead of degrading to an + * ordinary mutable-session send. */ + requiresCodexAppLedger?: boolean; + }; + } | { ok: false; error: string }; cliPath?: string; } = {}, @@ -1540,6 +1553,11 @@ export function startOutboxWatcher( if (trustedOrigin?.dispatchAttempt !== undefined) { requestEnv.BOTMUX_DISPATCH_ATTEMPT = String(trustedOrigin.dispatchAttempt); } + if (trustedOrigin?.requiresCodexAppLedger) { + requestEnv.BOTMUX_HOST_RELAY_REQUIRES_CODEX_APP_LEDGER = '1'; + } else { + delete requestEnv.BOTMUX_HOST_RELAY_REQUIRES_CODEX_APP_LEDGER; + } const child = spawn(process.execPath, [cli, 'send', ...hostArgs], { env: requestEnv }); let out = '', err = ''; child.stdout.on('data', d => { out += d; }); diff --git a/src/adapters/backend/tmux-backend.ts b/src/adapters/backend/tmux-backend.ts index 1580e23d6..0b158615d 100644 --- a/src/adapters/backend/tmux-backend.ts +++ b/src/adapters/backend/tmux-backend.ts @@ -370,8 +370,10 @@ export class TmuxBackend implements SessionBackend { * file's cwd field so a recycled PID can't mislead the resolver. */ cliCwd?: string; - write(data: string): void { - this.process?.write(data); + write(data: string): boolean { + if (!this.process) return false; + this.process.write(data); + return true; } /** diff --git a/src/adapters/backend/tmux-pipe-backend.ts b/src/adapters/backend/tmux-pipe-backend.ts index d4e697062..30f35d310 100644 --- a/src/adapters/backend/tmux-pipe-backend.ts +++ b/src/adapters/backend/tmux-pipe-backend.ts @@ -247,9 +247,9 @@ export class TmuxPipeBackend implements SessionBackend { } } - write(data: string): void { + write(data: string): boolean { // No PTY to write to — interpret as a literal send-keys. - this.sendText(data); + return this.sendText(data); } sendText(text: string): boolean { diff --git a/src/adapters/backend/types.ts b/src/adapters/backend/types.ts index 0c6e8298b..a41c74e31 100644 --- a/src/adapters/backend/types.ts +++ b/src/adapters/backend/types.ts @@ -41,7 +41,9 @@ export interface SpawnOpts { export interface SessionBackend { spawn(bin: string, args: string[], opts: SpawnOpts): void; - write(data: string): void; + /** Returns false only when the backend can prove the write was not accepted. + * Legacy implementations may return void on success. */ + write(data: string): void | boolean; resize(cols: number, rows: number): void; onData(cb: (data: string) => void): void; onExit(cb: (code: number | null, signal: string | null) => void): void; @@ -75,8 +77,42 @@ export interface SessionBackend { * so the follow-up lineage survives daemon restarts. `null` clears the * persisted lineage (follow-up failed → next message starts fresh). */ onTaskId?(cb: (taskId: string | null) => void): void; - /** Async-capable teardown: riff awaits the remote task-cancel here. */ - destroySession?(): void | Promise; + /** Async-capable teardown: Riff returns a confirmed cancellation result so + * daemon-driven explicit close can fence late create/follow-up races before + * publishing the durable closed row. Local backends remain synchronous. */ + destroySession?(): void | Promise; + /** Roll back a successful remote prepare when the daemon could not commit + * the durable closed row. The backend must restore write admission without + * discarding the last task lineage. */ + abortDestroySession?(): void | Promise; + /** Finalize a successful prepare after the durable row is closed. */ + commitDestroySession?(): void; + /** Graceful daemon shutdown for a remote-task backend. Fence only NEW + * writes, drain every write accepted before the fence, and return the exact + * final lineage without cancelling it. The daemon persists that lineage + * before telling the worker it may exit. */ + prepareShutdownDetach?(): Promise; + /** Restore admission/streaming when the daemon cannot complete a prepared + * shutdown detach (for example, lineage persistence failed). */ + abortShutdownDetach?(): SessionShutdownDetachResult | Promise; + /** Finalize a shutdown detach after exact lineage persistence has been + * acknowledged by the daemon. Shutdown refusal never cancels remote work. */ + commitShutdownDetach?(): void; +} + +export interface SessionDestroyResult { + ok: boolean; + /** Exact remote task that failed cancellation and must remain retryable. */ + taskId?: string; + error?: string; +} + +export interface SessionShutdownDetachResult { + ok: boolean; + /** Exact final lineage after all pre-fence writes have drained. `null` is + * authoritative and clears any stale durable parent. */ + taskId: string | null; + error?: string; } /** diff --git a/src/adapters/backend/zellij-backend.ts b/src/adapters/backend/zellij-backend.ts index b8ca9f272..899781dd2 100644 --- a/src/adapters/backend/zellij-backend.ts +++ b/src/adapters/backend/zellij-backend.ts @@ -201,20 +201,24 @@ export class ZellijBackend implements SessionBackend { // are forwarded verbatim to the focused pane — so every input path collapses // to pty.write(), exactly like TmuxBackend.write(). - write(data: string): void { - this.process?.write(data); + write(data: string): boolean { + if (!this.process) return false; + this.process.write(data); + return true; } /** Literal text, no Enter. */ - sendText(text: string): void { - this.process?.write(text); + sendText(text: string): boolean { + return this.write(text); } /** Special keys by tmux-style name (Enter, Escape, C-c, M-Enter, …). */ - sendSpecialKeys(...keys: string[]): void { + sendSpecialKeys(...keys: string[]): boolean { + if (!this.process) return false; for (const key of keys) { - this.process?.write(tmuxKeyToBytes(key)); + this.process.write(tmuxKeyToBytes(key)); } + return true; } /** Bracketed paste: wrap with \e[200~ … \e[201~ so TUIs (CoCo/Ink/Codex) diff --git a/src/adapters/backend/zellij-observe-backend.ts b/src/adapters/backend/zellij-observe-backend.ts index f4eb8e304..2eba19cc5 100644 --- a/src/adapters/backend/zellij-observe-backend.ts +++ b/src/adapters/backend/zellij-observe-backend.ts @@ -182,19 +182,22 @@ export class ZellijObserveBackend implements ObserveBackend { // ── Input: targeted `action` calls (focus-neutral, non-invasive) ── - write(data: string): void { - this.writeBytes(data); + write(data: string): boolean { + return this.writeBytes(data); } /** Literal text via write-chars (preserves UTF-8). */ - sendText(text: string): void { - if (!text) return; - this.action(['write-chars', '--pane-id', this.paneId, '--', text]); + sendText(text: string): boolean { + if (!text) return true; + return this.action(['write-chars', '--pane-id', this.paneId, '--', text]) !== null; } /** Special keys by tmux-style name → raw bytes → `action write`. */ - sendSpecialKeys(...keys: string[]): void { - for (const key of keys) this.writeBytes(tmuxKeyToBytes(key)); + sendSpecialKeys(...keys: string[]): boolean { + for (const key of keys) { + if (!this.writeBytes(tmuxKeyToBytes(key))) return false; + } + return true; } /** Bracketed paste — wrap so TUIs detect the boundary (mirrors paste-buffer -p). */ @@ -207,14 +210,15 @@ export class ZellijObserveBackend implements ObserveBackend { /** Write arbitrary bytes via `action write …` (handles control/escape). * Chunked so a large web-terminal paste can't blow the argv limit; zellij * serialises the writes in arrival order. */ - private writeBytes(data: string): void { - if (!data) return; + private writeBytes(data: string): boolean { + if (!data) return true; const buf = Buffer.from(data, 'utf-8'); const CHUNK = 512; for (let i = 0; i < buf.length; i += CHUNK) { const bytes = Array.from(buf.subarray(i, i + CHUNK), b => String(b)); - this.action(['write', '--pane-id', this.paneId, ...bytes]); + if (this.action(['write', '--pane-id', this.paneId, ...bytes]) === null) return false; } + return true; } /** Resize is a NO-OP in observe mode — the pane size is the user's, and diff --git a/src/adapters/cli/coco.ts b/src/adapters/cli/coco.ts index 2fa2602a4..98565af49 100644 --- a/src/adapters/cli/coco.ts +++ b/src/adapters/cli/coco.ts @@ -230,7 +230,7 @@ export function createCocoAdapter(pathOverride?: string): CliAdapter { // silently mask a real submit failure on a new install. if (!existsSync(HISTORY_PATH) && baseByte === 0) { if (await waitForHistoryAppend(HISTORY_PATH, baseByte, prefix, 1200)) { - return undefined; + return { submitted: true }; } if (!existsSync(HISTORY_PATH)) { return undefined; @@ -242,12 +242,12 @@ export function createCocoAdapter(pathOverride?: string): CliAdapter { for (let attempt = 0; attempt < 3; attempt++) { if (await waitForHistoryAppend(HISTORY_PATH, baseByte, prefix, 800)) { - return undefined; + return { submitted: true }; } if (!trySendEnter()) return { submitted: false }; } if (await waitForHistoryAppend(HISTORY_PATH, baseByte, prefix, 800)) { - return undefined; + return { submitted: true }; } // In-band budget exhausted. Hand the worker a recheck closure: a slow // CoCo (cold start, large initial prompt, heavy hooks) may still diff --git a/src/adapters/cli/codex-app.ts b/src/adapters/cli/codex-app.ts index dead8db3c..b198e173d 100644 --- a/src/adapters/cli/codex-app.ts +++ b/src/adapters/cli/codex-app.ts @@ -6,6 +6,14 @@ import type { CliAdapter, PtyHandle } from './types.js'; import { writeRunnerInput } from './runner-input.js'; function runnerPath(): string { + // Source-level worker integration tests execute through tsx and need the + // matching source runner rather than a possibly absent/stale ignored dist + // tree. Keep the override strictly test-scoped so production launch + // resolution remains canonical and cannot be redirected by ambient env. + const testOverride = process.env.NODE_ENV === 'test' + ? process.env.BOTMUX_TEST_CODEX_APP_RUNNER_PATH + : undefined; + if (testOverride) return resolve(testOverride); const here = dirname(fileURLToPath(import.meta.url)); const compiledSibling = resolve(here, '..', '..', 'codex-app-runner.js'); if (existsSync(compiledSibling)) return compiledSibling; diff --git a/src/adapters/cli/codex.ts b/src/adapters/cli/codex.ts index df05fb1d4..424926659 100644 --- a/src/adapters/cli/codex.ts +++ b/src/adapters/cli/codex.ts @@ -278,7 +278,7 @@ export function createCodexAdapter(pathOverride?: string): CliAdapter { if (match.found) { return match.cliSessionId ? { submitted: true, cliSessionId: match.cliSessionId } - : undefined; + : { submitted: true }; } if (!trySendEnter()) return { submitted: false }; } @@ -286,7 +286,7 @@ export function createCodexAdapter(pathOverride?: string): CliAdapter { if (match.found) { return match.cliSessionId ? { submitted: true, cliSessionId: match.cliSessionId } - : undefined; + : { submitted: true }; } // In-band budget exhausted. Hand the worker a recheck closure: a // slow-startup Codex (or one whose first turn is delayed by a heavy diff --git a/src/adapters/cli/read-isolation.ts b/src/adapters/cli/read-isolation.ts index 45d859f8f..64a013562 100644 --- a/src/adapters/cli/read-isolation.ts +++ b/src/adapters/cli/read-isolation.ts @@ -20,7 +20,13 @@ * plaintext). See the design doc for the two-layer rationale. */ -import { managedOriginCapabilityPath } from '../../core/managed-origin-capability.js'; +import { createHash } from 'node:crypto'; +import { + managedOriginAttestationDirectory, + managedOriginCapabilityPath, + managedOriginIsolationSentinelPath, + managedOriginRootLocatorPath, +} from '../../core/managed-origin-capability.js'; import { DEVICE_AUTHORITY_DIRECTORY, DEVICE_CREDENTIAL_FILE, @@ -107,6 +113,9 @@ export function buildCliExecutableReadCarveOuts(input: { export interface V2IsolationContext { /** The bot user's home directory. */ homeDir: string; + /** OS account home from getpwuid/userInfo, never the mutable HOME env. Used + * only for the fixed kernel-observable isolation sentinel. */ + osUserHomeDir?: string; /** BOTMUX_HOME root (e.g. `~/.botmux`). NOT denied wholesale — the botmux CLI the * agent runs (`botmux send`/`list`/`status`) needs broad read access to it (config, * daemon registry, pm2, session store). Only its cross-bot-SENSITIVE parts are denied. */ @@ -123,6 +132,8 @@ export interface V2IsolationContext { /** Current botmux session. Used only to carve back the exact rotating IPC * capability file from the otherwise denied read-isolation runtime dir. */ currentSessionId: string; + /** Unguessable authority channel bound to this exact persistent pane/profile. */ + currentOriginChannelId?: string; /** Per-bot extra deny paths (BotConfig.readDenyExtraPaths). */ extraDenyPaths?: string[]; } @@ -299,6 +310,9 @@ export function buildV2DenyPaths(ctx: V2IsolationContext): string[] { `${h}/.docker/config.json`, `${h}/.kube`, `${h}/Library/Keychains`, + // Fixed host-readable sentinel used by cmdSend to ask the kernel whether + // this process is inside Seatbelt. Never carve it back in. + managedOriginIsolationSentinelPath((ctx.osUserHomeDir ?? h).replace(/\/+$/, '')), // ── botmux SENSITIVE (surgical — leave config/registry/pm2/data structure readable) ── `${bh}/bots.json`, // ALL bots' secrets `${bh}/logs`, // daemon logs (cross-bot content) @@ -400,7 +414,8 @@ export function buildV2DenyRegexes(ctx: V2IsolationContext): string[] { const bh = ctx.botmuxHome.replace(/\/+$/, ''); const h = ctx.homeDir.replace(/\/+$/, ''); const defaultBh = (ctx.defaultBotmuxHome ?? `${h}/.botmux`).replace(/\/+$/, ''); - const botmuxRoots = dedupe([defaultBh, bh]); + const osBh = `${(ctx.osUserHomeDir ?? h).replace(/\/+$/, '')}/.botmux`; + const botmuxRoots = dedupe([defaultBh, bh, osBh]); return dedupe([ `^${escapeForRegex(sd)}/sessions-[^/]+\\.json$`, // Any `bots.json.` sidecar (backups/temp) — trailing dot so it matches @@ -450,6 +465,7 @@ export function buildReadIsolationProtectedWriteRules( const dashboardRoots = dedupe([ defaultBh, bh, + `${(ctx.osUserHomeDir ?? h).replace(/\/+$/, '')}/.botmux`, ...(extraRoots.dashboardRoots ?? []).map(root => root.replace(/\/+$/, '')), ]); const sessionDataDirs = dedupe([ @@ -458,6 +474,7 @@ export function buildReadIsolationProtectedWriteRules( ]); return { denyWritePaths: dedupe([ + managedOriginIsolationSentinelPath((ctx.osUserHomeDir ?? h).replace(/\/+$/, '')), ...dashboardRoots.flatMap(root => [ `${root}/.dashboard-secret`, `${root}/.dashboard-token`, @@ -520,7 +537,21 @@ export function buildV2CarveOuts(ctx: V2IsolationContext): { `${sd}/attachments/${self}`, // Parent read-isolation/ stays wholesale-denied; only this session's // rotating origin capability is visible to the confined CLI. - managedOriginCapabilityPath(sd, ctx.currentSessionId), + ...(ctx.currentOriginChannelId + ? [managedOriginCapabilityPath(sd, ctx.currentSessionId, ctx.currentOriginChannelId)] + : []), + // The CLI challenges the owning daemon with a random nonce and accepts + // only the corresponding host-written proof from this read-only carve. + // A fake loopback listener cannot manufacture files here. + ...(ctx.currentOriginChannelId + ? [managedOriginAttestationDirectory(sd, ctx.currentSessionId, ctx.currentOriginChannelId)] + : []), + // Fixed-home root locator binds mutable SESSION_DATA_DIR back to the + // daemon's true root. The legacy dashboard-secret regex denies the class; + // only this current-session leaf is reopened for reading. + ...(ctx.currentOriginChannelId + ? [managedOriginRootLocatorPath(ctx.osUserHomeDir ?? h, ctx.currentSessionId)] + : []), ], // file-read-metadata on the wholesale-denied parents so the CLI/skill can realpath() // through them WITHOUT `ls` (enumeration) leaking. @@ -889,18 +920,88 @@ function normalizeIsolationCapabilities( return ALL_ISOLATION_CAPABILITIES.filter(capability => requested.has(capability)); } +export interface IsolationPanePolicyInput { + readIsolation: boolean; + writeSandbox: boolean; + readDenyExtraPaths?: readonly string[]; + writeAllowExtraPaths?: readonly string[]; + workingDir?: string; + homeDir?: string; + osUserHomeDir?: string; + botmuxHome?: string; + sessionDataDir?: string; + currentAppId?: string; + cliId?: string; + resolvedBin?: string; +} + +/** Deterministic fingerprint of effective Darwin Seatbelt inputs that can + * change between worker forks while the pane survives. Arrays are normalized + * as sets because rule order does not change their final deny semantics. */ +export function isolationPanePolicyDigest(input: IsolationPanePolicyInput): string { + const normalizedExtra = (input.readDenyExtraPaths ?? []) + .map(normalizeIsolationPath) + .filter((value): value is string => !!value) + .sort(); + const normalizedWriteExtra = (input.writeAllowExtraPaths ?? []) + .map(normalizeIsolationPath) + .filter((value): value is string => !!value) + .sort(); + return createHash('sha256').update(JSON.stringify({ + domain: 'botmux.darwin-seatbelt-policy.v7', + readIsolation: input.readIsolation, + writeSandbox: input.writeSandbox, + readDenyExtraPaths: normalizedExtra, + writeAllowExtraPaths: normalizedWriteExtra, + workingDir: input.workingDir ?? '', + homeDir: input.homeDir ?? '', + osUserHomeDir: input.osUserHomeDir ?? '', + botmuxHome: input.botmuxHome ?? '', + sessionDataDir: input.sessionDataDir ?? '', + currentAppId: input.currentAppId ?? '', + cliId: input.cliId ?? '', + resolvedBin: input.resolvedBin ?? '', + })).digest('hex'); +} + /** Versioned marker written beside a freshly spawned persistent sandbox. */ export function isolationPaneMarkerContent( bootId: string, capabilities: readonly IsolationCapability[], + policy?: { + originChannelId: string; + readIsolation: boolean; + writeSandbox: boolean; + policyDigest: string; + }, ): string { + if (policy + && (!/^[a-f0-9]{64}$/.test(policy.originChannelId) + || !/^[a-f0-9]{64}$/.test(policy.policyDigest))) { + throw new Error('invalid Darwin isolation marker policy'); + } return JSON.stringify({ version: ISOLATION_PANE_MARKER_VERSION, bootId, capabilities: normalizeIsolationCapabilities(capabilities), + ...(policy ?? {}), }); } +export function isolatedPaneOriginChannel( + markerContent: string | null | undefined, +): string | undefined { + try { + const parsed = JSON.parse(markerContent ?? '') as { originChannelId?: unknown }; + return typeof parsed.originChannelId === 'string' + && /^[a-f0-9]{64}$/.test(parsed.originChannelId) + ? parsed.originChannelId + : undefined; + } catch { + return undefined; + } +} + /** * Decide whether a live persistent pane (tmux/zellij/herdr) may be reattached for * an isolated bot. Isolation is injected at CLI *spawn* time (the Seatbelt @@ -916,25 +1017,61 @@ export function isolationPaneMarkerContent( */ export function isolatedPaneReattachSafe( markerContent: string | null | undefined, - requiredCapabilities: readonly IsolationCapability[] = [], + expected: readonly IsolationCapability[] | { + requiredCapabilities: readonly IsolationCapability[]; + exactCapabilities?: boolean; + readIsolation?: boolean; + writeSandbox?: boolean; + requireOriginChannel?: boolean; + policyDigest?: string; + } = [], ): boolean { try { const parsed = JSON.parse(markerContent ?? '') as { version?: unknown; bootId?: unknown; capabilities?: unknown; + readIsolation?: unknown; + writeSandbox?: unknown; + originChannelId?: unknown; + policyDigest?: unknown; }; - if (!Array.isArray(parsed.capabilities) + if (parsed.version !== ISOLATION_PANE_MARKER_VERSION + || typeof parsed.bootId !== 'string' + || parsed.bootId.trim().length === 0 + || !Array.isArray(parsed.capabilities) || parsed.capabilities.some(capability => typeof capability !== 'string' || !ALL_ISOLATION_CAPABILITIES.includes(capability as IsolationCapability))) { return false; } + const expectedPolicy = Array.isArray(expected) + ? undefined + : expected as { + requiredCapabilities: readonly IsolationCapability[]; + exactCapabilities?: boolean; + readIsolation?: boolean; + writeSandbox?: boolean; + requireOriginChannel?: boolean; + policyDigest?: string; + }; + const requiredCapabilities: readonly IsolationCapability[] = expectedPolicy + ? expectedPolicy.requiredCapabilities + : expected as readonly IsolationCapability[]; const actual = new Set(parsed.capabilities as IsolationCapability[]); - return parsed.version === ISOLATION_PANE_MARKER_VERSION - && typeof parsed.bootId === 'string' - && parsed.bootId.trim().length > 0 - && requiredCapabilities.every(capability => actual.has(capability)); + if (!requiredCapabilities.every(capability => actual.has(capability))) return false; + if (expectedPolicy?.exactCapabilities + && actual.size !== new Set(requiredCapabilities).size) return false; + if (!expectedPolicy) return true; + if (expectedPolicy.readIsolation !== undefined + && parsed.readIsolation !== expectedPolicy.readIsolation) return false; + if (expectedPolicy.writeSandbox !== undefined + && parsed.writeSandbox !== expectedPolicy.writeSandbox) return false; + if (expectedPolicy.policyDigest !== undefined + && parsed.policyDigest !== expectedPolicy.policyDigest) return false; + return !expectedPolicy.requireOriginChannel + || (typeof parsed.originChannelId === 'string' + && /^[a-f0-9]{64}$/.test(parsed.originChannelId)); } catch { return false; } diff --git a/src/adapters/cli/runner-input.ts b/src/adapters/cli/runner-input.ts index 37b03520e..1b06185bf 100644 --- a/src/adapters/cli/runner-input.ts +++ b/src/adapters/cli/runner-input.ts @@ -1,4 +1,4 @@ -import type { PtyHandle } from './types.js'; +import type { PtyHandle, RunnerSubmissionDisposition } from './types.js'; import type { CodexAppTurnInput } from '../../types.js'; import { delay } from '../../utils/timing.js'; @@ -17,9 +17,9 @@ import { delay } from '../../utils/timing.js'; * webhook whose full MR JSON is embedded — ~16-21KB after base64) that single * injection overruns the pane pty's input buffer (N_TTY's ~4KB read buffer): * tmux's write blocks until the reader drains, which takes longer than - * execFileSync's 5s timeout, so the send-keys is killed and the keystroke is - * silently dropped — yet the old writeInput still reported `submitted: true`, - * wedging the session "busy" forever. (Compare claude-code, which throttles + * execFileSync's 5s timeout, so Botmux can no longer prove whether the + * keystroke landed — yet the old writeInput still reported `submitted: true`, + * potentially wedging the session "busy" forever. (Compare claude-code, which throttles * its send-keys for exactly this reason; codex-app/mira were the only naive * single-shot writers.) * @@ -61,11 +61,10 @@ export function chunkAscii(line: string, maxBytes: number): string[] { /** * Write one control line to a runner adapter's stdin, chunked + throttled. * - * Returns `{ submitted: false }` when any chunk (or the final Enter) fails to - * write — the tmux backend's send methods return `false` on a dropped-but-pane- - * alive keystroke, so a genuine drop is now surfaced to the worker (which raises - * a submit-failure notice + recheck so the user can retry) instead of being - * swallowed as a false success. + * Returns `{ submitted: false }` when any chunk (or the final Enter) cannot be + * confirmed. A tmux timeout is ambiguous: bytes may still have reached the + * pane. `submissionDisposition` therefore tells the worker whether the new + * frame was provably untouched or the runner generation must be fenced. * * Buffer-hygiene contract (the runner only clears its stdin buffer on a newline, * see handleInput in codex-app-runner.ts / mira-runner.ts — a half-written @@ -74,9 +73,10 @@ export function chunkAscii(line: string, maxBytes: number): string[] { * - Pre-flush: emit one Enter before writing, terminating any partial line a * prior failed write may have left behind (runner discards the fragment as * bad input; an empty buffer just ignores the blank line). - * - On a dropped chunk: emit a flush Enter so the partial we just wrote can't - * merge with the next message, then report non-submission (submit-failure). - * - Submit Enter is retried — a single dropped Enter would otherwise leave a + * - On an unconfirmed chunk: attempt a flush Enter so a partial frame is less + * likely to merge with the next message, then report an ambiguous dirty + * generation; the flush cannot make delivery proof retroactive. + * - Submit Enter is retried — a single unconfirmed Enter could otherwise leave a * COMPLETE but unsubmitted line in the buffer. */ export async function writeRunnerInput( @@ -84,18 +84,24 @@ export async function writeRunnerInput( markerPrefix: string, content: string, codexAppInput?: CodexAppTurnInput, -): Promise<{ submitted: boolean }> { +): Promise<{ submitted: boolean; submissionDisposition: RunnerSubmissionDisposition }> { const line = `${markerPrefix}${encodeRunnerInput(content, codexAppInput)}`; // Non-tmux fallback (raw PTY): a single write is fine — there's no send-keys // process to time out, and the PTY write isn't bounded the same way. if (!pty.sendText || !pty.sendSpecialKeys) { try { - pty.write(line + '\r'); + if (pty.write(line + '\r') === false) { + // SessionBackend.write(false) is a rejection, but its contract does not + // prove whether a lower layer accepted a prefix before reporting it. + return { submitted: false, submissionDisposition: 'dirty_unknown' }; + } } catch { - return { submitted: false }; + // A throwing PTY write does not prove whether the kernel accepted a + // prefix (or the complete line) before surfacing the error. + return { submitted: false, submissionDisposition: 'dirty_unknown' }; } - return { submitted: true }; + return { submitted: true, submissionDisposition: 'submitted' }; } const sendText = pty.sendText.bind(pty); @@ -117,24 +123,51 @@ export async function writeRunnerInput( // the user can retry); we never touch the buffer with a half write. (Idempotent // on the happy path: the previous message's // submit Enter already emptied the buffer, so this enqueues an ignored blank.) - if (!sendEnterWithRetry()) return { submitted: false }; + try { + if (!sendEnterWithRetry()) { + return { submitted: false, submissionDisposition: 'untouched' }; + } + } catch { + // The backend threw while attempting the pre-flush. No new frame bytes + // were intentionally written, but the Enter itself may have landed, so the + // runner generation is not proven clean enough for same-generation reuse. + return { submitted: false, submissionDisposition: 'dirty_unknown' }; + } const chunks = chunkAscii(line, RUNNER_INPUT_CHUNK_BYTES); for (let i = 0; i < chunks.length; i++) { - if (sendText(chunks[i]) === false) { + let chunkWritten: void | boolean; + try { + chunkWritten = sendText(chunks[i]); + } catch { + return { submitted: false, submissionDisposition: 'dirty_unknown' }; + } + if (chunkWritten === false) { // The chunks already written are a partial control line with no // terminating newline. Flush it (with retry) so it's less likely to // linger; even if every retry drops, the NEXT call's pre-flush gate above // refuses to write onto the dirty buffer, so no corruption-as-success can // slip through. - sendEnterWithRetry(); - return { submitted: false }; + try { sendEnterWithRetry(); } catch { /* disposition remains unknown */ } + // A tmux send-keys timeout reports false but cannot prove that the pane + // received zero bytes. In particular, a last-chunk timeout followed by a + // successful cleanup Enter may have submitted the complete valid frame. + // Never cancel attribution on this ambiguous boundary. + return { submitted: false, submissionDisposition: 'dirty_unknown' }; } if (i < chunks.length - 1) await delay(RUNNER_INPUT_THROTTLE_MS); } - // Submit (with retry — a single dropped Enter would leave a complete but + // Submit (with retry — a single unconfirmed Enter could leave a complete but // unsubmitted line in the buffer). - if (!sendEnterWithRetry()) return { submitted: false }; - return { submitted: true }; + try { + if (!sendEnterWithRetry()) { + // The complete, valid line may still be buffered. Retrying a successor + // in this generation could submit it against the wrong FIFO head. + return { submitted: false, submissionDisposition: 'dirty_unknown' }; + } + } catch { + return { submitted: false, submissionDisposition: 'dirty_unknown' }; + } + return { submitted: true, submissionDisposition: 'submitted' }; } diff --git a/src/adapters/cli/types.ts b/src/adapters/cli/types.ts index ea24d88cb..3a695da2a 100644 --- a/src/adapters/cli/types.ts +++ b/src/adapters/cli/types.ts @@ -1,14 +1,17 @@ import type { CodexAppTurnInput } from '../../types.js'; export interface PtyHandle { - write(data: string): void; + /** `false` means the backend rejected the write before it could confirm + * delivery. Callers must not silently promote that result to success. */ + write(data: string): void | boolean; /** Send text literally via tmux send-keys -l (tmux mode only). - * Returns `false` when the write was dropped (e.g. send-keys failed while the - * pane is still alive) so callers can surface a non-submission; `void`/`true` - * means the write was issued. Backends that can't tell return void. */ + * Returns `false` when the backend could not confirm the write (for example, + * send-keys timed out while the pane stayed alive). Delivery may still have + * occurred, so callers must treat false as ambiguous rather than proof that + * zero bytes landed. `void`/`true` means the write was issued. */ sendText?(text: string): void | boolean; /** Send special keys via tmux send-keys, e.g. 'Enter', 'Escape', 'C-c' (tmux mode only). - * Returns `false` on a dropped write (see sendText). */ + * Returns `false` on an unconfirmed write (see sendText). */ sendSpecialKeys?(...keys: string[]): void | boolean; /** Paste text via tmux load-buffer + paste-buffer (auto-brackets if terminal supports it). */ pasteText?(text: string): void; @@ -30,6 +33,19 @@ export type SubmitRecheckResult = boolean | { cliSessionId?: string; }; +/** What the adapter can prove about a failed runner-protocol write. + * + * Runner adapters write a framed line and then a newline. `submitted:false` + * alone is insufficient for recovery: the line may be untouched, may have + * been flushed as an invalid fragment, or may still be a complete valid frame + * waiting in the runner's stdin buffer. Only the first two dispositions are + * safe to cancel/retry in the same generation. */ +export type RunnerSubmissionDisposition = + | 'submitted' + | 'untouched' + | 'flushed_invalid' + | 'dirty_unknown'; + /** A session discovered on disk that botmux can resume (import) into a topic — * surfaced by `/adopt`'s second filter. Unlike an AdoptableSession (a live * tmux/zellij pane botmux *observes*), this is a stored transcript botmux @@ -193,6 +209,7 @@ export interface CliAdapter { ): Promise SubmitRecheckResult | Promise; }>; diff --git a/src/bot-registry.ts b/src/bot-registry.ts index effcc97f9..f5bbe3472 100644 --- a/src/bot-registry.ts +++ b/src/bot-registry.ts @@ -30,6 +30,16 @@ export type { VcMeetingConsumerProfileConfig, } from './types.js'; +/** Bound every official-SDK HTTP call so one stalled provider request cannot + * hold a bot-turn admission or maintenance mutation indefinitely. */ +export const LARK_REQUEST_TIMEOUT_MS = 15_000; + +export function configureLarkClientHttpTimeout(client: unknown): void { + const defaults = (client as { httpInstance?: { defaults?: { timeout?: number } } } | null) + ?.httpInstance?.defaults; + if (defaults) defaults.timeout = LARK_REQUEST_TIMEOUT_MS; +} + export type ChatReplyMode = 'chat' | 'new-topic' | 'shared' | 'chat-topic'; export type ContentTriggerScope = 'topic' | 'regularGroup' | 'both'; export type ContentTriggerMatchType = 'keyword' | 'regex'; @@ -1393,6 +1403,7 @@ export function registerBot(cfg: BotConfig): BotState { domain: sdkDomain(normalizeBrand(cfg.brand)), logger: larkLogger, }); + configureLarkClientHttpTimeout(client); const state: BotState = { config: cfg, client, diff --git a/src/cli.ts b/src/cli.ts index 79198fd61..0803ff27a 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -8,7 +8,10 @@ * botmux setup list|add|configure|edit|remove — scripted (non-TUI) bot management, see `botmux setup help` * botmux start — start daemon and auto plugin services * botmux stop [--with-plugin] — stop daemon (optionally stop auto plugin services) - * botmux restart [--include-pm2] [--with-plugin] — restart daemon, then ensure auto plugin services + * botmux restart [--include-pm2] [--with-plugin] — restart daemon, then ensure auto plugin services; + * --include-pm2 is a zero-live-God admission fence, not authority to signal an existing PM2 God + * botmux restart --bootstrap-shutdown-protocol --yes — operator-approved one-time retirement + * of a pre-protocol fleet after independently confirming all Session/Riff work is idle * botmux logs [--lines] — view daemon logs * botmux status — show daemon status * botmux upgrade|update — upgrade to latest version @@ -21,16 +24,20 @@ * botmux whiteboard status|enable|disable|current|list|read|update|write — local project whiteboard */ import { execSync, execFileSync, spawnSync, spawn } from 'node:child_process'; -import { existsSync, mkdirSync, copyFileSync, readFileSync, writeFileSync, renameSync, readdirSync, readlinkSync, appendFileSync, statSync, unlinkSync, rmSync, realpathSync } from 'node:fs'; +import { existsSync, mkdirSync, copyFileSync, readFileSync, writeFileSync, renameSync, readdirSync, appendFileSync, statSync, unlinkSync, rmSync, realpathSync } from 'node:fs'; import { atomicWriteFileSync } from './utils/atomic-write.js'; import { join, dirname, basename, resolve } from 'node:path'; -import { homedir } from 'node:os'; +import { homedir, userInfo } from 'node:os'; import { fileURLToPath } from 'node:url'; import { createInterface } from 'node:readline'; import { createRequire } from 'node:module'; import { randomBytes } from 'node:crypto'; import { validateWorkingDir } from './core/working-dir.js'; -import { resolveSessionContext } from './core/session-marker.js'; +import { + findAncestorSessionContext as findLiveAncestorSessionContext, + resolveSessionContext, +} from './core/session-marker.js'; +import { readSupervisorProcessStartIdentity } from './core/process-start-identity.js'; import { resolveBotmuxDataDir } from './core/data-dir.js'; import { dashboardSecretPath } from './core/dashboard-secret.js'; import { parseDispatchBotSpec, buildDispatchMessages, buildRepoPrimeText, buildReportContent, eligibleAutoMentionAliases, offTopicSubBotTopic, resolveReportTarget, resolveSendTarget } from './core/dispatch.js'; @@ -75,19 +82,65 @@ import { import { interactiveSelect, pickChoice, pickCliSelection } from './setup/interactive-select.js'; import { buildPreset, serializePreset, presetFilename } from './setup/agent-preset.js'; import type { CliId } from './adapters/cli/types.js'; +import type { CodexAppDispatchLedgerEntry } from './types.js'; +import { + validateCodexAppManagedSendOrigin, +} from './utils/codex-app-dispatch-ledger.js'; +import { hasProtectedSessionMutationOwnership } from './core/session-mutation-guard.js'; import { logger } from './utils/logger.js'; +import { withFileLock, withFileLockSync } from './utils/file-lock.js'; import { scrubSessionCliHomeEnv } from './utils/child-env.js'; import { scheduleTimeZone } from './utils/timezone.js'; import { expandHomePath, invalidWorkingDirs } from './utils/working-dir.js'; import { firstPositional } from './cli/arg-utils.js'; import { isColdResumeDormant, sessionListDisposition } from './cli/session-list-liveness.js'; +import type { BackendType } from './adapters/backend/types.js'; +import { cleanupExplicitSessionBacking } from './core/explicit-session-backing-cleanup.js'; +import { + FLEET_DAEMON_EXIT_WAIT_MS, + FLEET_SUCCESSOR_SETTLE_MS, + PM2_DAEMON_KILL_TIMEOUT_MS, + PM2_DAEMON_RESTART_DELAY_MS, +} from './core/shutdown-budgets.js'; +import { + isFleetEntryProvenFreeOfAutorestartTimer, + signalAndAwaitFleet, + type FleetProcessEntry, +} from './cli/fleet-shutdown.js'; +import { + startExactPm2ProcessIds, + type Pm2ExactStartClient, +} from './cli/pm2-exact-start.js'; import { dispatchPrimaryMessage, findStdinAliasAttachment, normalizeInteractiveCardInput, sendFileAttachments, sendVideoAttachments, shouldSendAsPureVideo, validateVideoAttachments } from './cli/send-dispatch.js'; import { dispatchDeferredTopicSend, type DeferredScheduleRunData } from './cli/deferred-topic-send.js'; import { resolveDaemonExternalHostEnv } from './cli/daemon-lifecycle-env.js'; import { buildPm2SpawnCommand } from './cli/pm2-command.js'; +import { + parseCanonicalPm2Id, + parsePm2JlistOutput, + parsePm2JlistOutputStrict, + parsePm2Integer, +} from './cli/pm2-jlist.js'; +import { assertLinuxPm2GodExecutableUsable } from './cli/pm2-preflight.js'; +import { assertNoUnregisteredLiveDaemonDescriptorsIn } from './cli/pm2-descriptor-guard.js'; +import { DAEMON_GRACEFUL_EXIT_CODE } from './core/supervisor-shutdown-protocol.js'; +import { assertPm2DaemonShutdownCapabilitiesIn } from './cli/pm2-shutdown-capability.js'; +import { assertIncludePm2RestartAdmission } from './cli/pm2-god-admission.js'; +import { + requestAttestedDaemonShutdown, + requestAttestedDaemonShutdownBatch, +} from './cli/supervisor-shutdown-client.js'; +import { + assertDaemonPm2GracefulExitPolicy, + assertConfiguredPm2FleetReady, + assertExactAttestedDaemonSet, + classifyStartBotFleetAdmission, + normalizeRawPm2StopExitCodes, + reconcileLatePm2StartPublication, + runBoundedPm2StartTransaction, +} from './cli/pm2-start-transaction.js'; import { callDashboard, type DashboardEndpoint, type DashboardResult } from './cli/dashboard-endpoint.js'; import { globalInstallUpdateLockTargetIn, installLatestBotmuxSync } from './core/maintenance.js'; -import { withFileLockSync } from './utils/file-lock.js'; import { formatGlobalInstallCommand, resolveGlobalInstallPlan, @@ -104,7 +157,19 @@ import { readWorkflowSessionRelayContext, } from './workflows/v3/session-relay-client.js'; import { fetchDaemonIpc, loadDaemonIpcSecret } from './core/daemon-ipc-auth.js'; -import { readManagedOriginCapability } from './core/managed-origin-capability.js'; +import { + hasManagedOriginIsolationMarker, + managedOriginDataRootProbeAccess, + managedOriginIsolationSentinelAccess, + managedOriginLegacyIsolationProbeAccess, + readManagedOriginRootLocator, + readManagedOriginCapability, +} from './core/managed-origin-capability.js'; +import { + attestManagedOrigin, + type ManagedOriginAttestation, + type ManagedOriginAttestationContext, +} from './core/managed-origin-attestation.js'; import { rejectLikelyWindowsStdinMojibake, decodeStdinBytes } from './cli/stdin-encoding.js'; import { formatBotInfoEntriesForCli, @@ -131,7 +196,14 @@ import { whiteboardPath, } from './services/whiteboard-store.js'; import { buildBridgeSendMarkerContent } from './services/bridge-fallback-gate.js'; -import { bindRestartLeaseTo, writeManualIntentIfAbsentTo } from './services/restart-intent-store.js'; +import { + commitRestartIntentAttemptTo, + bindRestartLeaseTo, + consumeRestartIntentTo, + removeRestartIntentAttemptTo, + writeRestartAttemptIntentTo, + type RestartIntent, +} from './services/restart-intent-store.js'; import { repairMissingChatScope, stripLegacyPendingCardFields } from './services/session-store.js'; import { evaluateVcMeetingManagedSend, @@ -190,6 +262,18 @@ const PM2_NAME = 'botmux'; * when those external pm2 installations get moved or removed. */ const PM2_HOME = join(CONFIG_DIR, 'pm2'); +const PM2_FLEET_MUTATION_LOCK_TARGET = join(CONFIG_DIR, 'pm2-fleet-mutation'); +const PM2_START_COMMAND_TIMEOUT_MS = 30_000; +const PM2_START_VERIFY_MIN_TIMEOUT_MS = 60_000; +const PM2_START_VERIFY_PER_PROCESS_MS = 2_000; +const PM2_START_LATE_PUBLICATION_SETTLE_MS = 10_000; + +function pm2StartVerifyTimeoutMs(processCount: number): number { + return Math.max( + PM2_START_VERIFY_MIN_TIMEOUT_MS, + Math.max(1, Math.floor(processCount)) * PM2_START_VERIFY_PER_PROCESS_MS, + ); +} // ─── Helpers ───────────────────────────────────────────────────────────────── @@ -237,54 +321,85 @@ function pm2Env(home: string = PM2_HOME): NodeJS.ProcessEnv { } function listPm2GodDaemonPids(home: string = PM2_HOME): number[] { - if (process.platform !== 'linux') return []; const marker = `God Daemon (${home})`; const pids: number[] = []; - try { - for (const ent of readdirSync('/proc')) { + if (process.platform === 'linux') { + let entries: string[]; + try { entries = readdirSync('/proc'); } + catch (err) { + throw new Error(`cannot inspect /proc for duplicate PM2 Gods: ${err instanceof Error ? err.message : err}`); + } + for (const ent of entries) { if (!/^\d+$/.test(ent)) continue; const pid = parseInt(ent, 10); if (!pid) continue; try { const cmd = readFileSync(`/proc/${pid}/cmdline`, 'utf-8').replace(/\u0000/g, ' ').trim(); if (cmd.includes('PM2 v') && cmd.includes(marker)) pids.push(pid); - } catch { /* ignore unreadable proc entries */ } + } catch { /* another user's or already-exited process */ } } - } catch { /* ignore proc scan failure */ } + return pids.sort((a, b) => a - b); + } + if (process.platform === 'win32') { + const windowsScan = spawnSync('powershell.exe', [ + '-NoProfile', + '-NonInteractive', + '-Command', + "$needle = \"God Daemon ($env:BOTMUX_PM2_SCAN_HOME)\"; " + + "Get-CimInstance Win32_Process | Where-Object { " + + "$_.CommandLine -and $_.CommandLine.Contains('PM2 v') " + + "-and $_.CommandLine.Contains($needle) } | ForEach-Object { $_.ProcessId }", + ], { + encoding: 'utf8', + stdio: ['ignore', 'pipe', 'pipe'], + timeout: 4_000, + env: { ...process.env, BOTMUX_PM2_SCAN_HOME: home }, + }); + if (windowsScan.status !== 0 || windowsScan.error) { + throw new Error( + `cannot inspect Windows process table for duplicate PM2 Gods: ` + + `${windowsScan.error?.message ?? String(windowsScan.stderr || `status ${windowsScan.status}`).trim()}`, + ); + } + for (const line of String(windowsScan.stdout).split(/\r?\n/)) { + const pid = parsePm2Integer(line.trim(), { nonNegative: true }); + if (pid && pid > 1) pids.push(pid); + } + return [...new Set(pids)].sort((a, b) => a - b); + } + const ps = spawnSync('ps', ['-axo', 'pid=,command='], { + encoding: 'utf8', + stdio: ['ignore', 'pipe', 'pipe'], + timeout: 2_000, + }); + if (ps.status !== 0 || ps.error) { + throw new Error( + `cannot inspect process table for duplicate PM2 Gods: ` + + `${ps.error?.message ?? String(ps.stderr || `status ${ps.status}`).trim()}`, + ); + } + for (const line of String(ps.stdout).split(/\r?\n/)) { + if (!line.includes('PM2 v') || !line.includes(marker)) continue; + const match = line.match(/^\s*(\d+)\s+/); + if (match) pids.push(Number(match[1])); + } return pids.sort((a, b) => a - b); } -function killDuplicatePm2GodDaemons(home: string = PM2_HOME): boolean { +function listSingletonPm2GodDaemonPidsForMutation(home: string = PM2_HOME): number[] { const pids = listPm2GodDaemonPids(home); - if (pids.length <= 1) return false; - - const pidFile = join(home, 'pm2.pid'); - let keepPid = 0; - if (existsSync(pidFile)) { - try { - const parsed = parseInt(readFileSync(pidFile, 'utf-8').trim(), 10); - if (pids.includes(parsed)) keepPid = parsed; - } catch { /* ignore malformed pid file */ } - } - if (!keepPid) keepPid = pids[pids.length - 1]; - - const dupes = pids.filter(pid => pid !== keepPid); - if (dupes.length === 0) return false; - - for (const pid of dupes) { - try { process.kill(pid, 'SIGTERM'); } catch { /* ignore */ } - try { - process.kill(pid, 0); - process.kill(pid, 'SIGKILL'); - } catch { /* ignore */ } - } - - try { - atomicWriteFileSync(pidFile, `${keepPid}\n`); - } catch { /* ignore */ } + if (pids.length <= 1) return pids; + // Never signal a duplicate God automatically. Its SIGTERM handler may + // serially stop/force-kill managed children, including a Riff generation + // whose lineage is not yet durable in the surviving God's registry. + throw new Error( + `refusing PM2 mutation: multiple PM2 God daemons share ${home} ` + + `(pids: ${pids.join(', ')}); no process was signalled`, + ); +} - console.warn(`⚠️ 检测到同一 PM2_HOME (${home}) 下存在多个 PM2 God Daemon，已清理重复实例；保留 pid ${keepPid}，移除: ${dupes.join(', ')}`); - return true; +function assertNoDuplicatePm2GodDaemons(home: string = PM2_HOME): void { + listSingletonPm2GodDaemonPidsForMutation(home); } function runPm2(args: string[], inherit = true, home: string = PM2_HOME, timeoutMs?: number): void { @@ -325,18 +440,70 @@ function pm2Capture(args: string[], home: string = PM2_HOME, timeoutMs = 10_000) return typeof r.stdout === 'string' ? r.stdout : ''; } -function parsePm2JlistOutput(output: string): any[] { +async function cmdInternalPm2StartExact(args: string[]): Promise { + const processIds = args.map(value => Number(value)); try { - const parsed = JSON.parse(output); - return Array.isArray(parsed) ? parsed : []; - } catch { - for (let start = output.lastIndexOf('['); start >= 0; start = output.lastIndexOf('[', start - 1)) { - try { - const parsed = JSON.parse(output.slice(start).trim()); - if (Array.isArray(parsed)) return parsed; - } catch { /* try an earlier '['; pm2 may prefix stdout with [PM2] logs */ } + const claimedParent = parsePm2Integer(process.env.BOTMUX_PM2_FLEET_LOCK_OWNER_PID, { + nonNegative: true, + }); + if (claimedParent !== process.ppid) { + throw new Error('internal exact PM2 start requires its live parent fleet-lock owner'); } - throw new Error('pm2_jlist_json_not_found'); + const lockPayload = readFileSync(`${PM2_FLEET_MUTATION_LOCK_TARGET}.lock`, 'utf8').trim(); + let lockPid: number | undefined; + try { + const parsed = JSON.parse(lockPayload) as unknown; + lockPid = parsed && typeof parsed === 'object' + ? parsePm2Integer((parsed as Record).pid, { nonNegative: true }) + : undefined; + } catch { + lockPid = parsePm2Integer(lockPayload, { nonNegative: true }); + } + if (lockPid !== process.ppid) { + throw new Error('internal exact PM2 start could not verify the parent fleet lock'); + } + const pm2 = require('pm2') as { Client: Pm2ExactStartClient }; + await startExactPm2ProcessIds(processIds, pm2.Client); + } catch (err) { + console.error(err instanceof Error ? err.message : String(err)); + process.exitCode = 1; + } +} + +function runExactPm2Starts( + entries: FleetProcessEntry[], + home: string, + timeoutMs: number, +): void { + const processIds = entries.map(entry => entry.pmId); + if (processIds.some(id => !Number.isInteger(id) || (id as number) < 0)) { + throw new Error('conditional PM2 compensation requires an exact pm_id for every entry'); + } + const boundedTimeoutMs = Math.floor(timeoutMs); + if (boundedTimeoutMs <= 0) { + throw new Error('fleet deadline exhausted before conditional PM2 compensation'); + } + // One bounded helper process opens one PM2 RPC connection and invokes + // God.startProcessId for every exact row concurrently. Public `pm2 start` + // routes existing names/ids through restartProcessId and is unsafe here. + const result = spawnSync( + process.execPath, + [__filename, '__pm2-start-exact', ...processIds.map(String)], + { + stdio: 'pipe', + env: { + ...pm2Env(home), + BOTMUX_PM2_FLEET_LOCK_OWNER_PID: String(process.pid), + }, + timeout: boundedTimeoutMs, + encoding: 'utf8', + }, + ); + if (result.status !== 0) { + const detail = result.error?.message + ?? result.stderr?.trim() + ?? `status ${result.status}`; + throw new Error(`conditional PM2 compensation failed: ${detail}`); } } @@ -395,9 +562,8 @@ function ensureUniqueBotProcessNames(bots: any[]): void { } } -function ecosystemConfig(): string { +function ecosystemConfig(bots: any[] = loadBotsJson()): string { const daemonScript = join(PKG_ROOT, 'dist', 'index-daemon.js'); - const bots = loadBotsJson(); ensureUniqueBotProcessNames(bots); const externalHostEnv = resolveDaemonExternalHostEnv( process.env, @@ -413,20 +579,19 @@ function ecosystemConfig(): string { cwd: CONFIG_DIR, autorestart: true, max_restarts: 10, - restart_delay: 3000, - // A graceful daemon shutdown exits 0 (SIGTERM/SIGINT → drain → process.exit(0)). - // Tell pm2 that exit 0 is intentional so it does NOT autorestart the daemon - // while `botmux restart` is tearing the fleet down — otherwise pm2 revives - // each daemon (after restart_delay) the instant our parallel SIGTERM drains - // it, and re-deleting those revivals one-by-one re-serializes the teardown - // (~13s of churn for 31 bots). Crashes (non-zero exit / killed by signal) - // are NOT in this list, so genuine crash-autorestart is preserved. - stop_exit_codes: [0], - // pm2's default kill_timeout (1.6s) is SHORTER than the daemon's own - // SHUTDOWN_GRACE_MS (3s), so any daemon pm2 has to signal directly gets - // SIGKILL'd mid-drain → orphaned (ppid=1) workers. Give pm2 headroom past - // the daemon's graceful-drain budget so it never force-kills mid-shutdown. - kill_timeout: 3500, + restart_delay: PM2_DAEMON_RESTART_DELAY_MS, + // PM2's God maps signal-only death to exit_code=0 before applying + // stop_exit_codes. Zero cannot be a graceful sentinel: SIGKILL/OOM during + // a prepared Riff drain would otherwise suppress autorestart and look safe + // to delete. Only shutdown()'s fully committed success exits the reserved + // non-zero code. All signal deaths and ordinary failures still restart. + stop_exit_codes: [DAEMON_GRACEFUL_EXIT_CODE], + // Riff graceful shutdown may spend 12s draining one bounded task-create, + // then 11s restoring admission after a refusal. A successful transaction + // instead has the ordinary 3s worker-exit backstop. The supervisor budget + // must remain outside every inner handshake or PM2 can SIGKILL a correct + // daemon mid-ACK. + kill_timeout: PM2_DAEMON_KILL_TIMEOUT_MS, log_date_format: 'YYYY-MM-DD HH:mm:ss', merge_logs: true, node_args: [ @@ -461,9 +626,10 @@ function ecosystemConfig(): string { cwd: PKG_ROOT, autorestart: true, max_restarts: 10, - restart_delay: 3000, - // Same rationale as the bot daemons: don't let pm2 revive on graceful exit-0 - // during a fleet teardown, and don't SIGKILL mid-shutdown. (See baseApp.) + restart_delay: PM2_DAEMON_RESTART_DELAY_MS, + // Dashboard owns no Session/Riff lineage, so its signal-only exit-0 has no + // prepare/commit ambiguity. Keep this explicitly separate from the daemon + // sentinel policy above and avoid reviving it during a fleet teardown. stop_exit_codes: [0], kill_timeout: 3500, error_file: join(LOG_DIR, 'dashboard-error.log'), @@ -1505,7 +1671,7 @@ async function cmdSetupScripted(argv: string[]): Promise { ); return; } - const live = ensureBotDaemonStarted(bot.larkAppId, { quiet: cmd.json }); + const live = await ensureBotDaemonStarted(bot.larkAppId, { quiet: cmd.json }); const next = live.ok ? 'live' : (live.reason === 'fleet_down' ? 'botmux start' : 'botmux restart'); if (cmd.json) { console.log(JSON.stringify({ @@ -1778,7 +1944,7 @@ async function cmdSetupScripted(argv: string[]): Promise { return; } // daemon 在跑就直接把新 bot 那一个进程拉起来，免整组 botmux restart。 - const live = ensureBotDaemonStarted(bot.larkAppId, { quiet: cmd.json }); + const live = await ensureBotDaemonStarted(bot.larkAppId, { quiet: cmd.json }); const next = live.ok ? 'live' : (live.reason === 'fleet_down' ? 'botmux start' : 'botmux restart'); if (cmd.json) { console.log(JSON.stringify({ @@ -2120,7 +2286,7 @@ async function cmdSetup(): Promise { console.log(`\n✅ 已添加机器人 ${newBot.larkAppId}，共 ${bots.length + 1} 个`); console.log(` 配置文件: ${BOTS_JSON_FILE}`); await finishOpenPlatformSetup(newBot.larkAppId, botBrand(newBot), { reuseOnly: hasSetupWebSession(newBot) }); - printAddBotLiveHint(newBot.larkAppId); + await printAddBotLiveHint(newBot.larkAppId); return; } @@ -2177,7 +2343,7 @@ async function cmdSetup(): Promise { console.log(` 配置文件: ${BOTS_JSON_FILE}`); console.log(` 旧配置已备份: ${ENV_FILE}.bak`); await finishOpenPlatformSetup(newBot.larkAppId, botBrand(newBot), { reuseOnly: hasSetupWebSession(newBot) }); - printAddBotLiveHint(newBot.larkAppId); + await printAddBotLiveHint(newBot.larkAppId); } else { // --- Fresh install --- @@ -2196,38 +2362,20 @@ async function cmdSetup(): Promise { * the daemon, but the error gets buried in pm2 logs and the user sees * silence. * - * Detects two cases and either auto-heals or aborts with a clear message: - * 1. pm2 god daemon's running binary is deleted → auto `pm2 kill` + * Detects two cases and aborts with a clear message: + * 1. pm2 god daemon's running binary is deleted → fail closed; an automatic + * kill could bypass a managed daemon's Riff shutdown protocol * 2. This package is installed under an nvm Node version that no longer * exists on disk → abort with reinstall instructions */ -function preflightNodeSanity(): void { - // Case 1: pm2 god is alive but its Node binary has been deleted. - const pm2PidFile = join(PM2_HOME, 'pm2.pid'); - if (existsSync(pm2PidFile)) { - let pm2Pid = 0; - try { pm2Pid = parseInt(readFileSync(pm2PidFile, 'utf-8').trim(), 10); } catch { /* ignore */ } - if (pm2Pid) { - let pm2Alive = false; - try { process.kill(pm2Pid, 0); pm2Alive = true; } catch { /* not alive */ } - if (pm2Alive && process.platform === 'linux') { - // On Linux, /proc//exe is a symlink to the running executable. - // readlink includes a " (deleted)" suffix when the on-disk file is gone. - try { - const exe = readlinkSync(`/proc/${pm2Pid}/exe`); - const cleanPath = exe.replace(/ \(deleted\)$/, ''); - const exeDeleted = exe.endsWith(' (deleted)') || !existsSync(cleanPath); - if (exeDeleted) { - console.warn(`⚠️ pm2 god daemon (pid ${pm2Pid}) 使用的 Node 二进制已失效: ${cleanPath}`); - console.warn(` 自动杀掉 pm2 god 以便用当前 Node 重启...`); - try { - runPm2(['kill'], false, PM2_HOME, 10_000); - } catch { - try { process.kill(pm2Pid, 'SIGKILL'); } catch { /* ignore */ } - } - } - } catch { /* /proc not readable, skip */ } - } +function preflightNodeSanity(home: string = PM2_HOME): void { + // Case 1: inspect every God actually enumerated for this PM2_HOME. `pm2.pid` + // is only a cache: it may be missing, stale, or point at a different + // generation while an older God still owns live children. + const actualGodPids = listPm2GodDaemonPids(home); + if (process.platform === 'linux') { + for (const pm2Pid of actualGodPids) { + assertLinuxPm2GodExecutableUsable(pm2Pid); } } @@ -2258,8 +2406,6 @@ async function cmdStart(): Promise { process.exit(1); } ensureConfigDir(); - killDuplicatePm2GodDaemons(); - preflightNodeSanity(); await ensureSystemDependencies(); // 启动前快速校验每个 bot 的凭证. Codex review 边界 #5: 凭证无效是 @@ -2295,9 +2441,73 @@ async function cmdStart(): Promise { } } - cleanupLegacyPm2(); - const cfg = ecosystemConfig(); - runPm2(['start', cfg]); + await withFileLock(PM2_FLEET_MUTATION_LOCK_TARGET, async () => { + await withFileLock(BOTS_JSON_FILE, async () => { + const lockedBots = loadBotsJson(); + if (JSON.stringify(lockedBots) !== JSON.stringify(botsForCheck)) { + throw new Error('[start] bots.json changed during credential preflight; retry with the new configuration'); + } + assertNoDuplicatePm2GodDaemons(); + preflightNodeSanity(); + cleanupLegacyPm2(); + const currentProjection = readVerifiedBotmuxPm2Projection('start'); + assertNoUnregisteredLiveDaemonDescriptors('start', currentProjection); + assertCanonicalUniquePm2Rows('start', currentProjection); + const configuredNames = configuredCoreProcessNames(lockedBots); + const verifyTimeoutMs = pm2StartVerifyTimeoutMs(configuredNames.length); + const cfg = ecosystemConfig(lockedBots); + const liveEntries = currentProjection.filter(isLivePm2Entry); + if (liveEntries.length > 0) { + try { + readAndAssertConfiguredFleetOnline( + 'start-idempotent-ready', + configuredNames, + PM2_HOME, + verifyTimeoutMs, + ); + // Idempotent means no PM2 mutation, not weaker readiness. An old + // in-memory daemon may be PM2-online while lacking the handler-ready + // protocol, so it must fail closed instead of being reported ready. + return; + } catch (error) { + throw new Error( + `[start] refusing PM2 start while a partial/live core fleet exists ` + + `(${liveEntries.map(entry => `${entry.name}:${entry.pid}`).join(', ')}): ` + + `${error instanceof Error ? error.message : String(error)}; ` + + 'use start-bot only for an exact one-missing-bot fleet, or restart', + ); + } + } + const unprovenDormant = currentProjection.filter( + entry => !isFleetEntryProvenFreeOfAutorestartTimer(entry), + ); + if (unprovenDormant.length > 0) { + throw new Error( + `[start] refusing PM2 start: dormant row(s) may still have a restart timer ` + + `(${unprovenDormant.map(entry => `${entry.name}:${entry.status ?? 'unknown'}`).join(', ')})`, + ); + } + runBoundedPm2StartTransaction( + 'start', + PM2_START_COMMAND_TIMEOUT_MS, + verifyTimeoutMs, + { + start: timeoutMs => { + assertBotsConfigSnapshotUnchanged('start', lockedBots); + assertNoDuplicatePm2GodDaemons(); + preflightNodeSanity(); + runPm2(['start', cfg], true, PM2_HOME, timeoutMs); + }, + verifyFresh: timeoutMs => readAndAssertConfiguredFleetOnline( + 'start-after-launch', configuredNames, PM2_HOME, timeoutMs, + ), + rollback: () => rollbackPm2StartAttempt( + 'start', currentProjection, configuredNames, + ), + }, + ); + }, { maxWaitMs: 5_000 }); + }, { maxWaitMs: 5_000 }); await reconcilePluginServicesForCli(undefined, { autoOnly: true }); const bots = loadBotsJson(); const count = bots.length || 1; @@ -2343,81 +2553,701 @@ function isBotmuxCoreProcessName(name: string): boolean { return name === PM2_NAME || (name.startsWith(`${PM2_NAME}-`) && !name.startsWith(`${PM2_NAME}-plugin-`)); } -function deleteAllBotmuxProcesses(home: string = PM2_HOME): void { - let entries: Array<{ name: string; pid: number; online: boolean }>; +function isBotmuxDaemonProcessName(name: string): boolean { + return isBotmuxCoreProcessName(name) && name !== 'botmux-dashboard'; +} + +type BotmuxPm2ProcessEntry = FleetProcessEntry; + +function toBotmuxPm2ProcessEntry(app: any): BotmuxPm2ProcessEntry { + const rawStopExitCodes = app?.pm2_env?.stop_exit_codes; + // PM2's top-level pm_id is the canonical registry identity used by RPC. + // Never revive it from a stale/mismatched pm2_env copy when the canonical + // field is absent or null; that would turn compensation into a guessed ID. + const pmId = parseCanonicalPm2Id(app); + const exitCode = parsePm2Integer(app?.pm2_env?.exit_code); + return { + name: String(app.name), + ...(pmId !== undefined ? { pmId } : {}), + pid: Number(app.pid) || 0, + online: app?.pm2_env?.status === 'online', + status: String(app?.pm2_env?.status ?? 'unknown'), + autorestart: app?.pm2_env?.autorestart, + stopExitCodes: normalizeRawPm2StopExitCodes(rawStopExitCodes), + ...(exitCode !== undefined ? { exitCode } : {}), + }; +} + +function readVerifiedBotmuxPm2Projection( + operation: string, + home: string = PM2_HOME, + timeoutMs = 10_000, +): BotmuxPm2ProcessEntry[] { try { - const apps = parsePm2JlistOutput(pm2Capture(['jlist'], home)); + const apps = parsePm2JlistOutputStrict(pm2Capture(['jlist'], home, timeoutMs)); + return apps + .filter(app => app && isBotmuxCoreProcessName(String(app.name))) + .map(toBotmuxPm2ProcessEntry); + } catch (err) { + throw new Error( + `[${operation}] pm2 jlist failed; refusing an unverified PM2 mutation: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + } +} + +/** A valid `jlist []` is not sufficient authority after PM2 registry loss. + * Fresh daemon descriptors are an independent, daemon-owned liveness view; + * refuse every core mutation if one points at a live PID absent from all PM2 + * projections participating in the operation. */ +function assertNoUnregisteredLiveDaemonDescriptors( + operation: string, + projections: BotmuxPm2ProcessEntry[], +): void { + assertNoUnregisteredLiveDaemonDescriptorsIn( + operation, + projections, + join(resolveDataDir(), 'dashboard-daemons'), + ); +} + +function duplicatePm2CoreNames(entries: BotmuxPm2ProcessEntry[]): string[] { + return [...new Set(entries.map(entry => entry.name))] + .filter(name => entries.filter(entry => entry.name === name).length > 1); +} + +function isLivePm2Entry(entry: BotmuxPm2ProcessEntry): boolean { + if (!Number.isInteger(entry.pid) || entry.pid <= 1) return false; + try { process.kill(entry.pid, 0); return true; } catch { return false; } +} + +function configuredCoreProcessNames(bots: any[] = loadBotsJson()): string[] { + return [ + ...bots.map((bot, index) => botProcessName(bot, index, PM2_NAME)), + 'botmux-dashboard', + ]; +} + +function assertBotsConfigSnapshotUnchanged(operation: string, snapshot: any[]): void { + if (JSON.stringify(loadBotsJson()) === JSON.stringify(snapshot)) return; + throw new Error(`[${operation}] bots.json generation changed before PM2 start; no launch attempted`); +} + +function readAndAssertConfiguredFleetOnline( + operation: string, + configuredNames: string[], + home: string = PM2_HOME, + timeoutMs: number = PM2_START_VERIFY_MIN_TIMEOUT_MS, +): BotmuxPm2ProcessEntry[] { + const deadline = Date.now() + Math.max(1, Math.floor(timeoutMs)); + let lastError: unknown; + while (Date.now() < deadline) { + try { + const remainingMs = Math.max(1, deadline - Date.now()); + const projection = readVerifiedBotmuxPm2Projection(operation, home, remainingMs); + assertNoUnregisteredLiveDaemonDescriptors(operation, projection); + assertConfiguredPm2FleetReady( + operation, + projection, + configuredNames, + pid => { try { process.kill(pid, 0); return true; } catch { return false; } }, + readyEntries => { + const daemonEntries = readyEntries + .filter(entry => isBotmuxDaemonProcessName(entry.name)); + assertDaemonPm2GracefulExitPolicy( + `${operation}-handler-ready-pm2-policy`, + daemonEntries, + ); + const attested = assertPm2DaemonShutdownCapabilitiesIn( + `${operation}-handler-ready`, + daemonEntries.map(entry => ({ name: entry.name, pid: entry.pid })), + join(resolveDataDir(), 'dashboard-daemons'), + ); + assertExactAttestedDaemonSet( + `${operation}-handler-ready`, + daemonEntries, + attested.map(entry => entry.pid), + pid => { try { process.kill(pid, 0); return true; } catch { return false; } }, + ); + for (const target of attested) { + if (readSupervisorProcessStartIdentity(target.pid) !== target.processStartIdentity) { + throw new Error( + `[${operation}-handler-ready] daemon generation changed after capability scan: ` + + `${target.name}/${target.pid}`, + ); + } + } + // Dashboard has no daemon capability endpoint; recheck its OS + // liveness after the potentially-long descriptor scan as well. + const dashboardRows = readyEntries.filter(entry => !isBotmuxDaemonProcessName(entry.name)); + if (dashboardRows.some(entry => !isLivePm2Entry(entry))) { + throw new Error(`[${operation}] dashboard exited during handler-ready verification`); + } + }, + ); + return projection; + } catch (error) { + lastError = error; + const remainingMs = deadline - Date.now(); + if (remainingMs <= 0) break; + sleepSyncMs(Math.min(100, remainingMs)); + } + } + throw new Error( + `[${operation}] configured fleet never reached PM2-online plus handler-ready capability ` + + `within ${timeoutMs}ms: ${lastError instanceof Error ? lastError.message : String(lastError)}`, + ); +} + +function assertCanonicalUniquePm2Rows( + operation: string, + entries: BotmuxPm2ProcessEntry[], +): void { + const duplicateNames = duplicatePm2CoreNames(entries); + const missingIds = entries.filter(entry => + !Number.isSafeInteger(entry.pmId) || (entry.pmId as number) < 0); + const duplicateIds = [...new Set(entries + .map(entry => entry.pmId) + .filter((id): id is number => Number.isSafeInteger(id)))] + .filter(id => entries.filter(entry => entry.pmId === id).length > 1); + const duplicateLivePids = [...new Set(entries + .map(entry => entry.pid) + .filter(pid => Number.isSafeInteger(pid) && pid > 1))] + .filter(pid => entries.filter(entry => entry.pid === pid).length > 1); + if (duplicateNames.length === 0 + && missingIds.length === 0 + && duplicateIds.length === 0 + && duplicateLivePids.length === 0) return; + throw new Error( + `[${operation}] refusing PM2 mutation: canonical registry identity is ambiguous` + + (duplicateNames.length > 0 ? ` (duplicate names: ${duplicateNames.join(', ')})` : '') + + (duplicateIds.length > 0 ? ` (duplicate pm_id: ${duplicateIds.join(', ')})` : '') + + (duplicateLivePids.length > 0 + ? ` (duplicate positive pid: ${duplicateLivePids.join(', ')})` + : '') + + (missingIds.length > 0 + ? ` (missing pm_id: ${missingIds.map(entry => entry.name).join(', ')})` + : ''), + ); +} + +/** Revalidate exact row identity and OS quiescence immediately before/after a + * PM2 registry mutation. Missing rows are already reaped; recreated/duplicate + * rows and any live generation fail closed. */ +function exactQuiescentRowsForMutation( + operation: string, + originals: BotmuxPm2ProcessEntry[], + fresh: BotmuxPm2ProcessEntry[], +): BotmuxPm2ProcessEntry[] { + const exact: BotmuxPm2ProcessEntry[] = []; + for (const original of originals) { + const rows = fresh.filter(entry => entry.name === original.name); + if (rows.length === 0) continue; + if (rows.length !== 1 || rows[0]!.pmId !== original.pmId) { + throw new Error( + `[${operation}] refusing PM2 mutation: registry row ${original.name} was recreated or duplicated`, + ); + } + if (isLivePm2Entry(rows[0]!)) { + throw new Error( + `[${operation}] refusing PM2 mutation: live generation appeared for ` + + `${original.name}:${rows[0]!.pid}`, + ); + } + if (!isFleetEntryProvenFreeOfAutorestartTimer(rows[0]!)) { + throw new Error( + `[${operation}] refusing PM2 mutation: ${original.name} may still publish a successor`, + ); + } + exact.push(rows[0]!); + } + return exact; +} + +/** The projection used to choose an exact PM2 id must be the last meaningful + * action before each stop/delete. Re-read descriptors and every original row + * for every individual mutation so a successor/recreated row discovered after + * a prior mutation cannot be hit by a stale batch id. */ +function revalidateExactQuiescentRowBeforeMutation( + operation: string, + original: BotmuxPm2ProcessEntry, + allOriginals: BotmuxPm2ProcessEntry[], + home: string = PM2_HOME, + additionalDescriptorAuthority: BotmuxPm2ProcessEntry[] = [], +): BotmuxPm2ProcessEntry | undefined { + // Legacy cleanup can target a different PM2_HOME. It gets the same + // duplicate-God fail-closed gate as the dedicated home before every core + // registry mutation. + assertNoDuplicatePm2GodDaemons(home); + const fresh = readVerifiedBotmuxPm2Projection(operation, home); + assertNoUnregisteredLiveDaemonDescriptors( + operation, + [...fresh, ...additionalDescriptorAuthority], + ); + assertCanonicalUniquePm2Rows(operation, fresh); + const exact = exactQuiescentRowsForMutation(operation, allOriginals, fresh); + return exact.find(entry => entry.name === original.name); +} + +/** Signal every online core process in parallel and wait for its own graceful + * shutdown decision. A daemon that cannot safely drain/cancel Riff remains + * alive; callers must propagate this error and MUST NOT invoke a PM2 mutation + * that would force-kill it after kill_timeout. */ +function signalAndAwaitBotmuxProcesses( + entries: BotmuxPm2ProcessEntry[], + operation: 'restart' | 'stop', + home: string = PM2_HOME, + additionalDescriptorAuthority: BotmuxPm2ProcessEntry[] = [], +): void { + assertCanonicalUniquePm2Rows(operation, entries); + const processNameByPid = new Map(); + const processEntryByPid = new Map(); + const processStartByPid = new Map(); + const rememberProcessIdentity = ( + identityOperation: string, + entry: BotmuxPm2ProcessEntry, + ): void => { + if (entry.pid <= 1) return; + const identity = readSupervisorProcessStartIdentity(entry.pid); + if (!identity) { + if (!isLivePm2Entry(entry)) return; + throw new Error( + `[${identityOperation}] cannot bind ${entry.name}/${entry.pid} to a process-start identity`, + ); + } + processNameByPid.set(entry.pid, entry.name); + processEntryByPid.set(entry.pid, entry); + processStartByPid.set(entry.pid, identity); + }; + for (const entry of entries) rememberProcessIdentity(`${operation}-initial-identity`, entry); + const assertShutdownCapability = ( + capabilityOperation: string, + targets: BotmuxPm2ProcessEntry[], + ) => { + const daemonTargets = targets + .filter(entry => isBotmuxDaemonProcessName(entry.name)); + assertDaemonPm2GracefulExitPolicy(capabilityOperation, daemonTargets); + return assertPm2DaemonShutdownCapabilitiesIn( + capabilityOperation, + daemonTargets.map(entry => ({ name: entry.name, pid: entry.pid })), + join(resolveDataDir(), 'dashboard-daemons'), + ); + }; + // Validate the whole initial live daemon set before the first signal, so a + // mixed new/old fleet cannot be partially retired before an old in-memory + // daemon is discovered. Dashboard has no sessions and needs no Riff protocol. + assertShutdownCapability( + `${operation}-shutdown-capability-preflight`, + entries.filter(entry => entry.online && entry.pid > 1), + ); + const list = (timeoutMs: number): BotmuxPm2ProcessEntry[] => { + const apps = parsePm2JlistOutputStrict(pm2Capture( + ['jlist'], + home, + Math.min(10_000, Math.max(1, Math.floor(timeoutMs))), + )); + const projection = (Array.isArray(apps) ? apps : []) + .filter(app => app && isBotmuxCoreProcessName(String(app.name))) + .map(toBotmuxPm2ProcessEntry); + assertNoUnregisteredLiveDaemonDescriptors( + `${operation}-successor-projection`, + [...projection, ...additionalDescriptorAuthority], + ); + assertCanonicalUniquePm2Rows(`${operation}-successor-projection`, projection); + for (const entry of projection) { + rememberProcessIdentity(`${operation}-successor-identity`, entry); + } + return projection; + }; + const shutdownRequestFailures: string[] = []; + const recordShutdownFailure = (name: string, pid: number, error: unknown): void => { + shutdownRequestFailures.push( + `${name}/${pid}: ${error instanceof Error ? error.message : String(error)}`, + ); + }; + const signalDashboardResidual = (name: string, pid: number): void => { + // Dashboard owns no session/Riff lineage and has no daemon IPC endpoint. + // Snapshot + fresh birth recheck narrows PID reuse, but process.kill remains + // PID-addressed. This is an explicit non-data-safety residual; the formal + // guarantee below applies only to daemon generations that own Riff lineage. + const expectedStart = processStartByPid.get(pid); + const currentStart = readSupervisorProcessStartIdentity(pid); + if (!currentStart) return; + if (!expectedStart || currentStart !== expectedStart) { + recordShutdownFailure(name, pid, 'dashboard process generation changed'); + return; + } + try { process.kill(pid, 'SIGTERM'); } + catch (error) { + if ((error as NodeJS.ErrnoException)?.code !== 'ESRCH') { + recordShutdownFailure(name, pid, error); + } + } + }; + + try { + signalAndAwaitFleet(entries, operation, FLEET_DAEMON_EXIT_WAIT_MS, { + signal: pid => { + const name = processNameByPid.get(pid); + if (!name) { + throw new Error(`[${operation}] refusing signal for unmapped PM2 daemon pid ${pid}`); + } + if (isBotmuxDaemonProcessName(name)) { + try { + // A successor gets a new exact descriptor/boot/birth authority and + // one authenticated request. The receiving process is the final + // generation check; no daemon is signalled by PID. + const successor = processEntryByPid.get(pid); + if (!successor) { + recordShutdownFailure(name, pid, 'successor PM2 policy projection is missing'); + return; + } + const authorized = assertShutdownCapability( + `${operation}-successor-immediately-before-request`, + [successor], + ); + const target = authorized.find(entry => entry.pid === pid); + if (!target) { + recordShutdownFailure(name, pid, 'daemon exited before exact IPC attestation'); + return; + } + requestAttestedDaemonShutdown(target, loadDaemonIpcSecret()); + } catch (error) { + recordShutdownFailure(name, pid, error); + } + return; + } + signalDashboardResidual(name, pid); + }, + signalInitial: targets => { + // Re-attest the full daemon set as one all-or-none admission immediately + // before dispatch. One child process then issues every exact IPC request + // concurrently, so 31 endpoints consume one bounded 4s window, not 31. + let authorized; + try { + authorized = assertShutdownCapability( + `${operation}-initial-immediately-before-batch-request`, + targets as BotmuxPm2ProcessEntry[], + ); + } catch (error) { + for (const target of targets.filter(entry => isBotmuxDaemonProcessName(entry.name))) { + recordShutdownFailure(target.name, target.pid, error); + } + return; + } + const expectedDaemonTargets = targets + .filter(entry => isBotmuxDaemonProcessName(entry.name)); + const authorizedPids = new Set(authorized.map(entry => entry.pid)); + for (const target of expectedDaemonTargets) { + if (!authorizedPids.has(target.pid)) { + recordShutdownFailure( + target.name, + target.pid, + 'daemon exited before initial exact IPC attestation', + ); + } + } + let attempts; + try { + attempts = requestAttestedDaemonShutdownBatch(authorized, loadDaemonIpcSecret()); + } catch (error) { + attempts = authorized.map(target => ({ target, ok: false, error: String(error) })); + } + for (const attempt of attempts) { + if (!attempt.ok) recordShutdownFailure( + attempt.target.name, + attempt.target.pid, + attempt.error ?? 'supervisor shutdown request refused', + ); + } + for (const target of targets.filter(entry => !isBotmuxDaemonProcessName(entry.name))) { + signalDashboardResidual(target.name, target.pid); + } + }, + assertSignalAuthorityComplete: () => { + if (shutdownRequestFailures.length > 0) { + throw new Error(shutdownRequestFailures.join('; ')); + } + }, + isAlive: pid => { try { process.kill(pid, 0); return true; } catch { return false; } }, + now: () => Date.now(), + sleep: sleepSyncMs, + startOffline: (offlineEntries, timeoutMs) => { + runExactPm2Starts(offlineEntries, home, Math.min(10_000, timeoutMs)); + }, + list, + successorSettleMs: FLEET_SUCCESSOR_SETTLE_MS, + }); + } catch (error) { + if (shutdownRequestFailures.length === 0) throw error; + throw new Error( + `${error instanceof Error ? error.message : String(error)}; ` + + `shutdown request refusal(s): ${shutdownRequestFailures.join('; ')}`, + ); + } +} + +/** Compensate only rows owned by the just-attempted start. Rows absent before + * the attempt are deleted; pre-existing dormant exact rows are returned to + * stopped state. Existing online peers (the start-bot case) remain authority + * for descriptor reconciliation and are never signalled. */ +function rollbackPm2StartAttempt( + operation: string, + before: BotmuxPm2ProcessEntry[], + candidateNames: string[], + home: string = PM2_HOME, +): void { + const candidateSet = new Set(candidateNames); + reconcileLatePm2StartPublication( + operation, + PM2_START_LATE_PUBLICATION_SETTLE_MS, + FLEET_DAEMON_EXIT_WAIT_MS + PM2_START_LATE_PUBLICATION_SETTLE_MS, + { + now: () => Date.now(), + sleep: sleepSyncMs, + reconcileOnce: () => { + assertNoDuplicatePm2GodDaemons(home); + const fresh = readVerifiedBotmuxPm2Projection(`${operation}-rollback-read`, home); + assertNoUnregisteredLiveDaemonDescriptors(`${operation}-rollback-read`, fresh); + assertCanonicalUniquePm2Rows(`${operation}-rollback-read`, fresh); + const attemptedRows = fresh.filter(entry => candidateSet.has(entry.name)); + + for (const row of attemptedRows) { + const priorRows = before.filter(entry => entry.name === row.name); + if (priorRows.length > 1 + || (priorRows.length === 1 && priorRows[0]!.pmId !== row.pmId) + || (priorRows.length === 1 && isLivePm2Entry(priorRows[0]!))) { + throw new Error( + `[${operation}] cannot prove ownership of partial-launch row ${row.name}/${row.pmId}`, + ); + } + } + + const rowsNeedingCompensation = attemptedRows.filter(row => { + const prior = before.find(entry => entry.name === row.name); + if (!prior) return true; + return isLivePm2Entry(row) || !isFleetEntryProvenFreeOfAutorestartTimer(row); + }); + if (rowsNeedingCompensation.length > 0) { + // A timed-out PM2 start can expose a live `launching` row. It is + // nevertheless owned by this transaction (exact absent/dormant identity + // proved above), so treat that generation as a signal target. A dead + // transitional row gets a synthetic initial safe label only to enter the + // successor loop; every subsequent raw projection must prove quiet. + const shutdownRows = rowsNeedingCompensation.map(entry => { + if (isLivePm2Entry(entry)) return { ...entry, online: true }; + if (isFleetEntryProvenFreeOfAutorestartTimer(entry)) return entry; + return { ...entry, online: false, status: 'stopped', autorestart: false }; + }); + signalAndAwaitBotmuxProcesses(shutdownRows, 'stop', home); + + for (const original of rowsNeedingCompensation) { + const exact = revalidateExactQuiescentRowBeforeMutation( + `${operation}-rollback-before-mutation`, + original, + rowsNeedingCompensation, + home, + ); + if (!exact) continue; + const existedBefore = before.some(entry => + entry.name === original.name && entry.pmId === original.pmId); + runPm2( + [existedBefore ? 'stop' : 'delete', String(exact.pmId)], + false, + home, + 10_000, + ); + } + return false; + } + + const restored = candidateNames.every(name => { + const prior = before.find(entry => entry.name === name); + const rows = attemptedRows.filter(entry => entry.name === name); + if (!prior) return rows.length === 0; + return rows.length === 1 + && rows[0]!.pmId === prior.pmId + && !isLivePm2Entry(rows[0]!) + && isFleetEntryProvenFreeOfAutorestartTimer(rows[0]!); + }); + if (!restored) { + throw new Error(`[${operation}] rollback could not prove the pre-start registry shape`); + } + return true; + }, + }, + ); +} + +function deleteAllBotmuxProcesses( + home: string = PM2_HOME, + additionalDescriptorAuthority: BotmuxPm2ProcessEntry[] = [], +): void { + assertNoDuplicatePm2GodDaemons(home); + let entries: BotmuxPm2ProcessEntry[]; + try { + const apps = parsePm2JlistOutputStrict(pm2Capture(['jlist'], home)); entries = (Array.isArray(apps) ? apps : []) .filter(a => a && isBotmuxCoreProcessName(String(a.name))) - .map(a => ({ name: String(a.name), pid: Number(a.pid) || 0, online: a?.pm2_env?.status === 'online' })); + .map(toBotmuxPm2ProcessEntry); } catch (e) { - console.error(`[restart] pm2 jlist failed (pm2 not running or no apps?): ${e instanceof Error ? e.message : e}`); - return; + throw new Error( + `[restart] pm2 jlist failed; refusing to start a second fleet without a safe shutdown view: ` + + `${e instanceof Error ? e.message : e}`, + ); } + assertNoUnregisteredLiveDaemonDescriptors( + 'restart', + [...entries, ...additionalDescriptorAuthority], + ); + assertCanonicalUniquePm2Rows('restart', entries); if (entries.length === 0) return; const names = entries.map(e => e.name); // Parallel graceful shutdown. pm2's own delete stops apps one-at-a-time // (async eachLimit, concurrency 1) and each botmux daemon's drain eats pm2's // full kill_timeout (~1.6s) → ~N×1.6s serial (~38s for 31 bots). Instead we - // SIGTERM every online daemon AT ONCE so their graceful drains overlap (the - // daemon's SIGTERM handler detaches workers within SHUTDOWN_GRACE_MS), wait + // SIGTERM every online daemon AT ONCE so their graceful drains overlap, wait // once for them all to exit, then let pm2 delete reap the now-dead entries // instantly. Orphan-safe: each daemon runs its FULL graceful drain and we wait // for real exit before pm2 touches it — avoiding the mid-drain SIGKILL the old - // path forced (pm2 kill_timeout 1.6s < daemon SHUTDOWN_GRACE_MS 3s). - const pids = entries.filter(e => e.online && e.pid > 0).map(e => e.pid); - for (const pid of pids) { - try { process.kill(pid, 'SIGTERM'); } catch { /* already gone */ } - } - // Poll until every signalled daemon has exited (bounded). SHUTDOWN_GRACE_MS is - // 3s; give headroom. Exits early the moment the last one dies. - const deadline = Date.now() + 5_000; - let alive = pids.slice(); - while (alive.length > 0 && Date.now() < deadline) { - sleepSyncMs(50); - alive = alive.filter(pid => { try { process.kill(pid, 0); return true; } catch { return false; } }); - } - - // Reap pm2 entries. Processes are already dead → each delete is instant, and - // ONE batched `pm2 delete name1 name2 …` collapses N pm2 CLI cold-boots - // (~315ms each) into one. A revived (autorestart, gated by restart_delay) - // instance is still removed by name. - const batchTimeout = Math.max(15_000, names.length * 2_500); - try { - runPm2(['delete', ...names], false, home, batchTimeout); - return; - } catch (e) { - // pm2's batched delete (async eachLimit) aborts on the first failed name, - // so a mid-batch failure can leave stragglers. Fall back to the resilient - // per-name loop that try/catches each name independently. - console.error(`[restart] batched pm2 delete failed, falling back to per-name: ${e instanceof Error ? e.message : e}`); - } - for (const name of names) { + // path forced (pm2's old 1.6s/our old 3.5s timeout was shorter than a Riff + // task-id materialization window). + signalAndAwaitBotmuxProcesses(entries, 'restart', home, additionalDescriptorAuthority); + + // Reap each exact id separately. Before every individual destructive call, + // obtain a new strict registry projection plus descriptor reconciliation and + // revalidate all not-yet-reaped originals. A batched delete would make the + // later ids stale while PM2 serially processed earlier ids. + const deleteErrors: string[] = []; + for (const entry of entries) { try { - runPm2(['delete', name], false, home, 10_000); + const exact = revalidateExactQuiescentRowBeforeMutation( + 'restart-before-delete', + entry, + entries, + home, + additionalDescriptorAuthority, + ); + if (!exact) continue; + runPm2(['delete', String(exact.pmId)], false, home, 10_000); } catch (e) { - // Don't swallow silently — a failed delete here used to leave the - // restart half-done with no trace. Surface it (the auto-restart - // driver captures stderr to ~/.botmux/logs/maintenance-restart.log). - console.error(`[restart] pm2 delete ${name} failed: ${e instanceof Error ? e.message : e}`); + const message = e instanceof Error ? e.message : String(e); + deleteErrors.push(`${entry.name}: ${message}`); + console.error(`[restart] pm2 delete ${entry.name}/${entry.pmId} failed: ${message}`); } } -} -function killPm2GodDaemon(home: string = PM2_HOME): void { + let remaining: string[]; try { - runPm2(['kill'], true, home, 15_000); - return; - } catch { - // Fall back to direct pid cleanup below. + const fresh = parsePm2JlistOutputStrict(pm2Capture(['jlist'], home)); + const targetNames = new Set(names); + const freshProjection = (Array.isArray(fresh) ? fresh : []) + .filter(app => app && isBotmuxCoreProcessName(String(app.name))) + .map(toBotmuxPm2ProcessEntry); + assertNoUnregisteredLiveDaemonDescriptors( + 'restart-after-delete', + [...freshProjection, ...additionalDescriptorAuthority], + ); + exactQuiescentRowsForMutation('restart-after-delete', entries, freshProjection); + remaining = freshProjection + .filter(entry => targetNames.has(entry.name)) + .map(entry => entry.name); + } catch (err) { + throw new Error( + `[restart] PM2 delete verification failed; refusing to report a clean fleet: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + } + if (remaining.length > 0) { + throw new Error( + `[restart] PM2 delete left registry entries: ${[...new Set(remaining)].join(', ')}` + + (deleteErrors.length > 0 ? ` (${deleteErrors.join('; ')})` : ''), + ); + } +} + +/** + * Explicit first-upgrade escape hatch for a live fleet that predates the + * authenticated shutdown protocol. This intentionally bypasses descriptor + * capability attestation, but only behind a named flag plus --yes. Every PM2 + * delete is still bound to the exact name + pm_id + PID + process birth read + * before the first mutation and revalidated immediately before that mutation. + */ +function bootstrapDeleteAllBotmuxProcesses( + operation: 'stop' | 'restart', + home: string = PM2_HOME, +): void { + assertNoDuplicatePm2GodDaemons(home); + const readProjection = (phase: string): BotmuxPm2ProcessEntry[] => { + const apps = parsePm2JlistOutputStrict(pm2Capture(['jlist'], home)); + const projection = (Array.isArray(apps) ? apps : []) + .filter(app => app && isBotmuxCoreProcessName(String(app.name))) + .map(toBotmuxPm2ProcessEntry); + assertCanonicalUniquePm2Rows(`${operation}-bootstrap-${phase}`, projection); + return projection; + }; + + const entries = readProjection('initial'); + const identities = new Map(); + for (const entry of entries) { + if (!Number.isInteger(entry.pmId)) { + throw new Error( + `[${operation}] bootstrap refused: ${entry.name} has no canonical PM2 id`, + ); + } + if (entry.pid > 1 && isLivePm2Entry(entry)) { + const identity = readSupervisorProcessStartIdentity(entry.pid); + if (!identity) { + throw new Error( + `[${operation}] bootstrap refused: cannot bind ${entry.name}/${entry.pid} to a process birth`, + ); + } + identities.set(entry.pmId!, identity); + } } - for (const pid of listPm2GodDaemonPids(home)) { - try { process.kill(pid, 'SIGTERM'); } catch { /* already gone */ } + for (const original of entries) { + const fresh = readProjection(`before-delete-${original.pmId}`); + const exact = fresh.filter(entry => entry.pmId === original.pmId); + if (exact.length !== 1 || exact[0]!.name !== original.name) { + throw new Error( + `[${operation}] bootstrap refused: PM2 row ${original.name}/${original.pmId} changed before delete`, + ); + } + const current = exact[0]!; + const expectedBirth = identities.get(original.pmId!); + if (expectedBirth) { + const currentBirth = current.pid > 1 + ? readSupervisorProcessStartIdentity(current.pid) + : undefined; + if (current.pid !== original.pid + || !isLivePm2Entry(current) + || currentBirth !== expectedBirth) { + throw new Error( + `[${operation}] bootstrap refused: process generation changed for ` + + `${original.name}/${original.pmId}`, + ); + } + } else if (isLivePm2Entry(current)) { + throw new Error( + `[${operation}] bootstrap refused: dormant PM2 row ${original.name}/${original.pmId} became live`, + ); + } + runPm2( + ['delete', String(current.pmId)], + false, + home, + FLEET_DAEMON_EXIT_WAIT_MS, + ); } - for (const pid of listPm2GodDaemonPids(home)) { - try { process.kill(pid, 0); process.kill(pid, 'SIGKILL'); } catch { /* already gone */ } + + const remaining = readProjection('after-delete'); + if (remaining.length > 0) { + throw new Error( + `[${operation}] bootstrap retirement incomplete: ` + + remaining.map(entry => `${entry.name}/${entry.pmId}`).join(', '), + ); } } @@ -2428,40 +3258,129 @@ function killPm2GodDaemon(home: string = PM2_HOME): void { * truth. Only touches processes named `botmux` or `botmux-*` — the user's * unrelated pm2 apps are left untouched. No-op on fresh installs. */ -function cleanupLegacyPm2(): boolean { +function cleanupLegacyPm2( + bootstrapOperation?: 'stop' | 'restart', +): boolean { const legacyHome = join(homedir(), '.pm2'); if (legacyHome === PM2_HOME) return false; - const legacyPidFile = join(legacyHome, 'pm2.pid'); - if (!existsSync(legacyPidFile)) return false; - - let legacyPid = 0; - try { legacyPid = parseInt(readFileSync(legacyPidFile, 'utf-8').trim(), 10); } catch { return false; } - if (!legacyPid) return false; - // If the legacy daemon isn't alive anymore there's nothing to clean. - try { process.kill(legacyPid, 0); } catch { return false; } - - deleteAllBotmuxProcesses(legacyHome); + // The pidfile is neither necessary nor sufficient authority: upgrades can + // leave an actual legacy God alive after that file is removed/replaced. + const legacyGodPids = listPm2GodDaemonPids(legacyHome); + if (legacyGodPids.length === 0) return false; + assertNoDuplicatePm2GodDaemons(legacyHome); + preflightNodeSanity(legacyHome); + + // A current dedicated-home daemon and a legacy-home daemon share the same + // descriptor directory. Union both verified PM2 views before judging a + // descriptor unregistered, then retire only the legacy rows. + assertNoDuplicatePm2GodDaemons(legacyHome); + if (bootstrapOperation) bootstrapDeleteAllBotmuxProcesses(bootstrapOperation, legacyHome); + else { + const currentProjection = readVerifiedBotmuxPm2Projection('legacy-cleanup-authority'); + deleteAllBotmuxProcesses(legacyHome, currentProjection); + } return true; } async function cmdStop(): Promise { const includePluginServices = process.argv.includes('--with-plugin'); - killDuplicatePm2GodDaemons(); - cleanupLegacyPm2(); - let stopped = false; + const bootstrapShutdownProtocol = process.argv.includes('--bootstrap-shutdown-protocol'); + const bootstrapConfirmed = process.argv.includes('--yes'); + if (bootstrapShutdownProtocol && !bootstrapConfirmed) { + throw new Error( + '[stop] --bootstrap-shutdown-protocol requires --yes after confirming every Session/Riff workload is idle', + ); + } + ensureConfigDir(); + await withFileLock(PM2_FLEET_MUTATION_LOCK_TARGET, async () => { + assertNoDuplicatePm2GodDaemons(); + cleanupLegacyPm2(bootstrapShutdownProtocol ? 'stop' : undefined); + if (bootstrapShutdownProtocol) { + bootstrapDeleteAllBotmuxProcesses('stop'); + cleanupStaleDaemonDescriptors(); + if (includePluginServices) await stopPluginServicesForCli(undefined, { autoOnly: true }); + console.log('daemon 已通过一次性 shutdown-protocol bootstrap 安全边界停止。'); + return; + } + let entries: BotmuxPm2ProcessEntry[] = []; try { const output = pm2Capture(['jlist']); - const apps = parsePm2JlistOutput(output); - for (const app of apps) { - if (isBotmuxCoreProcessName(String(app.name))) { - try { runPm2(['stop', app.name]); stopped = true; } catch { /* */ } - } + const apps = parsePm2JlistOutputStrict(output); + entries = (Array.isArray(apps) ? apps : []) + .filter(app => app && isBotmuxCoreProcessName(String(app.name))) + .map(toBotmuxPm2ProcessEntry); + } catch (err) { + throw new Error( + `[stop] pm2 jlist failed; refusing an unverified stop: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + } + assertNoUnregisteredLiveDaemonDescriptors('stop', entries); + assertCanonicalUniquePm2Rows('stop', entries); + if (entries.length === 0) { + cleanupStaleDaemonDescriptors(); + if (includePluginServices) await stopPluginServicesForCli(undefined, { autoOnly: true }); + console.log('daemon 未在运行。'); + return; + } + signalAndAwaitBotmuxProcesses(entries, 'stop'); + const beforeStop = readVerifiedBotmuxPm2Projection('stop-before-registry-mutation'); + assertNoUnregisteredLiveDaemonDescriptors('stop-before-registry-mutation', beforeStop); + exactQuiescentRowsForMutation( + 'stop-before-registry-mutation', + entries, + beforeStop, + ); + // Processes have already exited through their own safe path. These PM2 calls + // only preserve `stop`'s registry semantics; they can no longer interrupt a + // Riff drain or trigger PM2's force-kill timer. + const stopErrors: string[] = []; + for (const entry of entries) { + try { + const exact = revalidateExactQuiescentRowBeforeMutation( + 'stop-immediately-before-registry-mutation', + entry, + entries, + ); + if (!exact) continue; + runPm2(['stop', String(exact.pmId)], false, PM2_HOME, 10_000); } - } catch { /* */ } + catch (err) { + const message = err instanceof Error ? err.message : String(err); + stopErrors.push(`${entry.name}: ${message}`); + console.error(`[stop] pm2 stop ${entry.name} failed: ${message}`); + } + } + let nonStoppedResidual: string[]; + try { + const fresh = parsePm2JlistOutputStrict(pm2Capture(['jlist'])); + const targetNames = new Set(entries.map(entry => entry.name)); + const freshProjection = (Array.isArray(fresh) ? fresh : []) + .filter(app => app && isBotmuxCoreProcessName(String(app.name))) + .map(toBotmuxPm2ProcessEntry); + assertNoUnregisteredLiveDaemonDescriptors('stop-after-registry-mutation', freshProjection); + exactQuiescentRowsForMutation('stop-after-registry-mutation', entries, freshProjection); + nonStoppedResidual = freshProjection + .filter(entry => targetNames.has(entry.name) && entry.status !== 'stopped') + .map(entry => `${entry.name}:${entry.status ?? 'unknown'}`); + } catch (err) { + throw new Error( + `[stop] PM2 stop verification failed: ${err instanceof Error ? err.message : String(err)}`, + ); + } + if (stopErrors.length > 0 || nonStoppedResidual.length > 0) { + throw new Error( + `[stop] PM2 registry mutation incomplete` + + (nonStoppedResidual.length > 0 + ? ` (not stopped: ${[...new Set(nonStoppedResidual)].join(', ')})` + : '') + + (stopErrors.length > 0 ? ` (errors: ${stopErrors.join('; ')})` : ''), + ); + } // Wipe abandoned dashboard-daemon descriptors left behind by stopped daemons. cleanupStaleDaemonDescriptors(); if (includePluginServices) await stopPluginServicesForCli(undefined, { autoOnly: true }); - if (!stopped) console.log('daemon 未在运行。'); + }, { maxWaitMs: 5_000 }); } async function cmdRestart(): Promise { @@ -2483,31 +3402,107 @@ async function cmdRestart(): Promise { process.exit(1); } ensureConfigDir(); + await withFileLock(PM2_FLEET_MUTATION_LOCK_TARGET, async () => { const includePm2 = process.argv.includes('--include-pm2'); const includePluginServices = process.argv.includes('--with-plugin'); - // Drop a restart-intent breadcrumb so the fresh daemon knows this was an - // intentional restart and DMs the owner a summary. `IfAbsent` preserves a - // richer breadcrumb (update / auto-restart) already written by the - // maintenance timer that spawned this very `botmux restart`. A pm2 - // crash-autorestart bypasses this path → no breadcrumb → silent. + const bootstrapShutdownProtocol = process.argv.includes('--bootstrap-shutdown-protocol'); + const bootstrapConfirmed = process.argv.includes('--yes'); + if (bootstrapShutdownProtocol && !bootstrapConfirmed) { + throw new Error( + '[restart] --bootstrap-shutdown-protocol requires --yes after confirming every Session/Riff workload is idle', + ); + } + if (bootstrapShutdownProtocol && includePm2) { + throw new Error('[restart] --bootstrap-shutdown-protocol cannot be combined with --include-pm2'); + } + // Admission precedes breadcrumb consumption, dependency repair, any PM2 RPC, + // and every daemon signal. If an old God exists we cannot bind a signal to + // its generation, so the whole command is a guaranteed zero-fleet-mutation + // refusal rather than a half-completed restart. + if (includePm2) { + assertIncludePm2RestartAdmission(listPm2GodDaemonPids()); + } + const restartIntentDir = resolveDataDir(); + let stagedRestartIntent: RestartIntent | null = null; try { - const now = Date.now(); - writeManualIntentIfAbsentTo(resolveDataDir(), now, new Date(now).toISOString()); - } catch { /* breadcrumb is best-effort */ } - killDuplicatePm2GodDaemons(); + // Maintenance/dashboard update flows write their richer intent before + // spawning this CLI. Take it out of circulation while preflight/drain runs: + // any refusal then leaves no breadcrumb for an unrelated later crash. + stagedRestartIntent = consumeRestartIntentTo(restartIntentDir, Date.now()); + } catch { /* intent reporting is best-effort */ } + assertNoDuplicatePm2GodDaemons(); preflightNodeSanity(); await ensureSystemDependencies(); - const cfg = ecosystemConfig(); - cleanupLegacyPm2(); + cleanupLegacyPm2(bootstrapShutdownProtocol ? 'restart' : undefined); // Delete all botmux processes (handles both old single-process and new multi-process) - deleteAllBotmuxProcesses(); + if (bootstrapShutdownProtocol) bootstrapDeleteAllBotmuxProcesses('restart'); + else deleteAllBotmuxProcesses(); if (includePluginServices) await stopPluginServicesForCli(undefined, { autoOnly: true }); - if (includePm2) { - killPm2GodDaemon(); - } // Wipe abandoned dashboard-daemon descriptors left behind by killed daemons. cleanupStaleDaemonDescriptors(); - runPm2(['start', cfg]); + const retiredProjection = readVerifiedBotmuxPm2Projection('restart-start'); + assertNoUnregisteredLiveDaemonDescriptors('restart-start', retiredProjection); + if (retiredProjection.length > 0) { + throw new Error( + `[restart-start] new PM2 core row(s) appeared after verified retirement: ` + + retiredProjection.map(entry => `${entry.name}:${entry.pid}`).join(', '), + ); + } + await withFileLock(BOTS_JSON_FILE, async () => { + const restartBots = loadBotsJson(); + const cfg = ecosystemConfig(restartBots); + // Publish intent only after the old fleet is safely retired and every + // independent live-daemon descriptor is accounted for. A graceful refusal + // before this point must never poison a later crash-autorestart report. + // `IfAbsent` preserves a richer maintenance-written update breadcrumb. + const restartAttemptId = randomBytes(16).toString('hex'); + let restartIntentPrepared = false; + try { + const now = Date.now(); + writeRestartAttemptIntentTo( + restartIntentDir, + stagedRestartIntent ?? { kind: 'manual', at: new Date(now).toISOString() }, + now, + restartAttemptId, + ); + restartIntentPrepared = true; + } catch { /* breadcrumb is best-effort */ } + try { + const configuredNames = configuredCoreProcessNames(restartBots); + const verifyTimeoutMs = pm2StartVerifyTimeoutMs(configuredNames.length); + runBoundedPm2StartTransaction( + 'restart-start', + PM2_START_COMMAND_TIMEOUT_MS, + verifyTimeoutMs, + { + start: timeoutMs => { + assertBotsConfigSnapshotUnchanged('restart-start', restartBots); + assertNoDuplicatePm2GodDaemons(); + preflightNodeSanity(); + runPm2(['start', cfg], true, PM2_HOME, timeoutMs); + }, + verifyFresh: timeoutMs => readAndAssertConfiguredFleetOnline( + 'restart-after-launch', configuredNames, PM2_HOME, timeoutMs, + ), + rollback: () => rollbackPm2StartAttempt( + 'restart-start', retiredProjection, configuredNames, + ), + }, + ); + } catch (err) { + try { removeRestartIntentAttemptTo(restartIntentDir, restartAttemptId); } catch { /* best-effort */ } + throw err; + } + if (restartIntentPrepared) { + let committed = false; + try { committed = commitRestartIntentAttemptTo(restartIntentDir, restartAttemptId); } + catch { /* report breadcrumb remains best-effort after a verified healthy fleet */ } + if (!committed) { + try { removeRestartIntentAttemptTo(restartIntentDir, restartAttemptId); } catch { /* best-effort */ } + console.warn('⚠️ daemon 已完整启动，但重启摘要凭据未能提交；本次不会发送重启摘要。'); + } + } + }, { maxWaitMs: 5_000 }); // Default restart preserves running plugin services, then ensures every auto // service is online. --with-plugin changes only the pre-restart side above: // it explicitly stops auto services first, so this ensure becomes a restart. @@ -2516,21 +3511,7 @@ async function cmdRestart(): Promise { console.log(`autostart unit 已同步到当前 Node/cli.js 路径`); } await printDashboardHintWithRetry(); -} - -/** - * pm2 process list filtered to botmux entries (bot daemons + dashboard). Returns - * `[]` when pm2 isn't running or has no botmux apps at all. - */ -function listBotmuxPm2Apps(): Array<{ name: string; online: boolean }> { - try { - const apps = parsePm2JlistOutput(pm2Capture(['jlist'])); - return (Array.isArray(apps) ? apps : []) - .filter(a => a && (a.name === PM2_NAME || String(a.name).startsWith(`${PM2_NAME}-`))) - .map(a => ({ name: String(a.name), online: a?.pm2_env?.status === 'online' })); - } catch { - return []; - } + }, { maxWaitMs: 5_000 }); } export type StartBotLiveResult = @@ -2556,31 +3537,81 @@ export type StartBotLiveResult = * NOT start a lone bot; that case belongs to `botmux start`, which brings up the * entire ecosystem (all bots + dashboard). */ -function ensureBotDaemonStarted(appId: string, opts: { quiet?: boolean } = {}): StartBotLiveResult { +async function ensureBotDaemonStarted( + appId: string, + opts: { quiet?: boolean } = {}, +): Promise { + ensureConfigDir(); + try { + return await withFileLock(PM2_FLEET_MUTATION_LOCK_TARGET, async () => { + return await withFileLock(BOTS_JSON_FILE, async () => { + assertNoDuplicatePm2GodDaemons(); + preflightNodeSanity(); + cleanupLegacyPm2(); const bots = loadBotsJson(); const index = bots.findIndex(b => b?.larkAppId === appId); if (index < 0) { return { ok: false, reason: 'not_found', message: `appId ${appId} 不在 bots.json 中` }; } const processName = botProcessName(bots[index], index, PM2_NAME); - - const running = listBotmuxPm2Apps(); - if (running.length === 0) { - return { ok: false, reason: 'fleet_down', message: 'daemon 未在运行，请先 botmux start' }; - } - if (running.some(a => a.name === processName && a.online)) { + const configuredNames = configuredCoreProcessNames(bots); + const verifyTimeoutMs = pm2StartVerifyTimeoutMs(configuredNames.length); + + const running = readVerifiedBotmuxPm2Projection('start-bot'); + assertNoUnregisteredLiveDaemonDescriptors('start-bot', running); + const admission = classifyStartBotFleetAdmission( + 'start-bot', + running, + configuredNames, + processName, + pid => { try { process.kill(pid, 0); return true; } catch { return false; } }, + ); + if (admission.state === 'already-online') { + readAndAssertConfiguredFleetOnline( + 'start-bot-already-online-ready', + configuredNames, + PM2_HOME, + verifyTimeoutMs, + ); return { ok: true, state: 'already-online', processName }; } + if (admission.state === 'fleet-down') { + return { ok: false, reason: 'fleet_down', message: 'daemon 未在运行，请先 botmux start' }; + } - const cfg = ecosystemConfig(); - try { - // `--only ` filters the ecosystem to just this app, so pm2 starts only - // the new bot's daemon and never restarts the already-online ones. - runPm2(['start', cfg, '--only', processName], !opts.quiet); + const cfg = ecosystemConfig(bots); + runBoundedPm2StartTransaction( + 'start-bot', + PM2_START_COMMAND_TIMEOUT_MS, + verifyTimeoutMs, + { + start: timeoutMs => { + // `--only ` filters the ecosystem to just this app, so PM2 starts + // only the exact missing row and never restarts an existing peer. + assertBotsConfigSnapshotUnchanged('start-bot', bots); + assertNoDuplicatePm2GodDaemons(); + preflightNodeSanity(); + runPm2( + ['start', cfg, '--only', processName], + !opts.quiet, + PM2_HOME, + timeoutMs, + ); + }, + verifyFresh: timeoutMs => readAndAssertConfiguredFleetOnline( + 'start-bot-after-launch', configuredNames, PM2_HOME, timeoutMs, + ), + rollback: () => rollbackPm2StartAttempt( + 'start-bot', running, [processName], + ), + }, + ); + return { ok: true, state: 'started', processName }; + }, { maxWaitMs: 5_000 }); + }, { maxWaitMs: 5_000 }); } catch (e) { return { ok: false, reason: 'pm2_error', message: e instanceof Error ? e.message : String(e) }; } - return { ok: true, state: 'started', processName }; } /** @@ -2599,7 +3630,7 @@ async function cmdStartBot(argv: string[]): Promise { process.exit(1); } ensureConfigDir(); - const r = ensureBotDaemonStarted(appId, { quiet: wantsJson }); + const r = await ensureBotDaemonStarted(appId, { quiet: wantsJson }); if (wantsJson) { console.log(JSON.stringify(r, null, 2)); if (!r.ok) process.exitCode = 1; @@ -2621,8 +3652,8 @@ async function cmdStartBot(argv: string[]): Promise { /** Print the post-add "next step" line for interactive setup: auto-start the new * bot's own daemon when the fleet is up (no fleet-wide restart), else fall back * to the botmux start / restart hint. */ -function printAddBotLiveHint(appId: string): void { - const live = ensureBotDaemonStarted(appId); +async function printAddBotLiveHint(appId: string): Promise { + const live = await ensureBotDaemonStarted(appId); if (live.ok) { console.log(`✅ 已自动上线（${live.processName}），无需重启其它机器人。\n`); } else if (live.reason === 'fleet_down') { @@ -2699,7 +3730,6 @@ function warnIfLegacyBotmuxAlive(): void { } function cmdLogs(): void { - killDuplicatePm2GodDaemons(); warnIfLegacyBotmuxAlive(); const lines = process.argv.includes('--lines') ? process.argv[process.argv.indexOf('--lines') + 1] || '50' @@ -2741,7 +3771,6 @@ function cmdLogs(): void { } function cmdStatus(): void { - killDuplicatePm2GodDaemons(); warnIfLegacyBotmuxAlive(); runPm2(['status']); } @@ -2891,6 +3920,10 @@ interface SessionData { pid?: number; workingDir?: string; webPort?: number; + /** Backend frozen by the owning daemon at first spawn. */ + backendType?: BackendType; + /** Persisted Riff task lineage; retained until remote cancellation confirms. */ + riffParentTaskId?: string; larkAppId?: string; ownerOpenId?: string; creatorOpenId?: string; @@ -2898,8 +3931,13 @@ interface SessionData { /** Chat-scope quote chain — see Session.quoteTargetId in types.ts. */ quoteTargetId?: string; currentReplyTarget?: { rootMessageId: string; turnId: string; updatedAt: string; quoteOnly?: boolean; substitute?: boolean }; - /** Per-turn reply targets（见 Session.replyTargets in types.ts）——排队/并发轮次各自的回复锚点。 */ replyTargets?: Record; + codexAppDispatchLedger?: CodexAppDispatchLedgerEntry[]; + codexAppGenerationCommits?: unknown; + queued?: boolean; + queuedActivationPending?: boolean; + queuedActivationTail?: import('./types.js').QueuedActivationTailEntry[]; + pendingRepoSetup?: import('./types.js').PendingRepoSetup; /** 文档评论入口 per-turn 回复落点（见 Session.docCommentTargets in types.ts）。 */ docCommentTargets?: Record; quoteTargetSenderOpenId?: string; @@ -2966,85 +4004,274 @@ function loadSessions(): Map { } } catch { /* ignore */ } - // Migrate: remove sessions from legacy file if they have larkAppId (belong in per-bot files) - let legacyDirty = false; - for (const [k, v] of Object.entries(legacyData)) { - const s = v as SessionData; - if (s.larkAppId) { - delete legacyData[k]; - legacyDirty = true; - // Ensure the session exists in its per-bot file - const perBotFp = join(dataDir, `sessions-${s.larkAppId}.json`); - let perBotData: Record = {}; - if (existsSync(perBotFp)) { - try { perBotData = JSON.parse(readFileSync(perBotFp, 'utf-8')); } catch { /* */ } - } - // Only write if per-bot file doesn't already have this session - if (!perBotData[k]) { - perBotData[k] = s; - const tmpFp = perBotFp + '.tmp'; - writeFileSync(tmpFp, JSON.stringify(perBotData, null, 2), 'utf-8'); - renameSync(tmpFp, perBotFp); - } - } - } - if (legacyDirty) { - const tmpFp = legacyFp + '.tmp'; - writeFileSync(tmpFp, JSON.stringify(legacyData, null, 2), 'utf-8'); - renameSync(tmpFp, legacyFp); - } + // Deliberately read-only. Older CLIs opportunistically migrated legacy rows + // here, which made even `botmux list` a whole-file writer capable of racing a + // daemon ledger save. Durable migration belongs to daemon startup under the + // session-store lock; this generic discovery path only composes snapshots. return sessions; } -/** Save a single session back to its appropriate file based on larkAppId. */ -function loadSessionFresh(session: SessionData): SessionData | undefined { - return loadSessions().get(session.sessionId); -} - -function saveSession(session: SessionData): void { +/** Offline-only narrow session mutation. Callers must prefer the owning daemon + * while it is available; rereading the exact row here prevents stale CLI + * snapshots from copying FIFO fields during offline maintenance. */ +function mutateSessionOffline( + session: SessionData, + mutate: (current: SessionData) => boolean, +): SessionData | undefined { const dataDir = resolveDataDir(); const fileName = session.larkAppId ? `sessions-${session.larkAppId}.json` : 'sessions.json'; const fp = join(dataDir, fileName); + return withFileLockSync(fp, () => { + // The owning daemon uses the same session-file lock for every save. Check + // absence while holding it so two CLIs cannot race each other and an + // already-published daemon cannot be bypassed by the offline fallback. + if (session.larkAppId) { + try { if (findDaemon(session.larkAppId)) return undefined; } catch { /* offline */ } + } + let data: Record = {}; + if (existsSync(fp)) { + try { data = JSON.parse(readFileSync(fp, 'utf-8')); } catch { /* start fresh */ } + } + const current = data[session.sessionId]; + if (!current || !mutate(current)) return current; + data[session.sessionId] = current; + + // Clean up entries where file key doesn't match the entry's sessionId + // (data corruption), and strip removed placeholder-card fields. + for (const [key, val] of Object.entries(data)) { + if (val && typeof val === 'object' && 'sessionId' in val && (val as SessionData).sessionId !== key) { + delete data[key]; + continue; + } + if (val && typeof val === 'object') stripLegacyPendingCardFields(val as unknown as Record); + } + + // Recheck immediately before publication. A daemon that appears during + // the read/decision phase becomes authoritative; leave the file untouched. + if (session.larkAppId) { + try { if (findDaemon(session.larkAppId)) return undefined; } catch { /* offline */ } + } + const tmpFp = `${fp}.${process.pid}.${randomBytes(4).toString('hex')}.tmp`; + writeFileSync(tmpFp, JSON.stringify(data, null, 2), 'utf-8'); + renameSync(tmpFp, fp); + return current; + }); +} + +type OfflineAbandonResult = + | { ok: true; current: SessionData; cleanedBacking?: string } + | { ok: false; error: string }; + +/** Offline explicit abandon is a three-phase operation: read the exact row, + * stop and confirm the Botmux-owned worker first, re-read its final lineage, + * then destroy/cancel that exact backing resource before a final CAS close. + * Cleanup failure leaves an active, retryable worker-less row; a failed Riff + * cancellation never erases the task id. */ +async function abandonSessionOffline(session: SessionData): Promise { + let current = mutateSessionOffline(session, () => false); + if (!current) return { ok: false, error: 'owning_daemon_became_available' }; + + const originalPid = adoptedCliPid(current); + const ownedWorkerPid = current.pid && current.pid !== originalPid ? current.pid : undefined; + if (ownedWorkerPid) { + // Narrow the unavoidable descriptor race: do not signal a worker after an + // owning daemon has become visible. The locked write below repeats this. + if (current.larkAppId) { + try { + if (findDaemon(current.larkAppId)) { + return { ok: false, error: 'owning_daemon_became_available' }; + } + } catch { /* still offline */ } + } + if (isProcessAlive(ownedWorkerPid)) { + const signalled = killProcess(ownedWorkerPid); + if ((!signalled && isProcessAlive(ownedWorkerPid)) + || !(await waitForProcessExit(ownedWorkerPid))) { + return { ok: false, error: `worker_pid_kill_failed:${ownedWorkerPid}` }; + } + } - // Read current file, update session, write back - let data: Record = {}; - if (existsSync(fp)) { - try { data = JSON.parse(readFileSync(fp, 'utf-8')); } catch { /* start fresh */ } + // Persist worker-less state without touching FIFO/task authority. If a new + // daemon/generation changed the row while SIGTERM settled, fail before any + // provider or persistent-pane destruction. + let workerCleared = false; + const afterStop = mutateSessionOffline(current, latest => { + if (latest.pid !== ownedWorkerPid + || latest.backendType !== current!.backendType + || isAdoptedSession(latest) !== isAdoptedSession(current!)) return false; + delete latest.pid; + workerCleared = true; + return true; + }); + if (!afterStop || !workerCleared) { + return { ok: false, error: 'session_changed_while_stopping_worker' }; + } + current = afterStop; + } else { + // Even without a Botmux worker pid, re-read after the first authority check + // so the cleanup inputs are the newest locked backend/task lineage. + const refreshed = mutateSessionOffline(current, () => false); + if (!refreshed) return { ok: false, error: 'owning_daemon_became_available' }; + current = refreshed; } - data[session.sessionId] = session; - // Clean up entries where file key doesn't match the entry's sessionId (data - // corruption), and strip legacy placeholder-card fields so the file converges - // to clean (see stripLegacyPendingCardFields in services/session-store). - for (const [key, val] of Object.entries(data)) { - if (val && typeof val === 'object' && 'sessionId' in val && (val as SessionData).sessionId !== key) { - delete data[key]; - continue; + let riffConfig; + if (current.backendType === 'riff' && current.riffParentTaskId) { + try { + const { loadBotConfigs } = await import('./bot-registry.js'); + riffConfig = loadBotConfigs().find(cfg => cfg.larkAppId === current!.larkAppId)?.riff; + } catch (err) { + return { + ok: false, + error: `riff_config_unavailable: ${err instanceof Error ? err.message : String(err)}`, + }; } - if (val && typeof val === 'object') stripLegacyPendingCardFields(val as unknown as Record); } - const tmpFp = fp + '.tmp'; - writeFileSync(tmpFp, JSON.stringify(data, null, 2), 'utf-8'); - renameSync(tmpFp, fp); + // Config lookup may involve disk I/O. Re-check authority immediately before + // the destructive provider/backend call; the final locked publication below + // checks once more and refuses if a daemon appeared during cleanup. + if (current.larkAppId) { + try { + if (findDaemon(current.larkAppId)) { + return { ok: false, error: 'owning_daemon_became_available' }; + } + } catch { /* still offline */ } + } + + const cleanup = await cleanupExplicitSessionBacking({ + sessionId: current.sessionId, + backendType: current.backendType, + riffParentTaskId: current.riffParentTaskId, + adopted: isAdoptedSession(current), + riffConfig, + }); + if (!cleanup.ok) { + return { ok: false, error: cleanup.kind }; + } + + let applied = false; + const published = mutateSessionOffline(current, latest => { + // Another offline actor or a newly-published daemon may have changed which + // resource this row owns while cleanup was in flight. Never close that new + // generation using an old cancellation/kill result. + if (latest.backendType !== current.backendType + || isAdoptedSession(latest) !== isAdoptedSession(current)) return false; + if (cleanup.kind === 'cancelled_riff' + && latest.riffParentTaskId !== cleanup.taskId) return false; + if (latest.status === 'closed') { + applied = true; + return false; + } + if (cleanup.kind === 'cancelled_riff') delete latest.riffParentTaskId; + latest.status = 'closed'; + latest.closedAt = new Date().toISOString(); + delete latest.codexAppDispatchLedger; + delete latest.codexAppGenerationCommits; + applied = true; + return true; + }); + if (!published || !applied) { + return { ok: false, error: 'session_changed_during_offline_cleanup' }; + } - // Remove duplicate from legacy file if session moved to per-bot file (or vice versa) - const otherFile = session.larkAppId ? 'sessions.json' : null; - if (otherFile) { - const otherFp = join(dataDir, otherFile); - if (existsSync(otherFp)) { - try { - const otherData: Record = JSON.parse(readFileSync(otherFp, 'utf-8')); - if (otherData[session.sessionId]) { - delete otherData[session.sessionId]; - const otherTmp = otherFp + '.tmp'; - writeFileSync(otherTmp, JSON.stringify(otherData, null, 2), 'utf-8'); - renameSync(otherTmp, otherFp); - } - } catch { /* ignore */ } - } + const cleanedBacking = cleanup.kind === 'destroyed_persistent' + ? `${cleanup.backendType} ${cleanup.name}` + : cleanup.kind === 'cancelled_riff' + ? `riff task ${cleanup.taskId}` + : undefined; + return { ok: true, current: published, cleanedBacking }; +} + +function pruneSessionOfflineIfLedgerEmpty(session: SessionData): boolean { + let pruned = false; + mutateSessionOffline(session, current => { + if (hasProtectedSessionMutationOwnership(current)) return false; + current.status = 'closed'; + current.closedAt = new Date().toISOString(); + delete current.codexAppDispatchLedger; + delete current.codexAppGenerationCommits; + pruned = true; + return true; + }); + return pruned; +} + +function patchSessionWhiteboardOffline(session: SessionData, whiteboardId: string): boolean { + return !!mutateSessionOffline(session, current => { + current.whiteboardId = whiteboardId; + return true; + }); +} + +async function postOwningDaemonSessionMutation( + session: SessionData, + suffix: 'close' | 'prune' | 'whiteboard', + body?: Record, +): Promise<'applied' | 'refused' | 'unavailable'> { + if (!session.larkAppId) return 'unavailable'; + let daemon: ReturnType; + try { daemon = findDaemon(session.larkAppId); } catch { return 'unavailable'; } + if (!daemon) return 'unavailable'; + let secret: string; + try { secret = loadDaemonIpcSecret(); } catch { return 'unavailable'; } + const res = await fetchDaemonIpc( + daemon.ipcPort, + `/api/sessions/${encodeURIComponent(session.sessionId)}/${suffix}`, + { + method: 'POST', + ...(body + ? { + headers: { 'content-type': 'application/json' }, + body: JSON.stringify(body), + } + : {}), + }, + secret, + ); + if (suffix === 'prune' && res.status === 409) return 'refused'; + if (!res.ok) { + throw new Error(`owning daemon ${suffix} mutation failed: HTTP ${res.status}`); } + return 'applied'; +} + +type AuthoritativeAbandonResult = + | { ok: true; mode: 'daemon' } + | { ok: true; mode: 'offline'; current: SessionData; cleanedBacking?: string } + | { ok: false; error?: string }; + +async function abandonSessionAuthoritatively(session: SessionData): Promise { + const result = await postOwningDaemonSessionMutation(session, 'close'); + if (result === 'applied') return { ok: true, mode: 'daemon' }; + if (result === 'refused') return { ok: false }; + const offline = await abandonSessionOffline(session); + return offline.ok + ? { + ok: true, + mode: 'offline', + current: offline.current, + ...(offline.cleanedBacking ? { cleanedBacking: offline.cleanedBacking } : {}), + } + : { ok: false, error: offline.error }; +} + +async function pruneSessionAuthoritatively(session: SessionData): Promise { + const result = await postOwningDaemonSessionMutation(session, 'prune'); + if (result === 'applied') return true; + if (result === 'refused') return false; + return pruneSessionOfflineIfLedgerEmpty(session); +} + +async function patchSessionWhiteboardAuthoritatively( + session: SessionData, + whiteboardId: string, +): Promise { + const result = await postOwningDaemonSessionMutation(session, 'whiteboard', { whiteboardId }); + if (result === 'applied') return true; + if (result === 'refused') return false; + return patchSessionWhiteboardOffline(session, whiteboardId); } function isProcessAlive(pid: number): boolean { @@ -3065,6 +4292,15 @@ function killProcess(pid: number): boolean { } } +async function waitForProcessExit(pid: number, timeoutMs = 2_000): Promise { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + if (!isProcessAlive(pid)) return true; + await new Promise(resolve => setTimeout(resolve, 50)); + } + return !isProcessAlive(pid); +} + function formatDuration(ms: number): string { const seconds = Math.floor(ms / 1000); if (seconds < 60) return `${seconds}s`; @@ -3440,26 +4676,21 @@ function interactiveSessionPicker(active: SessionData[]): Promise { process.stdout.write('\x1b[?1049l'); // leave alt screen } - function deleteSession(idx: number): void { + async function deleteSession(idx: number): Promise { const r = rows[idx]; const s = r.session; - // Kill botmux's worker process. For adopted sessions, never kill the - // user's original CLI pid if an old record stored it in `pid`. - const originalPid = adoptedCliPid(s); - if (s.pid && s.pid !== originalPid && isProcessAlive(s.pid)) { - killProcess(s.pid); - } - - // Kill only botmux-owned tmux sessions. Adopted panes belong to the user. - if (!r.isAdopt && r.hasTmux) { - try { execSync(`tmux kill-session -t '${r.tmuxName}' 2>/dev/null`, { stdio: 'ignore', env: tmuxEnv() }); } catch { /* */ } + // Explicit delete is the abandon boundary. Prefer the owning daemon so + // its current in-memory FIFO is cleared atomically with close; offline + // fallback rereads the exact row and clears the latest ledger/commits. + const result = await abandonSessionAuthoritatively(s); + if (!result.ok) { + flashMsg = `\x1b[31m删除失败 ${s.sessionId.substring(0, 8)}${result.error ? `：${result.error}` : ''}；会话保持活跃，可重试\x1b[0m`; + return; } - - // Mark closed & persist - s.status = 'closed'; - s.closedAt = new Date().toISOString(); - saveSession(s); + // A live daemon owns process/pane teardown and has re-resolved the latest + // generation. Only the locked offline path may clean up resources, using + // the freshly-read row rather than this TUI's stale snapshot. // Remove from active list and TUI rows const activeIdx = active.indexOf(s); @@ -3470,12 +4701,12 @@ function interactiveSessionPicker(active: SessionData[]): Promise { flashMsg = `\x1b[32m✓ 已删除 ${s.sessionId.substring(0, 8)}\x1b[0m`; } - process.stdin.on('data', (key: string) => { + process.stdin.on('data', async (key: string) => { // Delete confirmation mode if (confirmDelete) { confirmDelete = false; if (key === 'y' || key === 'Y') { - deleteSession(cursor); + await deleteSession(cursor); } else { flashMsg = '\x1b[2m取消删除\x1b[0m'; } @@ -3581,18 +4812,26 @@ async function cmdList(): Promise { else if (disposition === 'prune_scratch') prunedScratch.push(s); else live.push(s); } - const closeNow = (arr: SessionData[]) => { + const closeNow = async (arr: SessionData[]): Promise => { + let closed = 0; for (const s of arr) { - s.status = 'closed'; - s.closedAt = new Date().toISOString(); - saveSession(s); + if (await pruneSessionAuthoritatively(s)) { + closed++; + } else { + // The owning daemon observed a newly unsettled FIFO after this CLI's + // liveness snapshot. Keep the owner visible instead of abandoning it. + live.push(s); + } } + return closed; }; // Scratches: close silently — they were placeholders, not dead sessions. - closeNow(prunedScratch); + await closeNow(prunedScratch); if (pruned.length > 0) { - closeNow(pruned); - console.log(`已自动清理 ${pruned.length} 个不可恢复的会话（进程已退出或无可恢复后端）`); + const prunedCount = await closeNow(pruned); + if (prunedCount > 0) { + console.log(`已自动清理 ${prunedCount} 个不可恢复的会话（进程已退出或无可恢复后端）`); + } } // Sort by creation time, newest first @@ -3613,7 +4852,7 @@ async function cmdList(): Promise { await interactiveSessionPicker(live); } -function cmdDelete(): void { +async function cmdDelete(): Promise { const target = process.argv[3]; if (!target) { console.error('用法: botmux delete '); @@ -3664,29 +4903,13 @@ function cmdDelete(): void { } for (const s of toDelete) { - const originalPid = adoptedCliPid(s); - - // Kill botmux's worker process if running. For adopted sessions, never - // kill the user's original CLI pid. - if (s.pid && s.pid !== originalPid && isProcessAlive(s.pid)) { - killProcess(s.pid); - console.log(` killed pid ${s.pid}`); + const result = await abandonSessionAuthoritatively(s); + if (!result.ok) { + throw new Error(`failed to close session ${s.sessionId}${result.error ? `: ${result.error}` : ''}`); } - - // Kill associated botmux-owned tmux session if it exists. Adopted panes - // belong to the user and must be left untouched. - const tmuxName = `bmx-${s.sessionId.substring(0, 8)}`; - if (!isAdoptedSession(s)) { - try { - execSync(`tmux kill-session -t '${tmuxName}' 2>/dev/null`, { stdio: 'ignore', env: tmuxEnv() }); - console.log(` killed tmux ${tmuxName}`); - } catch { /* no tmux session */ } + if (result.mode === 'offline') { + if (result.cleanedBacking) console.log(` killed ${result.cleanedBacking}`); } - - // Mark session as closed - s.status = 'closed'; - s.closedAt = new Date().toISOString(); - saveSession(s); console.log(`✓ ${s.sessionId.substring(0, 8)} ${s.title}`); } console.log(`\n已关闭 ${toDelete.length} 个会话`); @@ -4444,7 +5667,10 @@ botmux v${getVersion()} — IM ↔ AI 编程 CLI 桥接 默认使用 botmux 内置 Feishu Web QR 登录尝试自动导入权限/redirect/发布版本；可加 --no-open-platform-auto 跳过 start 启动 daemon，并启动 mode=auto 的插件 service stop 停止 daemon（默认不停止插件 service；--with-plugin 显式停止 mode=auto 的插件 service） - restart 重启 daemon（默认不停止插件 service，core 启动后确保 mode=auto 正在运行；--with-plugin 显式先停再启动 auto service；--include-pm2 同时重启 PM2 God） + restart 重启 daemon（默认不停止插件 service，core 启动后确保 mode=auto 正在运行；--with-plugin 显式先停再启动 auto service） + --include-pm2 仅允许“入场时没有 live PM2 God”的干净启动；若已有 live God，整条命令会在 fleet/breadcrumb 零改动处拒绝，且不会信号或重启现存 God + 首次升级若旧 daemon 缺少 shutdown protocol：先独立确认所有 Session/Riff 工作均 idle，再一次性运行 + botmux restart --bootstrap-shutdown-protocol --yes；普通 stop/restart 仍保持 fail-closed logs 查看 daemon 日志（--lines N, --bot <0-based-index|pm2-name|appId>） status 查看 daemon 状态 upgrade 升级到最新版本（别名：update） @@ -4830,7 +6056,10 @@ Context flags: --session-id, --lark-app-id, --chat-id, --working-dir/--repo`); let meta = ctx.session?.whiteboardId ? getWhiteboard(ctx.session.whiteboardId) : undefined; if (!meta && argFlag(rest, '--create')) { meta = ensureDefaultWhiteboard({ larkAppId: ctx.larkAppId, chatId: ctx.chatId, workingDir: ctx.workingDir, sessionId: ctx.sessionId }); - if (ctx.session) { ctx.session.whiteboardId = meta.id; saveSession(ctx.session); } + if (ctx.session) { + await patchSessionWhiteboardAuthoritatively(ctx.session, meta.id); + ctx.session.whiteboardId = meta.id; + } } if (!meta) { console.log(JSON.stringify({ enabled: true, current: null, hint: 'Run `botmux whiteboard current --create` to ensure the default board.' }, null, 2)); @@ -4844,7 +6073,10 @@ Context flags: --session-id, --lark-app-id, --chat-id, --working-dir/--repo`); requireWhiteboardEnabled(); const ctx = currentWhiteboardContext(rest); const meta = createWhiteboard({ id: argValue(rest, '--id'), title: argValue(rest, '--title'), larkAppId: ctx.larkAppId, chatId: ctx.chatId, workingDir: ctx.workingDir, sessionId: ctx.sessionId }); - if (ctx.session && !ctx.session.whiteboardId) { ctx.session.whiteboardId = meta.id; saveSession(ctx.session); } + if (ctx.session && !ctx.session.whiteboardId) { + await patchSessionWhiteboardAuthoritatively(ctx.session, meta.id); + ctx.session.whiteboardId = meta.id; + } console.log(JSON.stringify({ board: meta, path: whiteboardPath(meta.id) }, null, 2)); return; } @@ -4865,7 +6097,10 @@ Context flags: --session-id, --lark-app-id, --chat-id, --working-dir/--repo`); if (!id && whiteboardEnabled() && action === 'update') { const meta = ensureDefaultWhiteboard({ larkAppId: ctx.larkAppId, chatId: ctx.chatId, workingDir: ctx.workingDir, sessionId: ctx.sessionId }); id = meta.id; - if (ctx.session) { ctx.session.whiteboardId = id; saveSession(ctx.session); } + if (ctx.session) { + await patchSessionWhiteboardAuthoritatively(ctx.session, id); + ctx.session.whiteboardId = id; + } } if (!id) { console.error('No whiteboard id. Pass --id or run `botmux whiteboard current --create`.'); process.exit(1); } @@ -5510,6 +6745,7 @@ async function relaySend( resolveDataDir(), sid, relayDir, + process.env.BOTMUX_ORIGIN_CHANNEL_ID, )?.capability; // Structured request: the daemon-side watcher rebuilds the argv from these // validated fields (it NEVER executes raw argv — see buildRelayHostArgs). @@ -5739,14 +6975,100 @@ async function cmdSend(rest: string[]): Promise { ); process.exit(2); } - // Managed output attribution must come from the live process-tree marker, - // whose turn/attempt is refreshed by the worker. BOTMUX_TURN_ID is a - // spawn-time fallback and can be stale in a detached child after a later - // delivery starts, so it is never authority for receiver policy. - const liveMarkerCtx = resolveSessionContext( - resolveDataDir(), - process.env.BOTMUX_SESSION_ID, - ); + // Resolve isolation marker-first. A visible host marker always wins over a + // leftover capability. Linux bwrap keeps its host-execution outbox; macOS + // read isolation instead challenges the owning daemon and trusts only the + // matching host-written read-only proof sidecar. The capability file itself + // may survive worker SIGKILL and is never direct-send authority. + let sendDataDir = resolveDataDir(); + let liveMarkerCtx = findLiveAncestorSessionContext(sendDataDir); + const relayDir = process.env.BOTMUX_SEND_RELAY; + const sessionIdArg = argValue(rest, '--session-id'); + const inheritedSessionId = process.env.BOTMUX_SESSION_ID?.trim(); + const inheritedOriginChannelId = process.env.BOTMUX_ORIGIN_CHANNEL_ID?.trim(); + const isolationSessionCandidates = [...new Set([ + inheritedSessionId, + ancestorCtx?.sessionId, + sessionIdArg, + ].filter((value): value is string => !!value))]; + const isolationMarkerPresent = isolationSessionCandidates + .some(sessionId => !!inheritedOriginChannelId + && hasManagedOriginIsolationMarker( + sendDataDir, + sessionId, + inheritedOriginChannelId, + )); + let osUserHomeDir: string; + try { osUserHomeDir = userInfo().homedir; } + catch { + console.error('botmux send refused: OS account home unavailable for isolation classification'); + process.exit(2); + } + if (!osUserHomeDir) { + console.error('botmux send refused: OS account home unavailable for isolation classification'); + process.exit(2); + } + const kernelReadIsolationDetected = managedOriginLegacyIsolationProbeAccess(osUserHomeDir) + === 'sandbox_denied' + || managedOriginIsolationSentinelAccess(osUserHomeDir) === 'sandbox_denied'; + const isolatedSendRequired = !relayDir + && (kernelReadIsolationDetected + || process.env.BOTMUX_READ_ISOLATED === '1' + || (!liveMarkerCtx?.sessionId && isolationMarkerPresent)); + let isolatedBoundSessionId: string | undefined; + if (isolatedSendRequired) { + if (!inheritedOriginChannelId || !/^[a-f0-9]{64}$/.test(inheritedOriginChannelId)) { + console.error('botmux send refused: read-isolated pane authority channel is missing or invalid'); + process.exit(2); + } + const locators = isolationSessionCandidates.flatMap(sessionId => { + const locator = readManagedOriginRootLocator(osUserHomeDir, sessionId); + return locator ? [locator] : []; + }); + if (locators.length !== 1) { + console.error('botmux send refused: read-isolated owning data-root locator is missing or ambiguous'); + process.exit(2); + } + const locator = locators[0]!; + if ((sessionIdArg && sessionIdArg !== locator.sessionId) + || (inheritedSessionId && inheritedSessionId !== locator.sessionId)) { + console.error('botmux send refused: read-isolated session does not match the owning data-root locator'); + process.exit(2); + } + isolatedBoundSessionId = locator.sessionId; + sendDataDir = locator.dataDir; + // A locator is data, not authority. Require the kernel to deny the probe + // under the locator-selected *actual* Botmux root. Legacy Seatbelt profiles + // already denied this dashboard-secret basename class at their real root; + // a child-forged fake root remains readable and is therefore rejected. + if (managedOriginDataRootProbeAccess(sendDataDir, locator.sessionId) + !== 'sandbox_denied') { + console.error('botmux send refused: locator-selected data root is not protected by the active sandbox'); + process.exit(2); + } + // All later session/credential/marker reads in this short-lived command + // must use the host-bound root, never the child's mutable inherited value. + process.env.SESSION_DATA_DIR = sendDataDir; + liveMarkerCtx = findLiveAncestorSessionContext(sendDataDir); + } + const isolatedCapabilityCtx = !isolatedSendRequired && liveMarkerCtx?.sessionId + ? null + : readWorkflowSessionRelayContext({ + env: process.env, + dataDir: sendDataDir, + // Keep one marker snapshot for the whole decision. In particular, do + // not let resolveSessionContext's protected-capability fallback get + // mislabeled as a live process marker. + findMarker: () => isolatedSendRequired ? null : liveMarkerCtx, + }); + if (isolatedSendRequired + && (isolatedCapabilityCtx?.sessionId !== isolatedBoundSessionId + || isolatedCapabilityCtx?.originChannelId !== inheritedOriginChannelId)) { + console.error('botmux send refused: read-isolated capability is not bound to this session'); + process.exit(2); + } + let isolatedAttestationContext: ManagedOriginAttestationContext | undefined; + let isolatedManagedOriginCtx: ManagedOriginAttestation | undefined; const trustedHostRelay = process.env.BOTMUX_HOST_RELAY_AUTHORIZED === '1'; const trustedRelayAttemptRaw = Number(process.env.BOTMUX_DISPATCH_ATTEMPT); const trustedRelayCandidate = trustedHostRelay && process.env.BOTMUX_SESSION_ID @@ -5758,15 +7080,54 @@ async function cmdSend(rest: string[]): Promise { : undefined, } : undefined; - const sessionIdArg = argValue(rest, '--session-id'); // Inside bwrap the PID namespace makes host marker traversal impossible. // The relay watcher therefore binds a short-lived host-issued capability to // the worker's live turn and performs the authoritative policy check. - const relayDir = process.env.BOTMUX_SEND_RELAY; - if (relayDir) { + if (relayDir && isolatedCapabilityCtx) { await relaySend(rest, relayDir); return; } + if (relayDir && !liveMarkerCtx?.sessionId) { + // The child may delete or replace its writable outbox capability, while + // the immutable default snapshot remains visible. That snapshot is only a + // routing hint and can survive worker death; never fall through to direct + // Lark send when the live relay token is absent. + console.error('botmux send refused: managed host relay capability is stale or missing'); + process.exit(2); + } + if (!relayDir && isolatedSendRequired && !isolatedCapabilityCtx) { + // Isolation classification must not depend on successfully parsing the + // rotating token. A missing/corrupt/revoked capability is an authorization + // failure, never permission to downgrade into the ordinary direct path. + console.error('botmux send refused: read-isolated managed origin capability is stale, corrupt, or missing'); + process.exit(2); + } + if (!relayDir && isolatedCapabilityCtx) { + isolatedAttestationContext = { + sessionId: isolatedCapabilityCtx.sessionId, + channelId: isolatedCapabilityCtx.originChannelId!, + capability: isolatedCapabilityCtx.capability, + dataDir: sendDataDir, + ...(isolatedCapabilityCtx.larkAppId + ? { larkAppId: isolatedCapabilityCtx.larkAppId } + : {}), + ...(isolatedCapabilityCtx.ipcPortFallback !== undefined + ? { ipcPortFallback: isolatedCapabilityCtx.ipcPortFallback } + : {}), + }; + try { + isolatedManagedOriginCtx = await attestManagedOrigin({ + context: isolatedAttestationContext, + resolveIpcPort: (larkAppId) => { + try { return larkAppId ? findDaemon(larkAppId)?.ipcPort : undefined; } + catch { return undefined; } + }, + }); + } catch (err) { + console.error(`botmux send refused: ${err instanceof Error ? err.message : String(err)}`); + process.exit(2); + } + } // Silent is an execution policy, not a prompt suggestion. During a durable // meeting turn, refuse botmux-mediated sends before either the direct-Lark or // sandbox-relay path can run. Explicit human IM turns have no dispatchAttempt @@ -5785,10 +7146,45 @@ async function cmdSend(rest: string[]): Promise { ) ? trustedRelayCandidate : undefined; + // Keep origin authority independent from the requested destination. In + // particular, `--session-id ` may select where an ordinary Lark turn + // is sent, but it must never replace the session whose durable delivery sink + // governs this process. Session identity may fall back to the inherited env + // for detached commands; the turn tuple may not. `ancestorCtx` deliberately + // adds spawn-time BOTMUX_TURN_ID/ATTEMPT compatibility fallbacks, which can be + // stale after a later turn, so only a parent-bound host relay or a fresh + // marker/protected-capability tuple can authorize a durable dispatch. const originSessionId = trustedRelayCtx?.sessionId - || liveMarkerCtx?.sessionId - || ancestorCtx?.sessionId - || sessionIdArg; + ?? liveMarkerCtx?.sessionId + ?? isolatedManagedOriginCtx?.sessionId + ?? ancestorCtx?.sessionId + ?? process.env.BOTMUX_SESSION_ID; + const authoritativeOriginTurnCtx = trustedRelayCtx + ? (trustedRelayCtx.turnId ? trustedRelayCtx : undefined) + : (liveMarkerCtx?.turnId + ? liveMarkerCtx + : isolatedManagedOriginCtx?.turnId + ? isolatedManagedOriginCtx + : undefined); + const originTurnId = authoritativeOriginTurnCtx?.turnId; + const originDispatchAttempt = authoritativeOriginTurnCtx?.dispatchAttempt; + const originSession = originSessionId + ? sessionsForOrigin.get(originSessionId) + : undefined; + const hostRelayRequiresCodexAppLedger = !!trustedRelayCtx + && process.env.BOTMUX_HOST_RELAY_REQUIRES_CODEX_APP_LEDGER === '1'; + // The live worker authorizes a Codex App relay only while an exact durable + // dispatch remains unsettled. Recheck that prerequisite before any other + // managed-output policy or provider setup: terminal settlement can win the + // race between watcher authorization and this host child loading the store, + // and must never downgrade the send into an ordinary Lark message. + if ((hostRelayRequiresCodexAppLedger || isolatedManagedOriginCtx?.requiresCodexAppLedger) + && (originSession?.codexAppDispatchLedger?.length ?? 0) === 0) { + console.error( + `botmux send refused: authorized Codex App origin ${originSessionId}/${originTurnId} is no longer unsettled`, + ); + process.exit(2); + } let explicitVcMeetingImOrigin: ReturnType; let vcMeetingListenerOutputOwner: { listenerAppId: string; meetingId: string } | undefined; let vcMeetingManagedSendOrigin: VcMeetingManagedSendOrigin | undefined; @@ -5798,14 +7194,12 @@ async function cmdSend(rest: string[]): Promise { dispatchAttempt: number; } | undefined; if (originSessionId) { - const originSession = sessionsForOrigin.get(originSessionId); - const originTurnId = trustedRelayCtx?.turnId ?? liveMarkerCtx?.turnId; const imOrigin = resolveVcMeetingImTurnOrigin(originSession, originTurnId); const managedOrigin: VcMeetingManagedSendOrigin = { receiverSessionId: originSessionId, receiverSession: !!originSession?.vcMeetingReceiver, turnId: originTurnId, - dispatchAttempt: trustedRelayCtx?.dispatchAttempt ?? liveMarkerCtx?.dispatchAttempt, + dispatchAttempt: originDispatchAttempt, currentImTurnOrigin: imOrigin, }; const decision = evaluateVcMeetingManagedSend(resolveDataDir(), managedOrigin); @@ -5827,7 +7221,7 @@ async function cmdSend(rest: string[]): Promise { } } if (decision.kind === 'listener_thread' - && (trustedRelayCtx?.dispatchAttempt ?? liveMarkerCtx?.dispatchAttempt) === undefined) { + && originDispatchAttempt === undefined) { explicitVcMeetingImOrigin = imOrigin; } } @@ -5989,7 +7383,7 @@ async function cmdSend(rest: string[]): Promise { } const sessions = loadSessions(); - const currentTurnId = ancestorCtx?.turnId ?? process.env.BOTMUX_TURN_ID; + const currentTurnId = originTurnId; let s = sessions.get(sid); // Riff (remote backend) sandbox: no local daemon/sessions.json/bots.json. @@ -6020,10 +7414,167 @@ async function cmdSend(rest: string[]): Promise { let deferredMaterializedByThisCommand = false; let deferredTopicRootMessageIdForOutput: string | undefined; - // 本轮的回复锚点：优先按 turnId 精确取 per-turn 落点（排队/并发轮次不再被 - // 后到轮次覆盖 currentReplyTarget 而塌成群顶层 plain），无记录再退回单槽。 + // Prefer the exact per-turn reply anchor; the latest single slot is only a + // compatibility fallback for sessions persisted before replyTargets. const turnReplyTarget = pickTurnReplyTarget(s, currentTurnId); + const exactOriginDispatch = (() => { + const unsettledOriginDispatches = originSession?.codexAppDispatchLedger ?? []; + // The live worker authorized this host re-exec against an exact unsettled + // Codex App entry. If terminal settlement/revocation won the race before + // this child reloaded the store, never degrade into an ordinary Lark send + // against mutable session state. + if (unsettledOriginDispatches.length === 0) { + return undefined; + } + // A detached/backgrounded command can still inherit the correct session id + // but only a spawn-time (and now stale) turn env. If this session has any + // unsettled durable output, absence of a fresh marker/capability tuple must + // fail closed rather than silently falling through to ordinary Lark IM. + if (!originTurnId) { + console.error( + `botmux send refused: origin session ${originSessionId} has unsettled durable output but no fresh authoritative dispatch identity`, + ); + process.exit(2); + } + const turnMatches = unsettledOriginDispatches + .filter(entry => entry.turnId === originTurnId); + const exactMatches = originDispatchAttempt === undefined + ? turnMatches + : turnMatches.filter(entry => entry.dispatchAttempt === originDispatchAttempt); + // Once a durable turn exists, a supplied attempt must match it exactly. + // Falling back to another attempt would let a stale/replayed process select + // the wrong sink. With no attempt, multiple entries are equally unsafe. + if (exactMatches.length !== 1) { + const detail = originDispatchAttempt === undefined + ? `${turnMatches.length} ledger entries share the turn id` + : `${exactMatches.length} ledger entries match attempt ${originDispatchAttempt}`; + console.error( + `botmux send refused: origin dispatch ${originSessionId}/${originTurnId} is ambiguous (${detail})`, + ); + process.exit(2); + } + return exactMatches[0]; + })(); + + // Frozen reply routing applies only when the selected destination is the + // trusted origin session. A cross-session publish must not accidentally + // reuse a coincidentally-equal turn id from the destination ledger. + const frozenTurnDispatch = originSessionId === sid + ? exactOriginDispatch + : undefined; + const frozenTurnReplyTarget = frozenTurnDispatch?.replyTarget; + if (exactOriginDispatch?.deliverySink === 'http_wait' + || exactOriginDispatch?.deliverySink === 'http_async' + || exactOriginDispatch?.deliverySink === 'suppressed') { + console.error( + `botmux send refused: origin turn ${originTurnId} is bound to the ` + + `${exactOriginDispatch.deliverySink} host sink`, + ); + process.exit(2); + } + + // A proof is a point-in-time liveness check, not a five-second send lease. + // Re-challenge immediately before observable provider effects so lengthy + // local parsing/card preparation cannot carry an old capability across a + // worker restart, turn rotation, or Codex ledger settlement. + const revalidateIsolatedOriginBeforeEffect = async (): Promise => { + if (!isolatedAttestationContext || !isolatedManagedOriginCtx) return undefined; + const fresh = await attestManagedOrigin({ + context: isolatedAttestationContext, + resolveIpcPort: (larkAppId) => { + try { return larkAppId ? findDaemon(larkAppId)?.ipcPort : undefined; } + catch { return undefined; } + }, + }); + if (fresh.sessionId !== isolatedManagedOriginCtx.sessionId + || fresh.turnId !== isolatedManagedOriginCtx.turnId + || fresh.dispatchAttempt !== isolatedManagedOriginCtx.dispatchAttempt + || fresh.requiresCodexAppLedger !== isolatedManagedOriginCtx.requiresCodexAppLedger) { + throw new Error('managed origin changed before provider effect'); + } + const currentOriginSession = loadSessions().get(fresh.sessionId); + const ledgerDecision = validateCodexAppManagedSendOrigin( + currentOriginSession?.codexAppDispatchLedger, + fresh, + fresh.requiresCodexAppLedger, + ); + if (!ledgerDecision.ok + || ledgerDecision.requiresLedger !== fresh.requiresCodexAppLedger) { + throw new Error( + `managed origin ledger changed before provider effect${ledgerDecision.ok ? '' : `: ${ledgerDecision.error}`}`, + ); + } + return fresh; + }; + const fenceIsolatedOriginBeforeEffect = async (): Promise => { + await revalidateIsolatedOriginBeforeEffect(); + }; + const isolatedHookOrigin = isolatedAttestationContext?.ipcPortFallback + && isolatedManagedOriginCtx + ? { + ipcPort: isolatedAttestationContext.ipcPortFallback, + sessionId: isolatedManagedOriginCtx.sessionId, + capability: isolatedAttestationContext.capability, + turnId: isolatedManagedOriginCtx.turnId, + ...(isolatedManagedOriginCtx.dispatchAttempt !== undefined + ? { dispatchAttempt: isolatedManagedOriginCtx.dispatchAttempt } + : {}), + } + : undefined; + // Outbound hooks are a distinct post-provider effect. Preserve normal hook + // behavior, but bind it to a fresh challenge of this command's original + // protected claim. The Lark client treats fence failure as hook-only loss so + // an already-delivered primary is never reported failed and duplicated. + const outboundMessageOptions = (suppressHook = false) => + suppressHook + ? { suppressHook: true as const } + : isolatedAttestationContext + ? isolatedHookOrigin + ? { + beforeHook: fenceIsolatedOriginBeforeEffect, + hookOrigin: isolatedHookOrigin, + } + : { suppressHook: true as const } + : undefined; + + // A document-comment turn has exactly one supported observable effect: a + // plain text reply to its frozen origin target. Validate the complete shape + // before reading stdin/content/card files and before any TTS, upload, or Lark + // provider call. In particular, an explicit destination session is not an + // escape hatch from the origin sink. + const docTarget = originTurnId + ? originSession?.docCommentTargets?.[originTurnId] + : undefined; + // Codex App turns are governed exclusively by their exact dispatch ledger: + // a settled/missing ledger entry must never fall back to mutable session + // state. Other CLI adapters predate that ledger, so their frozen per-turn + // docCommentTargets entry remains the authoritative origin sink. + const isOriginDocCommentTurn = exactOriginDispatch?.deliverySink === 'doc_comment' + || (!exactOriginDispatch && originSession?.cliId !== 'codex-app' && !!docTarget); + if (isOriginDocCommentTurn) { + if (!docTarget || !originSession?.larkAppId) { + console.error('botmux send refused: this turn is bound to a document comment, but its exact origin target is no longer available'); + process.exit(2); + } + if (sid !== originSessionId + || sendTopLevel + || !!overrideChatId + || !!sendInto + || asVoice + || images.length > 0 + || files.length > 0 + || videoAttachments.length > 0 + || videoCovers.length > 0 + || customCardRequested + || attention.requested + || explicitQuote !== undefined + || noQuote) { + console.error('botmux send refused: a document-comment turn supports only its exact plain-text comment reply'); + process.exit(2); + } + } + // Read content from: --content-file > positional arg > stdin let content = ''; let customCard: Record | undefined; @@ -6125,8 +7676,12 @@ async function cmdSend(rest: string[]): Promise { const targetChatId = overrideChatId ?? s.chatId; let dir: string | undefined; try { - const out = await synthesizeVoiceOpus(appId, content); + await revalidateIsolatedOriginBeforeEffect(); + const out = await synthesizeVoiceOpus(appId, content, { + beforeProviderEffect: fenceIsolatedOriginBeforeEffect, + }); dir = out.dir; + await revalidateIsolatedOriginBeforeEffect(); const fileKey = await uploadFile(appId, out.path, { duration: out.durationMs }); const sentAtMs = Date.now(); const proposedOutput = { @@ -6150,9 +7705,7 @@ async function cmdSend(rest: string[]): Promise { messageId = prepared.messageId; } else { revalidateVcMeetingManagedSend(); - const managedProviderOptions = prepared - ? { suppressHook: true } - : undefined; + const managedProviderOptions = outboundMessageOptions(!!prepared); const deferred = !sendInto && (!overrideChatId || overrideChatId === s.chatId) ? await dispatchDeferredTopicSend({ dataDir: resolveDataDir(), @@ -6163,9 +7716,18 @@ async function cmdSend(rest: string[]): Promise { content: canonicalOutput.content, msgType: canonicalOutput.msgType, uuid: prepared?.providerKey, - sendRoot: (body, type, uuid) => sendMessage(appId, targetChatId, body, type, uuid, undefined, managedProviderOptions), - sendTitleSeed: (title, uuid) => sendMessage(appId, targetChatId, title, 'text', uuid), - replyRoot: (root, body, type, uuid) => replyMessage(appId, root, body, type, true, uuid, undefined, managedProviderOptions), + sendRoot: async (body, type, uuid) => { + await revalidateIsolatedOriginBeforeEffect(); + return sendMessage(appId, targetChatId, body, type, uuid, undefined, managedProviderOptions); + }, + sendTitleSeed: async (title, uuid) => { + await revalidateIsolatedOriginBeforeEffect(); + return sendMessage(appId, targetChatId, title, 'text', uuid); + }, + replyRoot: async (root, body, type, uuid) => { + await revalidateIsolatedOriginBeforeEffect(); + return replyMessage(appId, root, body, type, true, uuid, undefined, managedProviderOptions); + }, }) : { handled: false }; if (deferred.handled && deferred.messageId) { @@ -6173,17 +7735,20 @@ async function cmdSend(rest: string[]): Promise { deferredTopicRootMessageIdForOutput = deferred.rootMessageId; messageId = deferred.messageId; } else { - const canonicalTarget = resolveSendTarget({ - into: sendInto, - topLevel: sendTopLevel, - chatScope: s.scope === 'chat', - chatId: canonicalOutput.targetChatId, - rootMessageId: s.rootMessageId, - replyTargetRootId: turnReplyTarget?.rootMessageId, - replyTargetTurnId: turnReplyTarget?.turnId, - replyTargetQuoteOnly: turnReplyTarget?.quoteOnly, - currentTurnId, - }); + const canonicalTarget = !sendInto && !sendTopLevel && !overrideChatId && frozenTurnReplyTarget + ? frozenTurnReplyTarget + : resolveSendTarget({ + into: sendInto, + topLevel: sendTopLevel, + chatScope: s.scope === 'chat', + chatId: canonicalOutput.targetChatId, + rootMessageId: s.rootMessageId, + replyTargetRootId: turnReplyTarget?.rootMessageId, + replyTargetTurnId: turnReplyTarget?.turnId, + replyTargetQuoteOnly: turnReplyTarget?.quoteOnly, + currentTurnId, + }); + await revalidateIsolatedOriginBeforeEffect(); messageId = canonicalTarget.mode === 'plain' ? await sendMessage( appId, @@ -6242,28 +7807,22 @@ async function cmdSend(rest: string[]): Promise { } // ── 文档评论入口分流（/watch-comment / /subscribe-lark-doc）───────────────── - // 本轮若由飞书文档评论触发（daemon 已把落点写进 session.docCommentTargets[turnId]）， - // 把用户可见回复发表为飞书文档评论，而非发回飞书会话。绕过 @ 硬门（评论不 @ 飞书 - // 用户）。显式改路由（--top-level / --chat-id / --into）时不分流，让模型仍能主动 - // 用 BOTMUX_TURN_ID 从 per-turn map 取本轮落点；非文档轮的 turnId 不会命中 map， - // 天然不会误投。per-turn map 设计：并发评论之间互不覆盖，不会串线。 - const docTarget = currentTurnId ? s.docCommentTargets?.[currentTurnId] : undefined; - if (docTarget && !sendTopLevel && !overrideChatId && !sendInto) { - if (customCardRequested) { - console.error('botmux send: 文档评论回复不支持 --card-file/--card-json；请改用普通文本，或显式 --top-level/--chat-id 发到飞书群'); - process.exit(2); - } + // The authority and payload-shape checks ran before content/provider work. + // Use only the trusted origin session here; `--session-id` is a destination + // selector for ordinary Lark turns and cannot retarget a document turn. + if (isOriginDocCommentTurn) { + const exactDocTarget = docTarget!; + const exactDocSession = originSession!; const { registerBot, loadBotConfigs } = await import('./bot-registry.js'); try { for (const cfg of loadBotConfigs()) registerBot(cfg); } catch { /* */ } if (envPinnedRiffBot) { try { registerBot(envPinnedRiffBot); } catch { /* */ } } const { replyToDocComment, chunkCommentText, removeCommentReaction } = await import('./im/lark/doc-comment.js'); - const appId = s.larkAppId!; - const loc = localeForBot(appId); + const appId = exactDocSession.larkAppId!; try { // @ 落点：--mention-back → 回 @ 原评论人；--mention → @ 指定人； // 否则（--no-mention / 无）不 @。文档评论里靠 person 元素渲染 @，仅首块加。 let docMentionOpenId: string | undefined; - if (mentionBack) docMentionOpenId = docTarget.replyToOpenId; + if (mentionBack) docMentionOpenId = exactDocTarget.replyToOpenId; else if (mentionArgs.length > 0) { const first = mentionArgs[0]; const idx = first.indexOf(':'); @@ -6272,27 +7831,33 @@ async function cmdSend(rest: string[]): Promise { // 嵌套回复到用户那条评论 thread（已挂其下，无需 ↪ 前缀）。 const chunks = chunkCommentText(content); for (let i = 0; i < chunks.length; i++) { - await replyToDocComment(appId, { fileToken: docTarget.fileToken, fileType: docTarget.fileType }, docTarget.commentId, chunks[i], i === 0 ? docMentionOpenId : undefined); + await replyToDocComment( + appId, + { fileToken: exactDocTarget.fileToken, fileType: exactDocTarget.fileType }, + exactDocTarget.commentId, + chunks[i], + i === 0 ? docMentionOpenId : undefined, + { beforeProviderEffect: fenceIsolatedOriginBeforeEffect }, + ); } // 清理 "Typing" reaction（bot 已回复完毕）。 - if (docTarget.reactionId && docTarget.replyId) { + if (exactDocTarget.reactionId && exactDocTarget.replyId) { await removeCommentReaction(appId, - { fileToken: docTarget.fileToken, fileType: docTarget.fileType }, - docTarget.commentId, docTarget.replyId, docTarget.reactionId); + { fileToken: exactDocTarget.fileToken, fileType: exactDocTarget.fileType }, + exactDocTarget.commentId, exactDocTarget.replyId, exactDocTarget.reactionId, + { beforeProviderEffect: fenceIsolatedOriginBeforeEffect }); } // 写 bridge send marker → 抑制 worker 的 final_output 兜底（否则会再补一条评论）。 try { const markerDir = join(resolveDataDir(), 'turn-sends'); if (!existsSync(markerDir)) mkdirSync(markerDir, { recursive: true }); - appendFileSync(join(markerDir, `${sid}.jsonl`), JSON.stringify({ sentAtMs: Date.now(), messageId: `doc:${docTarget.commentId}`, contentLength: content.length }) + '\n'); + appendFileSync(join(markerDir, `${originSessionId}.jsonl`), JSON.stringify({ sentAtMs: Date.now(), messageId: `doc:${exactDocTarget.commentId}`, contentLength: content.length }) + '\n'); } catch { /* best-effort：漏记只多一条兜底 */ } - // 清理已消费的 per-turn 落点，避免 session 文件无限堆积。 - if (s.docCommentTargets && currentTurnId && s.docCommentTargets[currentTurnId]) { - delete s.docCommentTargets[currentTurnId]; - try { saveSession(s); } catch { /* best-effort */ } - } - console.error(`✓ 已回复文档评论 ${docTarget.commentId.slice(0, 12)}（${chunks.length} 条）`); - console.log(JSON.stringify({ success: true, commentId: docTarget.commentId, sessionId: sid, kind: 'doc-comment', chunks: chunks.length })); + // Do not write this startup snapshot back after the async provider calls: + // the daemon may have advanced the dispatch ledger or accepted another + // turn in the meantime. Daemon settlement owns exact target retirement. + console.error(`✓ 已回复文档评论 ${exactDocTarget.commentId.slice(0, 12)}（${chunks.length} 条）`); + console.log(JSON.stringify({ success: true, commentId: exactDocTarget.commentId, sessionId: originSessionId, kind: 'doc-comment', chunks: chunks.length })); } catch (e: any) { console.error(`文档评论发送失败：${e?.message ?? e}`); process.exit(1); @@ -6314,6 +7879,7 @@ async function cmdSend(rest: string[]): Promise { } } const replyTargetSenderOpenId = explicitVcMeetingImOrigin?.replyTargetSenderOpenId + ?? frozenTurnDispatch?.replyTargetSenderOpenId ?? s.quoteTargetSenderOpenId; // @ hard-gate (config.send.requireMentionDecision, default on): force the @@ -6414,8 +7980,10 @@ async function cmdSend(rest: string[]): Promise { }; // Dispatch helper: top-level / chat-scope send vs reply-in-thread, single // decision point. Used for file attachments (always plain in chat scope). - const sendTarget = resolveSendTarget({ into: sendInto, topLevel: sendTopLevel, chatScope: isChatScope, chatId: targetChatId, rootMessageId: s.rootMessageId, replyTargetRootId: turnReplyTarget?.rootMessageId, replyTargetTurnId: turnReplyTarget?.turnId, replyTargetQuoteOnly: turnReplyTarget?.quoteOnly, currentTurnId }); - const dispatch = async ( + const sendTarget = !sendInto && !sendTopLevel && !overrideChatId && frozenTurnReplyTarget + ? frozenTurnReplyTarget + : resolveSendTarget({ into: sendInto, topLevel: sendTopLevel, chatScope: isChatScope, chatId: targetChatId, rootMessageId: s.rootMessageId, replyTargetRootId: turnReplyTarget?.rootMessageId, replyTargetTurnId: turnReplyTarget?.turnId, replyTargetQuoteOnly: turnReplyTarget?.quoteOnly, currentTurnId }); + const dispatchAfterOriginGate = async ( content: string, msgType: string, uuid?: string, @@ -6434,29 +8002,38 @@ async function cmdSend(rest: string[]): Promise { content, msgType, uuid, - sendRoot: (body, type, rootUuid) => sendMessage( - appId, - targetChatId, - body, - type, - rootUuid, - hookContext, - suppressHook ? { suppressHook: true } : undefined, - ), + sendRoot: async (body, type, rootUuid) => { + await revalidateIsolatedOriginBeforeEffect(); + return sendMessage( + appId, + targetChatId, + body, + type, + rootUuid, + hookContext, + outboundMessageOptions(!!suppressHook), + ); + }, // The optional title is the root seed and the actual alert follows as // its first reply. Do not emit a user outbound hook for presentation- // only seed text; the alert itself still goes through the hook path. - sendTitleSeed: (title, rootUuid) => sendMessage(appId, targetChatId, title, 'text', rootUuid), - replyRoot: (root, body, type, replyUuid) => replyMessage( - appId, - root, - body, - type, - true, - replyUuid, - hookContext, - suppressHook ? { suppressHook: true } : undefined, - ), + sendTitleSeed: async (title, rootUuid) => { + await revalidateIsolatedOriginBeforeEffect(); + return sendMessage(appId, targetChatId, title, 'text', rootUuid); + }, + replyRoot: async (root, body, type, replyUuid) => { + await revalidateIsolatedOriginBeforeEffect(); + return replyMessage( + appId, + root, + body, + type, + true, + replyUuid, + hookContext, + outboundMessageOptions(!!suppressHook), + ); + }, }); if (deferred.handled && deferred.messageId) { deferredMaterializedByThisCommand ||= deferred.materializedNow === true; @@ -6464,17 +8041,18 @@ async function cmdSend(rest: string[]): Promise { return deferred.messageId; } } + await revalidateIsolatedOriginBeforeEffect(); return sendTarget.mode === 'plain' - ? sendMessage( + ? await sendMessage( appId, sendTarget.chatId, content, msgType, uuid, hookContext, - suppressHook ? { suppressHook: true } : undefined, + outboundMessageOptions(!!suppressHook), ) - : replyMessage( + : await replyMessage( appId, sendTarget.rootMessageId, content, @@ -6482,9 +8060,18 @@ async function cmdSend(rest: string[]): Promise { sendTarget.mode === 'thread', uuid, hookContext, - suppressHook ? { suppressHook: true } : undefined, + outboundMessageOptions(!!suppressHook), ); }; + const dispatch = async ( + content: string, + msgType: string, + uuid?: string, + suppressHook?: boolean, + ): Promise => { + await revalidateIsolatedOriginBeforeEffect(); + return dispatchAfterOriginGate(content, msgType, uuid, suppressHook); + }; const recordBridgeSendMarker = (sentAtMs: number, messageId: string, sentContent: string): void => { try { const markerDir = join(resolveDataDir(), 'turn-sends'); @@ -6508,11 +8095,17 @@ async function cmdSend(rest: string[]): Promise { // belong to another queued IM turn and is not part of the delivery action. sessionQuoteTargetId: vcMeetingDeliveryReplyOrigin ? undefined - : explicitVcMeetingImOrigin?.larkMessageId ?? s.quoteTargetId, + : explicitVcMeetingImOrigin?.larkMessageId + ?? frozenTurnDispatch?.quoteTargetId + ?? s.quoteTargetId, }); let primaryQuotedId: string | null = null; let vcMeetingListenerReplyReplay = false; - const dispatchPrimary = async (content: string, msgType: string): Promise => { + const dispatchPrimary = async ( + content: string, + msgType: string, + originAlreadyRevalidated = false, + ): Promise => { // `dispatchPrimaryMessage` may call replyMessage directly for a quote, so // fence immediately before preparing/performing that primary effect too. revalidateVcMeetingManagedSend(); @@ -6561,11 +8154,17 @@ async function cmdSend(rest: string[]): Promise { ...(prepared ? { suppressHook: true } : {}), hookContext, MessageWithdrawnError, + ...(isolatedHookOrigin + ? { + beforeHook: fenceIsolatedOriginBeforeEffect, + hookOrigin: isolatedHookOrigin, + } + : {}), // Explicit VC IM --into is rejected above, so an unquoted first send // retains the normal chat-scope dispatch semantics. On a mismatch the // frozen canonical target remains authoritative. dispatch: prepared?.outputMismatch - ? (body, type, uuid, suppressHook) => { + ? async (body, type, uuid, suppressHook) => { revalidateVcMeetingManagedSend(); return sendMessage( appId, @@ -6574,11 +8173,17 @@ async function cmdSend(rest: string[]): Promise { type, uuid, hookContext, - suppressHook ? { suppressHook: true } : undefined, + outboundMessageOptions(!!suppressHook), ); } - : dispatch, - beforeQuoteFallback: revalidateVcMeetingManagedSend, + : dispatchAfterOriginGate, + beforeEffect: originAlreadyRevalidated + ? undefined + : fenceIsolatedOriginBeforeEffect, + beforeQuoteFallback: async () => { + revalidateVcMeetingManagedSend(); + await revalidateIsolatedOriginBeforeEffect(); + }, onQuoteWithdrawn: (id) => { console.error(`引用目标 ${id} 已撤回，改为普通发送`); }, @@ -6625,8 +8230,10 @@ async function cmdSend(rest: string[]): Promise { // managed side-effect gate above. const imageKeys: string[] = []; if (images.length > 0) { - const results = await Promise.all(images.map(p => uploadImage(appId, p))); - imageKeys.push(...results); + for (const imagePath of images) { + await revalidateIsolatedOriginBeforeEffect(); + imageKeys.push(await uploadImage(appId, imagePath)); + } } // Auto-detect @BotName in text and inject as mentions, using the sender @@ -6731,11 +8338,18 @@ async function cmdSend(rest: string[]): Promise { // --no-mention 显式不 @ 任何人 → 连 footer 的"发送给/cc"寻址 也清空， // 否则 footer 仍会 @ 人，与 --no-mention 语义和"未@任何人"输出自相矛盾 // （Codex review P2）。--top-level 同样无特定收件人。 + const frozenFooterAddressingSource = explicitVcMeetingImOrigin?.replyTargetSenderOpenId + ? { ...s, lastCallerOpenId: explicitVcMeetingImOrigin.replyTargetSenderOpenId } + : frozenTurnDispatch?.replyTargetSenderOpenId + ? { + ...s, + lastCallerOpenId: frozenTurnDispatch.replyTargetSenderOpenId, + lastCallerIsBot: frozenTurnDispatch.replyTargetSenderIsBot, + } + : s; const footerAddressing = (sendTopLevel || noMention) ? { sendTo: undefined as string | undefined, cc: [] as string[] } - : buildFooterAddressing(explicitVcMeetingImOrigin?.replyTargetSenderOpenId - ? { ...s, lastCallerOpenId: explicitVcMeetingImOrigin.replyTargetSenderOpenId } - : s, { + : buildFooterAddressing(frozenFooterAddressingSource, { isOncall: !!oncallEntry, isSubstitute: turnReplyTarget?.turnId === currentTurnId && turnReplyTarget?.substitute === true, hasExplicitBotMention: explicitKnownBotMention, @@ -6794,8 +8408,9 @@ async function cmdSend(rest: string[]): Promise { { uploadFile, uploadImage, - dispatch, - primaryDispatch: dispatchPrimary, + dispatch: dispatchAfterOriginGate, + primaryDispatch: (body, type) => dispatchPrimary(body, type, true), + beforeEffect: fenceIsolatedOriginBeforeEffect, ...(vcMeetingManagedSendOrigin ? { maxMessages: 1 } : {}), }, appId, @@ -6937,10 +8552,10 @@ async function cmdSend(rest: string[]): Promise { // surface as command failure. if (!pureVideoSend && !vcMeetingListenerReplyReplay) { ({ failed: failedAttachments } = await sendFileAttachments( - { uploadFile, dispatch }, appId, files, + { uploadFile, dispatch: dispatchAfterOriginGate, beforeEffect: fenceIsolatedOriginBeforeEffect }, appId, files, )); const videoResult = await sendVideoAttachments( - { uploadFile, uploadImage, dispatch }, appId, videoAttachments, + { uploadFile, uploadImage, dispatch: dispatchAfterOriginGate, beforeEffect: fenceIsolatedOriginBeforeEffect }, appId, videoAttachments, ); failedVideoAttachments = videoResult.failed; } @@ -6971,13 +8586,29 @@ async function cmdSend(rest: string[]): Promise { let attentionError: string | undefined; if (attention.requested) { try { - const daemon = findDaemon(appId); - if (!daemon) throw new Error(`找不到 daemon (larkAppId=${appId})`); - const originCapability = readManagedOriginCapability( - resolveDataDir(), - sid, - process.env.BOTMUX_SEND_RELAY, - )?.capability; + // Raising attention is a second externally visible effect after the + // awaited Lark send. Bind it to a fresh proof of the same original + // turn instead of re-reading a stable path that may now belong to a + // successor worker generation. + const freshIsolatedOrigin = await revalidateIsolatedOriginBeforeEffect(); + const attentionOrigin = freshIsolatedOrigin ?? liveMarkerCtx; + const daemon = freshIsolatedOrigin ? undefined : findDaemon(appId); + const attentionPort = freshIsolatedOrigin + ? isolatedAttestationContext?.ipcPortFallback + : daemon?.ipcPort; + if (!attentionPort) { + throw new Error(freshIsolatedOrigin + ? '受保护的 managed-origin claim 未提供 owning daemon 端口' + : `找不到 daemon (larkAppId=${appId})`); + } + const originCapability = freshIsolatedOrigin + ? isolatedAttestationContext?.capability + : readManagedOriginCapability( + resolveDataDir(), + sid, + process.env.BOTMUX_SEND_RELAY, + process.env.BOTMUX_ORIGIN_CHANNEL_ID, + )?.capability; const request = { method: 'POST', headers: { 'content-type': 'application/json' }, @@ -6988,15 +8619,15 @@ async function cmdSend(rest: string[]): Promise { kind: attention.kind, reason: text.trim(), originCapability, - originTurnId: liveMarkerCtx?.turnId, - originDispatchAttempt: liveMarkerCtx?.dispatchAttempt, + originTurnId: attentionOrigin?.turnId, + originDispatchAttempt: attentionOrigin?.dispatchAttempt, }), } satisfies RequestInit; let secret: string | undefined; try { secret = loadDaemonIpcSecret(); } catch { /* Seatbelt/read-isolated CLI */ } const res = secret - ? await fetchDaemonIpc(daemon.ipcPort, '/api/attention', request, secret) - : await fetch(`http://127.0.0.1:${daemon.ipcPort}/api/attention`, request); + ? await fetchDaemonIpc(attentionPort, '/api/attention', request, secret) + : await fetch(`http://127.0.0.1:${attentionPort}/api/attention`, request); if (!res.ok) throw new Error(`daemon HTTP ${res.status}`); attentionRaised = true; console.error(`🙋 已举手：本会话已进 dashboard「需要你」列（用户回复后自动撤下）`); @@ -7582,6 +9213,7 @@ async function postAsk(body: Record): Promise { resolveDataDir(), askSessionId, askRelayDir, + process.env.BOTMUX_ORIGIN_CHANNEL_ID, )?.capability; const body = { sessionId: askSessionId, @@ -7973,6 +9606,7 @@ async function cmdSessionReady(): Promise { resolveDataDir(), sessionId, relayDir, + process.env.BOTMUX_ORIGIN_CHANNEL_ID, )?.capability; const liveOrigin = resolveSessionContext(resolveDataDir(), sessionId); const envAttempt = Number(process.env.BOTMUX_DISPATCH_ATTEMPT); @@ -9032,6 +10666,9 @@ async function runPluginCommandByName(rawCommand: string, commandArgs: string[]) } switch (command) { + case '__pm2-start-exact': + await cmdInternalPm2StartExact(process.argv.slice(3)); + break; case '--version': case '-v': console.log(getVersion()); break; case 'setup': { @@ -9066,7 +10703,7 @@ switch (command) { case 'ls': await cmdList(); break; case 'delete': case 'del': - case 'rm': cmdDelete(); break; + case 'rm': await cmdDelete(); break; case 'resume': await cmdResume(); break; case 'suspend': await cmdSuspend(); break; case 'slash': await cmdSlash(); break; diff --git a/src/cli/fleet-shutdown.ts b/src/cli/fleet-shutdown.ts new file mode 100644 index 000000000..d6ba8ebc7 --- /dev/null +++ b/src/cli/fleet-shutdown.ts @@ -0,0 +1,468 @@ +import { parsePm2Integer } from './pm2-jlist.js'; + +export type FleetProcessEntry = { + name: string; + /** Stable PM2 registry identity. Required for generation-safe compensation. */ + pmId?: number; + pid: number; + online: boolean; + status?: string; + autorestart?: boolean | string; + stopExitCodes?: unknown[]; + exitCode?: number; +}; + +export interface FleetShutdownRuntime { + signal(pid: number): void; + /** Optional one-shot initial dispatch. Used when each daemon owns an exact + * authenticated IPC endpoint: all requests are delivered concurrently inside + * one bounded helper. Individual refusals must return normally and remain + * live so the standard fresh compensation path can restore exited peers. */ + signalInitial?(entries: FleetProcessEntry[]): void; + /** Checked before declaring the quiet fleet successful. A missing exact IPC + * ACK remains a transaction failure even if that process later crashes; the + * helper then enters the same fresh compensation path as a live refuser. */ + assertSignalAuthorityComplete?(): void; + isAlive(pid: number): boolean; + now(): number; + sleep(ms: number): void; + /** Under the caller's cross-process fleet lock, conditionally start only the + * exact PM2 rows confirmed offline, within the remaining `timeoutMs`. */ + startOffline(entries: FleetProcessEntry[], timeoutMs: number): void; + /** Return a fresh PM2 projection, completing or failing within `timeoutMs`. */ + list(timeoutMs: number): FleetProcessEntry[]; + /** Must cover PM2 restart_delay so an old autorestart policy cannot publish + * a successor immediately after the original PID disappears. */ + successorSettleMs?: number; + pollMs?: number; +} + +function errorText(err: unknown): string { + return err instanceof Error ? err.message : String(err); +} + +function rowsForName( + entries: FleetProcessEntry[], + name: string, +): FleetProcessEntry[] { + return entries.filter(entry => entry.name === name); +} + +function isLiveGeneration( + entry: FleetProcessEntry, + runtime: FleetShutdownRuntime, +): boolean { + // OS liveness is authority. PM2 can expose a real launching/stopping PID + // while its status is not yet `online`; such a generation is never quiet. + return entry.pid > 0 && runtime.isAlive(entry.pid); +} + +export function isFleetEntryProvenFreeOfAutorestartTimer(entry: FleetProcessEntry): boolean { + if (entry.status === 'errored') return true; + // PM2 handleExit labels delayed post-exit rows `waiting restart` even when + // stop_exit_codes/autorestart suppresses creation of restart_task. A bare + // `stopped` status is not enough: handleExit can publish STOPPED before a + // zero-delay restart task is installed. Policy fields still do not make + // launching/stopping/online-dead rows quiescent. + if (entry.status !== 'waiting restart' && entry.status !== 'stopped') return false; + if (entry.autorestart === false || entry.autorestart === 'false') return true; + if (!Number.isFinite(entry.exitCode) || !Array.isArray(entry.stopExitCodes)) return false; + return entry.stopExitCodes.some(code => parsePm2Integer(code) === entry.exitCode); +} + +function rawStopCodeMatchesExitCode(code: unknown, exitCode: number): boolean { + return (typeof code === 'number' && Number.isSafeInteger(code) && code === exitCode) + || (typeof code === 'string' && code === String(exitCode)); +} + +/** Being timer-free is sufficient for initial admission and compensation, but + * not proof that a signalled generation completed its shutdown transaction. + * PM2 can mark an abrupt exit `errored` after max_restarts and suppress a + * successor even though exit_code=0 is not the daemon's graceful sentinel. */ +export function isFleetEntryProvenTerminalAfterSignal(entry: FleetProcessEntry): boolean { + if (entry.status !== 'errored' + && entry.status !== 'waiting restart' + && entry.status !== 'stopped') return false; + if (!Number.isSafeInteger(entry.exitCode) || !Array.isArray(entry.stopExitCodes)) return false; + return entry.stopExitCodes.some(code => rawStopCodeMatchesExitCode(code, entry.exitCode!)); +} + +function isProvenInitiallyQuiescent( + entry: FleetProcessEntry, + runtime: FleetShutdownRuntime, +): boolean { + if (isLiveGeneration(entry, runtime)) return false; + return isFleetEntryProvenFreeOfAutorestartTimer(entry); +} + +/** Signal every observed generation in parallel. Success requires a bounded + * quiet window with no online successor for any target name. If one generation + * refuses, restore only names proven truly offline and never touch a live + * refuser or an already-auto-restarted healthy successor. */ +export function signalAndAwaitFleet( + entries: FleetProcessEntry[], + operation: 'restart' | 'stop', + timeoutMs: number, + runtime: FleetShutdownRuntime, +): void { + const duplicateNames = [...new Set(entries.map(entry => entry.name))] + .filter(name => entries.filter(entry => entry.name === name).length > 1); + if (duplicateNames.length > 0) { + throw new Error( + `[${operation}] refusing PM2 mutation: duplicate registry row(s) for singleton botmux name(s): ` + + duplicateNames.join(', '), + ); + } + const uncertainLive = entries.filter(entry => + !entry.online && entry.pid > 0 && runtime.isAlive(entry.pid)); + if (uncertainLive.length > 0) { + throw new Error( + `[${operation}] refusing PM2 mutation: non-online registry rows still have live PID(s): ` + + uncertainLive.map(entry => `${entry.name}:${entry.pid}`).join(', '), + ); + } + const unverifiedDormant = entries.filter(entry => + !isLiveGeneration(entry, runtime) && !isProvenInitiallyQuiescent(entry, runtime)); + if (unverifiedDormant.length > 0) { + throw new Error( + `[${operation}] refusing PM2 mutation: dormant registry row(s) may still restart: ` + + unverifiedDormant.map(entry => `${entry.name}:${entry.status ?? 'unknown'}`).join(', '), + ); + } + const targets = entries.filter(entry => entry.online && entry.pid > 0); + + const deadline = runtime.now() + Math.max(0, timeoutMs); + const remainingMs = (): number => Math.max(0, Math.floor(deadline - runtime.now())); + const listWithinDeadline = ( + purpose: string, + capMs: number = Number.POSITIVE_INFINITY, + ): FleetProcessEntry[] => { + let lastError: unknown; + for (let attempt = 1; attempt <= 2; attempt++) { + const budgetMs = remainingMs(); + if (budgetMs <= 0) { + throw new Error(`fleet deadline exhausted before ${purpose}`); + } + try { + const projection = runtime.list(Math.max(1, Math.min(budgetMs, Math.floor(capMs)))); + if (remainingMs() <= 0) { + throw new Error(`fleet deadline exhausted during ${purpose}`); + } + return projection; + } catch (error) { + lastError = error; + if (remainingMs() <= 0 || attempt === 2) throw error; + } + } + throw lastError; + }; + + // Later PM2 stop/delete mutates every supplied registry name, including + // initially stopped rows. Every one must remain live-PID-free for the same + // successor settle window before that name mutation is safe. + const targetNames = new Set(entries.map(entry => entry.name)); + const signalledPids = new Set(); + const provenTerminalPids = new Set(); + const latestTrackedPidByName = new Map(); + const tracked = new Map(); + const signalGeneration = (entry: FleetProcessEntry): void => { + if (signalledPids.has(entry.pid)) return; + signalledPids.add(entry.pid); + tracked.set(entry.pid, entry); + latestTrackedPidByName.set(entry.name, entry.pid); + // The concrete runtime handles a proven ESRCH. Capability/generation + // authorization failures must escape; swallowing them here could turn a + // rollout fence into a partially signalled fleet. + runtime.signal(entry.pid); + }; + if (runtime.signalInitial) { + for (const target of targets) { + signalledPids.add(target.pid); + tracked.set(target.pid, target); + latestTrackedPidByName.set(target.name, target.pid); + } + runtime.signalInitial(targets); + if (remainingMs() <= 0) { + throw new Error( + `[${operation}] fleet deadline exhausted during bounded initial daemon dispatch; ` + + 'no later fleet action was attempted', + ); + } + } else { + for (const target of targets) { + if (remainingMs() <= 0) { + throw new Error( + `[${operation}] fleet deadline exhausted while signalling initial daemon generations; ` + + 'no later fleet action was attempted', + ); + } + signalGeneration(target); + // A runtime signal implementation is synchronous and may itself consume + // the remaining budget. Never signal the next target or begin polling + // after such a late return. + if (remainingMs() <= 0) { + throw new Error( + `[${operation}] fleet deadline exhausted during initial daemon signalling; ` + + 'no later fleet action was attempted', + ); + } + } + } + + const pollMs = Math.max(1, runtime.pollMs ?? 50); + const successorSettleMs = Math.max(0, runtime.successorSettleMs ?? 3_500); + // A live refuser is decided early enough to leave one explicitly partitioned + // compensation tail. All-dead fleets may continue their normal successor + // settle beyond this point; they need no compensation subprocesses. + const productionBudget = timeoutMs >= 10_000; + const compensationReserveMs = productionBudget + ? Math.min(25_000, Math.max(10_000, Math.floor(timeoutMs / 3))) + : Math.min(6_000, Math.max(5, Math.floor(timeoutMs / 5))); + const liveRefusalDeadline = Math.max(runtime.now(), deadline - compensationReserveMs); + const preProjectionCapMs = productionBudget + ? Math.max(5_000, Math.floor(compensationReserveMs / 4)) + : Math.max(1, Math.floor(compensationReserveMs / 5)); + const exactStartCapMs = productionBudget + ? Math.max(10_000, Math.floor(compensationReserveMs / 2)) + : Math.max(1, Math.floor(compensationReserveMs / 2)); + const postProjectionCapMs = preProjectionCapMs; + const successorProjectionCapMs = preProjectionCapMs; + let quietSince: number | null = null; + let verificationFailure: string | null = null; + let signalAuthorityFailure: string | null = null; + let terminalOutcomeFailure: string | null = null; + + while (runtime.now() < deadline) { + const aliveTracked = [...tracked.values()].filter(entry => runtime.isAlive(entry.pid)); + if (aliveTracked.length > 0) { + quietSince = null; + if (runtime.now() >= liveRefusalDeadline) break; + runtime.sleep(Math.min(pollMs, Math.max(1, liveRefusalDeadline - runtime.now()))); + continue; + } + + let projection: FleetProcessEntry[]; + try { + projection = listWithinDeadline( + 'PM2 successor verification', + successorProjectionCapMs, + ); + } catch (err) { + verificationFailure = `PM2 successor verification failed: ${errorText(err)}`; + break; + } + const successors = projection.filter(entry => + targetNames.has(entry.name) + && isLiveGeneration(entry, runtime) + && !signalledPids.has(entry.pid)); + + // PM2 restart overlimit is timer-free but not necessarily graceful. Before + // following a successor or starting the quiet window, prove the terminal + // outcome of every dead generation that this transaction signalled. A + // live replacement still carries the prior generation's exit_code, so it + // may prove that outcome before receiving its own shutdown request. + let terminalOutcomePending = false; + for (const [pid, trackedEntry] of tracked) { + const isLatestTrackedGeneration = latestTrackedPidByName.get(trackedEntry.name) === pid; + // A predecessor may be cached only after a fresh successor carrying its + // accepted exit_code was observed and then became the newly signalled + // generation. The latest generation must re-prove its terminal row on + // every quiet-window projection; a later missing row is never success. + if (runtime.isAlive(pid) + || (provenTerminalPids.has(pid) && !isLatestTrackedGeneration)) continue; + const sameNameRows = rowsForName(projection, trackedEntry.name); + const exactRows = Number.isInteger(trackedEntry.pmId) + ? sameNameRows.filter(row => row.pmId === trackedEntry.pmId) + : sameNameRows; + if (exactRows.length !== 1) { + terminalOutcomePending = true; + continue; + } + const exactState = exactRows[0]!; + const replacementPublished = exactState.pid > 0 && exactState.pid !== pid; + const liveReplacementPublished = replacementPublished + && isLiveGeneration(exactState, runtime); + const liveReplacementCarriesAcceptedExit = liveReplacementPublished + && Number.isSafeInteger(exactState.exitCode) + && Array.isArray(exactState.stopExitCodes) + && exactState.stopExitCodes.some(code => + rawStopCodeMatchesExitCode(code, exactState.exitCode!)); + if (liveReplacementCarriesAcceptedExit) { + // This cache becomes usable only after signalGeneration records the + // replacement as this name's latest tracked generation. + provenTerminalPids.add(pid); + continue; + } + // A dead different-PID row may already contain that replacement's own + // exit_code, so it can never prove the predecessor's terminal outcome. + if (!replacementPublished && isFleetEntryProvenTerminalAfterSignal(exactState)) continue; + const terminalStatus = exactState.status === 'errored' + || exactState.status === 'waiting restart' + || exactState.status === 'stopped'; + if (terminalStatus || replacementPublished) { + terminalOutcomeFailure = `${trackedEntry.name}/${pid} exited without an accepted ` + + `stop_exit_codes terminal (status=${exactState.status ?? 'unknown'}, ` + + `exit_code=${exactState.exitCode ?? 'missing'})`; + break; + } + terminalOutcomePending = true; + } + if (terminalOutcomeFailure) break; + if (terminalOutcomePending) { + quietSince = null; + if (runtime.now() >= liveRefusalDeadline) break; + runtime.sleep(Math.min(pollMs, Math.max(1, liveRefusalDeadline - runtime.now()))); + continue; + } + + if (successors.length > 0) { + quietSince = null; + if (runtime.now() >= liveRefusalDeadline) { + // This is a positively verified live refuser, not an unreadable fleet. + // Leave the late generation untouched, but retain it in the refusal + // accounting and compensate unrelated exact offline originals. + for (const successor of successors) tracked.set(successor.pid, successor); + break; + } + // A projection can consume its entire subprocess timeout. Never act on + // an observation returned at or beyond the absolute fleet deadline. + if (remainingMs() <= 0) { + verificationFailure = 'fleet deadline exhausted before signalling a PM2 successor'; + break; + } + for (const successor of successors) { + if (remainingMs() <= 0) { + verificationFailure = 'fleet deadline exhausted while signalling PM2 successors'; + break; + } + signalGeneration(successor); + } + if (verificationFailure) break; + continue; + } + + // No live PID is not enough: PM2 may still own a delayed restart_task or a + // launching/stopping transition. Do not begin the quiet window until every + // visible target row proves that no successor timer exists. + const unprovenDormant = projection.filter(entry => + targetNames.has(entry.name) + && !isLiveGeneration(entry, runtime) + && !isFleetEntryProvenFreeOfAutorestartTimer(entry)); + if (unprovenDormant.length > 0) { + quietSince = null; + if (runtime.now() >= liveRefusalDeadline) break; + runtime.sleep(Math.min(pollMs, Math.max(1, liveRefusalDeadline - runtime.now()))); + continue; + } + + const now = runtime.now(); + quietSince ??= now; + if (now - quietSince >= successorSettleMs) { + try { runtime.assertSignalAuthorityComplete?.(); } + catch (error) { + signalAuthorityFailure = errorText(error); + break; + } + return; + } + runtime.sleep(Math.min(pollMs, Math.max(1, deadline - now))); + } + + const liveSignalled = [...tracked.values()].filter(entry => runtime.isAlive(entry.pid)); + const refusal = `[${operation}] ${liveSignalled.length}/${targetNames.size} daemon generation(s) ` + + `refused or could not be verified within ${timeoutMs}ms` + + (signalAuthorityFailure ? ` (signal authority: ${signalAuthorityFailure})` : '') + + (terminalOutcomeFailure + ? ` (post-signal terminal proof: ${terminalOutcomeFailure})` + : ''); + if (verificationFailure) { + throw new Error( + `${refusal}; fleet state is unverified and no compensation was attempted ` + + `(${verificationFailure})`, + ); + } + + // A refusal path must fresh-read BEFORE any mutation. The old original-PID + // projection is not authority: PM2 may already have published a healthy + // successor for a peer that exited. + let beforeCompensation: FleetProcessEntry[]; + try { + beforeCompensation = listWithinDeadline( + 'PM2 verification before compensation', + preProjectionCapMs, + ); + } catch (err) { + throw new Error( + `${refusal}; fleet state is unverified and no compensation was attempted ` + + `(PM2 verification before compensation failed: ${errorText(err)})`, + ); + } + const offlineEntries = targets + .map(target => { + if (runtime.isAlive(target.pid)) return false; + const sameNameRows = rowsForName(beforeCompensation, target.name); + if (sameNameRows.some(state => isLiveGeneration(state, runtime))) return false; + // Compensation is allowed only against the exact original PM2 row. + // If it was deleted/recreated (or the projection omitted pm_id), there + // is no race-free way to recreate it by name. + if (!Number.isInteger(target.pmId)) return false; + if (sameNameRows.length !== 1 || sameNameRows[0]!.pmId !== target.pmId) return false; + const exactState = sameNameRows[0]!; + if (!exactState || !isFleetEntryProvenFreeOfAutorestartTimer(exactState)) return false; + // God.startProcessId does not clear restart_task. Restrict compensation + // to rows whose exit policy proves PM2 could not have scheduled one. + return exactState; + }) + .filter((entry): entry is FleetProcessEntry => !!entry); + const offlineNames = offlineEntries.map(entry => entry.name); + + const compensationErrors: string[] = []; + if (offlineEntries.length > 0) { + try { + const compensationBudgetMs = remainingMs(); + if (compensationBudgetMs <= 0) { + throw new Error('fleet deadline exhausted before compensation'); + } + runtime.startOffline( + offlineEntries, + Math.min(compensationBudgetMs, exactStartCapMs), + ); + if (remainingMs() <= 0) { + throw new Error('fleet deadline exhausted during compensation'); + } + } + catch (err) { compensationErrors.push(errorText(err)); } + } + + let afterCompensation: FleetProcessEntry[] = []; + try { + afterCompensation = listWithinDeadline( + 'PM2 verification after compensation', + postProjectionCapMs, + ); + } + catch (err) { + compensationErrors.push(`PM2 verification after compensation failed: ${errorText(err)}`); + } + const unavailable = targets + .filter(target => { + if (runtime.isAlive(target.pid)) return false; + return !rowsForName(afterCompensation, target.name) + .some(state => isLiveGeneration(state, runtime)); + }) + .map(target => target.name); + + if (unavailable.length > 0 || compensationErrors.length > 0) { + throw new Error( + `${refusal}; fleet is partially stopped` + + (unavailable.length > 0 ? ` (offline: ${unavailable.join(', ')})` : '') + + (compensationErrors.length > 0 + ? ` (restore errors: ${compensationErrors.join('; ')})` + : ''), + ); + } + throw new Error( + `${refusal}; restored ${offlineNames.length} offline PM2 ` + + `entr${offlineNames.length === 1 ? 'y' : 'ies'} and left every live generation untouched`, + ); +} diff --git a/src/cli/pm2-descriptor-guard.ts b/src/cli/pm2-descriptor-guard.ts new file mode 100644 index 000000000..a4b090c2a --- /dev/null +++ b/src/cli/pm2-descriptor-guard.ts @@ -0,0 +1,144 @@ +import { existsSync, readFileSync, readdirSync, statSync } from 'node:fs'; +import { join } from 'node:path'; +import { readSupervisorProcessStartIdentity } from '../core/process-start-identity.js'; + +export interface Pm2DescriptorProjectionEntry { + pid: number; +} + +export interface Pm2DescriptorGuardRuntime { + now(): number; + exists(path: string): boolean; + readdir(path: string): string[]; + read(path: string): string; + mtime(path: string): number; + isAlive(pid: number): boolean; + readStartIdentity(pid: number): string | undefined; +} + +const FRESH_MS = 90_000; + +const defaultRuntime: Pm2DescriptorGuardRuntime = { + now: () => Date.now(), + exists: path => existsSync(path), + readdir: path => readdirSync(path), + read: path => readFileSync(path, 'utf8'), + mtime: path => statSync(path).mtimeMs, + isAlive: pid => { try { process.kill(pid, 0); return true; } catch { return false; } }, + readStartIdentity: pid => readSupervisorProcessStartIdentity(pid), +}; + +/** Reconcile PM2 registry authority with daemon-owned descriptors before a + * core mutation. Fresh semantic corruption is fail-closed. A stale descriptor + * is not automatically harmless: an event-loop-frozen/orphan daemon can remain + * live after PM2 loses its registry. Ignore a semantically parseable stale + * record only after proving its PID dead or owned by a different process birth. */ +export function assertNoUnregisteredLiveDaemonDescriptorsIn( + operation: string, + projections: Pm2DescriptorProjectionEntry[], + registryDir: string, + runtime: Pm2DescriptorGuardRuntime = defaultRuntime, +): void { + const registeredPids = new Set( + projections.map(entry => entry.pid).filter(pid => Number.isSafeInteger(pid) && pid > 1), + ); + if (!runtime.exists(registryDir)) return; + const now = runtime.now(); + let names: string[]; + try { names = runtime.readdir(registryDir); } + catch (err) { + throw new Error( + `[${operation}] daemon descriptor registry is unreadable; refusing PM2 mutation: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + } + + const unregistered: Array<{ appId: string; pid: number }> = []; + for (const name of names) { + if (!name.endsWith('.json')) continue; + const path = join(registryDir, name); + let mtimeMs: number; + try { mtimeMs = runtime.mtime(path); } + catch (err) { + if ((err as NodeJS.ErrnoException).code === 'ENOENT') continue; + throw new Error( + `[${operation}] cannot inspect daemon descriptor ${name}; refusing PM2 mutation: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + } + const mtimeFresh = now - mtimeMs <= FRESH_MS; + + let value: unknown; + try { value = JSON.parse(runtime.read(path)); } + catch (err) { + if (!mtimeFresh) continue; + throw new Error( + `[${operation}] fresh daemon descriptor ${name} is unreadable or malformed; ` + + `refusing PM2 mutation: ${err instanceof Error ? err.message : String(err)}`, + ); + } + const record = value && typeof value === 'object' && !Array.isArray(value) + ? value as Record + : undefined; + const heartbeat = record?.lastHeartbeat; + const heartbeatValid = typeof heartbeat === 'number' && Number.isFinite(heartbeat); + const heartbeatFresh = heartbeatValid && now - heartbeat <= FRESH_MS; + const pid = record?.pid; + const appId = record?.larkAppId; + if (!mtimeFresh && !heartbeatFresh) { + // Unparseable old debris has no PID authority to reconcile. But once a + // stale record names a canonical PID, liveness + process birth decide; + // age alone must never authorize a second fleet. + if (!record || !Number.isSafeInteger(pid) || (pid as number) <= 1) continue; + if (!runtime.isAlive(pid as number)) continue; + const describedStart = record.processStartIdentity; + if (typeof describedStart !== 'string' || !describedStart) { + throw new Error( + `[${operation}] stale daemon descriptor ${name} still names live PID ${pid} ` + + 'but has no process-start identity; refusing PM2 mutation', + ); + } + const currentStart = runtime.readStartIdentity(pid as number); + if (!currentStart) { + if (!runtime.isAlive(pid as number)) continue; + throw new Error( + `[${operation}] cannot revalidate process-start identity for live stale descriptor ` + + `${name}/${pid}; refusing PM2 mutation`, + ); + } + if (currentStart !== describedStart) continue; + if (typeof appId !== 'string' || !appId.trim()) { + throw new Error( + `[${operation}] stale daemon descriptor ${name} matches live PID ${pid} ` + + 'but has no canonical app id; refusing PM2 mutation', + ); + } + if (!registeredPids.has(pid as number)) { + unregistered.push({ appId: appId.trim(), pid: pid as number }); + } + continue; + } + + if (!record + || !heartbeatValid + || now - heartbeat > FRESH_MS + || !Number.isSafeInteger(pid) + || (pid as number) <= 1 + || typeof appId !== 'string' + || !appId.trim()) { + throw new Error( + `[${operation}] fresh daemon descriptor ${name} has invalid pid/app id/heartbeat; ` + + 'refusing PM2 mutation', + ); + } + if (runtime.isAlive(pid as number) && !registeredPids.has(pid as number)) { + unregistered.push({ appId: appId.trim(), pid: pid as number }); + } + } + if (unregistered.length > 0) { + throw new Error( + `[${operation}] refusing PM2 mutation: daemon descriptor PID(s) are live but absent ` + + `from PM2 registry (${unregistered.map(item => `${item.appId}:${item.pid}`).join(', ')})`, + ); + } +} diff --git a/src/cli/pm2-exact-start.ts b/src/cli/pm2-exact-start.ts new file mode 100644 index 000000000..5a087e008 --- /dev/null +++ b/src/cli/pm2-exact-start.ts @@ -0,0 +1,77 @@ +export interface Pm2ExactStartClient { + launchRPC(callback: (error?: unknown) => void): void; + executeRemote( + method: 'startProcessId', + processId: number, + callback: (error?: unknown) => void, + ): void; + close(callback: (error?: unknown) => void): void; +} + +function errorText(error: unknown): string { + if (error instanceof Error) return error.message; + if (typeof error === 'string') return error; + try { return JSON.stringify(error); } catch { return String(error); } +} + +/** `God.startProcessId` rejects these states before executeApp and therefore + * cannot have killed or restarted a live generation. The final jlist remains + * authoritative about whether the fleet is available. */ +export function isNonMutatingPm2StartRefusal(error: unknown): boolean { + return /(?:\bid unknown\b|process already online|process already started|process with pid \d+ already exists)/i + .test(errorText(error)); +} + +function launchRpc(client: Pm2ExactStartClient): Promise { + return new Promise((resolve, reject) => { + client.launchRPC(error => error ? reject(error) : resolve()); + }); +} + +function closeRpc(client: Pm2ExactStartClient): Promise { + return new Promise(resolve => client.close(() => resolve())); +} + +function startExactProcessId( + client: Pm2ExactStartClient, + processId: number, +): Promise { + return new Promise(resolve => { + client.executeRemote('startProcessId', processId, error => { + if (!error || isNonMutatingPm2StartRefusal(error)) { + resolve(null); + return; + } + resolve(`pm_id ${processId}: ${errorText(error)}`); + }); + }); +} + +/** Connect once and concurrently issue PM2's conditional startProcessId RPC + * for distinct exact registry identities. The caller must hold Botmux's shared + * fleet-operation lock: PM2 checks a row before an async fork and is not itself + * a CAS across two clients. Unlike `pm2 start `, this never routes + * through restartProcessId or intentionally interrupts a live successor. */ +export async function startExactPm2ProcessIds( + processIds: number[], + client: Pm2ExactStartClient, +): Promise { + const uniqueIds = [...new Set(processIds)]; + if (uniqueIds.length !== processIds.length + || uniqueIds.some(id => !Number.isInteger(id) || id < 0)) { + throw new Error('exact PM2 compensation requires unique non-negative pm_id values'); + } + if (uniqueIds.length === 0) return; + + await launchRpc(client); + try { + const failures = (await Promise.all( + uniqueIds.map(id => startExactProcessId(client, id)), + )).filter((failure): failure is string => !!failure); + if (failures.length > 0) { + throw new Error(`conditional PM2 start failed (${failures.join('; ')})`); + } + } finally { + await closeRpc(client); + } +} diff --git a/src/cli/pm2-god-admission.ts b/src/cli/pm2-god-admission.ts new file mode 100644 index 000000000..ddaa81fca --- /dev/null +++ b/src/cli/pm2-god-admission.ts @@ -0,0 +1,27 @@ +/** + * Node's process.kill is PID-addressed and PM2's `kill` RPC is PM2_HOME/socket + * addressed; neither binds a signal to a PID+birth generation. Therefore a + * live God cannot be safely replaced automatically. This admission check must + * run before any fleet or breadcrumb mutation. + */ +export function assertIncludePm2RestartAdmission(pids: readonly number[]): void { + const canonical = [...new Set(pids)] + .filter(pid => Number.isSafeInteger(pid) && pid > 1) + .sort((a, b) => a - b); + if (canonical.length !== pids.length) { + throw new Error('[restart --include-pm2] PM2 God scan returned invalid/duplicate PIDs'); + } + if (canonical.length === 0) return; + if (canonical.length > 1) { + throw new Error( + `[restart --include-pm2] multiple PM2 God daemons are visible ` + + `(pids: ${canonical.join(', ')}); no process or breadcrumb was changed`, + ); + } + throw new Error( + `[restart --include-pm2] refusing before fleet mutation: live PM2 God pid ${canonical[0]} ` + + 'cannot be signalled with generation-bound authority on this platform; ' + + 'this option does not signal or restart an existing God; ' + + 'no process or breadcrumb was changed', + ); +} diff --git a/src/cli/pm2-jlist.ts b/src/cli/pm2-jlist.ts new file mode 100644 index 000000000..ce9f90218 --- /dev/null +++ b/src/cli/pm2-jlist.ts @@ -0,0 +1,123 @@ +type ParsedProjection = + | { kind: 'array'; value: any[] } + | { kind: 'non-array' } + | { kind: 'malformed' }; + +export function parsePm2Integer( + value: unknown, + options: { nonNegative?: boolean } = {}, +): number | undefined { + if (value === null || value === undefined) return undefined; + if (typeof value === 'string' && !/^-?\d+$/.test(value.trim())) return undefined; + if (typeof value !== 'number' && typeof value !== 'string') return undefined; + const parsed = Number(value); + if (!Number.isSafeInteger(parsed)) return undefined; + if (options.nonNegative && parsed < 0) return undefined; + return parsed; +} + +/** PM2 RPC addresses the canonical top-level pm_id. A nested pm2_env.pm_id can + * be a stale serialized copy and must never resurrect an absent/null identity. */ +export function parseCanonicalPm2Id(app: unknown): number | undefined { + if (!app || typeof app !== 'object' || Array.isArray(app)) return undefined; + return parsePm2Integer((app as Record).pm_id, { nonNegative: true }); +} + +function parseProjection(output: string): ParsedProjection { + try { + const parsed = JSON.parse(output); + return Array.isArray(parsed) + ? { kind: 'array', value: parsed } + : { kind: 'non-array' }; + } catch { + // PM2 can prefix stdout with informational `[PM2]` lines. Search backward + // for the final valid JSON array without accepting arbitrary JSON values. + for ( + let start = output.lastIndexOf('['); + start >= 0; + start = start === 0 ? -1 : output.lastIndexOf('[', start - 1) + ) { + try { + const parsed = JSON.parse(output.slice(start).trim()); + if (Array.isArray(parsed)) return { kind: 'array', value: parsed }; + } catch { /* try an earlier '[' */ } + } + return { kind: 'malformed' }; + } +} + +function isPlainRecord(value: unknown): value is Record { + return !!value && typeof value === 'object' && !Array.isArray(value); +} + +/** Validate the parts of a PM2 registry row that every mutating caller relies + * on. A syntactically valid JSON array is not authority when one of its rows + * would later be silently coerced to name="undefined", pid=0, or an absent + * canonical pm_id. */ +function assertSemanticPm2Rows(rows: any[]): void { + const ids = new Map(); + const positivePids = new Map(); + for (let index = 0; index < rows.length; index++) { + const row = rows[index] as unknown; + if (!isPlainRecord(row)) { + throw new Error(`pm2 jlist row ${index} is not an object`); + } + const name = row.name; + if (typeof name !== 'string' || !name.trim()) { + throw new Error(`pm2 jlist row ${index} has no non-empty name`); + } + const pmId = parseCanonicalPm2Id(row); + if (pmId === undefined) { + throw new Error(`pm2 jlist row ${index} (${name}) has no canonical non-negative pm_id`); + } + const priorName = ids.get(pmId); + if (priorName !== undefined) { + throw new Error( + `pm2 jlist has duplicate canonical pm_id ${pmId} across ${priorName} and ${name}`, + ); + } + ids.set(pmId, name); + + const pid = parsePm2Integer(row.pid, { nonNegative: true }); + if (pid === undefined) { + throw new Error(`pm2 jlist row ${index} (${name}) has no non-negative pid`); + } + if (pid > 1) { + const priorPidName = positivePids.get(pid); + if (priorPidName !== undefined) { + throw new Error( + `pm2 jlist has duplicate positive pid ${pid} across ${priorPidName} and ${name}`, + ); + } + positivePids.set(pid, name); + } + const env = row.pm2_env; + if (!isPlainRecord(env) + || typeof env.status !== 'string' + || !env.status.trim()) { + throw new Error(`pm2 jlist row ${index} (${name}) has no semantic pm2_env.status`); + } + } +} + +/** Forgiving parser for read-only/status surfaces. */ +export function parsePm2JlistOutput(output: string): any[] { + const parsed = parseProjection(output); + return parsed.kind === 'array' ? parsed.value : []; +} + +/** Shutdown authority must never interpret `{}`, `null`, or malformed output + * as an empty fleet: doing so would allow a later name-based PM2 mutation to + * bypass graceful shutdown of live daemons. */ +export function parsePm2JlistOutputStrict(output: string): any[] { + const parsed = parseProjection(output); + if (parsed.kind === 'array') { + assertSemanticPm2Rows(parsed.value); + return parsed.value; + } + throw new Error( + parsed.kind === 'non-array' + ? 'pm2 jlist returned non-array JSON' + : 'pm2 jlist returned malformed output', + ); +} diff --git a/src/cli/pm2-preflight.ts b/src/cli/pm2-preflight.ts new file mode 100644 index 000000000..678871716 --- /dev/null +++ b/src/cli/pm2-preflight.ts @@ -0,0 +1,29 @@ +import { existsSync, readlinkSync } from 'node:fs'; + +export interface Pm2ExecutableProbeRuntime { + readlink(path: string): string; + exists(path: string): boolean; +} + +/** Fail closed when a live Linux PM2 God is executing a deleted Node binary. + * Failure to inspect /proc remains non-authoritative and is skipped, but a + * successful read is evaluated outside that catch so the safety refusal can + * never be swallowed as an inspection error. */ +export function assertLinuxPm2GodExecutableUsable( + pm2Pid: number, + runtime: Pm2ExecutableProbeRuntime = { + readlink: path => readlinkSync(path), + exists: path => existsSync(path), + }, +): void { + let executable: string; + try { executable = runtime.readlink(`/proc/${pm2Pid}/exe`); } + catch { return; } + + const cleanPath = executable.replace(/ \(deleted\)$/, ''); + if (!executable.endsWith(' (deleted)') && runtime.exists(cleanPath)) return; + throw new Error( + `pm2 god daemon (pid ${pm2Pid}) 使用的 Node 二进制已失效: ${cleanPath}; ` + + '为避免强杀未完成 Riff 交接的 daemon，拒绝自动清理，请先人工核对进程归属', + ); +} diff --git a/src/cli/pm2-shutdown-capability.ts b/src/cli/pm2-shutdown-capability.ts new file mode 100644 index 000000000..df0393217 --- /dev/null +++ b/src/cli/pm2-shutdown-capability.ts @@ -0,0 +1,206 @@ +import { existsSync, readFileSync, readdirSync, statSync } from 'node:fs'; +import { join } from 'node:path'; +import { SUPERVISOR_SHUTDOWN_PROTOCOL } from '../core/supervisor-shutdown-protocol.js'; +import { readSupervisorProcessStartIdentity } from '../core/process-start-identity.js'; + +export interface Pm2ShutdownCapabilityTarget { + name: string; + pid: number; +} + +export interface AttestedPm2DaemonShutdownTarget extends Pm2ShutdownCapabilityTarget { + larkAppId: string; + ipcPort: number; + bootInstanceId: string; + processStartIdentity: string; +} + +export interface Pm2ShutdownCapabilityRuntime { + now(): number; + exists(path: string): boolean; + readdir(path: string): string[]; + read(path: string): string; + mtime(path: string): number; + isAlive(pid: number): boolean; + readStartIdentity(pid: number): string | undefined; +} + +const FRESH_MS = 90_000; + +const defaultRuntime: Pm2ShutdownCapabilityRuntime = { + now: () => Date.now(), + exists: path => existsSync(path), + readdir: path => readdirSync(path), + read: path => readFileSync(path, 'utf8'), + mtime: path => statSync(path).mtimeMs, + isAlive: pid => { try { process.kill(pid, 0); return true; } catch { return false; } }, + readStartIdentity: pid => { + // Lazy import is unnecessary here: this pure helper has no daemon state. + // The concrete function is injected below to keep tests deterministic. + return readSupervisorProcessStartIdentity(pid); + }, +}; + +function failure(operation: string, detail: string): Error { + return new Error( + `[${operation}] refusing to signal daemon generation(s): ${detail}. ` + + `The live daemon may predate shutdown protocol ${SUPERVISOR_SHUTDOWN_PROTOCOL}; ` + + 'normal stop/restart intentionally fails closed on this first-upgrade boundary. ' + + 'After confirming every Session/Riff workload is idle, run ' + + '`botmux restart --bootstrap-shutdown-protocol --yes` for the operator-approved one-time bootstrap; ' + + 'automatic update must not be reported as applied until the new handler-ready fleet is verified', + ); +} + +/** + * Require every still-live target daemon PID to own one fresh, daemon-written + * descriptor advertising the exact safe shutdown protocol. This is a rollout + * boundary, not a package-version guess: installing a new CLI does not upgrade + * an already-running old daemon in memory. + */ +export function assertPm2DaemonShutdownCapabilitiesIn( + operation: string, + targets: readonly Pm2ShutdownCapabilityTarget[], + registryDir: string, + runtime: Pm2ShutdownCapabilityRuntime = defaultRuntime, +): AttestedPm2DaemonShutdownTarget[] { + const invalid = targets.filter(target => + !target.name.trim() || !Number.isSafeInteger(target.pid) || target.pid <= 1); + if (invalid.length > 0) { + throw failure(operation, 'shutdown target has no canonical name/live PID'); + } + const duplicatePids = [...new Set(targets.map(target => target.pid))] + .filter(pid => targets.filter(target => target.pid === pid).length > 1); + if (duplicatePids.length > 0) { + throw failure(operation, `multiple shutdown targets share PID(s) ${duplicatePids.join(', ')}`); + } + + // A generation that exited before attestation needs no signal. Any successor + // is discovered from a fresh PM2 projection and must attest independently. + const liveTargets = targets.filter(target => runtime.isAlive(target.pid)); + if (liveTargets.length === 0) return []; + if (!runtime.exists(registryDir)) { + throw failure(operation, 'daemon descriptor registry is missing'); + } + + let names: string[]; + try { names = runtime.readdir(registryDir); } + catch (error) { + throw failure( + operation, + `daemon descriptor registry is unreadable (${error instanceof Error ? error.message : String(error)})`, + ); + } + + const now = runtime.now(); + const targetPids = new Set(liveTargets.map(target => target.pid)); + const descriptors = new Map>(); + for (const name of names) { + if (!name.endsWith('.json')) continue; + const path = join(registryDir, name); + let mtimeMs: number; + try { mtimeMs = runtime.mtime(path); } + catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') continue; + throw failure( + operation, + `cannot inspect daemon descriptor ${name} (${error instanceof Error ? error.message : String(error)})`, + ); + } + const mtimeFresh = now - mtimeMs <= FRESH_MS; + + let value: unknown; + try { value = JSON.parse(runtime.read(path)); } + catch (error) { + if (!mtimeFresh) continue; + throw failure( + operation, + `fresh daemon descriptor ${name} is unreadable or malformed ` + + `(${error instanceof Error ? error.message : String(error)})`, + ); + } + const record = value && typeof value === 'object' && !Array.isArray(value) + ? value as Record + : undefined; + const heartbeat = record?.lastHeartbeat; + const heartbeatFresh = typeof heartbeat === 'number' + && Number.isFinite(heartbeat) + && now - heartbeat <= FRESH_MS; + if (!mtimeFresh && !heartbeatFresh) continue; + + const pid = record?.pid; + if (!Number.isSafeInteger(pid) || (pid as number) <= 1) { + throw failure(operation, `fresh daemon descriptor ${name} has no canonical PID`); + } + if (!targetPids.has(pid as number)) continue; + if (!heartbeatFresh) { + throw failure(operation, `daemon descriptor ${name} has no fresh semantic heartbeat`); + } + if (!runtime.isAlive(pid as number)) continue; + if (descriptors.has(pid as number)) { + throw failure(operation, `multiple fresh descriptors claim live PID ${pid}`); + } + descriptors.set(pid as number, record!); + } + + const authorized: AttestedPm2DaemonShutdownTarget[] = []; + for (const target of liveTargets) { + const descriptor = descriptors.get(target.pid); + if (!descriptor) { + // Re-check process birth: a generation that exited during the registry + // scan needs no signal, but a same-PID successor must not inherit a stale + // capability. + if (!runtime.readStartIdentity(target.pid) && !runtime.isAlive(target.pid)) continue; + throw failure( + operation, + `${target.name}/${target.pid} has no matching fresh daemon descriptor`, + ); + } + if (descriptor.supervisorShutdownProtocol !== SUPERVISOR_SHUTDOWN_PROTOCOL) { + throw failure( + operation, + `${target.name}/${target.pid} does not attest ${SUPERVISOR_SHUTDOWN_PROTOCOL}`, + ); + } + const larkAppId = descriptor.larkAppId; + const ipcPort = descriptor.ipcPort; + const bootInstanceId = descriptor.bootInstanceId; + if (typeof larkAppId !== 'string' || !larkAppId.trim() + || !Number.isSafeInteger(ipcPort) || (ipcPort as number) < 1 || (ipcPort as number) > 65_535 + || typeof bootInstanceId !== 'string' || !bootInstanceId) { + throw failure( + operation, + `${target.name}/${target.pid} descriptor has no canonical app/port/boot identity`, + ); + } + const describedStart = descriptor.processStartIdentity; + if (typeof describedStart !== 'string' || !describedStart) { + throw failure( + operation, + `${target.name}/${target.pid} descriptor has no process-start identity`, + ); + } + const currentStart = runtime.readStartIdentity(target.pid); + if (!currentStart) { + if (!runtime.isAlive(target.pid)) continue; + throw failure( + operation, + `cannot revalidate process-start identity for ${target.name}/${target.pid}`, + ); + } + if (currentStart !== describedStart) { + throw failure( + operation, + `${target.name}/${target.pid} process-start identity does not match its descriptor`, + ); + } + authorized.push({ + ...target, + larkAppId: larkAppId.trim(), + ipcPort: ipcPort as number, + bootInstanceId, + processStartIdentity: describedStart, + }); + } + return authorized; +} diff --git a/src/cli/pm2-start-transaction.ts b/src/cli/pm2-start-transaction.ts new file mode 100644 index 000000000..6a1694563 --- /dev/null +++ b/src/cli/pm2-start-transaction.ts @@ -0,0 +1,313 @@ +import type { FleetProcessEntry } from './fleet-shutdown.js'; +import { DAEMON_GRACEFUL_EXIT_CODE } from '../core/supervisor-shutdown-protocol.js'; + +/** Preserve PM2's raw stop_exit_codes elements for exact policy validation. + * PM2 applies parseInt to string elements at exit time, so lossy numeric + * projection would hide restart-suppressing extras such as "0foo". */ +export function normalizeRawPm2StopExitCodes(value: unknown): unknown[] { + return Array.isArray(value) ? [...value] : [value]; +} + +function errorText(error: unknown): string { + return error instanceof Error ? error.message : String(error); +} + +function assertUniqueConfiguredNames(names: string[]): void { + const unique = new Set(names); + if (unique.size !== names.length || names.some(name => !name.trim())) { + throw new Error('configured PM2 fleet names must be unique and non-empty'); + } +} + +function assertProjectionIdentities( + operation: string, + entries: FleetProcessEntry[], +): void { + const names = new Set(); + const ids = new Map(); + const positivePids = new Map(); + for (const entry of entries) { + if (names.has(entry.name)) { + throw new Error(`[${operation}] duplicate singleton PM2 row for ${entry.name}`); + } + names.add(entry.name); + if (!Number.isSafeInteger(entry.pmId) || (entry.pmId as number) < 0) { + throw new Error(`[${operation}] ${entry.name} has no canonical pm_id`); + } + const prior = ids.get(entry.pmId as number); + if (prior !== undefined) { + throw new Error( + `[${operation}] duplicate canonical pm_id ${entry.pmId} across ${prior} and ${entry.name}`, + ); + } + ids.set(entry.pmId as number, entry.name); + if (Number.isSafeInteger(entry.pid) && entry.pid > 1) { + const priorPidName = positivePids.get(entry.pid); + if (priorPidName !== undefined) { + throw new Error( + `[${operation}] duplicate positive pid ${entry.pid} across ${priorPidName} and ${entry.name}`, + ); + } + positivePids.set(entry.pid, entry.name); + } + } +} + +function assertNoUnexpectedRows( + operation: string, + entries: FleetProcessEntry[], + configuredNames: string[], +): void { + const configured = new Set(configuredNames); + const unexpected = entries.filter(entry => !configured.has(entry.name)); + if (unexpected.length > 0) { + throw new Error( + `[${operation}] unexpected PM2 core row(s): ${unexpected.map(entry => entry.name).join(', ')}`, + ); + } +} + +function isOnlineAndLive( + entry: FleetProcessEntry, + isAlive: (pid: number) => boolean, +): boolean { + return entry.online + && Number.isSafeInteger(entry.pid) + && entry.pid > 1 + && isAlive(entry.pid); +} + +/** The in-memory shutdown capability is insufficient when PM2 still owns an + * old registry policy. In particular `stop_exit_codes:[0]` suppresses restart + * after PM2 normalizes SIGKILL/OOM to exit_code 0. Require the exact daemon + * policy that makes only shutdown()'s reserved sentinel terminal. */ +export function assertDaemonPm2GracefulExitPolicy( + operation: string, + entries: FleetProcessEntry[], +): void { + const unsafe = entries.filter(entry => { + const codes = entry.stopExitCodes; + const exactSentinel = Array.isArray(codes) + && codes.length === 1 + && (codes[0] === DAEMON_GRACEFUL_EXIT_CODE + || codes[0] === String(DAEMON_GRACEFUL_EXIT_CODE)); + const restartEnabled = entry.autorestart === true || entry.autorestart === 'true'; + return !exactSentinel || !restartEnabled; + }); + if (unsafe.length > 0) { + throw new Error( + `[${operation}] daemon PM2 policy does not prove signal-death autorestart ` + + `(expected autorestart=true and stop_exit_codes=[${DAEMON_GRACEFUL_EXIT_CODE}]; unsafe: ` + + `${unsafe.map(entry => entry.name).join(', ')}). ` + + 'For a one-time pre-protocol upgrade, first independently confirm every Session/Riff ' + + 'workload is idle, then run: botmux restart --bootstrap-shutdown-protocol --yes', + ); + } +} + +/** Require one exact, online, OS-live registry row for every configured core + * process and no stale/foreign core rows. This is the postcondition for every + * public fleet start surface. */ +export function assertConfiguredPm2FleetOnline( + operation: string, + entries: FleetProcessEntry[], + configuredNames: string[], + isAlive: (pid: number) => boolean, +): void { + assertUniqueConfiguredNames(configuredNames); + assertProjectionIdentities(operation, entries); + assertNoUnexpectedRows(operation, entries, configuredNames); + + const unavailable = configuredNames.filter(name => { + const row = entries.find(entry => entry.name === name); + return !row || !isOnlineAndLive(row, isAlive); + }); + if (unavailable.length > 0 || entries.length !== configuredNames.length) { + throw new Error( + `[${operation}] configured PM2 fleet is not fully online` + + (unavailable.length > 0 ? ` (unavailable: ${unavailable.join(', ')})` : ''), + ); + } +} + +/** PM2 `online` is published before a daemon's shutdown endpoint/handler-ready + * capability. A public start is complete only after both authorities agree. */ +export function assertConfiguredPm2FleetReady( + operation: string, + entries: TEntry[], + configuredNames: string[], + isAlive: (pid: number) => boolean, + assertDaemonCapabilities: (entries: TEntry[]) => void, +): void { + assertConfiguredPm2FleetOnline(operation, entries, configuredNames, isAlive); + assertDaemonCapabilities(entries); +} + +/** Capability scanners used by shutdown may legitimately omit a target that + * exited during their read. Start verification may not: require an exact + * attested PID set and recheck OS liveness after the capability scan. */ +export function assertExactAttestedDaemonSet( + operation: string, + daemonEntries: FleetProcessEntry[], + attestedPids: readonly number[], + isAlive: (pid: number) => boolean, +): void { + const expected = daemonEntries.map(entry => entry.pid).sort((a, b) => a - b); + const actual = [...attestedPids].sort((a, b) => a - b); + if (expected.length !== actual.length + || expected.some((pid, index) => pid !== actual[index])) { + throw new Error( + `[${operation}] handler-ready capability set is incomplete ` + + `(expected pids: ${expected.join(', ') || 'none'}; attested: ${actual.join(', ') || 'none'})`, + ); + } + const deadAfterAttestation = daemonEntries.filter(entry => !isAlive(entry.pid)); + if (deadAfterAttestation.length > 0) { + throw new Error( + `[${operation}] daemon exited after capability attestation: ` + + deadAfterAttestation.map(entry => `${entry.name}/${entry.pid}`).join(', '), + ); + } +} + +export type StartBotFleetAdmission = + | { state: 'already-online' } + | { state: 'start-eligible' } + | { state: 'fleet-down' }; + +/** `start-bot` is safe only for the append-one-bot case: either the entire + * configured fleet is already online, or exactly the requested bot row is + * absent while every other configured bot and the dashboard are online. */ +export function classifyStartBotFleetAdmission( + operation: string, + entries: FleetProcessEntry[], + configuredNames: string[], + targetName: string, + isAlive: (pid: number) => boolean, +): StartBotFleetAdmission { + assertUniqueConfiguredNames(configuredNames); + if (!configuredNames.includes(targetName)) { + throw new Error(`[${operation}] target ${targetName} is not configured`); + } + assertProjectionIdentities(operation, entries); + assertNoUnexpectedRows(operation, entries, configuredNames); + if (entries.length === 0) return { state: 'fleet-down' }; + + const targetRows = entries.filter(entry => entry.name === targetName); + const unavailablePeers = configuredNames + .filter(name => name !== targetName) + .filter(name => { + const row = entries.find(entry => entry.name === name); + return !row || !isOnlineAndLive(row, isAlive); + }); + if (unavailablePeers.length > 0) { + throw new Error( + `[${operation}] refusing single-bot start because configured peer(s) are unavailable: ` + + unavailablePeers.join(', '), + ); + } + if (targetRows.length === 0) { + if (entries.length !== configuredNames.length - 1) { + throw new Error(`[${operation}] fleet is not the exact one-missing-bot shape`); + } + return { state: 'start-eligible' }; + } + const target = targetRows[0]!; + if (!isOnlineAndLive(target, isAlive)) { + throw new Error( + `[${operation}] refusing start-bot for existing non-live/transitional row ${targetName}`, + ); + } + if (entries.length !== configuredNames.length) { + throw new Error(`[${operation}] fleet is not the exact fully-configured shape`); + } + return { state: 'already-online' }; +} + +export interface Pm2StartTransactionRuntime { + start(timeoutMs: number): void; + /** Must obtain a new projection and validate the complete expected state. */ + verifyFresh(timeoutMs: number): TProjection; + /** Must independently re-read authority before compensating partial launch. */ + rollback(): void; +} + +/** Run one bounded PM2 launch and make the fresh fleet projection—not the CLI + * exit code—the success authority. Any incomplete/unverified launch is + * compensated before the error escapes. */ +export function runBoundedPm2StartTransaction( + operation: string, + startTimeoutMs: number, + verifyTimeoutMs: number, + runtime: Pm2StartTransactionRuntime, +): TProjection { + if (!Number.isFinite(startTimeoutMs) || startTimeoutMs <= 0 + || !Number.isFinite(verifyTimeoutMs) || verifyTimeoutMs <= 0) { + throw new Error(`[${operation}] PM2 start/verification budgets must be positive`); + } + + let startFailure: unknown; + try { runtime.start(Math.floor(startTimeoutMs)); } + catch (error) { startFailure = error; } + + try { + // A timed-out client can race with a God RPC that already completed. A + // complete fresh projection is therefore stronger evidence than the + // launcher exit code and is safe to accept. + return runtime.verifyFresh(Math.floor(verifyTimeoutMs)); + } catch (verifyFailure) { + let rollbackFailure: unknown; + try { runtime.rollback(); } + catch (error) { rollbackFailure = error; } + throw new Error( + `[${operation}] PM2 start transaction did not reach a verified complete fleet` + + (startFailure ? ` (start: ${errorText(startFailure)})` : '') + + ` (verify: ${errorText(verifyFailure)})` + + (rollbackFailure + ? `; partial-launch rollback failed: ${errorText(rollbackFailure)}` + : '; partial launch was rolled back'), + ); + } +} + +export interface LatePm2StartRollbackRuntime { + now(): number; + sleep(ms: number): void; + /** Re-read authority and compensate anything currently published. Return + * true only when this observation exactly matches the pre-start state. */ + reconcileOnce(): boolean; +} + +/** A killed/timed-out PM2 client does not cancel work already queued in God. + * Therefore an empty first rollback projection is not success: require one + * continuous restored window, resetting it whenever a late row appears. */ +export function reconcileLatePm2StartPublication( + operation: string, + settleMs: number, + timeoutMs: number, + runtime: LatePm2StartRollbackRuntime, +): void { + if (!Number.isFinite(settleMs) || settleMs < 0 + || !Number.isFinite(timeoutMs) || timeoutMs <= settleMs) { + throw new Error(`[${operation}] invalid late-publication rollback budgets`); + } + const deadline = runtime.now() + Math.floor(timeoutMs); + let restoredSince: number | undefined; + while (runtime.now() < deadline) { + const restored = runtime.reconcileOnce(); + const now = runtime.now(); + if (now >= deadline) break; + if (!restored) { + restoredSince = undefined; + runtime.sleep(Math.min(100, Math.max(1, deadline - now))); + continue; + } + restoredSince ??= now; + const settledFor = now - restoredSince; + if (settledFor >= settleMs) return; + runtime.sleep(Math.min(100, settleMs - settledFor, Math.max(1, deadline - now))); + } + throw new Error( + `[${operation}] partial-launch rollback remains uncertain after the late-publication settle window`, + ); +} diff --git a/src/cli/send-dispatch.ts b/src/cli/send-dispatch.ts index e8c15adcf..e19ea619f 100644 --- a/src/cli/send-dispatch.ts +++ b/src/cli/send-dispatch.ts @@ -1,4 +1,5 @@ import { extname } from 'node:path'; +import type { ManagedHookOrigin } from '../services/hook-runner.js'; export type SendMessageFn = ( larkAppId: string, @@ -7,7 +8,7 @@ export type SendMessageFn = ( msgType?: string, uuid?: string, hookContext?: Record, - options?: { suppressHook?: boolean }, + options?: { suppressHook?: boolean; beforeHook?: () => void | Promise; hookOrigin?: ManagedHookOrigin }, ) => Promise; export type ReplyMessageFn = ( @@ -18,7 +19,7 @@ export type ReplyMessageFn = ( replyInThread?: boolean, uuid?: string, hookContext?: Record, - options?: { suppressHook?: boolean }, + options?: { suppressHook?: boolean; beforeHook?: () => void | Promise; hookOrigin?: ManagedHookOrigin }, ) => Promise; export type DispatchPrimaryDeps = { @@ -48,6 +49,7 @@ export function findStdinAliasAttachment(paths: readonly string[]): string | nul export type SendFileAttachmentsDeps = { uploadFile: (appId: string, path: string) => Promise; dispatch: (content: string, msgType: string) => Promise; + beforeEffect?: () => void | Promise; }; export type SendFileAttachmentsResult = { @@ -72,7 +74,9 @@ export async function sendFileAttachments( const failed: { path: string; error: string }[] = []; for (const fp of files) { try { + await deps.beforeEffect?.(); const fileKey = await deps.uploadFile(appId, fp); + await deps.beforeEffect?.(); sent.push(await deps.dispatch(JSON.stringify({ file_key: fileKey }), 'file')); } catch (err: any) { failed.push({ path: fp, error: err?.message ?? String(err) }); @@ -324,6 +328,7 @@ export type SendVideoAttachmentsDeps = { * replies set this to one because only the primary media message has a durable * action/provider identity; later bare media sends would duplicate on replay. */ maxMessages?: number; + beforeEffect?: () => void | Promise; }; export type SendVideoAttachmentsResult = { @@ -349,7 +354,9 @@ export async function sendVideoAttachments( let primaryUsed = false; for (const video of videos) { try { + await deps.beforeEffect?.(); const fileKey = await deps.uploadFile(appId, video.videoPath); + await deps.beforeEffect?.(); const imageKey = await deps.uploadImage(appId, video.coverPath); const content = JSON.stringify({ file_key: fileKey, @@ -357,6 +364,7 @@ export async function sendVideoAttachments( duration: video.durationMs, }); const send = (!primaryUsed && deps.primaryDispatch) ? deps.primaryDispatch : deps.dispatch; + await deps.beforeEffect?.(); const messageId = await send(content, 'media'); primaryUsed = true; sent.push(messageId); @@ -384,9 +392,14 @@ export type DispatchPrimaryOptions = { dispatch: (content: string, msgType: string, uuid?: string, suppressHook?: boolean) => Promise; /** Provider UUID reconciliation must not repeat the local outbound hook. */ suppressHook?: boolean; + /** Revalidate immediately before the distinct post-provider hook effect. */ + beforeHook?: () => void | Promise; + hookOrigin?: ManagedHookOrigin; /** Revalidate any side-effect authority after an awaited quote failure and * immediately before the fallback creates a top-level message. */ beforeQuoteFallback?: () => void | Promise; + /** Revalidate managed authority immediately before each provider call. */ + beforeEffect?: () => void | Promise; onQuoteWithdrawn?: (messageId: string) => void; }; @@ -400,6 +413,7 @@ export async function dispatchPrimaryMessage( opts: DispatchPrimaryOptions, ): Promise { if (!opts.quoteTargetId) { + await opts.beforeEffect?.(); return { messageId: await (opts.suppressHook ? opts.dispatch(opts.content, opts.msgType, opts.uuid, true) @@ -409,6 +423,7 @@ export async function dispatchPrimaryMessage( } try { + await opts.beforeEffect?.(); const args = [ opts.appId, opts.quoteTargetId, @@ -418,13 +433,26 @@ export async function dispatchPrimaryMessage( opts.uuid, opts.hookContext, ] as const; - const messageId = opts.suppressHook - ? await deps.replyMessage(...args, { suppressHook: true }) + const hookOptions = opts.suppressHook + ? { suppressHook: true as const } + : opts.beforeHook + ? { + beforeHook: opts.beforeHook, + ...(opts.hookOrigin ? { hookOrigin: opts.hookOrigin } : {}), + } + : undefined; + const messageId = hookOptions + ? await deps.replyMessage(...args, hookOptions) : await deps.replyMessage(...args); return { messageId, primaryQuotedId: opts.quoteTargetId }; } catch (err: any) { if (err instanceof opts.MessageWithdrawnError) { - await opts.beforeQuoteFallback?.(); + // A quote failure is an awaited provider boundary. Revalidate once + // immediately before the fallback effect: callers may provide a + // fallback-specific composite fence (for example VC + managed-origin), + // otherwise reuse the ordinary per-effect fence. + if (opts.beforeQuoteFallback) await opts.beforeQuoteFallback(); + else await opts.beforeEffect?.(); opts.onQuoteWithdrawn?.(opts.quoteTargetId); return { messageId: await (opts.suppressHook @@ -437,14 +465,27 @@ export async function dispatchPrimaryMessage( opts.hookContext, { suppressHook: true }, ) - : deps.sendMessage( - opts.appId, - opts.targetChatId, - opts.content, - opts.msgType, - opts.uuid, - opts.hookContext, - )), + : opts.beforeHook + ? deps.sendMessage( + opts.appId, + opts.targetChatId, + opts.content, + opts.msgType, + opts.uuid, + opts.hookContext, + { + beforeHook: opts.beforeHook, + ...(opts.hookOrigin ? { hookOrigin: opts.hookOrigin } : {}), + }, + ) + : deps.sendMessage( + opts.appId, + opts.targetChatId, + opts.content, + opts.msgType, + opts.uuid, + opts.hookContext, + )), primaryQuotedId: null, }; } diff --git a/src/cli/session-list-liveness.ts b/src/cli/session-list-liveness.ts index c212b365d..810cadd63 100644 --- a/src/cli/session-list-liveness.ts +++ b/src/cli/session-list-liveness.ts @@ -3,6 +3,7 @@ export interface SessionListMarkers { cliId?: unknown; lastCliInput?: unknown; adoptedFrom?: unknown; + codexAppDispatchLedger?: readonly unknown[]; } export type SessionListDisposition = 'keep' | 'prune_real' | 'prune_scratch'; @@ -22,6 +23,7 @@ export function sessionListDisposition( ): SessionListDisposition { if (runtime.hasPid || runtime.hasBackingSession) return 'keep'; if (session.suspendedColdResume === true) return 'keep'; + if ((session.codexAppDispatchLedger?.length ?? 0) > 0) return 'keep'; return session.cliId || session.lastCliInput || session.adoptedFrom ? 'prune_real' : 'prune_scratch'; diff --git a/src/cli/supervisor-shutdown-client.ts b/src/cli/supervisor-shutdown-client.ts new file mode 100644 index 000000000..6b22775ee --- /dev/null +++ b/src/cli/supervisor-shutdown-client.ts @@ -0,0 +1,180 @@ +import { spawnSync } from 'node:child_process'; +import { daemonIpcAuthHeaders } from '../core/daemon-ipc-auth.js'; +import { readSupervisorProcessStartIdentity } from '../core/process-start-identity.js'; +import { SUPERVISOR_SHUTDOWN_ROUTE } from '../core/supervisor-shutdown-ipc.js'; +import type { AttestedPm2DaemonShutdownTarget } from './pm2-shutdown-capability.js'; + +export interface SupervisorShutdownHttpResult { + status?: number; + bodyRaw?: string; + error?: string; +} + +export interface SupervisorShutdownHttpInput { + port: number; + path: string; + headers: Record; + bodyRaw: string; + timeoutMs: number; +} + +export interface SupervisorShutdownClientRuntime { + readStartIdentity(pid: number): string | undefined; + postMany(inputs: SupervisorShutdownHttpInput[]): SupervisorShutdownHttpResult[]; +} + +const SYNC_HTTP_SCRIPT = String.raw` +const http = require('node:http'); +const inputs = JSON.parse(process.env.BOTMUX_SUPERVISOR_HTTP_REQUESTS); +function post(input) { + return new Promise((resolve) => { + let settled = false; + const finish = (value) => { if (!settled) { settled = true; resolve(value); } }; + const req = http.request({ + host: '127.0.0.1', port: input.port, path: input.path, method: 'POST', headers: input.headers, + }, (res) => { + const chunks = []; + res.on('data', chunk => chunks.push(Buffer.from(chunk))); + res.on('end', () => finish({ status: res.statusCode || 0, bodyRaw: Buffer.concat(chunks).toString('utf8') })); + }); + req.setTimeout(input.timeoutMs, () => req.destroy(new Error('supervisor shutdown request timed out'))); + req.on('error', err => finish({ error: String(err && err.message || err) })); + req.end(input.bodyRaw); + }); +} +Promise.all(inputs.map(post)).then(results => process.stdout.write(JSON.stringify(results))); +`; + +const defaultRuntime: SupervisorShutdownClientRuntime = { + readStartIdentity: readSupervisorProcessStartIdentity, + postMany(inputs): SupervisorShutdownHttpResult[] { + if (inputs.length === 0) return []; + const timeoutMs = Math.max(...inputs.map(input => input.timeoutMs)); + const result = spawnSync(process.execPath, ['-e', SYNC_HTTP_SCRIPT], { + encoding: 'utf8', + stdio: ['ignore', 'pipe', 'pipe'], + timeout: timeoutMs + 1_000, + env: { + ...process.env, + BOTMUX_SUPERVISOR_HTTP_REQUESTS: JSON.stringify(inputs), + }, + }); + if (result.status !== 0 || result.error) { + throw new Error( + result.error?.message + ?? String(result.stderr || `synchronous HTTP helper exited ${result.status}`).trim(), + ); + } + try { + const parsed = JSON.parse(String(result.stdout)) as SupervisorShutdownHttpResult[]; + if (!Array.isArray(parsed) || parsed.length !== inputs.length) { + throw new Error('invalid helper result'); + } + return parsed; + } catch (error) { + throw new Error(`invalid supervisor shutdown response transport: ${String(error)}`); + } + }, +}; + +export interface SupervisorShutdownAttempt { + target: AttestedPm2DaemonShutdownTarget; + ok: boolean; + error?: string; +} + +function requestInput( + target: AttestedPm2DaemonShutdownTarget, + secret: string, +): SupervisorShutdownHttpInput { + const bodyRaw = JSON.stringify({ + larkAppId: target.larkAppId, + bootInstanceId: target.bootInstanceId, + processStartIdentity: target.processStartIdentity, + }); + const headers = daemonIpcAuthHeaders({ + secret, + port: target.ipcPort, + method: 'POST', + path: SUPERVISOR_SHUTDOWN_ROUTE, + headers: { 'content-type': 'application/json' }, + }); + return { + port: target.ipcPort, + path: SUPERVISOR_SHUTDOWN_ROUTE, + headers: Object.fromEntries(headers.entries()), + bodyRaw, + timeoutMs: 4_000, + }; +} + +function exactAck( + target: AttestedPm2DaemonShutdownTarget, + response: SupervisorShutdownHttpResult, +): boolean { + if (response.status !== 202 || typeof response.bodyRaw !== 'string') return false; + let body: Record | undefined; + try { body = JSON.parse(response.bodyRaw) as Record; } + catch { return false; } + return body.ok === true + && body.accepted === true + && body.larkAppId === target.larkAppId + && body.bootInstanceId === target.bootInstanceId + && body.processStartIdentity === target.processStartIdentity; +} + +/** Dispatch one bounded helper containing concurrent HTTP requests for the + * whole initial fleet. One hung endpoint cannot delay request delivery to its + * peers or consume N times the fleet budget. */ +export function requestAttestedDaemonShutdownBatch( + targets: readonly AttestedPm2DaemonShutdownTarget[], + secret: string, + runtime: SupervisorShutdownClientRuntime = defaultRuntime, +): SupervisorShutdownAttempt[] { + const attempts = targets.map(target => ({ target, ok: false } as SupervisorShutdownAttempt)); + const eligible: Array<{ index: number; target: AttestedPm2DaemonShutdownTarget }> = []; + for (const [index, target] of targets.entries()) { + const currentStart = runtime.readStartIdentity(target.pid); + if (!currentStart) { + attempts[index]!.error = `daemon ${target.name}/${target.pid} exited before shutdown request`; + } else if (currentStart !== target.processStartIdentity) { + attempts[index]!.error = `daemon ${target.name}/${target.pid} process generation changed before shutdown request`; + } else { + eligible.push({ index, target }); + } + } + if (eligible.length === 0) return attempts; + + let responses: SupervisorShutdownHttpResult[]; + try { + responses = runtime.postMany(eligible.map(({ target }) => requestInput(target, secret))); + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + for (const { index } of eligible) attempts[index]!.error = message; + return attempts; + } + for (let i = 0; i < eligible.length; i++) { + const { index, target } = eligible[i]!; + const response = responses[i] ?? { error: 'missing helper response' }; + if (exactAck(target, response)) { + attempts[index] = { target, ok: true }; + } else { + attempts[index]!.error = response.error + ?? `daemon ${target.name}/${target.pid} rejected exact supervisor shutdown ` + + `(status ${response.status ?? 'transport-error'})`; + } + } + return attempts; +} + +/** Send a trusted-host request to the exact daemon boot/birth generation. + * The receiving process performs the decisive in-process comparison; a + * successor inheriting the port rejects rather than inheriting authority. */ +export function requestAttestedDaemonShutdown( + target: AttestedPm2DaemonShutdownTarget, + secret: string, + runtime: SupervisorShutdownClientRuntime = defaultRuntime, +): void { + const attempt = requestAttestedDaemonShutdownBatch([target], secret, runtime)[0]!; + if (!attempt.ok) throw new Error(attempt.error ?? `daemon ${target.name}/${target.pid} shutdown failed`); +} diff --git a/src/cli/vc-agent.ts b/src/cli/vc-agent.ts index 0ff95f508..f50ead449 100644 --- a/src/cli/vc-agent.ts +++ b/src/cli/vc-agent.ts @@ -297,6 +297,7 @@ async function cmdRequestOutput(args: string[]): Promise { config.session.dataDir, receiverSessionId, relayDir, + process.env.BOTMUX_ORIGIN_CHANNEL_ID, )?.capability; const discoveredReceiver = receiverAppId ? findOnlineDaemon(receiverAppId) : null; const receiverPort = Number.isSafeInteger(receiverPortRaw) && receiverPortRaw > 0 diff --git a/src/codex-app-runner.ts b/src/codex-app-runner.ts index b801c87e8..df416bd5e 100644 --- a/src/codex-app-runner.ts +++ b/src/codex-app-runner.ts @@ -1,6 +1,8 @@ #!/usr/bin/env node import { spawn, spawnSync, type ChildProcessWithoutNullStreams } from 'node:child_process'; import { Buffer } from 'node:buffer'; +import { createConnection, type Socket } from 'node:net'; +import type { KeyObject } from 'node:crypto'; import type { CodexAppTurnInput } from './types.js'; import { buildCodexAppTurnStartParams, @@ -10,6 +12,21 @@ import { type CodexVersion, } from './adapters/cli/codex-app-turn.js'; import { RunnerControlWriter } from './adapters/cli/runner-control-channel.js'; +import { + CODEX_APP_CONTROL_BOOTSTRAP_ENV, + CODEX_APP_CONTROL_FINAL_CHUNK_BYTES, + CODEX_APP_CONTROL_FINAL_MAX_BYTES, + CodexAppControlEndpointTracker, + CodexAppControlLineDecoder, + CodexAppControlRunnerHandshake, + armCodexAppControlHandshakeTimeout, + armCodexAppControlStartupTimeout, + consumeCodexAppControlBootstrap, + encodeCodexAppControlAuth, + encodeCodexAppSignedControlMarker, + parseCodexAppControlWireRecord, + takeCodexAppControlLocatorEndpoint, +} from './utils/codex-app-control.js'; type JsonObject = Record; @@ -17,6 +34,10 @@ interface Args { sessionId: string; codexBin: string; cwd: string; + controlGeneration: string; + controlPrivateKey: KeyObject; + controlSocketPath?: string; + controlLocatorPath?: string; threadId?: string; botName?: string; botOpenId?: string; @@ -27,6 +48,7 @@ interface PendingRequest { resolve: (value: any) => void; reject: (error: Error) => void; method: string; + timer?: NodeJS.Timeout; } interface ActiveTurn { @@ -34,8 +56,21 @@ interface ActiveTurn { * notifications from the server; botmux routing uses the stable client * message id carried alongside the queued input. */ nativeTurnId?: string; + /** Immutable Botmux/Lark identity sent through app-server. Mismatched native + * completions may be adopted only when their full items contain this exact + * client id. */ + clientUserMessageId?: string; + epoch: number; + reconciliation?: Promise; + identityConflictReported: boolean; + completed: boolean; + requestKind: 'start' | 'steer'; + requestAccepted: boolean; + pendingCompletions: JsonObject[]; + pendingNotifications: JsonObject[]; serverStarted: boolean; startedAtMs: number; + lastActivityMarkerAtMs: number; finalText: string; allAgentText: string; itemText: Map; @@ -49,16 +84,46 @@ interface QueuedInput { } const output = new RunnerControlWriter(); +const DEFAULT_REQUEST_TIMEOUT_MS = 30_000; +const RECONCILIATION_TIMEOUT_MS = 5_000; +const RECONCILIATION_PAGE_LIMIT = 3; +const RECONCILIATION_PAGE_SIZE = 50; + +class AppServerRpcError extends Error { + constructor( + readonly method: string, + readonly code: number | undefined, + readonly data: unknown, + message: string, + ) { + super(`${method}: ${message}`); + this.name = 'AppServerRpcError'; + } +} + +class AppServerRequestTimeoutError extends Error { + constructor(readonly method: string, readonly timeoutMs: number) { + super(`${method}: timed out after ${timeoutMs}ms; request acceptance is unknown`); + this.name = 'AppServerRequestTimeoutError'; + } +} function asError(value: unknown): Error { return value instanceof Error ? value : new Error(String(value)); } function parseArgs(argv: string[]): Args { + const controlBootstrapPath = process.env[CODEX_APP_CONTROL_BOOTSTRAP_ENV]; + // app-server and every model-launched tool inherit process.env. Remove even + // the non-secret bootstrap path before either can start; private key material + // was never present in env/argv/layout. + delete process.env[CODEX_APP_CONTROL_BOOTSTRAP_ENV]; const out: Args = { sessionId: '', codexBin: 'codex', cwd: process.cwd(), + controlGeneration: '', + controlPrivateKey: undefined as unknown as KeyObject, }; for (let i = 0; i < argv.length; i++) { const key = argv[i]; @@ -72,13 +137,15 @@ function parseArgs(argv: string[]): Args { else if (key === '--locale' && val !== undefined) { out.locale = val; i++; } } if (!out.sessionId) throw new Error('--session-id is required'); + if (!controlBootstrapPath) throw new Error(`${CODEX_APP_CONTROL_BOOTSTRAP_ENV} is required`); + const control = consumeCodexAppControlBootstrap(controlBootstrapPath, out.sessionId); + out.controlGeneration = control.generation; + out.controlPrivateKey = control.privateKey; + out.controlSocketPath = control.socketPath; + out.controlLocatorPath = control.locatorPath; return out; } -function emitMarker(kind: string, payload: unknown): void { - output.marker(kind, payload); -} - function writeLine(text = ''): void { output.line(text); } @@ -158,22 +225,40 @@ class AppServerClient { this.requestHandlers.push(handler); } - async initialize(): Promise { + async initialize(timeoutMs = DEFAULT_REQUEST_TIMEOUT_MS): Promise { await this.request('initialize', { clientInfo: { name: 'botmux-codex-app', version: '0.0.0' }, capabilities: { experimentalApi: true }, - }); + }, { timeoutMs }); this.notify('initialized'); } - request(method: string, params: unknown): Promise { + request( + method: string, + params: unknown, + options: { timeoutMs?: number } = {}, + ): Promise { const id = this.nextId++; return new Promise((resolve, reject) => { - this.pending.set(id, { resolve, reject, method }); + const timeoutMs = options.timeoutMs; + const pending: PendingRequest = { resolve, reject, method }; + if (timeoutMs !== undefined) { + if (!Number.isFinite(timeoutMs) || timeoutMs <= 0) { + reject(new AppServerRequestTimeoutError(method, Math.max(0, timeoutMs))); + return; + } + pending.timer = setTimeout(() => { + if (!this.pending.delete(id)) return; + reject(new AppServerRequestTimeoutError(method, timeoutMs)); + }, timeoutMs); + pending.timer.unref?.(); + } + this.pending.set(id, pending); try { this.write({ jsonrpc: '2.0', id, method, params }); } catch (err) { this.pending.delete(id); + if (pending.timer) clearTimeout(pending.timer); reject(asError(err)); } }); @@ -201,7 +286,10 @@ class AppServerClient { private failAll(err: Error): void { this.fatalError = this.fatalError ?? err; const fatal = this.fatalError; - for (const pending of this.pending.values()) pending.reject(fatal); + for (const pending of this.pending.values()) { + if (pending.timer) clearTimeout(pending.timer); + pending.reject(fatal); + } this.pending.clear(); } @@ -228,7 +316,15 @@ class AppServerClient { const pending = this.pending.get(msg.id); if (!pending) return; this.pending.delete(msg.id); - if (msg.error) pending.reject(new Error(`${pending.method}: ${JSON.stringify(msg.error)}`)); + if (pending.timer) clearTimeout(pending.timer); + if (msg.error) { + pending.reject(new AppServerRpcError( + pending.method, + typeof msg.error.code === 'number' ? msg.error.code : undefined, + msg.error.data, + typeof msg.error.message === 'string' ? msg.error.message : JSON.stringify(msg.error), + )); + } else pending.resolve(msg.result); return; } @@ -255,18 +351,217 @@ try { process.exit(2); } -const client = new AppServerClient(args.codexBin, args.cwd); +let controlSeq = 0; +let controlAckedSeq = 0; +let controlSentThrough = 0; +const controlQueue: Array<{ seq: number; kind: string; payload: JsonObject }> = []; +let controlFatal = false; +let controlSocket: Socket | undefined; +let controlChallenge: string | undefined; +let controlAccepted = false; +let controlAcceptanceCount = 0; +let controlReconnectTimer: NodeJS.Timeout | undefined; +let resolveControlReady!: () => void; +const controlReady = new Promise(resolve => { resolveControlReady = resolve; }); +const CONTROL_QUEUE_MAX_RECORDS = 2_048; +const controlEndpoints = new CodexAppControlEndpointTracker(); + +function scheduleControlReconnect(): void { + if (controlFatal || controlReconnectTimer) return; + controlReconnectTimer = setTimeout(() => { + controlReconnectTimer = undefined; + connectControlSocket(); + }, 250); +} + +function flushControlQueue(): void { + const socket = controlSocket; + const challenge = controlChallenge; + if (controlFatal || !socket || socket.destroyed || !controlAccepted || !challenge) return; + try { + for (const marker of controlQueue) { + if (marker.seq <= controlSentThrough) continue; + socket.write(`${encodeCodexAppSignedControlMarker( + args.controlPrivateKey, + args.sessionId, + args.controlGeneration, + challenge, + marker.seq, + marker.kind, + marker.payload, + )}\n`); + controlSentThrough = marker.seq; + } + } catch (err: any) { + controlFatal = true; + console.error(`Codex App control channel failed closed: ${err?.message ?? err}`); + process.exit(2); + } +} + +function nextControlEndpoint(): { endpoint: string; epoch?: string } | undefined { + if (args.controlSocketPath) return { endpoint: args.controlSocketPath }; + if (!args.controlLocatorPath) return undefined; + return takeCodexAppControlLocatorEndpoint({ + locatorPath: args.controlLocatorPath, + sessionId: args.sessionId, + tracker: controlEndpoints, + }); +} + +function connectControlSocket(): void { + if (controlFatal || (controlSocket && !controlSocket.destroyed)) return; + const target = nextControlEndpoint(); + if (!target) { + scheduleControlReconnect(); + return; + } + // A never-accepted locator endpoint may retry with the existing 250ms + // reconnect backoff; the protected 256-bit locator epoch is still required + // before acceptance. Once accepted, its endpoint is permanently burned and + // only a newly published locator can be used. + const socket = createConnection(target.endpoint); + const handshakeTimer = armCodexAppControlHandshakeTimeout(() => { + socket.destroy(new Error('Codex App control endpoint handshake timed out')); + }); + handshakeTimer.unref?.(); + const decoder = new CodexAppControlLineDecoder(); + const handshake = new CodexAppControlRunnerHandshake( + args.sessionId, + args.controlGeneration, + target.epoch, + ); + controlSocket = socket; + controlChallenge = undefined; + controlAccepted = false; + controlSentThrough = controlAckedSeq; + socket.setNoDelay(true); + socket.on('data', chunk => { + const decoded = decoder.push(chunk); + if (decoded.droppedMalformed) { + socket.destroy(new Error('oversized Codex App control response')); + return; + } + for (const line of decoded.lines) { + if (controlSocket !== socket) { + socket.destroy(new Error('unexpected Codex App control response')); + return; + } + const action = handshake.handle(parseCodexAppControlWireRecord(line), controlSentThrough); + if (action.type === 'authenticate') { + controlChallenge = action.challenge; + socket.write(`${encodeCodexAppControlAuth( + args.controlPrivateKey, + args.sessionId, + args.controlGeneration, + action.challenge, + )}\n`); + } else if (action.type === 'accepted') { + // `accepted` is intentionally unsigned. Its authority is the protected + // locator's independent epoch plus the already-bound random endpoint. + // Ed25519 still authenticates every runner marker to the worker. + controlAccepted = true; + clearTimeout(handshakeTimer); + if (args.controlLocatorPath) controlEndpoints.noteAccepted(target.endpoint); + controlAcceptanceCount++; + resolveControlReady(); + // The first acceptance happens before app-server initialization; it is + // not a ready boundary. Re-authentication can publish the live state + // only after initialization has completed. + if (controlAcceptanceCount > 1 && runnerReady) emitRunnerState(); + flushControlQueue(); + } else if (action.type === 'ack' && controlAccepted) { + if (action.seq > controlAckedSeq) controlAckedSeq = action.seq; + while (controlQueue[0] && controlQueue[0].seq <= controlAckedSeq) controlQueue.shift(); + } else { + socket.destroy(new Error('out-of-order Codex App control response')); + return; + } + } + }); + socket.on('error', () => { /* close schedules a retry */ }); + socket.on('close', () => { + clearTimeout(handshakeTimer); + if (controlSocket === socket) { + controlSocket = undefined; + controlChallenge = undefined; + controlAccepted = false; + } + scheduleControlReconnect(); + }); +} + +function emitMarker(kind: string, payload: JsonObject): void { + if (controlQueue.length >= CONTROL_QUEUE_MAX_RECORDS) { + controlFatal = true; + console.error('Codex App control queue exceeded its fail-closed bound'); + process.exit(2); + return; + } + controlQueue.push({ seq: ++controlSeq, kind, payload }); + flushControlQueue(); +} + +function emitFinalMarker(payload: JsonObject): void { + const original = Buffer.from(String(payload.content ?? ''), 'utf8'); + const truncated = original.length > CODEX_APP_CONTROL_FINAL_MAX_BYTES; + const content = truncated + ? Buffer.concat([ + original.subarray(0, CODEX_APP_CONTROL_FINAL_MAX_BYTES - 64), + Buffer.from('\n\n[botmux: final output truncated at control limit]', 'utf8'), + ]) + : original; + const id = `${String(payload.turnId ?? 'turn')}:${String(payload.completedAtMs ?? Date.now())}`; + const total = Math.ceil(content.length / CODEX_APP_CONTROL_FINAL_CHUNK_BYTES); + const { content: _content, ...metadata } = payload; + emitMarker('final-start', { id, total, truncated, ...metadata }); + for (let index = 0; index < total; index++) { + const start = index * CODEX_APP_CONTROL_FINAL_CHUNK_BYTES; + emitMarker('final-chunk', { + id, + index, + data: content.subarray(start, start + CODEX_APP_CONTROL_FINAL_CHUNK_BYTES).toString('base64'), + }); + } + emitMarker('final-end', { id, total }); +} + +connectControlSocket(); + +let client!: AppServerClient; let threadId = args.threadId; let threadReady = false; let activeTurn: ActiveTurn | null = null; +let activeTurnEpoch = 0; +/** App-server may start a Goal continuation without a Botmux input. Keep that + * native lifecycle separate from `activeTurn`; otherwise the next Lark input + * is incorrectly sent with turn/start and its completion can be discarded as + * belonging to an "unexpected" native turn. */ +let nativeActiveTurnId: string | undefined; const queue: QueuedInput[] = []; let inputBuffer = ''; let processing = false; +let runnerReady = false; let cleanInputUnsupported = false; let codexVersionChecked = false; let codexVersion: CodexVersion | undefined; let cleanVersionWarningShown = false; +function emitRunnerState( + busy = processing || queue.length > 0 || nativeActiveTurnId !== undefined, + tracksTurn = activeTurn !== null, +): void { + emitMarker('state', { + busy, + atMs: Date.now(), + // Input is accepted only after the runner has initialized and emitted this + // signed state. The worker uses this field as a runtime type-ahead gate; + // authentication alone never releases a prompt. + acceptingInput: runnerReady, + ...(busy && !tracksTurn ? { tracksTurn: false } : {}), + }); +} + function detectedCodexVersion(): CodexVersion | undefined { if (codexVersionChecked) return codexVersion; codexVersionChecked = true; @@ -284,11 +579,20 @@ function detectedCodexVersion(): CodexVersion | undefined { return codexVersion; } -function makeTurn(): ActiveTurn { +function makeTurn(clientUserMessageId: string | undefined, requestKind: 'start' | 'steer'): ActiveTurn { let resolveDone!: () => void; const done = new Promise(resolve => { resolveDone = resolve; }); return { + ...(clientUserMessageId ? { clientUserMessageId } : {}), + epoch: ++activeTurnEpoch, + identityConflictReported: false, + completed: false, + requestKind, + requestAccepted: false, + pendingCompletions: [], + pendingNotifications: [], startedAtMs: Date.now(), + lastActivityMarkerAtMs: 0, serverStarted: false, finalText: '', allAgentText: '', @@ -298,6 +602,25 @@ function makeTurn(): ActiveTurn { }; } +const TURN_ACTIVITY_MARKER_MIN_INTERVAL_MS = 5_000; + +/** + * Expose app-server lifecycle activity to the parent worker without polluting + * the visible terminal. Progress markers are throttled because token-delta + * notifications can arrive many times per second; submitted/completed edges + * are always emitted. + */ +function emitTurnActivity(turn: ActiveTurn, phase: 'submitted' | 'progress' | 'completed', force = false): void { + const atMs = Date.now(); + if (!force && atMs - turn.lastActivityMarkerAtMs < TURN_ACTIVITY_MARKER_MIN_INTERVAL_MS) return; + turn.lastActivityMarkerAtMs = atMs; + emitMarker('activity', { + phase, + atMs, + ...(turn.nativeTurnId ? { turnId: turn.nativeTurnId } : {}), + }); +} + function handleServerRequest(msg: JsonObject): boolean { const method = msg.method; if (method === 'item/commandExecution/requestApproval') { @@ -331,16 +654,236 @@ function handleServerRequest(msg: JsonObject): boolean { return false; } -function handleNotification(msg: JsonObject): void { +function exactClientItemIndexes(turn: JsonObject, clientUserMessageId: string): number[] { + if (turn?.itemsView !== 'full' || !Array.isArray(turn?.items)) return []; + const indexes: number[] = []; + for (let index = 0; index < turn.items.length; index++) { + const item = turn.items[index]; + if (item?.type === 'userMessage' && item.clientId === clientUserMessageId) indexes.push(index); + } + return indexes; +} + +function isTerminalNativeTurn(turn: JsonObject): boolean { + return turn?.status === undefined + || turn.status === 'completed' + || turn.status === 'failed' + || turn.status === 'interrupted'; +} + +/** Rebuild only from content causally after the exact user item. Never reuse + * streamed text from a different native turn during identity reconciliation. */ +function rebuildReconciledFinal(turn: JsonObject, userItemIndex: number): string { + const following = Array.isArray(turn.items) ? turn.items.slice(userItemIndex + 1) : []; + const finalAnswers = following.filter( + (item: JsonObject) => item?.type === 'agentMessage' && item.phase === 'final_answer', + ); + if (finalAnswers.length > 0) return String(finalAnswers.at(-1)?.text ?? ''); + if (turn?.error?.message) return `Codex App turn failed: ${String(turn.error.message)}`; + return ''; +} + +function completeActiveTurnFromNative(turn: ActiveTurn, nativeTurn: JsonObject, exactIndex?: number): void { + if (activeTurn !== turn || turn.completed || !isTerminalNativeTurn(nativeTurn)) return; + if (exactIndex !== undefined) { + turn.finalText = rebuildReconciledFinal(nativeTurn, exactIndex); + turn.allAgentText = ''; + } else if (nativeTurn?.error?.message && !turn.finalText) { + turn.finalText = `Codex App turn failed: ${String(nativeTurn.error.message)}`; + } + if (typeof nativeTurn?.id === 'string') { + turn.nativeTurnId = nativeTurn.id; + if (nativeActiveTurnId === nativeTurn.id) nativeActiveTurnId = undefined; + } + turn.completed = true; + emitTurnActivity(turn, 'completed', true); + turn.resolveDone(); +} + +function reportIdentityConflict(turn: ActiveTurn, observedNativeTurnId?: string, reason = 'no exact client id match'): void { + if (activeTurn !== turn || turn.identityConflictReported) return; + turn.identityConflictReported = true; + const stableTurnId = turn.clientUserMessageId; + const message = `Codex App native turn identity conflict (${reason}); refusing to attribute a completion without an exact clientUserMessageId match`; + writeLine(`[codex-app] ${message}`); + emitMarker('diagnostic', { + code: 'native_turn_identity_conflict', + message, + ...(stableTurnId ? { turnId: stableTurnId } : {}), + ...(turn.nativeTurnId ? { expectedNativeTurnId: turn.nativeTurnId } : {}), + ...(observedNativeTurnId ? { observedNativeTurnId } : {}), + atMs: Date.now(), + }); + // Attribution failed closed, but the logical Botmux turn must still settle. + // Leaving turn.done unresolved permanently poisons the runner FIFO and keeps + // the session "busy" forever. Emit one explicit error final under the stable + // client turn id; never reuse untrusted native text. + turn.finalText = `Codex App turn failed: ${message}`; + turn.allAgentText = ''; + if (turn.nativeTurnId && nativeActiveTurnId === turn.nativeTurnId) { + nativeActiveTurnId = undefined; + } + turn.completed = true; + emitTurnActivity(turn, 'completed', true); + turn.resolveDone(); +} + +async function reconcileCompletedTurn(turn: ActiveTurn, observedNativeTurnId?: string): Promise { + if (turn.reconciliation || turn.completed) return turn.reconciliation; + const clientUserMessageId = turn.clientUserMessageId; + if (!clientUserMessageId || !threadId) { + reportIdentityConflict(turn, observedNativeTurnId, 'legacy input has no clientUserMessageId'); + return; + } + const epoch = turn.epoch; + const deadlineAtMs = Date.now() + RECONCILIATION_TIMEOUT_MS; + turn.reconciliation = (async () => { + const matches: Array<{ turn: JsonObject; itemIndex: number }> = []; + let cursor: string | null | undefined; + for (let page = 0; page < RECONCILIATION_PAGE_LIMIT; page++) { + const remaining = deadlineAtMs - Date.now(); + if (remaining <= 0) break; + const result = await client.request('thread/turns/list', { + threadId, + ...(cursor ? { cursor } : {}), + limit: RECONCILIATION_PAGE_SIZE, + sortDirection: 'desc', + itemsView: 'full', + }, { timeoutMs: remaining }); + for (const candidate of Array.isArray(result?.data) ? result.data : []) { + if (!isTerminalNativeTurn(candidate)) continue; + const indexes = exactClientItemIndexes(candidate, clientUserMessageId); + if (indexes.length === 1) matches.push({ turn: candidate, itemIndex: indexes[0] }); + else if (indexes.length > 1) { + reportIdentityConflict(turn, observedNativeTurnId, 'client id appears more than once in one turn'); + return; + } + } + cursor = typeof result?.nextCursor === 'string' ? result.nextCursor : null; + if (!cursor) break; + } + if (activeTurn !== turn || turn.epoch !== epoch || turn.completed) return; + if (matches.length === 1) { + completeActiveTurnFromNative(turn, matches[0].turn, matches[0].itemIndex); + return; + } + reportIdentityConflict( + turn, + observedNativeTurnId, + matches.length === 0 ? 'bounded history lookup found no match' : 'bounded history lookup found multiple matches', + ); + })().catch(err => { + if (activeTurn === turn && !turn.completed) { + reportIdentityConflict(turn, observedNativeTurnId, `bounded history lookup failed: ${asError(err).message}`); + } + }); + await turn.reconciliation; +} + +function handleNotification(msg: JsonObject, replayedAfterResponse = false): void { const params = msg.params ?? {}; - if (!activeTurn || params.threadId !== threadId) return; - if (activeTurn.nativeTurnId && params.turnId && params.turnId !== activeTurn.nativeTurnId) return; + if (params.threadId !== threadId) return; + const notificationTurnId = params.turnId ?? params.turn?.id; if (msg.method === 'turn/started') { - activeTurn.serverStarted = true; - activeTurn.nativeTurnId = params.turn?.id ?? params.turnId ?? activeTurn.nativeTurnId; + const startedId = typeof notificationTurnId === 'string' ? notificationTurnId : undefined; + if (startedId && (!replayedAfterResponse + || nativeActiveTurnId === undefined + || nativeActiveTurnId === startedId)) nativeActiveTurnId = startedId; + const turn = activeTurn; + if (turn && startedId) { + const exact = turn.clientUserMessageId + ? exactClientItemIndexes(params.turn, turn.clientUserMessageId) + : []; + if (!turn.nativeTurnId && exact.length === 1) turn.nativeTurnId = startedId; + if (turn.nativeTurnId === startedId) { + turn.serverStarted = true; + emitTurnActivity(turn, 'progress', true); + return; + } + // app-server is allowed to publish turn/started before replying to + // turn/start. Without the response we do not yet know whether this is + // our native turn, but dropping it also loses the first (and sometimes + // only) progress edge. Replay it after the RPC binds nativeTurnId. + if (!turn.requestAccepted && turn.requestKind === 'start') { + const alreadyBufferedStart = turn.pendingNotifications.some( + notification => notification.method === 'turn/started' + && (notification.params?.turnId ?? notification.params?.turn?.id) === startedId, + ); + if (!alreadyBufferedStart) turn.pendingNotifications.push(msg); + return; + } + } + // A Goal continuation is native work, not a Botmux turn. Keep the worker + // busy while explicitly advertising that the initialized runner can accept + // a Lark follow-up through turn/steer. + if (runnerReady) emitRunnerState(true, false); + return; + } + + if (msg.method === 'turn/completed') { + const nativeTurn = params.turn ?? {}; + const completedId = typeof notificationTurnId === 'string' ? notificationTurnId : undefined; + if (completedId && nativeActiveTurnId === completedId) nativeActiveTurnId = undefined; + const turn = activeTurn; + if (!turn) { + if (runnerReady) emitRunnerState(); + return; + } + if (turn.nativeTurnId && completedId === turn.nativeTurnId) { + const exact = turn.clientUserMessageId + ? exactClientItemIndexes(nativeTurn, turn.clientUserMessageId) + : []; + if (exact.length === 1) { + completeActiveTurnFromNative(turn, nativeTurn, exact[0]); + return; + } + // A captured Goal turn can finish while turn/steer is still in flight. + // Until the RPC succeeds, native-id equality proves only that the old + // autonomous work completed; it does not prove this Lark input landed. + if (!turn.requestAccepted) { + turn.pendingCompletions.push(nativeTurn); + return; + } + if (turn.requestKind === 'steer') { + void reconcileCompletedTurn(turn, completedId); + return; + } + completeActiveTurnFromNative(turn, nativeTurn); + return; + } + const exact = turn.clientUserMessageId + ? exactClientItemIndexes(nativeTurn, turn.clientUserMessageId) + : []; + if (exact.length === 1) { + completeActiveTurnFromNative(turn, nativeTurn, exact[0]); + return; + } + if (!turn.requestAccepted) { + turn.pendingCompletions.push(nativeTurn); + return; + } + void reconcileCompletedTurn(turn, completedId); + return; + } + + const turn = activeTurn; + if (!turn) return; + if (!turn.requestAccepted) { + // turn/start notifications may beat their response. Buffer only that + // request's candidate native events and replay after the response chooses + // the authoritative id. For turn/steer, pre-response events can be old + // autonomous output and are deliberately never promoted into the final. + if (turn.requestKind === 'start' && typeof notificationTurnId === 'string') { + turn.pendingNotifications.push(msg); + } return; } + if (turn.nativeTurnId && notificationTurnId && notificationTurnId !== turn.nativeTurnId) return; + + // Every notification for the active app-server turn is evidence of forward + // progress, including reasoning/status events that do not render text. + emitTurnActivity(turn, 'progress'); if (msg.method === 'item/started') { const item = params.item; @@ -355,8 +898,8 @@ function handleNotification(msg: JsonObject): void { if (msg.method === 'item/agentMessage/delta') { const delta = String(params.delta ?? ''); const itemId = String(params.itemId ?? ''); - activeTurn.itemText.set(itemId, (activeTurn.itemText.get(itemId) ?? '') + delta); - activeTurn.allAgentText += delta; + turn.itemText.set(itemId, (turn.itemText.get(itemId) ?? '') + delta); + turn.allAgentText += delta; output.display(delta); return; } @@ -369,25 +912,35 @@ function handleNotification(msg: JsonObject): void { if (msg.method === 'item/completed') { const item = params.item; if (item?.type === 'agentMessage') { - if (item.phase === 'final_answer') activeTurn.finalText = String(item.text ?? ''); - else if (!activeTurn.itemText.has(item.id) && item.text) { - activeTurn.allAgentText += String(item.text); + if (item.phase === 'final_answer') turn.finalText = String(item.text ?? ''); + else if (!turn.itemText.has(item.id) && item.text) { + turn.allAgentText += String(item.text); } } return; } +} - if (msg.method === 'turn/completed') { - const turn = params.turn; - if (turn?.id && activeTurn.nativeTurnId && turn.id !== activeTurn.nativeTurnId) return; - if (turn?.error?.message && !activeTurn.finalText) { - activeTurn.finalText = `Codex App turn failed: ${turn.error.message}`; - } - activeTurn.resolveDone(); - } +function startupRequestTimeout(deadlineAtMs: number | undefined, method: string): number { + if (deadlineAtMs === undefined) return DEFAULT_REQUEST_TIMEOUT_MS; + const remaining = deadlineAtMs - Date.now(); + if (remaining <= 0) throw new AppServerRequestTimeoutError(method, 0); + return remaining; +} + +function isExplicitMissingThread(error: unknown): boolean { + if (!(error instanceof AppServerRpcError)) return false; + return /(thread|rollout|conversation).*(not found|does not exist|missing|unknown)|not found.*(thread|rollout|conversation)/i + .test(error.message); } -async function ensureThread(): Promise { +function isExplicitExpectedTurnInactive(error: unknown): boolean { + return error instanceof AppServerRpcError + && /(expected|active).*(turn).*(not active|no longer active|mismatch|does not match)|(turn).*(not active|no longer active).*(expected)/i + .test(error.message); +} + +async function ensureThread(startupDeadlineAtMs?: number): Promise { if (threadReady && threadId) return threadId; if (threadId) { @@ -403,13 +956,17 @@ async function ensureThread(): Promise { // Keep Codex App's rich history in sync with turns created by this // external runner so the desktop UI can render follow-up messages. persistExtendedHistory: true, - }); + }, { timeoutMs: startupRequestTimeout(startupDeadlineAtMs, 'thread/resume') }); const resumedThreadId = String(resumed.thread.id); threadId = resumedThreadId; threadReady = true; emitMarker('thread', { threadId: resumedThreadId }); return resumedThreadId; } catch (err: any) { + // A transport error or timeout is an ambiguous acceptance boundary. It + // must never fork history by silently creating a fresh thread. Only an + // explicit app-server "missing thread" rejection permits fallback. + if (!isExplicitMissingThread(err)) throw err; writeLine(`[codex-app] resume failed, starting a fresh thread: ${err?.message ?? err}`); threadId = undefined; threadReady = false; @@ -428,24 +985,31 @@ async function ensureThread(): Promise { // Keep Codex App's rich history in sync with turns created by this // external runner so the desktop UI can render follow-up messages. persistExtendedHistory: true, - }); + }, { timeoutMs: startupRequestTimeout(startupDeadlineAtMs, 'thread/start') }); const startedThreadId = String(started.thread.id); threadId = startedThreadId; threadReady = true; emitMarker('thread', { threadId: startedThreadId }); - try { - await client.request('thread/name/set', { + void client.request('thread/name/set', { threadId: startedThreadId, name: `botmux ${args.sessionId.slice(0, 8)}`, - }); - } catch { /* naming is cosmetic */ } + }, { timeoutMs: 2_000 }).catch(() => { /* naming is cosmetic */ }); return startedThreadId; } async function runTurn(message: QueuedInput): Promise { const tid = await ensureThread(); - const turn = makeTurn(); + const stableTurnId = message.codexAppInput?.clientUserMessageId; + let expectedSteerTurnId = nativeActiveTurnId; + const turn = makeTurn(stableTurnId, expectedSteerTurnId ? 'steer' : 'start'); + if (expectedSteerTurnId) { + turn.nativeTurnId = expectedSteerTurnId; + turn.serverStarted = true; + } activeTurn = turn; + // This edge proves the runner decoded and dequeued Botmux's control line, + // even if app-server stalls before acknowledging turn/start. + emitTurnActivity(turn, 'submitted', true); const version = message.codexAppInput ? detectedCodexVersion() : undefined; let built = buildCodexAppTurnStartParams({ threadId: tid, @@ -468,46 +1032,122 @@ async function runTurn(message: QueuedInput): Promise { writeLine(built.structured && message.codexAppInput ? message.codexAppInput.text : message.content); writeLine(); - let result; - try { - result = await client.request('turn/start', built.params); - } catch (err) { - if (!built.structured || turn.serverStarted || !isCleanInputCapabilityError(err)) throw err; - // The app-server explicitly rejected the experimental field before a turn - // started. Disable structured input for this runner lifetime and retry the - // preserved legacy prompt exactly once. - cleanInputUnsupported = true; - writeLine('[codex-app] clean input unsupported by app-server; retrying this turn with the legacy prompt'); - built = buildCodexAppTurnStartParams({ - threadId: tid, - cwd: args.cwd, - legacyContent: message.content, - codexAppInput: message.codexAppInput, - codexVersion: version, - structuredDisabled: true, + const requestBuiltTurn = (candidate: typeof built): Promise => { + if (!expectedSteerTurnId) return client.request('turn/start', candidate.params); + const { threadId, input, clientUserMessageId, additionalContext } = candidate.params; + return client.request('turn/steer', { + threadId, + input, + expectedTurnId: expectedSteerTurnId, + ...(clientUserMessageId ? { clientUserMessageId } : {}), + ...(additionalContext ? { additionalContext } : {}), }); - result = await client.request('turn/start', built.params); + }; + + let result: any; + let capabilityRetried = false; + let inactiveSteerFallback = false; + for (;;) { + try { + result = await requestBuiltTurn(built); + break; + } catch (err) { + if (expectedSteerTurnId + && !inactiveSteerFallback + && nativeActiveTurnId === undefined + && isExplicitExpectedTurnInactive(err)) { + // turn/steer was explicitly rejected before acceptance and the signed + // native completion already proved the captured Goal turn is gone. + // Starting the same client-id input is safe; timeout/transport/generic + // errors never enter this branch. + inactiveSteerFallback = true; + expectedSteerTurnId = undefined; + turn.requestKind = 'start'; + turn.nativeTurnId = undefined; + turn.serverStarted = false; + turn.pendingCompletions.length = 0; + turn.pendingNotifications.length = 0; + turn.finalText = ''; + turn.allAgentText = ''; + turn.itemText.clear(); + writeLine('[codex-app] captured native turn completed before steer acceptance; starting the same client-id input as a new turn'); + continue; + } + if (capabilityRetried + || !built.structured + || (!expectedSteerTurnId && turn.serverStarted) + || !isCleanInputCapabilityError(err)) throw err; + // The app-server explicitly rejected the experimental field before a turn + // started. Disable structured input for this runner lifetime and retry the + // preserved legacy prompt exactly once. + capabilityRetried = true; + cleanInputUnsupported = true; + writeLine('[codex-app] clean input unsupported by app-server; retrying this turn with the legacy prompt'); + built = buildCodexAppTurnStartParams({ + threadId: tid, + cwd: args.cwd, + legacyContent: message.content, + codexAppInput: message.codexAppInput, + codexVersion: version, + structuredDisabled: true, + }); + } + } + turn.nativeTurnId = result.turn?.id ?? result.turnId ?? turn.nativeTurnId; + turn.requestAccepted = true; + // A response may arrive after its turn completed and a Goal continuation B + // already started. Never let late response A overwrite the newer global + // native lifecycle. If no newer turn exists, temporarily restoring A is safe + // because the buffered A completion below clears it immediately. + if (turn.nativeTurnId + && (nativeActiveTurnId === undefined || nativeActiveTurnId === turn.nativeTurnId)) { + nativeActiveTurnId = turn.nativeTurnId; + } + const pendingNotifications = turn.pendingNotifications.splice(0); + for (const notification of pendingNotifications) { + const notificationTurnId = notification.params?.turnId ?? notification.params?.turn?.id; + if (notificationTurnId === turn.nativeTurnId) handleNotification(notification, true); + } + const pendingCompletions = turn.pendingCompletions.splice(0); + if (pendingCompletions.length > 0 && !turn.completed) { + const exactMatches = stableTurnId + ? pendingCompletions.flatMap(completion => { + const indexes = exactClientItemIndexes(completion, stableTurnId); + return indexes.length === 1 ? [{ completion, itemIndex: indexes[0] }] : []; + }) + : []; + const nativeMatches = pendingCompletions.filter( + completion => completion?.id === turn.nativeTurnId, + ); + if (exactMatches.length === 1) { + completeActiveTurnFromNative(turn, exactMatches[0].completion, exactMatches[0].itemIndex); + } else if (exactMatches.length > 1 || nativeMatches.length > 1) { + reportIdentityConflict(turn, turn.nativeTurnId, 'multiple pre-response completions matched one request'); + } else if (turn.requestKind === 'steer' && nativeMatches.length === 1) { + void reconcileCompletedTurn(turn, turn.nativeTurnId); + } else if (turn.requestKind === 'start' && nativeMatches.length === 1) { + completeActiveTurnFromNative(turn, nativeMatches[0]); + } } - turn.nativeTurnId = result.turn?.id ?? turn.nativeTurnId; await turn.done; const finalText = (turn.finalText || turn.allAgentText).trim(); const completedAtMs = Date.now(); - if (finalText) { - // clientUserMessageId is the daemon-frozen botmux/Lark turn identity. The - // app-server generates a different id for the same logical turn; exposing - // that native id as `turnId` breaks daemon wait maps, VC suppression and - // reply routing. When no structured sidecar exists, omit turnId so the - // worker deliberately falls back to its current botmux turn attribution. - const stableTurnId = message.codexAppInput?.clientUserMessageId; - emitMarker('final', { - ...(stableTurnId ? { turnId: stableTurnId } : {}), - ...(turn.nativeTurnId ? { nativeTurnId: turn.nativeTurnId } : {}), - content: finalText, - startedAtMs: turn.startedAtMs, - completedAtMs, - }); - } + // Every dequeued runner input gets one complete final transaction, including + // an empty model answer. Without that zero-chunk boundary the worker cannot + // advance its attribution FIFO safely before the next queued turn finishes. + // clientUserMessageId is the daemon-frozen botmux/Lark turn identity. The + // app-server generates a different id for the same logical turn; exposing + // that native id as `turnId` breaks daemon wait maps, VC suppression and + // reply routing. When no structured sidecar exists, omit turnId so the + // worker resolves it from its own FIFO head. + emitFinalMarker({ + ...(stableTurnId ? { turnId: stableTurnId } : {}), + ...(turn.nativeTurnId ? { nativeTurnId: turn.nativeTurnId } : {}), + content: finalText, + startedAtMs: turn.startedAtMs, + completedAtMs, + }); writeLine(); activeTurn = null; } @@ -526,7 +1166,7 @@ async function drainQueue(): Promise { const stableTurnId = next.codexAppInput?.clientUserMessageId; const nativeTurnId = activeTurn?.nativeTurnId; writeLine(message); - emitMarker('final', { + emitFinalMarker({ ...(stableTurnId ? { turnId: stableTurnId } : {}), ...(nativeTurnId ? { nativeTurnId } : {}), content: message, @@ -535,7 +1175,16 @@ async function drainQueue(): Promise { }); activeTurn = null; } - prompt(); + // Do not publish a transient idle boundary between inputs already queued + // in the serial runner. Once the queue is truly empty, append signed + // busy:false only AFTER completed + every final fragment. The worker can + // therefore become ready even if the terminal prompt is lost, while its + // IPC order remains final_output before prompt_ready. + if (queue.length === 0) { + const nativeBusy = nativeActiveTurnId !== undefined; + emitRunnerState(nativeBusy, !nativeBusy); + if (!nativeBusy) prompt(); + } } } finally { processing = false; @@ -586,11 +1235,31 @@ function handleInput(data: Buffer): void { } async function main(): Promise { + const testTimeout = process.env.NODE_ENV === 'test' + ? Number(process.env.BOTMUX_TEST_CODEX_APP_STARTUP_TIMEOUT_MS) + : Number.NaN; + const startupTimeoutMs = Number.isFinite(testTimeout) && testTimeout > 0 + ? testTimeout + : 90_000; + const startupDeadlineAtMs = Date.now() + startupTimeoutMs; + const authTimeout = armCodexAppControlStartupTimeout(() => { + console.error('Codex App startup timed out before the first signed runner state'); + process.exit(2); + }, startupTimeoutMs); + await controlReady; + client = new AppServerClient(args.codexBin, args.cwd); client.onRequest(handleServerRequest); client.onNotification(handleNotification); - await client.initialize(); - await ensureThread(); + await client.initialize(startupRequestTimeout(startupDeadlineAtMs, 'initialize')); + await ensureThread(startupDeadlineAtMs); writeLine('Codex App connected.'); + runnerReady = true; + // Initial readiness is signed too; never rely on terminal rendering as the + // only path that releases the worker's first-prompt gate. + emitRunnerState(false); + // Authentication is deliberately insufficient. Keep the absolute startup + // timer armed through initialize + resume/start and the first signed state. + clearTimeout(authTimeout); if (process.stdin.isTTY) process.stdin.setRawMode(true); process.stdin.resume(); process.stdin.on('data', handleInput); @@ -598,16 +1267,24 @@ async function main(): Promise { } process.on('SIGTERM', () => { - client.close(); + if (controlReconnectTimer) clearTimeout(controlReconnectTimer); + controlSocket?.destroy(); + client?.close(); process.exit(0); }); process.on('SIGINT', () => { - client.close(); + if (controlReconnectTimer) clearTimeout(controlReconnectTimer); + controlSocket?.destroy(); + client?.close(); process.exit(130); }); main().catch(err => { + if (!runnerReady && err instanceof AppServerRequestTimeoutError) { + output.error('Codex App startup timed out before the first signed runner state\n'); + process.exit(2); + } output.error(`${err?.stack ?? err?.message ?? err}\n`); process.exit(1); }); diff --git a/src/codex-rpc-engine.ts b/src/codex-rpc-engine.ts index 97f705fd0..c331a226a 100644 --- a/src/codex-rpc-engine.ts +++ b/src/codex-rpc-engine.ts @@ -75,6 +75,30 @@ export interface CodexRpcEngineOpts { * worker uses it to kill the now-orphaned `codex --remote` pane so the normal * exit→daemon-refork→resume path re-engages RPC on a fresh app-server (P1). */ onDead?: () => void; + /** Authoritative native turn terminal. `turn/start` returns a native Codex + * turn id which this engine binds to the exact Botmux delivery attempt before + * resolving the ack. The worker uses that identity to release only the + * matching rpcActive bridge entry. */ + onTurnTerminal?: (terminal: CodexRpcTurnTerminal) => void; +} + +export interface CodexRpcTurnIdentity { + turnId: string; + dispatchAttempt?: number; +} + +export type CodexRpcTurnTerminalStatus = + | 'completed' + | 'failed' + | 'aborted' + | 'engine-dead' + | 'stopped'; + +export interface CodexRpcTurnTerminal { + identity: CodexRpcTurnIdentity; + nativeTurnId: string; + status: CodexRpcTurnTerminalStatus; + errorCode?: string; } /** Server→client requests are auto-answered so codex never blocks on a human; @@ -103,7 +127,20 @@ export class CodexRpcEngine { private child?: ChildProcess; private ws?: WebSocket; private nextId = 1; - private pending = new Map void; reject: (e: Error) => void; timer: ReturnType }>(); + private pending = new Map void; + reject: (e: Error) => void; + timer: ReturnType; + method: string; + turnIdentity?: CodexRpcTurnIdentity; + }>(); + private readonly turnOwners = new Map(); + private readonly nativeTurnByOwner = new Map(); + private readonly terminalNativeTurns = new Set(); + private readonly deferredUnownedTerminals = new Map(); private port = 0; private threadId?: string; private closed = false; @@ -119,6 +156,97 @@ export class CodexRpcEngine { get activeThreadId(): string | undefined { return this.threadId; } get appServerPid(): number | undefined { return this.child?.pid; } + private ownerKey(identity: CodexRpcTurnIdentity): string { + return `${identity.turnId}\0${identity.dispatchAttempt ?? ''}`; + } + + private takeNativeTurnId(identity: CodexRpcTurnIdentity): string | undefined { + const ownerKey = this.ownerKey(identity); + const nativeTurnId = this.nativeTurnByOwner.get(ownerKey); + this.nativeTurnByOwner.delete(ownerKey); + return nativeTurnId; + } + + private bindNativeTurn(nativeTurnId: string, identity: CodexRpcTurnIdentity): void { + if (!nativeTurnId) throw new Error('turn/start returned no native turn id'); + const existingOwner = this.turnOwners.get(nativeTurnId); + if (existingOwner && this.ownerKey(existingOwner) !== this.ownerKey(identity)) { + throw new Error(`native turn ${nativeTurnId} was rebound to a different Botmux attempt`); + } + const ownerKey = this.ownerKey(identity); + const existingNative = this.nativeTurnByOwner.get(ownerKey); + if (existingNative && existingNative !== nativeTurnId) { + throw new Error(`Botmux attempt ${identity.turnId} was rebound to a different native turn`); + } + // A terminal notification may precede the turn/start response. In that + // ordering the terminal has already retired this native id; the response + // still needs to hand the id to the awaiting sender, but must not resurrect + // it as active (which would leak ownership until engine teardown). + if (!this.terminalNativeTurns.has(nativeTurnId)) { + this.turnOwners.set(nativeTurnId, { ...identity }); + } + this.nativeTurnByOwner.set(ownerKey, nativeTurnId); + const deferred = this.deferredUnownedTerminals.get(nativeTurnId); + if (deferred) { + this.deferredUnownedTerminals.delete(nativeTurnId); + this.emitTurnTerminal(nativeTurnId, deferred.status, deferred.errorCode); + } + } + + private emitTurnTerminal( + nativeTurnId: string, + status: CodexRpcTurnTerminalStatus, + errorCode?: string, + ): void { + if (!nativeTurnId || this.terminalNativeTurns.has(nativeTurnId)) return; + const identity = this.turnOwners.get(nativeTurnId); + if (!identity) { + // An attached TUI can broadcast turns not submitted by this engine. Never + // guess their Botmux owner from "one pending request". Buffer the terminal + // by native id so a later successful turn/start response can bind and + // replay it exactly; a response error/timeout leaves it unowned. + if (!this.deferredUnownedTerminals.has(nativeTurnId)) { + this.deferredUnownedTerminals.set(nativeTurnId, { + status, + ...(errorCode ? { errorCode } : {}), + }); + } + if (this.deferredUnownedTerminals.size > 1024) { + const oldest = this.deferredUnownedTerminals.keys().next().value as string | undefined; + if (oldest) this.deferredUnownedTerminals.delete(oldest); + } + this.log(`[codex-rpc] buffered terminal for unowned native turn ${nativeTurnId}`); + return; + } + this.terminalNativeTurns.add(nativeTurnId); + if (this.terminalNativeTurns.size > 1024) { + const oldest = this.terminalNativeTurns.values().next().value as string | undefined; + if (oldest) this.terminalNativeTurns.delete(oldest); + } + this.turnOwners.delete(nativeTurnId); + // Keep owner→native through the request continuation. A fast app-server can + // send turn/completed in the same socket read as the turn/start response; + // deleting here would make the awaiting sender falsely conclude that the + // accepted response carried no native id. + try { + this.opts.onTurnTerminal?.({ + identity: { ...identity }, + nativeTurnId, + status, + ...(errorCode ? { errorCode } : {}), + }); + } catch { /* worker callback is best-effort; engine transport must continue */ } + } + + private emitAllTurnTerminals( + status: Extract, + errorCode: string, + ): void { + for (const nativeTurnId of [...this.turnOwners.keys()]) { + this.emitTurnTerminal(nativeTurnId, status, errorCode); + } + } + /** Spawn the app-server, connect, and complete the initialize handshake. */ async start(): Promise { this.reapStaleAppServer(); @@ -198,7 +326,11 @@ export class CodexRpcEngine { * opts.fatalOnTimeout=false makes a timeout reject only THIS request instead of * tearing the engine down — used for the fresh first turn, whose ambiguity is * then resolved against rollout persistence (see sendFirstTurn). */ - async sendTurn(content: string, clientUserMessageId?: string, opts?: { timeoutMs?: number; fatalOnTimeout?: boolean }): Promise { + async sendTurn( + content: string, + identity: CodexRpcTurnIdentity, + opts?: { timeoutMs?: number; fatalOnTimeout?: boolean }, + ): Promise<{ nativeTurnId: string }> { if (!this.threadId) throw new Error('sendTurn before startThread/resumeThread'); const params: Json = { threadId: this.threadId, @@ -207,8 +339,15 @@ export class CodexRpcEngine { approvalPolicy: 'never', sandboxPolicy: { type: 'dangerFullAccess' }, }; - if (clientUserMessageId) params.clientUserMessageId = clientUserMessageId; - await this.request('turn/start', params, opts); + params.clientUserMessageId = identity.turnId; + try { + await this.request('turn/start', params, opts, undefined, identity); + } catch (err) { + throw err; + } + const nativeTurnId = this.takeNativeTurnId(identity); + if (!nativeTurnId) throw new Error('turn/start ack did not bind a native turn id'); + return { nativeTurnId }; } /** Deliver the FRESH first turn and resolve its outcome as one of THREE states, @@ -228,7 +367,11 @@ export class CodexRpcEngine { * case is ambiguous, and a timeout is non-fatal so the engine survives to serve * the accepted/ambiguous cases. `rolloutProbe` is the ground-truth positive * check (matches this turn's user_message in the persisted rollout). */ - async sendFirstTurn(content: string, clientUserMessageId: string | undefined, rolloutProbe: (threadId: string) => Promise): Promise<'accepted' | 'not-sent' | 'ambiguous'> { + async sendFirstTurn( + content: string, + identity: CodexRpcTurnIdentity, + rolloutProbe: (threadId: string) => Promise, + ): Promise<{ outcome: 'accepted' | 'not-sent' | 'ambiguous'; nativeTurnId?: string }> { const threadId = this.threadId; if (!threadId) throw new Error('sendFirstTurn before startThread'); let dispatched = false; @@ -239,26 +382,45 @@ export class CodexRpcEngine { approvalPolicy: 'never', sandboxPolicy: { type: 'dangerFullAccess' }, }; - if (clientUserMessageId) params.clientUserMessageId = clientUserMessageId; + params.clientUserMessageId = identity.turnId; try { - await this.request('turn/start', params, { timeoutMs: this.opts.requestTimeoutMs ?? 15_000, fatalOnTimeout: false }, () => { dispatched = true; }); - return 'accepted'; // ack received + await this.request( + 'turn/start', + params, + { timeoutMs: this.opts.requestTimeoutMs ?? 15_000, fatalOnTimeout: false }, + () => { dispatched = true; }, + identity, + ); + return { outcome: 'accepted', nativeTurnId: this.takeNativeTurnId(identity) }; } catch (err) { if (!dispatched) { this.log(`[codex-rpc] first turn/start not dispatched (${(err as Error).message}); safe to paste`); - return 'not-sent'; + return { outcome: 'not-sent' }; } // Dispatched but no ack — the ONLY safe resolution is positive rollout // evidence; absence is NOT proof it didn't run (it may persist >window or be // queued server-side), so no-evidence stays ambiguous. this.log(`[codex-rpc] first turn ack lost after dispatch (${(err as Error).message}); checking rollout for positive evidence`); - const landed = await rolloutProbe(threadId); - return landed ? 'accepted' : 'ambiguous'; + let landed = false; + try { + landed = await rolloutProbe(threadId); + } catch (probeErr) { + // The frame crossed the socket boundary, so a failed evidence probe is + // still ambiguous. Never bubble this into engageCodexRpc's paste + // fallback: doing so could execute an already-landed first turn twice. + this.log(`[codex-rpc] first-turn rollout probe failed (${(probeErr as Error).message}); keeping delivery ambiguous`); + } + return landed + ? { outcome: 'accepted', nativeTurnId: undefined } + : { outcome: 'ambiguous' }; } } stop(): void { + if (this.closed) return; this.closed = true; + this.emitAllTurnTerminals('stopped', 'rpc_engine_stopped'); + this.deferredUnownedTerminals.clear(); try { this.ws?.close(); } catch { /* already gone */ } const pid = this.child?.pid; if (pid) { @@ -378,7 +540,11 @@ export class CodexRpcEngine { params: unknown, opts?: { timeoutMs?: number; fatalOnTimeout?: boolean }, onDispatch?: () => void, + turnIdentity?: CodexRpcTurnIdentity, ): Promise { + if (turnIdentity && [...this.pending.values()].some(p => p.method === 'turn/start')) { + return Promise.reject(new Error('concurrent turn/start requests are not allowed')); + } const timeoutMs = opts?.timeoutMs ?? this.opts.requestTimeoutMs ?? REQUEST_TIMEOUT_MS; const fatalOnTimeout = opts?.fatalOnTimeout !== false; // default fatal const id = this.nextId++; @@ -396,12 +562,20 @@ export class CodexRpcEngine { } else { // Non-fatal (the fresh first turn): reject only THIS request and keep // the engine alive, so its ambiguity can be resolved against rollout - // persistence and the viewer can still resume if the turn landed (P1-1). + // persistence and the viewer can still resume if the turn landed + // (P1-1). Pre-response native notifications are deliberately not + // correlated here because an attached TUI can start unrelated turns. this.pending.delete(id); reject(err); } }, timeoutMs); timer.unref?.(); - this.pending.set(id, { resolve, reject, timer }); + this.pending.set(id, { + resolve, + reject, + timer, + method, + ...(turnIdentity ? { turnIdentity: { ...turnIdentity } } : {}), + }); // onDispatch fires ONLY after send() succeeds (ws was OPEN + no throw) — the // frame is then on the socket, so any later failure is "dispatched" and must // be treated as ambiguous, never not-sent (Codex P1-1 boundary). @@ -432,8 +606,20 @@ export class CodexRpcEngine { if (!p) return; this.pending.delete(msg.id); clearTimeout(p.timer); - if (msg.error) p.reject(new Error(typeof msg.error === 'object' ? JSON.stringify(msg.error) : String(msg.error))); - else p.resolve(msg.result); + if (msg.error) { + p.reject(new Error(typeof msg.error === 'object' ? JSON.stringify(msg.error) : String(msg.error))); + } else { + try { + if (p.turnIdentity) { + const nativeTurnId = String(msg.result?.turn?.id ?? ''); + this.bindNativeTurn(nativeTurnId, p.turnIdentity); + } + p.resolve(msg.result); + } catch (err) { + p.reject(err as Error); + this.failAll(err as Error); + } + } return; } // Server→client request (approvals / elicitations): auto-answer. @@ -441,8 +627,44 @@ export class CodexRpcEngine { this.respond(msg.id, autoApproval(msg.method)); return; } - // Notifications (turn/item/mcp events) are ignored here — the attached TUI - // renders them; botmux reads the pane as usual. + if (typeof msg.method === 'string') { + const params = msg.params ?? {}; + const nativeTurnId = String(params.turn?.id ?? params.turnId ?? ''); + if (msg.method === 'turn/started' && nativeTurnId && !this.turnOwners.has(nativeTurnId)) { + // Pre-response start notifications are not necessarily ours: the + // attached TUI can start an unrelated local turn on the same thread. + // Only the turn/start response (or transcript evidence for the special + // fresh-first path) may establish Botmux ownership. + this.log(`[codex-rpc] observed unowned turn/started ${nativeTurnId}; awaiting exact response binding`); + return; + } + if (msg.method === 'turn/completed' && nativeTurnId) { + const turn = params.turn ?? {}; + const rawStatus = String(turn.status ?? '').toLowerCase(); + const errorCode = String(turn.error?.code ?? turn.error?.message ?? ''); + const failed = !!turn.error || ['failed', 'error'].includes(rawStatus); + const aborted = ['aborted', 'cancelled', 'canceled', 'interrupted'].includes(rawStatus); + this.emitTurnTerminal( + nativeTurnId, + aborted ? 'aborted' : failed ? 'failed' : 'completed', + errorCode || undefined, + ); + return; + } + if (['turn/aborted', 'turn/cancelled', 'turn/canceled'].includes(msg.method) && nativeTurnId) { + this.emitTurnTerminal(nativeTurnId, 'aborted', msg.method.replace('turn/', 'rpc_turn_')); + return; + } + if (['turn/failed', 'turn/error'].includes(msg.method) && nativeTurnId) { + this.emitTurnTerminal( + nativeTurnId, + 'failed', + String(params.error?.code ?? params.error?.message ?? 'rpc_turn_failed'), + ); + return; + } + } + // Other notifications (item/mcp events) are rendered by the attached TUI. } private failAll(err: Error): void { @@ -451,6 +673,7 @@ export class CodexRpcEngine { this.pending.clear(); if (!this.closed && !this.deadNotified) { this.deadNotified = true; + this.emitAllTurnTerminals('engine-dead', 'rpc_engine_dead'); try { this.opts.onDead?.(); } catch { /* best effort */ } } } diff --git a/src/codex-rpc-lifecycle.ts b/src/codex-rpc-lifecycle.ts index 72c8ce5f9..f51f2b083 100644 --- a/src/codex-rpc-lifecycle.ts +++ b/src/codex-rpc-lifecycle.ts @@ -12,6 +12,64 @@ type InitCfg = Extract; * diverges (--resume flag) and needs its own verification before inclusion. */ export const RPC_CAPABLE_CLIS = new Set(['codex', 'traex']); +/** Retry cadence for native turn/completed → rollout visibility hydration. + * Total window is 11.55s: bounded, but intentionally aligned with the 12s + * first-turn persistence probe so a fast terminal does not discard fallback + * output merely because the rollout filesystem is a few seconds behind. */ +export const CODEX_RPC_TERMINAL_HYDRATION_DELAYS_MS = [ + 50, + 100, + 200, + 400, + 800, + 1_600, + 2_400, + 3_000, + 3_000, +] as const; + +/** Ordinary transcript ingest must not advance past a turn/start ACK that has + * not installed its exact bridge mark yet. Native-terminal hydration for an + * older owner is different: it must keep draining that owner's final output + * even while a successor waits for ACK. Any successor events reached by that + * drain stay in CodexBridgeQueue's unmatched replay buffer until activation. + * + * A matching awaiting owner still blocks hydration, because consuming its + * transcript before the exact mark exists would retire the same logical + * delivery against incomplete local ownership. */ +export function rpcTranscriptIngestBlockedByAwaitingActivation( + awaitingOwnerKeys: Iterable, + hydrationOwnerKey?: string, +): boolean { + for (const ownerKey of awaitingOwnerKeys) { + if (hydrationOwnerKey === undefined || ownerKey === hydrationOwnerKey) { + return true; + } + } + return false; +} + +/** Monotonic fence for async app-server engagement. Worker IPC handlers are not + * serialized, so restart/close can invalidate an engage while it is awaiting + * /readyz, thread creation, or first-turn rollout evidence. Only the lease + * returned by the latest begin() may publish process-global engine state. */ +export class RpcEngagementFence { + private epoch = 0; + + begin(): number { + this.epoch += 1; + return this.epoch; + } + + invalidate(): void { + this.epoch += 1; + } + + isCurrent(lease: number): boolean { + return lease === this.epoch; + } +} + /** All fail-closed gates for codex-family RPC input in ONE place so the worker's * pane-branching and engageCodexRpc agree. Every excluded case degrades to the * normal paste path — never a silent capability/security change: @@ -99,13 +157,15 @@ export interface PaneProbes { * - 'not-engaged' — setup failed OR fresh frame never dispatched → paste. */ export type EngageOutcome = 'accepted' | 'ambiguous' | 'resumed' | 'not-engaged'; -/** Whether the FRESH first turn should PRE-mark the codex bridge (so the reply is - * attributed even if the model skips `botmux send`). ONLY 'accepted' — a - * confirmed turn whose prompt is not re-queued, so exactly one mark exists and - * the reply consumes it. NOT 'ambiguous' (no positive evidence the turn ran → an - * unstarted queue head would wedge every later turn's drain — Codex P1) and NOT - * 'not-sent'/'resumed' (those never reach the pre-mark; not-sent's paste flush - * marks once, resume flushes its queued prompt). */ +/** Whether the FRESH first turn should use the normal confirmed pre-mark path + * (so the reply is attributed even if the model skips `botmux send`). ONLY + * 'accepted' — a confirmed turn whose prompt is not re-queued. The worker also + * gives 'ambiguous' its own attribution-only mark plus a fail-closed owner: + * structured terminal retires it when visible, while exact engine teardown is + * the intentional fallback when no native owner can ever be mapped. That + * separate path never resends the prompt and prevents a permanent queue head. + * 'not-sent'/'resumed' never reach either pre-mark path; not-sent's paste flush + * marks once, and resume flushes its queued prompt. */ export function shouldPreMarkFirstTurn(outcome: EngageOutcome): boolean { return outcome === 'accepted'; } diff --git a/src/core/bot-turn-mutation-gate.ts b/src/core/bot-turn-mutation-gate.ts new file mode 100644 index 000000000..317b706b5 --- /dev/null +++ b/src/core/bot-turn-mutation-gate.ts @@ -0,0 +1,296 @@ +/** + * Per-bot admission/drain gate for mutations that replace every live worker + * generation (currently read-isolation). JavaScript's single thread makes the + * check+increment and close+publish edges atomic, while the waiters cover all + * intervening awaits in inbound/HTTP turn preparation. + */ +import { AsyncLocalStorage } from 'node:async_hooks'; + +type GateState = { + activeAdmissions: number; + mutating: boolean; + openWaiters: Array; + drainWaiters: Array; +}; + +type GateWaiter = { wake: () => void }; + +const gates = new Map(); +type AdmissionLease = { active: boolean; ownerFinished: boolean }; +type MutationLease = { active: boolean }; +const admissionContext = new AsyncLocalStorage>(); +const mutationContext = new AsyncLocalStorage>(); + +function stateFor(larkAppId: string): GateState { + let state = gates.get(larkAppId); + if (!state) { + state = { activeAdmissions: 0, mutating: false, openWaiters: [], drainWaiters: [] }; + gates.set(larkAppId, state); + } + return state; +} + +function waitUntilOpen(state: GateState): Promise { + return state.mutating + ? new Promise(resolve => state.openWaiters.push({ wake: resolve })) + : Promise.resolve(); +} + +function waitUntilDrained(state: GateState): Promise { + return state.activeAdmissions > 0 + ? new Promise(resolve => state.drainWaiters.push({ wake: resolve })) + : Promise.resolve(); +} + +function wakeAll(waiters: GateWaiter[]): void { + const pending = waiters.splice(0); + for (const waiter of pending) waiter.wake(); +} + +/** Cancellable condition wait used only by bounded acquisition. Timeout + * removes the exact waiter, so it cannot resume later and run a stale shutdown + * after its caller already reported refusal. */ +function waitUntilBefore( + waiters: GateWaiter[], + ready: () => boolean, + deadlineMs: number, +): Promise { + if (Date.now() >= deadlineMs) return Promise.resolve(false); + if (ready()) return Promise.resolve(true); + const remaining = deadlineMs - Date.now(); + if (remaining <= 0) return Promise.resolve(false); + return new Promise(resolve => { + let settled = false; + let timer: NodeJS.Timeout; + const waiter: GateWaiter = { + wake: () => { + if (settled) return; + settled = true; + clearTimeout(timer); + resolve(Date.now() < deadlineMs); + }, + }; + waiters.push(waiter); + timer = setTimeout(() => { + if (settled) return; + settled = true; + const index = waiters.indexOf(waiter); + if (index >= 0) waiters.splice(index, 1); + resolve(false); + }, remaining); + }); +} + +export async function withBotTurnAdmission( + larkAppId: string, + action: () => Promise | T, +): Promise { + // A mutation already owns the bot exclusively. Re-entering an admission + // boundary from that mutation must not wait for the gate it owns. + const inheritedMutation = mutationContext.getStore()?.get(larkAppId); + if (inheritedMutation?.active) return action(); + const inherited = admissionContext.getStore(); + // Some admitted event paths delegate to triggerSessionTurn, which is also a + // public admission boundary. Treat that nested call as part of the outer + // lease; otherwise a mutation that closed the gate while the outer handler + // awaited would make the handler wait on itself and deadlock the drain. + const inheritedLease = inherited?.get(larkAppId); + if (inheritedLease?.active) return action(); + const state = stateFor(larkAppId); + // A mutation may finish and another queued mutation may close the gate again + // before this continuation runs, so re-check after every wake. + while (state.mutating) await waitUntilOpen(state); + state.activeAdmissions++; + const lease: AdmissionLease = { active: true, ownerFinished: false }; + try { + const next = new Map(inherited ?? []); + next.set(larkAppId, lease); + return await admissionContext.run(next, action); + } finally { + // AsyncLocalStorage descendants can outlive the awaited action when code + // starts fire-and-forget work. They retain this object, so revoke it before + // decrementing the drain count; a detached continuation must acquire a new + // admission instead of impersonating a still-live nested call. + // An admitted handler may upgrade itself to a mutation. The upgrade + // temporarily releases and later reacquires this exact lease; only the + // still-active owner decrements it here. + lease.ownerFinished = true; + if (lease.active) { + lease.active = false; + state.activeAdmissions--; + if (state.activeAdmissions === 0) { + wakeAll(state.drainWaiters); + } + } + } +} + +export async function withBotTurnMutation( + larkAppId: string, + action: () => Promise | T, +): Promise { + const inheritedMutation = mutationContext.getStore()?.get(larkAppId); + if (inheritedMutation?.active) return action(); + const state = stateFor(larkAppId); + const inheritedAdmission = admissionContext.getStore()?.get(larkAppId); + + // Explicit abandon actions are discovered inside already-admitted message + // and card handlers. Upgrade that admission instead of nesting and + // deadlocking: release only this handler's lease, drain every other turn, + // perform the mutation, then atomically reacquire the outer lease before the + // gate is reopened. The caller may safely finish delivery/logging under its + // original admission after the exclusive state change. + const upgrading = inheritedAdmission?.active === true; + if (upgrading) { + inheritedAdmission.active = false; + state.activeAdmissions--; + if (state.activeAdmissions === 0) { + wakeAll(state.drainWaiters); + } + } + + while (state.mutating) await waitUntilOpen(state); + state.mutating = true; + await waitUntilDrained(state); + const mutationLease: MutationLease = { active: true }; + try { + const next = new Map(mutationContext.getStore() ?? []); + next.set(larkAppId, mutationLease); + return await mutationContext.run(next, action); + } finally { + // Detached descendants retain the AsyncLocalStorage map. Revoke the + // mutable lease before reopening the gate so they cannot impersonate the + // completed mutation later. + mutationLease.active = false; + // Reacquire before waking queued admissions/mutations. This keeps the + // remainder of an upgraded outer handler inside the admission lifetime and + // prevents its finally block from double-decrementing the gate. + if (upgrading && !inheritedAdmission.ownerFinished) { + state.activeAdmissions++; + inheritedAdmission.active = true; + } + state.mutating = false; + wakeAll(state.openWaiters); + // Keep the state object stable. Resolved admission waiters resume in later + // promise jobs and still hold this object; deleting it here would let a new + // caller create a second gate and bypass those about-to-reacquire waiters. + // The key space is bounded by configured bot ids. + } +} + +export type BoundedBotTurnMutationResult = + | { acquired: true; value: T } + | { acquired: false; reason: 'timeout' | 'upgrade_conflict' }; + +/** Acquire one exact mutation lease within a single absolute deadline. + * + * Unlike Promise.race around `withBotTurnMutation`, this removes a timed-out + * waiter and rolls back a gate already closed while admissions were draining. + * Therefore `action` can never run after `{ acquired:false }` was returned. + * + * A same-bot admission may upgrade only when no other mutation already owns + * the gate. Returning `upgrade_conflict` leaves that admission untouched; this + * keeps the bounded path safe without resuming an admission concurrently with + * a mutation it had temporarily released for. */ +export async function tryWithBotTurnMutation( + larkAppId: string, + acquireTimeoutMs: number, + action: () => Promise | T, +): Promise> { + const inheritedMutation = mutationContext.getStore()?.get(larkAppId); + if (inheritedMutation?.active) { + return { acquired: true, value: await action() }; + } + const state = stateFor(larkAppId); + const inheritedAdmission = admissionContext.getStore()?.get(larkAppId); + const upgrading = inheritedAdmission?.active === true; + if (upgrading && state.mutating) { + return { acquired: false, reason: 'upgrade_conflict' }; + } + + const deadlineMs = Date.now() + Math.max(0, acquireTimeoutMs); + if (upgrading) { + inheritedAdmission.active = false; + state.activeAdmissions--; + if (state.activeAdmissions === 0) wakeAll(state.drainWaiters); + } else { + while (state.mutating) { + if (!await waitUntilBefore(state.openWaiters, () => !state.mutating, deadlineMs)) { + return { acquired: false, reason: 'timeout' }; + } + } + } + + if (Date.now() >= deadlineMs) { + if (upgrading && !inheritedAdmission.ownerFinished) { + state.activeAdmissions++; + inheritedAdmission.active = true; + } + return { acquired: false, reason: 'timeout' }; + } + + state.mutating = true; + const drained = await waitUntilBefore( + state.drainWaiters, + () => state.activeAdmissions === 0, + deadlineMs, + ); + if (!drained || Date.now() >= deadlineMs) { + if (upgrading && !inheritedAdmission.ownerFinished) { + state.activeAdmissions++; + inheritedAdmission.active = true; + } + state.mutating = false; + wakeAll(state.openWaiters); + return { acquired: false, reason: 'timeout' }; + } + + const mutationLease: MutationLease = { active: true }; + try { + const next = new Map(mutationContext.getStore() ?? []); + next.set(larkAppId, mutationLease); + if (Date.now() >= deadlineMs) { + return { acquired: false, reason: 'timeout' }; + } + const value = await mutationContext.run(next, action); + return { acquired: true, value }; + } finally { + mutationLease.active = false; + if (upgrading && !inheritedAdmission.ownerFinished) { + state.activeAdmissions++; + inheritedAdmission.active = true; + } + state.mutating = false; + wakeAll(state.openWaiters); + } +} + +/** + * Start an independent admission branch from fire-and-forget code. Normal + * same-bot nesting is intentionally reentrant and MUST be awaited by its + * caller; JavaScript cannot infer whether a returned Promise was floated. + * Detached callers clear both contexts and acquire a fresh counted root lease. + */ +export function runDetachedBotTurnAdmission( + larkAppId: string, + action: () => Promise | T, +): Promise { + return admissionContext.run(new Map(), () => + mutationContext.run(new Map(), () => withBotTurnAdmission(larkAppId, action))); +} + +/** Fire-and-forget counterpart for exclusive lifecycle mutations. A detached + * mutation inside an admission waits for that admission to drain; one inside a + * mutation waits for the parent mutation to reopen the gate. Do not await this + * helper from the parent scope whose lease it must wait on. */ +export function runDetachedBotTurnMutation( + larkAppId: string, + action: () => Promise | T, +): Promise { + return admissionContext.run(new Map(), () => + mutationContext.run(new Map(), () => withBotTurnMutation(larkAppId, action))); +} + +export function __testOnly_resetBotTurnMutationGates(): void { + gates.clear(); +} diff --git a/src/core/command-handler.ts b/src/core/command-handler.ts index 290682e99..972dd19b3 100644 --- a/src/core/command-handler.ts +++ b/src/core/command-handler.ts @@ -25,17 +25,8 @@ import { chatAppLink, normalizeBrand } from '../im/lark/lark-hosts.js'; import { claimPairing } from '../services/pairing-store.js'; import { logger } from '../utils/logger.js'; import { scheduleTimeZone } from '../utils/timezone.js'; -import { killWorker, suspendWorker, forkWorker, forkAdoptWorker, getCurrentCliVersion, postFreshStreamingCard, postPrivateSnapshotCard, resolvePrivateCardAudience, deliverEphemeralOrReply, deliverWritableTerminalCardTo } from './worker-pool.js'; -import { - expandHome, - getSessionWorkingDir, - getProjectScanDir, - getProjectScanDirs, - rememberLastCliInput, - buildNewTopicCliInput, - ensureSessionWhiteboard, - getAvailableBots, -} from './session-manager.js'; +import { killWorker, suspendWorker, forkWorker, forkAdoptWorker, getCurrentCliVersion, postFreshStreamingCard, postPrivateSnapshotCard, resolvePrivateCardAudience, deliverEphemeralOrReply, deliverWritableTerminalCardTo, closeSession as closeWorkerPoolSession, withActiveSessionKeyLock } from './worker-pool.js'; +import { expandHome, getSessionWorkingDir, getProjectScanDir, getProjectScanDirs, rememberLastCliInput } from './session-manager.js'; import { discoverSlashCommandsForAdapter, listMcpServerNames, supportsFilesystemCommandDiscovery } from './command-discovery.js'; import { validateWorkingDir } from './working-dir.js'; import { repinSessionWorkingDir } from './session-cwd.js'; @@ -85,14 +76,16 @@ import { readRoleProfileEntry, writeRoleProfileEntry, } from '../services/role-profile-store.js'; -import type { LarkMessage, DaemonToWorker } from '../types.js'; -import { sessionKey, sessionAnchorId, markRepoCardConsumed, claimCurrentRepoCard } from './types.js'; +import type { LarkMessage, DaemonToWorker, CodexAppTurnInput } from '../types.js'; +import { activeSessionKey, sessionKey, sessionAnchorId, markRepoCardConsumed } from './types.js'; import type { DaemonSession } from './types.js'; import { t, localeForBot, type Locale } from '../i18n/index.js'; import { runSkillsImCommand } from './skills/im-command.js'; import { fetchDaemonIpc } from './daemon-ipc-auth.js'; import { updateSessionTitle } from './session-title.js'; import { requestAgentSessionRename } from './session-rename.js'; +import { hasProtectedSessionMutationOwnership } from './session-mutation-guard.js'; +import { withBotTurnMutation } from './bot-turn-mutation-gate.js'; // ─── Exported constants ────────────────────────────────────────────────────── @@ -1251,21 +1244,44 @@ export async function handleCommand( switch (cmd) { case '/close': { if (ds) { - // Capture the closed-session card BEFORE killWorker/closeSession — - // it reads the live session's identity off `ds`. - const card = buildClosedSessionCard(ds, loc); - killWorker(ds); - sessionStore.closeSession(ds.session.sessionId); - activeSessions.delete(sessionKey(rootId, larkAppId!)); + const targetSessionId = ds.session.sessionId; + const closed = await withBotTurnMutation(ds.larkAppId, async () => { + // Re-resolve the exact session after all peer admissions drain. A + // relay may have moved it to another key while this command was + // admitted; closeSession removes its current identity, never the + // stale root key from this message. + const current = [...activeSessions.values()].find( + candidate => candidate.session.sessionId === targetSessionId, + ); + if (!current) return undefined; + const card = buildClosedSessionCard(current, localeForBot(current.larkAppId)); + const result = await closeWorkerPoolSession(targetSessionId); + if (!result.ok) return { status: 'failed' as const, current, failure: result }; + return { status: 'closed' as const, current, card }; + }); + if (!closed) { + await sessionReply(rootId, t('cmd.no_active_session', undefined, loc)); + break; + } + if (closed.status === 'failed') { + await sessionReply( + rootId, + 'Riff 远端任务取消失败，会话仍保持活跃且保留任务 ID；请稍后重试 /close。', + ); + logger.warn( + `[${logTag}] /close refused: ${closed.failure.error} task=${closed.failure.taskId}`, + ); + break; + } // 「会话已关闭」卡片优先「仅自己可见」：普通群里走 ephemeral 只发给执行 // /close 的本人；话题群不支持 ephemeral(18053) 时回退为正常的群内可见回复 // ——与流式卡片上「关闭会话」按钮的送达方式保持一致。 await deliverEphemeralOrReply( - ds, + closed.current, message.senderId, - card, + closed.card, 'interactive', - () => sessionReply(rootId, card, 'interactive'), + () => sessionReply(rootId, closed.card, 'interactive'), ); logger.info(`[${logTag}] Session closed by /close command`); } else { @@ -1347,9 +1363,18 @@ export async function handleCommand( break; } const closedSessionId = ds.session.sessionId; - killWorker(ds); - sessionStore.closeSession(closedSessionId); - activeSessions.delete(sessionKey(rootId, larkAppId!)); + const detached = await withBotTurnMutation(ds.larkAppId, async () => { + const current = [...activeSessions.values()].find( + candidate => candidate.session.sessionId === closedSessionId, + ); + if (!current || !current.adoptedFrom) return false; + await closeWorkerPoolSession(closedSessionId); + return true; + }); + if (!detached) { + await sessionReply(rootId, t('cmd.no_active_session', undefined, loc)); + break; + } await sessionReply(rootId, t('cmd.detach.success', undefined, loc)); logger.info(`[${logTag}] Detached (adopt) by ${cmd} command`); break; @@ -1357,15 +1382,49 @@ export async function handleCommand( case '/restart': { if (ds) { - if (ds.worker && !ds.worker.killed) { - ds.worker.send({ type: 'restart' } as DaemonToWorker); - const cliName = getCliDisplayName(getBot(ds.larkAppId).config.cliId); - await sessionReply(rootId, t('cmd.restart.in_progress', { cliName }, loc)); - } else { - killWorker(ds); - const cliName = getCliDisplayName(getBot(ds.larkAppId).config.cliId); - await sessionReply(rootId, t('cmd.restart.terminated', { cliName }, loc)); + const targetSessionId = ds.session.sessionId; + const restart = await withBotTurnMutation(ds.larkAppId, async () => { + const current = [...activeSessions.values()].find( + candidate => candidate.session.sessionId === targetSessionId + && candidate.session.status === 'active', + ); + if (!current) return { status: 'gone' as const }; + if (hasProtectedSessionMutationOwnership(current)) { + return { status: 'pending' as const }; + } + const cliName = getCliDisplayName(getBot(current.larkAppId).config.cliId); + if ((current.initConfig?.backendType ?? current.session.backendType) === 'riff' + || current.session.cliId === 'riff') { + return { status: 'riff_unsupported' as const, cliName }; + } + if (current.worker && !current.worker.killed) { + current.worker.send({ type: 'restart', reason: 'operator' } as DaemonToWorker); + return { status: 'restarting' as const, cliName }; + } + killWorker(current); + return { status: 'terminated' as const, cliName }; + }); + if (restart.status === 'pending') { + await sessionReply( + rootId, + '当前 Codex App 仍有未结算消息，暂不能重启；请等待本轮完成或关闭会话。', + ); + break; } + if (restart.status === 'gone') { + await sessionReply(rootId, t('cmd.no_active_session', undefined, loc)); + break; + } + if (restart.status === 'riff_unsupported') { + await sessionReply(rootId, t('cmd.restart.riff_unsupported', undefined, loc)); + break; + } + await sessionReply( + rootId, + restart.status === 'restarting' + ? t('cmd.restart.in_progress', { cliName: restart.cliName }, loc) + : t('cmd.restart.terminated', { cliName: restart.cliName }, loc), + ); logger.info(`[${logTag}] Restart by /restart command`); } else { await sessionReply(rootId, t('cmd.no_active_session', undefined, loc)); @@ -1383,21 +1442,68 @@ export async function handleCommand( await sessionReply(rootId, t('cmd.no_active_session', undefined, loc)); break; } + // A live Riff sandbox is pinned to its task lineage. Detaching before + // an in-flight create publishes its task id can orphan the new task; + // even an idle follow-up keeps the old sandbox/repo, so changing this + // host-side cwd would be misleading. Refuse before auto-creating paths. + if ((ds.initConfig?.backendType ?? ds.session.backendType) === 'riff') { + await sessionReply(rootId, t('cmd.cd.riff_unsupported', undefined, loc)); + break; + } + // Cheap preflight avoids creating a requested directory when the + // current FIFO already makes the switch impossible. The mutation + // below repeats this check after draining peer admissions. + if (hasProtectedSessionMutationOwnership(ds)) { + await sessionReply( + rootId, + '当前 Codex App 仍有未结算消息，暂不能切换工作目录；请等待本轮完成或关闭会话。', + ); + break; + } const validation = validateWorkingDir(targetPath, loc, { autoCreate: true }); if (!validation.ok) { await sessionReply(rootId, validation.error); break; } const resolvedPath = validation.resolvedPath; - // Persistent backends can cold-resume without treating a cwd switch as - // a real session close. This also preserves a sandbox upper long enough - // for the next worker to relocate Claude's cwd-scoped transcript. - // Adopted sessions are observers of a user-owned CLI and keep the old - // detach/kill behavior; PTY falls back to killWorker, while its normal - // on-disk transcript remains available for the resume-sync preflight. - const suspended = !ds.adoptedFrom && suspendWorker(ds, 'working_dir_changed'); - if (!suspended) killWorker(ds); - repinSessionWorkingDir(ds, resolvedPath); + const targetSessionId = ds.session.sessionId; + const switched = await withBotTurnMutation(ds.larkAppId, async () => { + const current = [...activeSessions.values()].find( + candidate => candidate.session.sessionId === targetSessionId + && candidate.session.status === 'active', + ); + if (!current) return 'gone' as const; + if (hasProtectedSessionMutationOwnership(current)) { + return 'pending' as const; + } + if ((current.initConfig?.backendType ?? current.session.backendType) === 'riff') { + return 'riff_unsupported' as const; + } + const suspended = !current.adoptedFrom + && suspendWorker(current, 'working_dir_changed'); + if (!suspended) killWorker(current); + repinSessionWorkingDir(current, resolvedPath); + // cwd 变了，riff 多仓 stamp（选择卡多选时写入）随之失效——保留会让下次 + // refork 仍按旧仓库组合推导、无视新目录。 + current.session.riffRepoDirs = undefined; + sessionStore.updateSession(current.session); + return 'switched' as const; + }); + if (switched === 'pending') { + await sessionReply( + rootId, + '当前 Codex App 仍有未结算消息，暂不能切换工作目录；请等待本轮完成或关闭会话。', + ); + break; + } + if (switched === 'gone') { + await sessionReply(rootId, t('cmd.no_active_session', undefined, loc)); + break; + } + if (switched === 'riff_unsupported') { + await sessionReply(rootId, t('cmd.cd.riff_unsupported', undefined, loc)); + break; + } if (validation.created) { await sessionReply(rootId, t('cmd.cd.created_switched', { path: resolvedPath }, loc)); } else { @@ -1440,141 +1546,150 @@ export async function handleCommand( case '/repo': { const repoArg = message.content.replace(/^\/repo\s*/, '').trim(); + if (ds && !ds.pendingRepo + && hasProtectedSessionMutationOwnership(ds)) { + await sessionReply( + rootId, + '当前 Codex App 仍有未结算消息，暂不能切换仓库；请等待本轮完成或关闭会话。', + ); + break; + } // First-spawn fork: consume the buffered prompt/attachments and start the // CLI in whatever workingDir is currently set on the session. Shared by // `commitRepoSelection` (a repo was named) and the bare-`/repo` launch // (use the default workingDir) — both only run while `pendingRepo`. - const forkPendingCli = async (replyText: string, claimAlready = false): Promise => { - if (!claimAlready) { - if (ds!.pendingRepoCommitInFlight) { - await sessionReply(rootId, t('cmd.repo.worktree_in_progress', undefined, loc)); - return false; + const forkPendingCli = async ( + replyText: string, + selection?: { path: string; riffRepoDirs?: string[] }, + ) => { + const targetSessionId = ds!.session.sessionId; + const started = await withBotTurnMutation(ds!.larkAppId, async () => { + const current = [...activeSessions.values()].find( + candidate => candidate.session.sessionId === targetSessionId + && candidate.session.status === 'active', + ); + if (!current || current !== ds || !current.pendingRepo) return false; + if (selection) { + current.workingDir = selection.path; + current.session.workingDir = selection.path; + current.session.riffRepoDirs = selection.riffRepoDirs; + sessionStore.updateSession(current.session); } - ds!.pendingRepoCommitInFlight = true; - } - const selfBot = getBot(ds!.larkAppId); - const botCfg = selfBot.config; - const commitGenSessionId = ds!.session.sessionId; - let committed = false; - try { - // Keep pendingRepo=true while prompt context is awaited. Incoming - // messages stay in the same buffer and card selections see the - // shared in-flight claim instead of treating this as a repo switch. - const initiallyBuffered = - (ds!.pendingPrompt?.trim().length ?? 0) > 0 || - (ds!.pendingAttachments?.length ?? 0) > 0 || - (ds!.pendingFollowUps?.length ?? 0) > 0; - const availableBots = initiallyBuffered - ? await getAvailableBots(ds!.larkAppId, ds!.chatId) - : []; - const stillActive = activeSessions.get(sessionKey(rootId, larkAppId!)) === ds; - if (!stillActive || ds!.session.sessionId !== commitGenSessionId || - !ds!.pendingRepo || (ds!.worker && !ds!.worker.killed)) { - logger.warn(`[${logTag}] Session changed while preparing the pending CLI (${commitGenSessionId} → ${ds!.session.sessionId}, active=${stillActive}, pending=${!!ds!.pendingRepo}, worker=${!!ds!.worker})`); - return false; + const selfBot = getBot(current.larkAppId); + const botCfg = selfBot.config; + const pendingPrompt = current.pendingPrompt ?? ''; + const pendingRawInput = current.pendingRawInput; + const hasBufferedInput = pendingPrompt.trim().length > 0 + || current.pendingCodexAppText !== undefined + || (current.pendingAttachments?.length ?? 0) > 0 + || (current.pendingFollowUps?.length ?? 0) > 0; + let wrappedInput: { content: string; codexAppInput?: CodexAppTurnInput } | undefined; + if (hasBufferedInput) { + const { buildNewTopicCliInput: buildInput, ensureSessionWhiteboard, getAvailableBots } = await import('./session-manager.js'); + ensureSessionWhiteboard(current); + const availableBots = await getAvailableBots(current.larkAppId, current.chatId); + // Detached lifecycle work can still close/replace while the + // roster lookup awaits. Never fork the captured generation. + if (current.session.status !== 'active' + || [...activeSessions.values()].find(candidate => candidate.session.sessionId === targetSessionId) !== current + || !current.pendingRepo) return false; + wrappedInput = buildInput( + pendingPrompt, + current.session.sessionId, + current.session.cliId ?? botCfg.cliId, + current.session.cliPathOverride ?? botCfg.cliPathOverride, + current.pendingAttachments, + current.pendingMentions, + availableBots, + current.pendingFollowUps, + { name: selfBot.botName, openId: selfBot.botOpenId }, + loc, + current.pendingSender, + { + larkAppId, + chatId: current.chatId, + whiteboardId: current.session.whiteboardId, + substituteTrigger: current.pendingSubstituteTrigger, + codexAppText: current.pendingCodexAppText, + codexAppApplicationContext: current.pendingCodexAppApplicationContext, + codexAppMessageContext: current.pendingCodexAppMessageContext, + codexAppFollowUps: current.pendingCodexAppFollowUps, + codexAppFollowUpContexts: current.pendingCodexAppFollowUpContexts, + }, + ); } - - const pendingPrompt = ds!.pendingPrompt ?? ''; - const pendingRawInput = ds!.pendingRawInput; - const hasBufferedInput = - pendingPrompt.trim().length > 0 || - (ds!.pendingAttachments?.length ?? 0) > 0 || - (ds!.pendingFollowUps?.length ?? 0) > 0; - if (hasBufferedInput) ensureSessionWhiteboard(ds!); - const wrappedInput = hasBufferedInput - ? buildNewTopicCliInput( - pendingPrompt, - ds!.session.sessionId, - ds!.session.cliId ?? botCfg.cliId, - ds!.session.cliPathOverride ?? botCfg.cliPathOverride, - ds!.pendingAttachments, - ds!.pendingMentions, - availableBots, - ds!.pendingFollowUps, - { name: selfBot.botName, openId: selfBot.botOpenId }, - loc, - ds!.pendingSender, - { - larkAppId, - chatId: ds!.chatId, - whiteboardId: ds!.session.whiteboardId, - substituteTrigger: ds!.pendingSubstituteTrigger, - codexAppText: ds!.pendingCodexAppText, - codexAppApplicationContext: ds!.pendingCodexAppApplicationContext, - codexAppMessageContext: ds!.pendingCodexAppMessageContext, - codexAppFollowUps: ds!.pendingCodexAppFollowUps, - codexAppFollowUpContexts: ds!.pendingCodexAppFollowUpContexts, - }, - ) - : { content: '' }; - const pendingTurnId = ds!.pendingTurnId; - const previousPendingFollowUpInput = ds!.pendingFollowUpInput; - if (pendingRawInput && hasBufferedInput) { - ds!.pendingFollowUpInput = { - userPrompt: ds!.pendingCodexAppText !== undefined || ds!.pendingCodexAppFollowUps - ? [ds!.pendingCodexAppText ?? '', ...(ds!.pendingCodexAppFollowUps ?? [])].filter(Boolean).join('\n\n') - : pendingPrompt || ds!.pendingFollowUps?.join('\n\n') || '', + if (pendingRawInput && hasBufferedInput && wrappedInput) { + current.pendingFollowUpInput = { + userPrompt: current.pendingCodexAppText !== undefined || current.pendingCodexAppFollowUps + ? [current.pendingCodexAppText ?? '', ...(current.pendingCodexAppFollowUps ?? [])].filter(Boolean).join('\n\n') + : pendingPrompt || current.pendingFollowUps?.join('\n\n') || '', cliInput: wrappedInput.content, - ...(ds!.pendingFollowUpTurnId ? { turnId: ds!.pendingFollowUpTurnId } : {}), - ...((ds!.session.cliId ?? botCfg.cliId) === 'codex-app' && botCfg.codexAppCleanInput === true && wrappedInput.codexAppInput + ...((current.pendingFollowUpTurnIds?.at(-1) ?? current.pendingFollowUpTurnId) + ? { turnId: current.pendingFollowUpTurnIds?.at(-1) ?? current.pendingFollowUpTurnId } + : {}), + ...((current.session.cliId ?? botCfg.cliId) === 'codex-app' && botCfg.codexAppCleanInput === true && wrappedInput.codexAppInput ? { codexAppInput: wrappedInput.codexAppInput } : {}), codexAppInputGateFrozen: true, }; } - - ds!.pendingRepo = false; - publishAttentionPatch(ds!); - try { - if (pendingRawInput) forkWorker(ds!, '', false); - else if (pendingTurnId && hasBufferedInput) forkWorker(ds!, wrappedInput, { turnId: pendingTurnId }); - else if (hasBufferedInput) forkWorker(ds!, wrappedInput); - else forkWorker(ds!, '', false); - } catch (e) { - ds!.pendingRepo = true; - ds!.pendingFollowUpInput = previousPendingFollowUpInput; - publishAttentionPatch(ds!); - throw e; - } - if (pendingRawInput || hasBufferedInput) { - rememberLastCliInput(ds!, pendingRawInput ?? pendingPrompt, pendingRawInput ?? wrappedInput); - } - ds!.pendingPrompt = undefined; - ds!.pendingCodexAppText = undefined; - ds!.pendingCodexAppApplicationContext = undefined; - ds!.pendingCodexAppMessageContext = undefined; - ds!.pendingAttachments = undefined; - ds!.pendingMentions = undefined; - ds!.pendingSubstituteTrigger = undefined; - ds!.pendingSender = undefined; - ds!.pendingFollowUps = undefined; - ds!.pendingCodexAppFollowUps = undefined; - ds!.pendingCodexAppFollowUpContexts = undefined; - ds!.pendingFollowUpTurnId = undefined; - ds!.pendingTurnId = undefined; - committed = true; - - // Local one-shot consume BEFORE network awaits. Feishu withdraw is - // best-effort; stale card clicks must be rejected via this mark even - // if sessionReply throws or deleteMessage lags/fails. - const cardToWithdraw = ds!.repoCardMessageId; - markRepoCardConsumed(ds!, cardToWithdraw); - ds!.repoCardMessageId = undefined; - - // Hold the claim through confirmation + best-effort card withdraw. + if (pendingRawInput) rememberLastCliInput(current, pendingRawInput, pendingRawInput); + else if (hasBufferedInput && wrappedInput) rememberLastCliInput(current, pendingPrompt, wrappedInput); + + // forkWorker performs the synchronous pre-accept/write-ahead work. + // Keep the opening reservation and every buffered field intact if + // that step throws, so a failed launch cannot silently consume the + // first user turn or expose this worker:null owner as scratch. + const pendingTurnId = current.pendingTurnId + ?? current.session.pendingRepoSetup?.turnId; + forkWorker( + current, + pendingRawInput ? '' : (wrappedInput ?? ''), + !pendingRawInput && pendingTurnId ? { turnId: pendingTurnId } : false, + ); + current.pendingRepo = false; + current.pendingRepoCommitInFlight = true; + // Queued activation ownership lasts through adapter submission. + // These source buffers were folded into opening N; clear them but + // keep the gate so later inbounds enter the exact post-ACK FIFO. + current.initialStartPending = current.session.queuedActivationPending === true; + publishAttentionPatch(current); + current.pendingPrompt = undefined; + current.pendingCodexAppText = undefined; + current.pendingCodexAppApplicationContext = undefined; + current.pendingCodexAppMessageContext = undefined; + current.pendingAttachments = undefined; + current.pendingMentions = undefined; + current.pendingSubstituteTrigger = undefined; + current.pendingSender = undefined; + current.pendingFollowUps = undefined; + current.pendingFollowUpTurnId = undefined; + current.pendingFollowUpTurnIds = undefined; + current.pendingCodexAppFollowUps = undefined; + current.pendingCodexAppFollowUpContexts = undefined; + current.pendingCodexAppFollowUpGateAccepted = undefined; + current.pendingTurnId = undefined; + const cardToWithdraw = current.repoCardMessageId; + markRepoCardConsumed(current, cardToWithdraw); + current.repoCardMessageId = undefined; + return { current, cardToWithdraw }; + }); + if (!started) return false; + try { try { await sessionReply(rootId, replyText); } catch (e) { logger.warn(`[${logTag}] Confirm reply after pending repo commit failed: ${e instanceof Error ? e.message : e}`); } - if (cardToWithdraw) { - try { await deleteMessage(ds!.larkAppId, cardToWithdraw); } catch { /* best-effort */ } + if (started.cardToWithdraw) { + try { await deleteMessage(started.current.larkAppId, started.cardToWithdraw); } + catch { /* best-effort */ } } } finally { - ds!.pendingRepoCommitInFlight = false; + started.current.pendingRepoCommitInFlight = false; } - return committed; + return true; }; // Shared commit path for an already-resolved repo: update the session's @@ -1583,26 +1698,13 @@ export async function handleCommand( // numeric `/repo ` form and the `/repo ` form. const commitRepoSelection = async (selectedPath: string, displayName: string, how: string): Promise => { if (ds!.pendingRepo) { - if (ds!.pendingRepoCommitInFlight) { - await sessionReply(rootId, t('cmd.repo.worktree_in_progress', undefined, loc)); - return false; - } - ds!.pendingRepoCommitInFlight = true; - try { - // Claim before changing cwd so a card selection cannot overwrite - // this choice while the pending prompt is prepared. - ds!.workingDir = selectedPath; - ds!.session.workingDir = selectedPath; - ds!.session.riffRepoDirs = undefined; - sessionStore.updateSession(ds!.session); - // forkPendingCli owns claim release + confirm reply + card withdraw. - if (!await forkPendingCli(t('cmd.repo.selected_in_pending', { name: displayName }, loc), true)) { - return false; - } - } catch (e) { - ds!.pendingRepoCommitInFlight = false; - throw e; - } + // First spawn: the cwd pin and fork are one exclusive commit. Two + // simultaneous selections cannot make A reply while forking B's cwd. + const started = await forkPendingCli( + t('cmd.repo.selected_in_pending', { name: displayName }, loc), + { path: selectedPath, riffRepoDirs: undefined }, + ); + if (!started) return false; } else { // Safety net: a mid-session `/repo` switch closes the running // session and spawns a fresh one on the SAME anchor. Without a @@ -1623,53 +1725,104 @@ export async function handleCommand( // card), so `claude --resume` later would reopen the old context in // the new repo's cwd. The new repo is pinned onto the fresh session // below instead. - const claimedCard = claimCurrentRepoCard(ds!, undefined); - const closedCard = buildClosedSessionCard(ds!, loc); - killWorker(ds!); - sessionStore.closeSession(ds!.session.sessionId); + const targetSessionId = ds!.session.sessionId; + const switched = await withBotTurnMutation(ds!.larkAppId, async () => { + const candidate = [...activeSessions.values()].find( + candidate => candidate.session.sessionId === targetSessionId, + ); + if (!candidate || candidate !== ds || candidate.session.status !== 'active') { + return { ok: false as const, error: 'session_replaced' as const }; + } + const key = activeSessionKey(candidate); + return withActiveSessionKeyLock(activeSessions, key, async () => { + // Resume/scheduler/dashboard creators use this same key lock + // without joining the bot admission gate. Re-resolve after the + // lock wait, then keep it across close -> replacement publish. + const current = [...activeSessions.values()].find( + owner => owner.session.sessionId === targetSessionId, + ); + if (!current || current !== candidate + || activeSessions.get(key) !== current + || current.session.status !== 'active') { + return { ok: false as const, error: 'session_replaced' as const }; + } + if (hasProtectedSessionMutationOwnership(current)) { + return { ok: false as const, error: 'dispatch_pending' as const }; + } + const closedCard = buildClosedSessionCard(current, loc); + const oldSession = current.session; + const closeResult = await closeWorkerPoolSession(targetSessionId); + if (!closeResult.ok) { + return { ok: false as const, error: 'close_failed' as const, closeResult }; + } + // The key lock excludes every sanctioned creator. A direct + // lifecycle callback may still have published unexpectedly; + // fail closed instead of overwriting that first owner. + if (activeSessions.get(key) === current) activeSessions.delete(key); + if (activeSessions.has(key)) { + return { ok: false as const, error: 'session_replaced' as const }; + } + const cardToWithdraw = current.repoCardMessageId; + markRepoCardConsumed(current, cardToWithdraw); + current.repoCardMessageId = undefined; + + const session = sessionStore.createSession( + current.chatId, + current.scope === 'chat' ? oldSession.rootMessageId : rootId, + displayName, + current.chatType, + current.scope, + ); + current.session = session; + current.lastUserPrompt = undefined; + current.lastCliInput = undefined; + current.workingDir = selectedPath; + session.workingDir = selectedPath; + session.larkAppId = current.larkAppId; + session.chatDisplayName = oldSession.chatDisplayName; + session.ownerOpenId = oldSession.ownerOpenId; + session.creatorOpenId = oldSession.creatorOpenId; + session.lastCallerOpenId = oldSession.lastCallerOpenId; + sessionStore.updateSession(session); + current.hasHistory = false; + activeSessions.set(key, current); + forkWorker(current, '', false); + return { ok: true as const, current, closedCard, cardToWithdraw }; + }); + }); + if (!switched.ok) { + if (switched.error === 'dispatch_pending') { + await sessionReply( + rootId, + '当前 Codex App 仍有未结算消息，暂不能切换仓库；请等待本轮完成或关闭会话。', + ); + } else if (switched.error === 'close_failed') { + await sessionReply( + rootId, + 'Riff 远端任务取消失败，原会话仍保持活跃；未切换仓库，请重试。', + ); + } else { + logger.warn(`[${logTag}] Repo switch aborted because the session was replaced`); + } + return false; + } await deliverEphemeralOrReply( - ds!, + switched.current, message.senderId, - closedCard, + switched.closedCard, 'interactive', - () => sessionReply(rootId, closedCard, 'interactive'), - ); - - const oldSession = ds!.session; - // `rootId` is the routing anchor. For chat-scope sessions it is the - // `oc_...` chat id, not the traceable `om_...` message root stored on - // Session. Preserve the old identity and explicitly persist scope so - // repo switches cannot turn a chat session into a legacy scope-less - // record that the CLI later mistakes for a thread. - const session = sessionStore.createSession( - ds!.chatId, - ds!.scope === 'chat' ? oldSession.rootMessageId : rootId, - displayName, - ds!.chatType, - ds!.scope, + () => sessionReply(rootId, switched.closedCard, 'interactive'), ); - ds!.session = session; - ds!.lastUserPrompt = undefined; - ds!.lastCliInput = undefined; - ds!.workingDir = selectedPath; - ds!.session.workingDir = selectedPath; - ds!.session.larkAppId = ds!.larkAppId; - ds!.session.chatDisplayName = oldSession.chatDisplayName; - ds!.session.ownerOpenId = oldSession.ownerOpenId; - ds!.session.creatorOpenId = oldSession.creatorOpenId; - ds!.session.lastCallerOpenId = oldSession.lastCallerOpenId; - sessionStore.updateSession(ds!.session); - ds!.hasHistory = false; - forkWorker(ds!, '', false); - try { - await sessionReply(rootId, t('cmd.repo.switched_to', { name: displayName }, loc)); - } catch (e) { - logger.warn(`[${logTag}] Confirm reply after mid-session repo switch failed: ${e instanceof Error ? e.message : e}`); - } - if (claimedCard) { - try { await deleteMessage(ds!.larkAppId, claimedCard); } catch { /* best-effort */ } + await sessionReply(rootId, t('cmd.repo.switched_to', { name: displayName }, loc)); + if (switched.cardToWithdraw) { + try { await deleteMessage(switched.current.larkAppId, switched.cardToWithdraw); } + catch { /* best-effort */ } } } + if (ds!.repoCardMessageId) { + deleteMessage(ds!.larkAppId, ds!.repoCardMessageId); + ds!.repoCardMessageId = undefined; + } logger.info(`[${logTag}] Repo selected via ${how}: ${selectedPath}`); return true; }; @@ -2880,6 +3033,11 @@ export async function handleCommand( await sessionReply(rootId, t('cmd.relay.adopt_not_relayable', undefined, loc)); break; } + if ((ds.initConfig?.backendType ?? ds.session.backendType) === 'riff' + || ds.session.cliId === 'riff') { + await sessionReply(rootId, t('cmd.relay.riff_not_relayable', undefined, loc)); + break; + } if (ds.pendingRepo) { await sessionReply(rootId, t('cmd.relay.not_started_yet', undefined, loc)); break; @@ -3426,23 +3584,43 @@ export async function startCodexAppThreadSession( deps.sessionReply(rid, content, msgType, larkAppId); const loc: Locale = localeForBot(ds.larkAppId ?? larkAppId); const title = codexAppThreadTitle(thread); - - ds.adoptedFrom = undefined; - ds.workingDir = thread.cwd; - ds.hasHistory = true; - ds.currentTurnTitle = undefined; - ds.lastScreenContent = undefined; - ds.lastScreenStatus = undefined; - - ds.session.workingDir = thread.cwd; - ds.session.title = `Codex App: ${title}`; - ds.session.cliId = 'codex-app'; - ds.session.cliSessionId = thread.threadId; - ds.session.adoptedFrom = undefined; - sessionStore.updateSession(ds.session); - - forkWorker(ds, '', true); - await sessionReply(sessionAnchorId(ds), t('cmd.codex_app_adopt.success', { title }, loc)); + const targetSessionId = ds.session.sessionId; + const switched = await withBotTurnMutation(ds.larkAppId, async () => { + const current = [...deps.activeSessions.values()].find( + candidate => candidate.session.sessionId === targetSessionId + && candidate.session.status === 'active', + ); + if (!current || current !== ds) return { status: 'gone' as const }; + if (hasProtectedSessionMutationOwnership(current)) { + return { status: 'pending' as const, anchor: sessionAnchorId(current) }; + } + current.adoptedFrom = undefined; + current.workingDir = thread.cwd; + current.hasHistory = true; + current.currentTurnTitle = undefined; + current.lastScreenContent = undefined; + current.lastScreenStatus = undefined; + current.session.workingDir = thread.cwd; + current.session.title = `Codex App: ${title}`; + current.session.cliId = 'codex-app'; + current.session.cliSessionId = thread.threadId; + current.session.adoptedFrom = undefined; + sessionStore.updateSession(current.session); + forkWorker(current, '', true); + return { status: 'switched' as const, anchor: sessionAnchorId(current) }; + }); + if (switched.status === 'gone') { + await sessionReply(sessionAnchorId(ds), t('cmd.no_active_session', undefined, loc)); + return; + } + if (switched.status === 'pending') { + await sessionReply( + switched.anchor, + '当前 Codex App 仍有未结算消息，不能切换原生 thread；请等待本轮完成或先关闭会话。', + ); + return; + } + await sessionReply(switched.anchor, t('cmd.codex_app_adopt.success', { title }, loc)); } export async function startAdoptSession( @@ -3466,34 +3644,55 @@ export async function startAdoptSession( const project = target.cwd.split('/').pop() || target.cwd; const pane = zellij ? `${target.zellijSession}/${target.zellijPaneId}` : adoptTargetLabel(target); - - ds.workingDir = target.cwd; - ds.session.workingDir = target.cwd; - ds.session.title = `Adopt: ${project}`; - ds.adoptedFrom = { - source: zellij ? 'zellij' : target.source, - tmuxTarget: zellij ? undefined : target.tmuxTarget, - zellijSession: zellij ? target.zellijSession : undefined, - zellijPaneId: zellij ? target.zellijPaneId : undefined, - herdrSessionName: zellij ? undefined : target.herdrSessionName, - herdrTarget: zellij ? undefined : target.herdrTarget, - herdrPaneId: zellij ? undefined : target.herdrPaneId, - herdrAgentName: zellij ? undefined : target.herdrAgentName, - herdrTerminalId: zellij ? undefined : target.herdrTerminalId, - originalCliPid: target.cliPid, - sessionId: target.sessionId, - cliId: target.cliId, - cwd: target.cwd, - paneCols: target.paneCols, - paneRows: target.paneRows, - }; - ds.session.adoptedFrom = { ...ds.adoptedFrom }; - sessionStore.updateSession(ds.session); - - forkAdoptWorker(ds); + const targetSessionId = ds.session.sessionId; + const adopted = await withBotTurnMutation(ds.larkAppId, async () => { + const current = [...deps.activeSessions.values()].find( + candidate => candidate.session.sessionId === targetSessionId + && candidate.session.status === 'active', + ); + if (!current || current !== ds) return { status: 'gone' as const }; + if (hasProtectedSessionMutationOwnership(current)) { + return { status: 'pending' as const, anchor: sessionAnchorId(current) }; + } + current.workingDir = target.cwd; + current.session.workingDir = target.cwd; + current.session.title = `Adopt: ${project}`; + current.adoptedFrom = { + source: zellij ? 'zellij' : target.source, + tmuxTarget: zellij ? undefined : target.tmuxTarget, + zellijSession: zellij ? target.zellijSession : undefined, + zellijPaneId: zellij ? target.zellijPaneId : undefined, + herdrSessionName: zellij ? undefined : target.herdrSessionName, + herdrTarget: zellij ? undefined : target.herdrTarget, + herdrPaneId: zellij ? undefined : target.herdrPaneId, + herdrAgentName: zellij ? undefined : target.herdrAgentName, + herdrTerminalId: zellij ? undefined : target.herdrTerminalId, + originalCliPid: target.cliPid, + sessionId: target.sessionId, + cliId: target.cliId, + cwd: target.cwd, + paneCols: target.paneCols, + paneRows: target.paneRows, + }; + current.session.adoptedFrom = { ...current.adoptedFrom }; + sessionStore.updateSession(current.session); + forkAdoptWorker(current); + return { status: 'adopted' as const, anchor: sessionAnchorId(current) }; + }); + if (adopted.status === 'gone') { + await sessionReply(sessionAnchorId(ds), t('cmd.no_active_session', undefined, loc)); + return; + } + if (adopted.status === 'pending') { + await sessionReply( + adopted.anchor, + '当前 Codex App 仍有未结算消息，不能切换到外部会话；请等待本轮完成或先关闭会话。', + ); + return; + } const cliName = getCliDisplayName(target.cliId); - await sessionReply(sessionAnchorId(ds), t('cmd.adopt.success', { cliName, project, pane }, loc)); + await sessionReply(adopted.anchor, t('cmd.adopt.success', { cliName, project, pane }, loc)); } /** Discover the sessions resumable from disk for `cliId`, excluding any whose @@ -3543,19 +3742,39 @@ export async function startResumeImportSession( deps.sessionReply(rid, content, msgType, larkAppId); const loc: Locale = localeForBot(ds.larkAppId ?? larkAppId); const project = target.cwd.split('/').pop() || target.cwd; - - ds.workingDir = target.cwd; - ds.session.workingDir = target.cwd; - ds.session.cliSessionId = target.cliSessionId; - ds.session.title = target.title || `Import: ${project}`; - // Resume sandbox decision is left to forkWorker (resume=true → not sandboxed, - // matching restore semantics). Mark history so the session is treated as a - // resume, not a fresh spawn. - ds.hasHistory = true; - sessionStore.updateSession(ds.session); - - forkWorker(ds, '', true); + const targetSessionId = ds.session.sessionId; + const resumed = await withBotTurnMutation(ds.larkAppId, async () => { + const current = [...deps.activeSessions.values()].find( + candidate => candidate.session.sessionId === targetSessionId + && candidate.session.status === 'active', + ); + if (!current || current !== ds) return { status: 'gone' as const }; + if (hasProtectedSessionMutationOwnership(current)) { + return { status: 'pending' as const, anchor: sessionAnchorId(current) }; + } + current.workingDir = target.cwd; + current.session.workingDir = target.cwd; + current.session.cliSessionId = target.cliSessionId; + current.session.title = target.title || `Import: ${project}`; + // Resume sandbox decision is left to forkWorker (resume=true → not + // sandboxed, matching restore semantics). Mark history so this is a resume. + current.hasHistory = true; + sessionStore.updateSession(current.session); + forkWorker(current, '', true); + return { status: 'resumed' as const, anchor: sessionAnchorId(current) }; + }); + if (resumed.status === 'gone') { + await sessionReply(sessionAnchorId(ds), t('cmd.no_active_session', undefined, loc)); + return; + } + if (resumed.status === 'pending') { + await sessionReply( + resumed.anchor, + '当前 Codex App 仍有未结算消息，不能导入外部会话；请等待本轮完成或先关闭会话。', + ); + return; + } const cliName = getCliDisplayName(getBot(ds.larkAppId).config.cliId); - await sessionReply(sessionAnchorId(ds), t('cmd.adopt.resume_success', { cliName, project, title: target.title || target.cliSessionId.slice(0, 8) }, loc)); + await sessionReply(resumed.anchor, t('cmd.adopt.resume_success', { cliName, project, title: target.title || target.cliSessionId.slice(0, 8) }, loc)); } diff --git a/src/core/daemon-heartbeat.ts b/src/core/daemon-heartbeat.ts index d756635d0..496ec6377 100644 --- a/src/core/daemon-heartbeat.ts +++ b/src/core/daemon-heartbeat.ts @@ -24,6 +24,15 @@ export interface Heartbeat { * Daemons should write well within this window (≈ every 15s). */ export const HEARTBEAT_FRESH_MS = 60_000; +/** + * A stalled turn is still potentially executing model/tool side effects. Keep + * it inside the fleet-wide maintenance gate just like working; changing the + * user-facing projection must not make a global restart newly eligible. + */ +export function isMaintenanceBusyStatus(status: string | undefined): boolean { + return status === 'working' || status === 'stalled'; +} + export function heartbeatDirIn(dir: string): string { return join(dir, 'heartbeats'); } diff --git a/src/core/dashboard-ipc-server.ts b/src/core/dashboard-ipc-server.ts index 16e882dad..c68818586 100644 --- a/src/core/dashboard-ipc-server.ts +++ b/src/core/dashboard-ipc-server.ts @@ -1,6 +1,6 @@ // src/core/dashboard-ipc-server.ts import { createServer, type IncomingMessage, type ServerResponse, type Server } from 'node:http'; -import { readFileSync } from 'node:fs'; +import { readFileSync, unlinkSync } from 'node:fs'; import { homedir } from 'node:os'; import { join } from 'node:path'; import { logger } from '../utils/logger.js'; @@ -9,6 +9,12 @@ import { WORKFLOW_DAEMON_IPC_ROUTE_PREFIX } from '../workflows/v3/daemon-ipc-aut import { V3_SESSION_RUN_MUTATION_ROUTE_PREFIX } from '../workflows/v3/session-relay.js'; import { listenWithProbe } from '../utils/listen-with-probe.js'; import { dashboardSecretPath } from './dashboard-secret.js'; +import { + MANAGED_ORIGIN_ATTEST_ROUTE, + MANAGED_ORIGIN_PROOF_DOMAIN, + MANAGED_ORIGIN_PROOF_TTL_MS, + writeManagedOriginAttestationProof, +} from './managed-origin-attestation.js'; import * as sessionStore from '../services/session-store.js'; import * as scheduleStore from '../services/schedule-store.js'; import * as groupsStore from '../services/groups-store.js'; @@ -20,6 +26,7 @@ import * as backendTypeStore from '../services/backend-type-store.js'; import { isValidRiffBaseUrl, isValidRiffSandboxCluster } from '../adapters/backend/riff-backend.js'; import { ensureBackendAvailable } from '../services/backend-availability.js'; import type { BackendType } from '../adapters/backend/types.js'; +import * as persistentBackend from './persistent-backend.js'; import * as cardPrefsStore from '../services/card-prefs-store.js'; import * as substituteModeStore from '../services/substitute-mode-store.js'; import { createCliAdapterSync } from '../adapters/cli/registry.js'; @@ -68,7 +75,7 @@ import { listActiveSessions, findActiveBySessionId, closeSession, getActiveSessi import { listOnlineDaemons } from '../utils/daemon-discovery.js'; import { getChatMode, replyMessage, sendMessage, resolveUnionIdFromOpenId, listThreadMessages, listChatMessages, listChatBotMembers, getUserProfile, getUserProfileStrict, resolveAllowedUsersWithMap, type ChatBotMember } from '../im/lark/client.js'; import { parseApiMessage, cardContentHasUpgradeFallback, resolveMergedCardContent } from '../im/lark/message-parser.js'; -import { resumeSession, spawnDashboardSession, activateQueuedSession, closeCliMismatchedSessionsForBot, suspendActiveSessionsForBot } from './session-manager.js'; +import { resumeSession, spawnDashboardSession, activateQueuedSession, closeCliMismatchedSessionsForBot } from './session-manager.js'; import { parseSpawnRequest } from './session-create.js'; import { cleanupMaterializedDashboardImages, materializeDashboardImages } from './dashboard-images.js'; import { getCliDisplayName } from '../im/lark/card-builder.js'; @@ -92,6 +99,18 @@ import { validateTriggerRequest, type TriggerResponse } from '../services/trigge import { resolveCliSelection, selectionKeyForBot } from '../setup/cli-selection.js'; import { checkCliAvailability } from '../setup/cli-availability.js'; import { enrichHistorySenders, type HistoryBotInfo } from '../dashboard/history-senders.js'; +import { + validateCodexAppManagedSendOrigin, +} from '../utils/codex-app-dispatch-ledger.js'; +import { withBotTurnAdmission, withBotTurnMutation } from './bot-turn-mutation-gate.js'; +import { + protectedSessionMutationReasons, +} from './session-mutation-guard.js'; +import { + SUPERVISOR_SHUTDOWN_ROUTE, + isExactSupervisorShutdownRequest, + type SupervisorShutdownIdentity, +} from './supervisor-shutdown-ipc.js'; // 机器人真·改名 renamer，由 daemon 启动时注册（开放平台自动化 + daemon 侧 // botName/descriptor/bots-info 同步都在 daemon 的闭包里做）。未注册（测试环境） @@ -113,6 +132,16 @@ let botAvatarChanger: ((image: Buffer) => Promise) | null = nu export function setBotAvatarChanger(fn: ((image: Buffer) => Promise) | null): void { botAvatarChanger = fn; } + +type SupervisorShutdownRegistration = SupervisorShutdownIdentity & { + shutdown: () => Promise; +}; +let supervisorShutdownRegistration: SupervisorShutdownRegistration | null = null; +export function setSupervisorShutdownHandler( + registration: SupervisorShutdownRegistration | null, +): void { + supervisorShutdownRegistration = registration; +} import { composeRowFromActive, composeRowFromClosed, @@ -187,6 +216,77 @@ export function jsonRes(res: ServerResponse, status: number, body: unknown): voi res.end(JSON.stringify(body)); } +function rejectProtectedSessionMutation( + res: ServerResponse, + values: readonly (DaemonSession | Session)[], +): boolean { + const bySessionId = new Map; + }>(); + for (const value of values) { + const session = 'session' in value ? value.session : value; + const reasons = protectedSessionMutationReasons(value); + if (reasons.length === 0) continue; + const existing = bySessionId.get(session.sessionId); + if (existing) { + existing.reasons = [...new Set([...existing.reasons, ...reasons])]; + continue; + } + bySessionId.set(session.sessionId, { + sessionId: session.sessionId, + ...(session.cliId ? { cliId: session.cliId } : {}), + reasons, + }); + } + const blockingSessions = [...bySessionId.values()]; + if (blockingSessions.length === 0) return false; + const codexDispatchOnly = blockingSessions.every(blocker => + blocker.reasons.every(reason => reason === 'codex_app_dispatch')); + jsonRes(res, 409, { + ok: false, + error: codexDispatchOnly + ? 'codex_app_dispatch_pending' + : 'session_mutation_pending', + blockingSessions, + }); + return true; +} + +ipcRoute('POST', SUPERVISOR_SHUTDOWN_ROUTE, async (req, res) => { + // The production server-wide HMAC gate records trusted requests here. Keep + // an explicit route-local check: shutdown is never a bare loopback API. + if (!isTrustedHostIpcRequest(req)) { + return jsonRes(res, 403, { ok: false, error: 'supervisor_shutdown_unauthorized' }); + } + const registration = supervisorShutdownRegistration; + if (!registration) { + return jsonRes(res, 503, { ok: false, error: 'supervisor_shutdown_not_ready' }); + } + let body: unknown; + try { body = await readJsonBody(req); } + catch { return jsonRes(res, 400, { ok: false, error: 'invalid_json' }); } + if (!isExactSupervisorShutdownRequest(registration, body)) { + return jsonRes(res, 409, { ok: false, error: 'supervisor_shutdown_generation_mismatch' }); + } + // ACK means this exact in-memory generation accepted the request; the CLI + // still proves OS/PM2 quiescence. Start after flushing the ACK so a long Riff + // drain cannot turn a valid request into an ambiguous transport timeout. + jsonRes(res, 202, { + ok: true, + accepted: true, + larkAppId: registration.larkAppId, + bootInstanceId: registration.bootInstanceId, + processStartIdentity: registration.processStartIdentity, + }); + setImmediate(() => { + void registration.shutdown().catch(error => { + logger.error(`supervisor shutdown failed: ${error instanceof Error ? error.message : String(error)}`); + }); + }); +}); + export async function readJsonBody(req: IncomingMessage): Promise { const chunks: Buffer[] = []; for await (const c of req) chunks.push(c as Buffer); @@ -194,6 +294,86 @@ export async function readJsonBody(req: IncomingMessage): Promise( + req: IncomingMessage, + maxBytes: number, + timeoutMs: number, +): Promise { + const contentLength = req.headers['content-length']; + if (typeof contentLength === 'string') { + if (!/^\d+$/.test(contentLength) || Number(contentLength) > maxBytes) { + throw new IpcBodyTooLargeError('request body too large'); + } + } + return await new Promise((resolve, reject) => { + const chunks: Buffer[] = []; + let total = 0; + let settled = false; + const cleanup = () => { + clearTimeout(timer); + req.off('data', onData); + req.off('end', onEnd); + req.off('error', onError); + req.off('aborted', onAborted); + }; + const fail = (err: Error) => { + if (settled) return; + settled = true; + req.pause(); + cleanup(); + reject(err); + }; + const onData = (raw: Buffer | string) => { + const chunk = Buffer.isBuffer(raw) ? raw : Buffer.from(raw); + total += chunk.length; + if (total > maxBytes) { + fail(new IpcBodyTooLargeError('request body too large')); + return; + } + chunks.push(chunk); + }; + const onEnd = () => { + if (settled) return; + settled = true; + cleanup(); + try { + resolve(chunks.length === 0 + ? {} as T + : JSON.parse(Buffer.concat(chunks, total).toString('utf8')) as T); + } catch (err) { + reject(err); + } + }; + const onError = (err: Error) => fail(err); + const onAborted = () => fail(new Error('request aborted')); + const timer = setTimeout( + () => fail(new IpcBodyTimeoutError('request body timed out')), + timeoutMs, + ); + timer.unref?.(); + req.on('data', onData); + req.once('end', onEnd); + req.once('error', onError); + req.once('aborted', onAborted); + }); +} + +function closeUntrustedRequestAfterResponse(req: IncomingMessage, res: ServerResponse): void { + res.setHeader('connection', 'close'); + res.once('finish', () => req.destroy()); + // Drain whatever is already buffered until the response has been flushed; + // the finish hook then closes a partial/slow body instead of reusing it as a + // keep-alive request. + req.resume(); +} + // ─── Trusted-host auth (loopback + route-bound HMAC) ──────────────────────── // // Production start enables a server-wide gate: loopback is connectivity, not @@ -250,6 +430,10 @@ function routeHasNarrowUntrustedAuth(method: string, pathname: string): boolean if (method === 'POST' && /^\/api\/sessions\/[^/]+\/(?:slash|cd)$/.test(pathname)) return true; if (method === 'POST' && pathname === '/api/hooks/emit') return true; if (method === 'POST' && pathname === '/api/attention') return true; + // macOS read-isolated `botmux send` presents a rotating worker capability; + // the handler writes the authoritative tuple into a host-owned read-only + // proof sidecar, so loopback response spoofing cannot confer authority. + if (method === 'POST' && pathname === MANAGED_ORIGIN_ATTEST_ROUTE) return true; // Workflow v3 mutations carry their own domain-separated full-envelope // protocol (request signature over method/path/exact body with nonce // anti-replay + boot audience, signed response), keyed on the same host @@ -293,6 +477,168 @@ ipcRoute('GET', '/__health', (_req, res) => { jsonRes(res, 200, { ok: true }); }); +const MANAGED_ORIGIN_ATTEST_BODY_MAX_BYTES = 2 * 1024; +const MANAGED_ORIGIN_ATTEST_BODY_TIMEOUT_MS = 1_000; +const MANAGED_ORIGIN_ATTEST_MAX_PREAUTH_IN_FLIGHT = 128; +const MANAGED_ORIGIN_ATTEST_MAX_OUTSTANDING_PER_SESSION = 64; +const managedOriginOutstandingProofs = new Map(); +let managedOriginPreauthInFlight = 0; + +async function handleManagedOriginAttestation( + req: IncomingMessage, + res: ServerResponse, +): Promise { + let body: { + sessionId?: unknown; + originChannelId?: unknown; + channelId?: unknown; + originCapability?: unknown; + nonce?: unknown; + }; + try { + body = await readBoundedJsonBody( + req, + MANAGED_ORIGIN_ATTEST_BODY_MAX_BYTES, + MANAGED_ORIGIN_ATTEST_BODY_TIMEOUT_MS, + ); + } + catch (err) { + if (err instanceof IpcBodyTooLargeError || err instanceof IpcBodyTimeoutError) { + closeUntrustedRequestAfterResponse(req, res); + } + return jsonRes( + res, + err instanceof IpcBodyTooLargeError + ? 413 + : err instanceof IpcBodyTimeoutError + ? 408 + : 400, + { + ok: false, + error: err instanceof IpcBodyTooLargeError + ? 'body_too_large' + : err instanceof IpcBodyTimeoutError + ? 'body_timeout' + : 'bad_json', + }, + ); + } + if (!body || typeof body !== 'object' || Array.isArray(body)) { + return jsonRes(res, 400, { ok: false, error: 'bad_attestation_request' }); + } + const sessionId = typeof body.sessionId === 'string' && body.sessionId.length <= 256 + ? body.sessionId + : ''; + const capability = typeof body.originCapability === 'string' + && /^[a-f0-9]{32,128}$/i.test(body.originCapability) + ? body.originCapability + : ''; + const channelId = typeof (body.originChannelId ?? body.channelId) === 'string' + && /^[a-f0-9]{64}$/.test((body.originChannelId ?? body.channelId) as string) + ? (body.originChannelId ?? body.channelId) as string + : ''; + const nonce = typeof body.nonce === 'string' && /^[a-f0-9]{64}$/.test(body.nonce) + ? body.nonce + : ''; + if (!sessionId || !channelId || !capability || !nonce) { + return jsonRes(res, 400, { ok: false, error: 'bad_attestation_request' }); + } + const ds = findActiveBySessionId(sessionId); + const worker = ds?.worker; + let workerPidLive = false; + if (worker && Number.isSafeInteger(worker.pid) && (worker.pid ?? 0) > 0) { + try { + process.kill(worker.pid!, 0); + workerPidLive = true; + } catch { /* ESRCH/EPERM/invalid pid all fail closed */ } + } + const workerLive = !!worker + && worker.connected === true + && !worker.killed + && worker.exitCode === null + && worker.signalCode === null + && workerPidLive; + const liveTurnId = ds?.managedTurnOrigin?.turnId; + const verified = authorizeSessionScopedIpc({ + trustedHost: false, + sessionExists: !!ds, + receiverSession: !!ds?.session.vcMeetingReceiver, + allowReceiver: true, + sessionId, + liveOrigin: ds?.managedTurnOrigin, + claimedCapability: capability, + }); + if (!verified.ok || !ds?.managedTurnOrigin || !liveTurnId || !workerLive) { + return jsonRes(res, 403, { ok: false, error: 'origin_unproven' }); + } + const origin = ds.managedTurnOrigin; + if (!origin.originChannelId || !/^[a-f0-9]{64}$/.test(origin.originChannelId)) { + return jsonRes(res, 403, { ok: false, error: 'origin_channel_unproven' }); + } + if (channelId !== origin.originChannelId) { + return jsonRes(res, 403, { ok: false, error: 'origin_channel_unproven' }); + } + const codexDecision = validateCodexAppManagedSendOrigin( + ds.session.codexAppDispatchLedger, + origin, + ds.initConfig?.cliId === 'codex-app' || ds.session.cliId === 'codex-app', + ); + if (!codexDecision.ok) { + return jsonRes(res, 409, { ok: false, error: 'origin_not_sendable' }); + } + const outstanding = managedOriginOutstandingProofs.get(sessionId) ?? 0; + if (outstanding >= MANAGED_ORIGIN_ATTEST_MAX_OUTSTANDING_PER_SESSION) { + return jsonRes(res, 429, { ok: false, error: 'too_many_attestations' }); + } + let proofPath: string; + try { + proofPath = writeManagedOriginAttestationProof({ + dataDir: config.session.dataDir, + proof: { + domain: MANAGED_ORIGIN_PROOF_DOMAIN, + version: 1, + nonce, + channelId: origin.originChannelId, + sessionId, + turnId: liveTurnId, + ...(origin.dispatchAttempt !== undefined + ? { dispatchAttempt: origin.dispatchAttempt } + : {}), + requiresCodexAppLedger: codexDecision.requiresLedger, + issuedAtMs: Date.now(), + }, + }); + } catch (err) { + logger.warn(`[managed-origin] could not write attestation proof: ${err}`); + return jsonRes(res, 409, { ok: false, error: 'proof_unavailable' }); + } + managedOriginOutstandingProofs.set(sessionId, outstanding + 1); + const cleanupTimer = setTimeout(() => { + try { unlinkSync(proofPath); } catch { /* expired/already gone */ } + const remaining = (managedOriginOutstandingProofs.get(sessionId) ?? 1) - 1; + if (remaining > 0) managedOriginOutstandingProofs.set(sessionId, remaining); + else managedOriginOutstandingProofs.delete(sessionId); + }, MANAGED_ORIGIN_PROOF_TTL_MS + 1_000); + cleanupTimer.unref?.(); + return jsonRes(res, 200, { ok: true }); +} + +ipcRoute('POST', MANAGED_ORIGIN_ATTEST_ROUTE, async (req, res) => { + // This counter is acquired before parsing or capability lookup. Per-session + // proof quotas cannot protect the unauthenticated slow-body phase because a + // session id is not trustworthy until the complete request has been read. + if (managedOriginPreauthInFlight >= MANAGED_ORIGIN_ATTEST_MAX_PREAUTH_IN_FLIGHT) { + closeUntrustedRequestAfterResponse(req, res); + return jsonRes(res, 429, { ok: false, error: 'too_many_attestation_requests' }); + } + managedOriginPreauthInFlight += 1; + try { + await handleManagedOriginAttestation(req, res); + } finally { + managedOriginPreauthInFlight -= 1; + } +}); + // ─── Session list / detail ───────────────────────────────────────────────── // Row shape + composers live in dashboard-rows.ts so worker-pool can publish // SessionRow events without importing this module (which would create a cycle: @@ -359,8 +705,41 @@ ipcRoute('GET', '/api/sessions/:sessionId', (_req, res, params) => { }); ipcRoute('POST', '/api/sessions/:sessionId/close', async (_req, res, params) => { - const r = await closeSession(params.sessionId); - jsonRes(res, 200, r); + const initial = findSessionRecord(params.sessionId); + if (!initial) return jsonRes(res, 200, { ok: true, alreadyClosed: true }); + const larkAppId = initial.larkAppId || cachedLarkAppId; + if (!larkAppId) return jsonRes(res, 503, { ok: false, error: 'bot_not_found' }); + return withBotTurnMutation(larkAppId, async () => { + // Re-resolve only after every earlier admitted turn has drained. Explicit + // close is the abandon boundary and may clear accepted FIFO, but it must + // never clear a stale pre-drain snapshot while a turn is still preparing. + const current = findSessionRecord(params.sessionId); + if (!current || current.status === 'closed') { + return jsonRes(res, 200, { ok: true, alreadyClosed: true }); + } + const r = await closeSession(params.sessionId); + // A worker-less Riff owner is not closed until its remote task confirms + // cancellation. Preserve the active row/task id and expose a retryable + // failure instead of returning a false 200 success while the agent runs. + jsonRes(res, r.ok ? 200 : 502, r); + }); +}); + +// `botmux list` zombie pruning is maintenance, not explicit abandon. Serialize +// against inbound admission and refuse if any backend-neutral durable owner +// became unsettled after the CLI took its liveness snapshot. +ipcRoute('POST', '/api/sessions/:sessionId/prune', async (_req, res, params) => { + const initial = findSessionRecord(params.sessionId); + if (!initial) return jsonRes(res, 404, { ok: false, error: 'session_not_found' }); + const larkAppId = initial.larkAppId || cachedLarkAppId; + if (!larkAppId) return jsonRes(res, 503, { ok: false, error: 'bot_not_found' }); + return withBotTurnMutation(larkAppId, async () => { + const current = findSessionRecord(params.sessionId); + if (!current) return jsonRes(res, 404, { ok: false, error: 'session_not_found' }); + if (rejectProtectedSessionMutation(res, [current])) return; + const r = await closeSession(params.sessionId); + jsonRes(res, r.ok ? 200 : 502, r); + }); }); /** Post a scope-aware "restarting" notice into the session's Lark thread/chat, @@ -389,7 +768,10 @@ function postRestartNotice(ds: DaemonSession, fresh: boolean): void { } } -ipcRoute('POST', '/api/sessions/:sessionId/restart', (_req, res, params) => { +ipcRoute('POST', '/api/sessions/:sessionId/restart', async (_req, res, params) => { + const initial = findActiveBySessionId(params.sessionId); + if (!initial) return jsonRes(res, 404, { ok: false, error: 'session_not_active' }); + return withBotTurnMutation(initial.larkAppId, () => { const ds = findActiveBySessionId(params.sessionId); if (!ds) return jsonRes(res, 404, { ok: false, error: 'session_not_active' }); // Adopt/observed sessions: botmux never owned the CLI — restarting would kill @@ -397,11 +779,16 @@ ipcRoute('POST', '/api/sessions/:sessionId/restart', (_req, res, params) => { if (ds.adoptedFrom || ds.initConfig?.adoptMode) { return jsonRes(res, 409, { ok: false, error: 'adopt_restart_unsupported' }); } + if ((ds.initConfig?.backendType ?? ds.session.backendType) === 'riff' + || ds.session.cliId === 'riff') { + return jsonRes(res, 409, { ok: false, error: 'riff_restart_unsupported' }); + } + if (rejectProtectedSessionMutation(res, [ds])) return; const cliId = ds.session.cliId ?? 'unknown'; if (ds.worker && !ds.worker.killed) { // Live worker → in-place CLI restart (kills the CLI, respawns with --resume). try { - ds.worker.send({ type: 'restart' } as DaemonToWorker); + ds.worker.send({ type: 'restart', reason: 'operator' } as DaemonToWorker); } catch (err) { return jsonRes(res, 502, { ok: false, error: String(err) }); } @@ -416,6 +803,7 @@ ipcRoute('POST', '/api/sessions/:sessionId/restart', (_req, res, params) => { forkWorker(ds, '', ds.hasHistory); postRestartNotice(ds, true); jsonRes(res, 200, { ok: true, sessionId: params.sessionId, cliId, revived: true }); + }); }); /** Manually suspend one active session: kill the worker + CLI/pane, session @@ -423,7 +811,10 @@ ipcRoute('POST', '/api/sessions/:sessionId/restart', (_req, res, params) => { * the same semantics the idle-worker sweeper applies over the live cap. * Primary use: `botmux suspend --isolated` after a credential rotation, so * isolated bots' next cold spawn re-provisions the freshest creds. */ -ipcRoute('POST', '/api/sessions/:sessionId/suspend', (_req, res, params) => { +ipcRoute('POST', '/api/sessions/:sessionId/suspend', async (_req, res, params) => { + const initial = findActiveBySessionId(params.sessionId); + if (!initial) return jsonRes(res, 404, { ok: false, error: 'session_not_active' }); + return withBotTurnMutation(initial.larkAppId, () => { const ds = findActiveBySessionId(params.sessionId); if (!ds) return jsonRes(res, 404, { ok: false, error: 'session_not_active' }); // Adopt/observed sessions: botmux never owned the CLI — suspending would kill @@ -431,6 +822,7 @@ ipcRoute('POST', '/api/sessions/:sessionId/suspend', (_req, res, params) => { if (ds.adoptedFrom || ds.initConfig?.adoptMode) { return jsonRes(res, 409, { ok: false, error: 'adopt_suspend_unsupported' }); } + if (rejectProtectedSessionMutation(res, [ds])) return; if (!ds.worker || ds.worker.killed) { // Worker already gone (idle-suspended / crash-stopped) — the goal state is // already reached, so report idempotent success without a live kill. @@ -442,6 +834,7 @@ ipcRoute('POST', '/api/sessions/:sessionId/suspend', (_req, res, params) => { return jsonRes(res, 409, { ok: false, error: 'backend_not_suspendable' }); } jsonRes(res, 200, { ok: true, sessionId: params.sessionId, suspended: true }); + }); }); /** 会话级 CLI IPC（slash/cd）的调用方证明：trusted-host（.dashboard-secret HMAC， @@ -617,41 +1010,89 @@ ipcRoute('POST', '/api/sessions/:sessionId/board', async (req, res, params) => { if (!column && position === null) return jsonRes(res, 400, { ok: false, error: 'bad_request' }); const session = findSessionRecord(params.sessionId); if (!session) return jsonRes(res, 404, { ok: false, error: 'session_not_found' }); + const larkAppId = session.larkAppId || cachedLarkAppId; + if (!larkAppId) return jsonRes(res, 503, { ok: false, error: 'bot_not_found' }); + return withBotTurnAdmission(larkAppId, async () => { + const currentSession = findSessionRecord(params.sessionId); + if (!currentSession) return jsonRes(res, 404, { ok: false, error: 'session_not_found' }); // 待办池(queued)会话被拖到「进行中」= 激活：把暂存内容当首轮发给 CLI 开跑。 // activateQueuedSession 内部会清 queued + 把列设成 in_progress + forkWorker。 const activeDs = findActiveBySessionId(params.sessionId); + let activationTransferred = false; if (column === 'in_progress' && activeDs?.session.queued) { - await activateQueuedSession(activeDs); + const activated = await activateQueuedSession(activeDs); + if (!activated.ok) return jsonRes(res, 500, activated); + activationTransferred = true; } else if (column) { - session.kanbanColumn = column; + currentSession.kanbanColumn = column; + } + if (position !== null) currentSession.kanbanPosition = position; + try { + sessionStore.updateSession(currentSession); + } catch (err) { + if (!activationTransferred) throw err; + logger.error( + `[dashboard] board metadata persistence failed after queued activation ownership transferred: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); } - if (position !== null) session.kanbanPosition = position; - sessionStore.updateSession(session); dashboardEventBus.publish({ type: 'session.update', body: { sessionId: params.sessionId, // queued 一并下发：激活后 session.queued 已为 false，前端浅合并若不带这个字段 // 会残留 queued=true（卡片仍显示「开始」、再点 409）。!!session.queued 始终反映现态。 - patch: { kanbanColumn: session.kanbanColumn, kanbanPosition: session.kanbanPosition, queued: !!session.queued }, + patch: { kanbanColumn: currentSession.kanbanColumn, kanbanPosition: currentSession.kanbanPosition, queued: !!currentSession.queued }, }, }); jsonRes(res, 200, { ok: true }); + }); +}); + +// Narrow CLI whiteboard binding mutation. Keeping this daemon-side avoids a +// short-lived `botmux whiteboard` process rewriting a stale whole Session row +// over a concurrent Codex App FIFO transition. +ipcRoute('POST', '/api/sessions/:sessionId/whiteboard', async (req, res, params) => { + let body: { whiteboardId?: unknown }; + try { body = await readJsonBody(req); } + catch { return jsonRes(res, 400, { ok: false, error: 'bad_json' }); } + if (typeof body.whiteboardId !== 'string' + || body.whiteboardId.length === 0 + || body.whiteboardId.length > 256) { + return jsonRes(res, 400, { ok: false, error: 'bad_whiteboard_id' }); + } + const session = findSessionRecord(params.sessionId); + if (!session) return jsonRes(res, 404, { ok: false, error: 'session_not_found' }); + const larkAppId = session.larkAppId || cachedLarkAppId; + if (!larkAppId) return jsonRes(res, 503, { ok: false, error: 'bot_not_found' }); + return withBotTurnAdmission(larkAppId, async () => { + const current = findSessionRecord(params.sessionId); + if (!current) return jsonRes(res, 404, { ok: false, error: 'session_not_found' }); + current.whiteboardId = body.whiteboardId as string; + sessionStore.updateSession(current); + jsonRes(res, 200, { ok: true, whiteboardId: current.whiteboardId }); + }); }); // 待办池会话「开始」：把 parked 会话激活（发首轮、起 CLI），与拖到「进行中」同义。 ipcRoute('POST', '/api/sessions/:sessionId/start', async (_req, res, params) => { + const initial = findActiveBySessionId(params.sessionId); + if (!initial) return jsonRes(res, 404, { ok: false, error: 'session_not_found' }); + return withBotTurnAdmission(initial.larkAppId, async () => { const ds = findActiveBySessionId(params.sessionId); if (!ds) return jsonRes(res, 404, { ok: false, error: 'session_not_found' }); if (!ds.session.queued) return jsonRes(res, 409, { ok: false, error: 'not_queued' }); const r = await activateQueuedSession(ds); if (!r.ok) return jsonRes(res, 500, r); - sessionStore.updateSession(ds.session); dashboardEventBus.publish({ type: 'session.update', - body: { sessionId: params.sessionId, patch: { kanbanColumn: ds.session.kanbanColumn, queued: false } }, + body: { + sessionId: params.sessionId, + patch: { kanbanColumn: ds.session.kanbanColumn, queued: !!ds.session.queued }, + }, }); jsonRes(res, 200, { ok: true }); + }); }); // Dashboard「创建会话」spawn：在新建的群里为本 daemon 的 bot 拉起/暂存一条 chat-scope @@ -665,6 +1106,7 @@ ipcRoute('POST', '/api/sessions/spawn', async (req, res) => { const parsed = parseSpawnRequest(body); if (!parsed.ok) return jsonRes(res, 400, { ok: false, error: parsed.error }); const postBanner = !!(body as any).postBanner; + return withBotTurnAdmission(cachedLarkAppId, async () => { let attachments; try { attachments = materializeDashboardImages(cachedLarkAppId, parsed.value.images); @@ -690,6 +1132,7 @@ ipcRoute('POST', '/api/sessions/spawn', async (req, res) => { return jsonRes(res, r.error === 'session_exists' ? 409 : 500, r); } jsonRes(res, 200, r); + }); }); ipcRoute('POST', '/api/chat-reply-mode', async (req, res) => { @@ -1031,6 +1474,13 @@ ipcRoute('POST', '/api/sessions/:sessionId/sandbox-land/:action', (_req, res, pa */ ipcRoute('POST', '/api/sessions/:sessionId/resume', async (req, res, params) => { const sessionId = params.sessionId; + const sourceSession = findSessionRecord(sessionId); + if (!sourceSession) return jsonRes(res, 404, { ok: false, error: 'not_found' }); + // Legacy persisted sessions may carry an empty larkAppId and are hydrated + // with this daemon's identity by resumeSession. Use the same fallback for + // admission instead of rejecting an otherwise valid recovery record. + const larkAppId = sourceSession.larkAppId || cachedLarkAppId || '__legacy_unbound__'; + return withBotTurnAdmission(larkAppId, async () => { const reg = getActiveSessionsRegistry(); if (!reg) return jsonRes(res, 503, { ok: false, error: 'registry_unavailable' }); const result = await resumeSession(sessionId, reg); @@ -1097,6 +1547,7 @@ ipcRoute('POST', '/api/sessions/:sessionId/resume', async (req, res, params) => workingDir: ds.session.workingDir, cliId, }); + }); }); /** @@ -2305,6 +2756,7 @@ ipcRoute('PUT', '/api/bot-avatar', async (req, res) => { // path; this covers the hot-switch path). ipcRoute('PUT', '/api/bot-agent', async (req, res) => { if (!cachedLarkAppId) return jsonRes(res, 503, { error: 'larkAppId_not_set' }); + const larkAppId = cachedLarkAppId; let body: { cliId?: unknown; model?: unknown }; try { body = await readJsonBody<{ cliId?: unknown; model?: unknown }>(req); } catch { return jsonRes(res, 400, { ok: false, error: 'bad_json' }); } @@ -2318,7 +2770,7 @@ ipcRoute('PUT', '/api/bot-agent', async (req, res) => { return jsonRes(res, 400, { ok: false, error: 'invalid_cli', message: err?.message ?? String(err) }); } const model = typeof body.model === 'string' ? body.model.trim() : ''; - const currentBotConfig = getBot(cachedLarkAppId).config; + const currentBotConfig = getBot(larkAppId).config; const availability = checkCliAvailability({ cliId: selected.cliId, wrapperCli: selected.wrapperCli, @@ -2331,12 +2783,28 @@ ipcRoute('PUT', '/api/bot-agent', async (req, res) => { ? undefined : `配置已保存，但所选 Agent 当前无法启动：${availability.reason ?? '本地启动依赖不可用'}。请先在 daemon 所在机器安装或修正 PATH / CLI 路径。`; - // If the new CLI/wrapper can no longer enforce a currently-on read isolation, - // auto-clear the flag here so the next session doesn't fail-close on it. (The - // read-isolation toggle validates at enable time; changing the agent afterwards - // is the other way a bot could end up configured-but-unenforceable.) - let readIsolationCleared = false; - const r = await rmwBotEntry(cachedLarkAppId, (entry) => { + return withBotTurnMutation(larkAppId, async () => { + // Agent selection can replace every live worker generation and may also + // auto-clear readIsolation. Close admission and drain in-flight acceptance + // before inspecting both the registry and restart source of truth. A + // settings mutation is not an explicit abandon boundary: an unsettled + // Codex App FIFO must survive unchanged for recovery. + const activeBotSessions = listActiveSessions().filter(ds => ds.larkAppId === larkAppId); + const persistedActiveBotSessions = sessionStore.listSessions().filter(session => + session.status === 'active' + && (session.larkAppId === larkAppId || !session.larkAppId), + ); + if (rejectProtectedSessionMutation(res, [ + ...activeBotSessions, + ...persistedActiveBotSessions, + ])) return; + + // If the new CLI/wrapper can no longer enforce a currently-on read isolation, + // auto-clear the flag here so the next session doesn't fail-close on it. (The + // read-isolation toggle validates at enable time; changing the agent afterwards + // is the other way a bot could end up configured-but-unenforceable.) + let readIsolationCleared = false; + const r = await rmwBotEntry(larkAppId, (entry) => { entry.cliId = selected.cliId; if (selected.wrapperCli) entry.wrapperCli = selected.wrapperCli; else delete entry.wrapperCli; @@ -2357,42 +2825,43 @@ ipcRoute('PUT', '/api/bot-agent', async (req, res) => { delete entry.backendType; } return { write: true, result: null }; - }); - if (!r.ok) return jsonRes(res, 400, { ok: false, error: r.reason }); - - const bot = getBot(cachedLarkAppId); - bot.config.cliId = selected.cliId; - if (selected.wrapperCli) bot.config.wrapperCli = selected.wrapperCli; - else bot.config.wrapperCli = undefined; - bot.config.model = model || undefined; - if (readIsolationCleared) bot.config.readIsolation = false; - if (selected.cliId === 'riff') { - bot.config.backendType = 'riff'; - } else if (bot.config.backendType === 'riff') { - bot.config.backendType = undefined; - } + }); + if (!r.ok) return jsonRes(res, 400, { ok: false, error: r.reason }); + + const bot = getBot(larkAppId); + bot.config.cliId = selected.cliId; + if (selected.wrapperCli) bot.config.wrapperCli = selected.wrapperCli; + else bot.config.wrapperCli = undefined; + bot.config.model = model || undefined; + if (readIsolationCleared) bot.config.readIsolation = false; + if (selected.cliId === 'riff') { + bot.config.backendType = 'riff'; + } else if (bot.config.backendType === 'riff') { + bot.config.backendType = undefined; + } - // 热切后立刻清掉本 bot 名下失配的存量会话——否则它们冻结的旧 CLI 会被下一条 - // 消息 lazy resume 复活，要等下次 daemon 重启才被 restore 守卫清理。 - const closedMismatchedSessions = await closeCliMismatchedSessionsForBot(cachedLarkAppId); + // 热切后立刻清掉本 bot 名下失配的存量会话——否则它们冻结的旧 CLI 会被下一条 + // 消息 lazy resume 复活，要等下次 daemon 重启才被 restore 守卫清理。 + const closedMismatchedSessions = await closeCliMismatchedSessionsForBot(larkAppId); - const selectionKey = selectionKeyForBot(selected.cliId, selected.wrapperCli); - jsonRes(res, 200, { - ok: true, - cliId: selected.cliId, - wrapperCli: selected.wrapperCli ?? null, - model: model || null, - selectionKey, - closedMismatchedSessions, - // Report the (possibly auto-cleared) read-isolation state + whether the new - // agent can still enforce it, so the dashboard updates its toggle immediately - // instead of showing a stale enabled/supported state until a full refetch. - readIsolation: bot.config.readIsolation === true, - readIsolationSupported: readIsolationEnforceableFor(bot.config), - readIsolationCleared, - agentAvailable: availability.available, - availabilityWarning, - requiredCommand: availability.command, + const selectionKey = selectionKeyForBot(selected.cliId, selected.wrapperCli); + jsonRes(res, 200, { + ok: true, + cliId: selected.cliId, + wrapperCli: selected.wrapperCli ?? null, + model: model || null, + selectionKey, + closedMismatchedSessions, + // Report the (possibly auto-cleared) read-isolation state + whether the new + // agent can still enforce it, so the dashboard updates its toggle immediately + // instead of showing a stale enabled/supported state until a full refetch. + readIsolation: bot.config.readIsolation === true, + readIsolationSupported: readIsolationEnforceableFor(bot.config), + readIsolationCleared, + agentAvailable: availability.available, + availabilityWarning, + requiredCommand: availability.command, + }); }); }); @@ -2648,6 +3117,9 @@ ipcRoute('PUT', '/api/bot-sandbox', async (req, res) => { let body: { enabled?: unknown }; try { body = await readJsonBody<{ enabled?: unknown }>(req); } catch { return jsonRes(res, 400, { ok: false, error: 'bad_json' }); } + // File-sandbox policy is frozen onto each Session at creation and reused on + // restore; this toggle is intentionally next-session-only and cannot mutate + // a live pane's profile. const r = await sandboxStore.updateBotSandbox(cachedLarkAppId, body.enabled === true); if (!r.ok) return jsonRes(res, 400, { ok: false, error: r.reason }); jsonRes(res, 200, { ok: true, sandbox: r.sandbox }); @@ -2658,26 +3130,104 @@ ipcRoute('PUT', '/api/bot-sandbox', async (req, res) => { // unreadable). The macOS counterpart of the file sandbox above. ipcRoute('PUT', '/api/bot-read-isolation', async (req, res) => { if (!cachedLarkAppId) return jsonRes(res, 503, { error: 'larkAppId_not_set' }); + const larkAppId = cachedLarkAppId; let body: { enabled?: unknown }; try { body = await readJsonBody<{ enabled?: unknown }>(req); } catch { return jsonRes(res, 400, { ok: false, error: 'bad_json' }); } const enable = body.enabled === true; - // The worker FAIL-CLOSES (refuses to start the session) for a configured - // readIsolation that can't be enforced: non-darwin, an adapter without - // supportsReadIsolation, or a wrapperCli gateway. Reject enabling it in exactly - // those cases so the toggle can never brick the bot's next session. (Turning it - // OFF is always allowed — recovers a flag that became unenforceable.) - if (enable && !readIsolationEnforceable(cachedLarkAppId)) { - return jsonRes(res, 400, { ok: false, error: 'read_isolation_unenforceable' }); - } - const r = await sandboxStore.updateBotReadIsolation(cachedLarkAppId, enable); - if (!r.ok) return jsonRes(res, 400, { ok: false, error: r.reason }); - // Read isolation only takes effect at COLD spawn (provisionIsolatedBotHome + - // Seatbelt wrapper run then). Suspend this bot's active sessions so the next - // message cold-restarts under the new state — otherwise close+resume would keep - // running the old, un-provisioned state and the toggle would silently no-op. - const suspendedSessions = await suspendActiveSessionsForBot(cachedLarkAppId); - jsonRes(res, 200, { ok: true, readIsolation: r.readIsolation, suspendedSessions }); + return withBotTurnMutation(larkAppId, async () => { + // An idempotent request changes neither the durable policy nor any pane. + // Return before pending/active/teardown guards so a dashboard refresh that + // repeats the authoritative value cannot be rejected merely because the + // bot is doing work. + if (sandboxStore.getBotReadIsolation(larkAppId) === enable) { + return jsonRes(res, 200, { + ok: true, + readIsolation: enable, + suspendedSessions: 0, + changed: false, + }); + } + // Close admission first and drain handlers that may already be awaiting + // downloads/noteTurnReceived. Ledger preflight alone cannot see those + // pre-accept turns; draining prevents a post-sweep send into a killed ds. + const activeBotSessions = listActiveSessions().filter(ds => ds.larkAppId === larkAppId); + // Registry state alone is insufficient: partial restore, an anchor + // collision, or a failed staggered reattach can omit a durable active row + // while its persistent pane still survives. Consult the same persisted + // session source a restart will hydrate. Legacy unscoped active rows are + // conservatively treated as this daemon's until explicitly closed. + const persistedBotSessions = sessionStore.listSessions().filter(session => + session.larkAppId === larkAppId || !session.larkAppId, + ); + const persistedActiveBotSessions = persistedBotSessions.filter(session => + session.status === 'active', + ); + if (rejectProtectedSessionMutation(res, [ + ...activeBotSessions, + ...persistedActiveBotSessions, + ])) return; + // Crash-transactional safety boundary: bots.json is the restart source of + // truth, while a live tmux/herdr/zellij pane retains its old in-memory + // Seatbelt profile. Persisting the new flag before tearing those panes down + // creates an unrecoverable crash window because the restart path cannot + // prove which exact read/write isolation profile a surviving pane runs. + // Require explicit close first; with no active logical session there is no + // owned pane a restart can reattach under the newly persisted policy. + if (activeBotSessions.length > 0 || persistedActiveBotSessions.length > 0) { + return jsonRes(res, 409, { + ok: false, + error: 'read_isolation_active_sessions', + }); + } + // `/close` intentionally returns after sending worker close IPC and marking + // the row closed; persistent-pane destruction can lag. A closed row's pid + // is deliberately not probed: PID alone has no birth identity and may have + // been reused by an unrelated process. closeSession clears it atomically. + // For current rows, the stamped persistent backend is the teardown proof. + // Pre-stamp closed rows are not synchronously probed across three CLIs here: + // that legacy shell fan-out blocks the daemon event loop, while any active + // legacy row has already failed the active-session guard above. + for (const session of persistedBotSessions) { + if (session.adoptedFrom || session.title?.startsWith('Adopt:')) continue; + const backendTypes: persistentBackend.PersistentBackendType[] = + persistentBackend.isSuspendableBackendType(session.backendType) + ? [session.backendType] + : []; + for (const backendType of backendTypes) { + const backingName = persistentBackend.persistentSessionName( + backendType, + session.sessionId, + ); + if (persistentBackend.probePersistentSession(backendType, backingName) !== 'missing') { + return jsonRes(res, 409, { + ok: false, + error: 'read_isolation_teardown_unverified', + }); + } + } + } + // The worker FAIL-CLOSES (refuses to start the session) for a configured + // readIsolation that cannot be enforced. Check this after the active-session + // safety boundary so even an unsupported enable cannot obscure a surviving + // old-policy pane with a less important validation error. + if (enable && !readIsolationEnforceable(larkAppId)) { + return jsonRes(res, 400, { ok: false, error: 'read_isolation_unenforceable' }); + } + // With the gate closed and no active logical session, persistence is the + // only mutation. updateBotReadIsolation writes bots.json atomically and + // then publishes the same value to the daemon runtime before resolving. + // A crash at any point can only lead to a cold spawn under the old or new + // durable policy; there is no owned pane to reattach. + const r = await sandboxStore.updateBotReadIsolation(larkAppId, enable); + if (!r.ok) return jsonRes(res, 400, { ok: false, error: r.reason }); + jsonRes(res, 200, { + ok: true, + readIsolation: r.readIsolation, + suspendedSessions: 0, + changed: true, + }); + }); }); // Per-bot session backend override (pty | tmux | herdr | zellij), or clear it @@ -3023,6 +3573,10 @@ export function startIpcServer(opts: { * deliberate secret repair cannot strand a daemon on a stale cached key. * Tests that omit this option retain the lightweight in-process server. */ authRequired?: boolean; + /** Daemon restore barrier. The socket/health route may come up early so its + * descriptor is discoverable, but every state-bearing route waits until all + * durable session owners have been registered. */ + ready?: Promise; }): Promise { let boundPort = opts.port; const server: Server = createServer(async (req, res) => { @@ -3042,6 +3596,7 @@ export function startIpcServer(opts: { return jsonRes(res, 401, { ok: false, error: 'unauthorized', reason: auth.reason }); } } + if (!publicRoute && opts.ready) await opts.ready; for (const r of routes) { if (r.method !== req.method) continue; const m = r.pattern.exec(url.pathname); diff --git a/src/core/explicit-session-backing-cleanup.ts b/src/core/explicit-session-backing-cleanup.ts new file mode 100644 index 000000000..d5b560194 --- /dev/null +++ b/src/core/explicit-session-backing-cleanup.ts @@ -0,0 +1,110 @@ +import type { BackendType } from '../adapters/backend/types.js'; +import type { RiffBackendConfig } from '../adapters/backend/riff-backend.js'; +import { cancelRiffTaskById } from '../adapters/backend/riff-backend.js'; +import { + isSuspendableBackendType, + killPersistentSession, + persistentSessionName, + probePersistentSession, +} from './persistent-backend.js'; + +export type ExplicitSessionBackingCleanupResult = + | { ok: true; kind: 'skipped_adopted' | 'no_backing' } + | { ok: true; kind: 'destroyed_persistent'; backendType: 'tmux' | 'herdr' | 'zellij'; name: string } + | { ok: true; kind: 'cancelled_riff'; taskId: string } + | { + ok: false; + kind: 'persistent_destroy_failed' | 'riff_config_missing' | 'riff_cancel_failed'; + backendType: BackendType; + taskId?: string; + error?: string; + }; + +export interface ExplicitSessionBackingCleanupInput { + sessionId: string; + backendType?: BackendType; + riffParentTaskId?: string; + /** Adopted panes/agents are user-owned. Explicit Botmux close only detaches + * its logical row and must never destroy the observed backing resource. */ + adopted?: boolean; + /** Current authoritative config for the row's bot. Required only when the + * row is frozen to Riff and still carries a remote task id. */ + riffConfig?: RiffBackendConfig; +} + +export interface ExplicitSessionBackingCleanupDeps { + cancelRiffTask?: typeof cancelRiffTaskById; + killPersistent?: typeof killPersistentSession; + probePersistent?: typeof probePersistentSession; +} + +/** + * Destroy the Botmux-owned backing resource before an offline/worker-less + * explicit close is published. + * + * This helper never mutates the session record. In particular, callers may + * erase `riffParentTaskId` only after `kind === 'cancelled_riff'`; a failed or + * unconfigurable cancellation therefore retains the durable retry handle. + */ +export async function cleanupExplicitSessionBacking( + input: ExplicitSessionBackingCleanupInput, + deps: ExplicitSessionBackingCleanupDeps = {}, +): Promise { + if (input.adopted) return { ok: true, kind: 'skipped_adopted' }; + + if (input.backendType === 'riff') { + const taskId = input.riffParentTaskId; + if (!taskId) return { ok: true, kind: 'no_backing' }; + if (!input.riffConfig?.baseUrl) { + return { ok: false, kind: 'riff_config_missing', backendType: 'riff', taskId }; + } + const cancel = deps.cancelRiffTask ?? cancelRiffTaskById; + try { + const cancelled = await cancel(input.riffConfig, taskId); + return cancelled + ? { ok: true, kind: 'cancelled_riff', taskId } + : { ok: false, kind: 'riff_cancel_failed', backendType: 'riff', taskId }; + } catch (err) { + return { + ok: false, + kind: 'riff_cancel_failed', + backendType: 'riff', + taskId, + error: err instanceof Error ? err.message : String(err), + }; + } + } + + if (!isSuspendableBackendType(input.backendType)) { + // Explicit pty and legacy unstamped rows have no deterministic persistent + // backing session to destroy. Never guess that an unstamped row was tmux. + return { ok: true, kind: 'no_backing' }; + } + + const name = persistentSessionName(input.backendType, input.sessionId); + const kill = deps.killPersistent ?? killPersistentSession; + const probe = deps.probePersistent ?? probePersistentSession; + try { + kill(input.backendType, name); + // Backend kill helpers are intentionally idempotent and historically + // swallow command errors. Offline explicit abandon needs a stronger + // contract: only publish closed after a post-kill probe confirms absence. + const after = probe(input.backendType, name); + if (after !== 'missing') { + return { + ok: false, + kind: 'persistent_destroy_failed', + backendType: input.backendType, + error: after === 'exists' ? 'backing_session_still_exists' : 'backing_session_state_unknown', + }; + } + return { ok: true, kind: 'destroyed_persistent', backendType: input.backendType, name }; + } catch (err) { + return { + ok: false, + kind: 'persistent_destroy_failed', + backendType: input.backendType, + error: err instanceof Error ? err.message : String(err), + }; + } +} diff --git a/src/core/idle-worker-sweeper.ts b/src/core/idle-worker-sweeper.ts index bbb44e679..43dac96de 100644 --- a/src/core/idle-worker-sweeper.ts +++ b/src/core/idle-worker-sweeper.ts @@ -1,6 +1,7 @@ import type { DaemonSession } from './types.js'; import { suspendWorker } from './worker-pool.js'; import { isSuspendableBackendType } from './persistent-backend.js'; +import { tryWithBotTurnMutation } from './bot-turn-mutation-gate.js'; /** * Default per-bot live-session cap applied when a bot has no explicit @@ -11,6 +12,7 @@ import { isSuspendableBackendType } from './persistent-backend.js'; * ('botDefaults.maxLiveWorkers*' i18n) — keep them in sync. */ export const DEFAULT_MAX_LIVE_WORKERS = 30; +export const IDLE_WORKER_SWEEP_MUTATION_ACQUIRE_TIMEOUT_MS = 1_000; export interface IdleWorkerSweepOptions { /** @@ -20,6 +22,11 @@ export interface IdleWorkerSweepOptions { * never suspend). */ maxLiveWorkers?: number; + /** + * Bound how long a detached sweep may wait for already-admitted turns. + * A wedged admission must not hold the bot-wide mutation gate forever. + */ + mutationAcquireTimeoutMs?: number; } export interface IdleWorkerSweepResult { @@ -80,3 +87,28 @@ export function sweepIdleWorkers( } return suspended; } + +/** + * Run a cap sweep only after every already-admitted inbound turn has either + * durably accepted its input or finished. A worker spawn can put the bot over + * cap while another handler is paused in sender/reaction setup, before that + * handler has changed its screen status or dispatch ledger. A synchronous + * sweep could otherwise mistake that handler's worker for idle, suspend it, + * and make the subsequent send fail before acceptance. + * + * Spawn/idle callbacks commonly run inside an admission, so the mutation gate + * upgrades the current lease when possible. Otherwise acquisition is bounded: + * a wedged admission skips this sweep instead of freezing every later turn for + * the bot. The next idle/spawn callback retries. + */ +export function sweepIdleWorkersAfterTurnDrain( + larkAppId: string, + activeSessions: Map, + opts: IdleWorkerSweepOptions = {}, +): Promise { + return tryWithBotTurnMutation( + larkAppId, + opts.mutationAcquireTimeoutMs ?? IDLE_WORKER_SWEEP_MUTATION_ACQUIRE_TIMEOUT_MS, + () => sweepIdleWorkers(activeSessions, opts), + ).then(result => result.acquired ? result.value : []); +} diff --git a/src/core/inflight-input-tracker.ts b/src/core/inflight-input-tracker.ts index 8eaa611c5..b578c3e56 100644 --- a/src/core/inflight-input-tracker.ts +++ b/src/core/inflight-input-tracker.ts @@ -26,7 +26,10 @@ export type InflightItem = { content: string; logicalContent?: string; turnId?: string; + replyTurnId?: string; dispatchAttempt?: number; + codexAppDispatchId?: string; + queuedActivationToken?: string; vcMeetingImTurnOrigin?: VcMeetingImTurnOrigin; codexAppInput?: CodexAppTurnInput; }; @@ -40,6 +43,14 @@ export class InflightInputTracker { this.unacked.push(item); } + /** Remove one exact item after a definitive local write failure before the + * caller re-queues it. Covers both the live in-flight set and a synchronous + * backend-exit handoff that may already have moved it to carryOver. */ + forget(item: InflightItem): void { + this.unacked = this.unacked.filter(candidate => candidate !== item); + this.carryOver = this.carryOver.filter(candidate => candidate !== item); + } + /** CLI is back at its idle prompt — everything written has been consumed * (answered, steered into the active turn, or drained from the TUI's own * type-ahead queue). Nothing is in flight anymore. */ diff --git a/src/core/managed-origin-attestation.ts b/src/core/managed-origin-attestation.ts new file mode 100644 index 000000000..e25711203 --- /dev/null +++ b/src/core/managed-origin-attestation.ts @@ -0,0 +1,268 @@ +import { randomBytes } from 'node:crypto'; +import { + closeSync, + constants as fsConstants, + fstatSync, + openSync, + readSync, + unlinkSync, + writeSync, +} from 'node:fs'; +import { + ensureManagedOriginAttestationDirectory, + managedOriginAttestationProofPath, +} from './managed-origin-capability.js'; + +export const MANAGED_ORIGIN_ATTEST_ROUTE = '/api/session-origin/attest'; +export const MANAGED_ORIGIN_PROOF_DOMAIN = 'botmux.managed-origin-attestation.v1'; +export const MANAGED_ORIGIN_PROOF_TTL_MS = 5_000; +const MAX_PROOF_BYTES = 8 * 1024; + +export interface ManagedOriginAttestationContext { + sessionId: string; + channelId: string; + capability: string; + dataDir: string; + /** Routing hints only; the daemon derives identity from its live registry. */ + larkAppId?: string; + ipcPortFallback?: number; +} + +export interface ManagedOriginAttestation { + sessionId: string; + turnId: string; + dispatchAttempt?: number; + requiresCodexAppLedger: boolean; +} + +export interface ManagedOriginAttestationProof extends ManagedOriginAttestation { + domain: typeof MANAGED_ORIGIN_PROOF_DOMAIN; + version: 1; + nonce: string; + channelId: string; + issuedAtMs: number; +} + +export class ManagedOriginAttestationError extends Error { + constructor(message: string, options?: ErrorOptions) { + super(message, options); + this.name = 'ManagedOriginAttestationError'; + } +} + +/** Daemon-side proof creation. The nonce is strict and the destination is + * derived only from the authenticated session + daemon data root. O_EXCL and + * O_NOFOLLOW make a pre-existing leaf fail closed rather than follow it. */ +export function writeManagedOriginAttestationProof(input: { + dataDir: string; + proof: ManagedOriginAttestationProof; +}): string { + const dir = ensureManagedOriginAttestationDirectory( + input.dataDir, + input.proof.sessionId, + input.proof.channelId, + ); + const path = managedOriginAttestationProofPath( + input.dataDir, + input.proof.sessionId, + input.proof.channelId, + input.proof.nonce, + ); + let fd: number | undefined; + let created = false; + try { + fd = openSync( + path, + fsConstants.O_WRONLY | fsConstants.O_CREAT | fsConstants.O_EXCL + | (fsConstants.O_NOFOLLOW ?? 0), + 0o600, + ); + created = true; + const stat = fstatSync(fd); + const expectedUid = process.getuid?.(); + if (!stat.isFile() || stat.isSymbolicLink() || stat.nlink !== 1 + || (stat.mode & 0o777) !== 0o600 + || (expectedUid !== undefined && stat.uid !== expectedUid)) { + throw new Error(`unsafe managed origin proof file under ${dir}`); + } + const body = Buffer.from(JSON.stringify(input.proof), 'utf8'); + if (body.length === 0 || body.length > MAX_PROOF_BYTES) { + throw new Error('managed origin proof exceeds size limit'); + } + let offset = 0; + while (offset < body.length) { + const written = writeSync(fd, body, offset, body.length - offset, null); + if (written <= 0) throw new Error('managed origin proof write made no progress'); + offset += written; + } + return path; + } catch (err) { + if (created) { + try { unlinkSync(path); } catch { /* best effort */ } + } + throw err; + } finally { + if (fd !== undefined) { + try { closeSync(fd); } catch { /* best effort */ } + } + } +} + +function readManagedOriginAttestationProof(path: string): unknown { + let fd: number | undefined; + try { + fd = openSync( + path, + fsConstants.O_RDONLY | (fsConstants.O_NOFOLLOW ?? 0) | fsConstants.O_NONBLOCK, + ); + const stat = fstatSync(fd); + const expectedUid = process.getuid?.(); + if (!stat.isFile() || stat.isSymbolicLink() || stat.nlink !== 1 + || (stat.mode & 0o777) !== 0o600 + || (expectedUid !== undefined && stat.uid !== expectedUid) + || stat.size <= 0 || stat.size > MAX_PROOF_BYTES) { + return undefined; + } + const body = Buffer.alloc(stat.size); + let offset = 0; + while (offset < body.length) { + const read = readSync(fd, body, offset, body.length - offset, null); + if (read <= 0) return undefined; + offset += read; + } + return JSON.parse(body.toString('utf8')) as unknown; + } catch { + return undefined; + } finally { + if (fd !== undefined) { + try { closeSync(fd); } catch { /* best effort */ } + } + } +} + +function validateProof(input: { + value: unknown; + context: ManagedOriginAttestationContext; + nonce: string; + nowMs: number; +}): ManagedOriginAttestation | undefined { + if (!input.value || typeof input.value !== 'object' || Array.isArray(input.value)) return undefined; + const proof = input.value as Record; + if (proof.domain !== MANAGED_ORIGIN_PROOF_DOMAIN || proof.version !== 1 + || proof.nonce !== input.nonce + || proof.sessionId !== input.context.sessionId + || proof.channelId !== input.context.channelId + || typeof proof.turnId !== 'string' || proof.turnId.length === 0 || proof.turnId.length > 256 + || typeof proof.issuedAtMs !== 'number' || !Number.isFinite(proof.issuedAtMs) + || proof.issuedAtMs > input.nowMs + 1_000 + || input.nowMs - proof.issuedAtMs > MANAGED_ORIGIN_PROOF_TTL_MS + || typeof proof.requiresCodexAppLedger !== 'boolean') { + return undefined; + } + const dispatchAttempt = proof.dispatchAttempt === undefined + ? undefined + : typeof proof.dispatchAttempt === 'number' + && Number.isSafeInteger(proof.dispatchAttempt) + && proof.dispatchAttempt > 0 + ? proof.dispatchAttempt + : null; + if (dispatchAttempt === null) return undefined; + return { + sessionId: input.context.sessionId, + turnId: proof.turnId, + ...(dispatchAttempt !== undefined ? { dispatchAttempt } : {}), + requiresCodexAppLedger: proof.requiresCodexAppLedger, + }; +} + +/** + * Exchange a rotating capability for a host-file proof of the daemon's exact + * live tuple. HTTP is transport only: a stale child can bind a released port + * and forge a response, but Seatbelt prevents it from creating the random + * nonce proof in the host-owned read-only directory. + */ +export async function attestManagedOrigin(input: { + context: ManagedOriginAttestationContext; + resolveIpcPort?: (larkAppId: string | undefined) => number | undefined; + fetchImpl?: typeof fetch; + timeoutMs?: number; + nonce?: string; + now?: () => number; + wait?: (delayMs: number) => Promise; +}): Promise { + const discovered = input.resolveIpcPort?.(input.context.larkAppId); + // The capability leaf is host-written and read-only in Seatbelt. Daemon + // discovery descriptors remain ordinary filesystem data that the confined + // child may poison, so they are only a compatibility fallback when an older + // capability lacks the owning port. + const port = input.context.ipcPortFallback ?? discovered; + if (!Number.isSafeInteger(port) || !port || port <= 0) { + throw new ManagedOriginAttestationError( + '找不到目标 daemon 端口；无法验证当前 worker 的 managed origin', + ); + } + const nonce = input.nonce ?? randomBytes(32).toString('hex'); + if (!/^[a-f0-9]{64}$/.test(nonce)) { + throw new ManagedOriginAttestationError('managed origin challenge nonce 无效'); + } + const proofPath = managedOriginAttestationProofPath( + input.context.dataDir, + input.context.sessionId, + input.context.channelId, + nonce, + ); + const timeoutMs = input.timeoutMs ?? 3_000; + const now = input.now ?? Date.now; + const wait = input.wait ?? (delayMs => new Promise(resolve => setTimeout(resolve, delayMs))); + const deadline = now() + timeoutMs; + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), timeoutMs); + timer.unref?.(); + let response: Response; + try { + response = await (input.fetchImpl ?? fetch)( + `http://127.0.0.1:${port}${MANAGED_ORIGIN_ATTEST_ROUTE}`, + { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ + sessionId: input.context.sessionId, + channelId: input.context.channelId, + originCapability: input.context.capability, + nonce, + }), + signal: controller.signal, + }, + ); + } catch (err) { + throw new ManagedOriginAttestationError( + controller.signal.aborted + ? 'daemon managed-origin 验证超时' + : `无法连接 owning daemon 验证 managed origin: ${err instanceof Error ? err.message : String(err)}`, + { cause: err }, + ); + } finally { + clearTimeout(timer); + } + if (!response.ok) { + try { await response.body?.cancel(); } catch { /* transport body is untrusted */ } + throw new ManagedOriginAttestationError(`daemon 拒绝 managed origin: HTTP ${response.status}`); + } + // Ignore the response body completely. Only a valid protected proof is + // authority, and a fake loopback listener cannot write one. + try { await response.body?.cancel(); } catch { /* authority never comes from HTTP */ } + for (;;) { + const nowMs = now(); + const proof = validateProof({ + value: readManagedOriginAttestationProof(proofPath), + context: input.context, + nonce, + nowMs, + }); + if (proof) return proof; + if (nowMs >= deadline) { + throw new ManagedOriginAttestationError('daemon 未生成有效的 managed-origin host proof'); + } + await wait(Math.min(20, Math.max(1, deadline - nowMs))); + } +} diff --git a/src/core/managed-origin-capability.ts b/src/core/managed-origin-capability.ts index 921dcb79a..a4d78ff24 100644 --- a/src/core/managed-origin-capability.ts +++ b/src/core/managed-origin-capability.ts @@ -1,16 +1,39 @@ import { createHash, randomBytes } from 'node:crypto'; import { - lstatSync, mkdirSync, readFileSync, renameSync, unlinkSync, writeFileSync, + chmodSync, closeSync, constants as fsConstants, fstatSync, lstatSync, mkdirSync, + openSync, opendirSync, readSync, realpathSync, renameSync, rmSync, unlinkSync, + writeFileSync, } from 'node:fs'; -import { dirname, join } from 'node:path'; +import { basename, dirname, join } from 'node:path'; export const RELAY_ORIGIN_CAPABILITY_BASENAME = '.botmux-origin-capability.json'; +export const MANAGED_ORIGIN_ISOLATION_MARKER_BASENAME = '.botmux-read-isolated-v1'; +export const MANAGED_ORIGIN_ISOLATION_SENTINEL_BASENAME = '.botmux-read-isolation-sentinel-v1'; +const MANAGED_ORIGIN_STALE_PROOF_MS = 60_000; export interface ManagedOriginCapabilityClaim { sessionId: string; + channelId?: string; capability: string; turnId?: string; dispatchAttempt?: number; + /** Current daemon port, host-written on every capability rotation. */ + ipcPort?: number; +} + +function assertManagedOriginChannelId(channelId: string): string { + if (!/^[a-f0-9]{64}$/.test(channelId)) { + throw new Error('invalid managed origin channel id'); + } + return channelId; +} + +function managedOriginChannelDigest(sessionId: string, channelId: string): string { + return createHash('sha256') + .update(sessionId) + .update('\0') + .update(assertManagedOriginChannelId(channelId)) + .digest('hex'); } /** @@ -20,11 +43,431 @@ export interface ManagedOriginCapabilityClaim { * The parent is denied wholesale by the Seatbelt profile; only this exact file * is carved back in for the owning session. */ -export function managedOriginCapabilityPath(dataDir: string, sessionId: string): string { - const digest = createHash('sha256').update(sessionId).digest('hex'); +export function managedOriginCapabilityPath( + dataDir: string, + sessionId: string, + channelId: string, +): string { + const digest = managedOriginChannelDigest(sessionId, channelId); return join(dataDir, 'read-isolation', `origin-${digest}.json`); } +/** Host-written, child-read-only proof directory for live daemon attestation. */ +export function managedOriginAttestationDirectory( + dataDir: string, + sessionId: string, + channelId: string, +): string { + const digest = managedOriginChannelDigest(sessionId, channelId); + return join(dataDir, 'read-isolation', `attest-${digest}`); +} + +export function managedOriginAttestationProofPath( + dataDir: string, + sessionId: string, + channelId: string, + nonce: string, +): string { + if (!/^[a-f0-9]{64}$/.test(nonce)) throw new Error('invalid managed origin attestation nonce'); + return join(managedOriginAttestationDirectory(dataDir, sessionId, channelId), `${nonce}.json`); +} + +export function managedOriginIsolationMarkerPath( + dataDir: string, + sessionId: string, + channelId: string, +): string { + return join( + managedOriginAttestationDirectory(dataDir, sessionId, channelId), + MANAGED_ORIGIN_ISOLATION_MARKER_BASENAME, + ); +} + +export function managedOriginIsolationSentinelPath(homeDir: string): string { + return join(homeDir, MANAGED_ORIGIN_ISOLATION_SENTINEL_BASENAME); +} + +export function ensureManagedOriginIsolationSentinel(homeDir: string): string { + const path = managedOriginIsolationSentinelPath(homeDir); + replaceManagedOriginCapabilityFile(path, JSON.stringify({ + domain: 'botmux.read-isolation-sentinel.v1', + })); + chmodSync(path, 0o600); + return path; +} + +/** Fixed-OS-home locator whose basename is covered by the legacy + * `.dashboard-secret(?:\.|$)` read/write deny. New profiles carve only the + * exact current-session leaf back in for reads while the final write deny + * remains in force. This gives cmdSend a data-root binding independent of + * mutable HOME/SESSION_DATA_DIR. */ +export function managedOriginRootLocatorPath( + osUserHomeDir: string, + sessionId: string, +): string { + const digest = createHash('sha256').update(sessionId).digest('hex'); + return join( + osUserHomeDir, + '.botmux', + `.dashboard-secret.origin-root-${digest}.json`, + ); +} + +export interface ManagedOriginRootLocator { + sessionId: string; + dataDir: string; +} + +/** Probe placed inside the locator-selected data root's legacy-denied + * `read-isolation/` namespace, but outside every current exact carve-out. The + * exact directory location matters: a forged sibling dataDir must not inherit + * a parent-level dashboard-secret deny and masquerade as the real root. */ +export function managedOriginDataRootProbePath( + dataDir: string, + sessionId: string, +): string { + if (!dataDir.startsWith('/')) throw new Error('managed origin data root must be absolute'); + const digest = createHash('sha256') + .update(sessionId) + .update('\0') + .update(dataDir) + .digest('hex'); + return join(dataDir, 'read-isolation', `.origin-root-probe-${digest}.json`); +} + +export function ensureManagedOriginDataRootProbe( + dataDir: string, + sessionId: string, +): string { + const canonicalDataDir = realpathSync(dataDir); + const path = managedOriginDataRootProbePath(canonicalDataDir, sessionId); + replaceManagedOriginCapabilityFile(path, JSON.stringify({ + domain: 'botmux.managed-origin-root-probe.v1', + sessionId, + dataDir: canonicalDataDir, + })); + chmodSync(path, 0o600); + return path; +} + +export function managedOriginDataRootProbeAccess( + dataDir: string, + sessionId: string, +): 'host_accessible' | 'sandbox_denied' | 'missing_or_unsafe' { + const path = managedOriginDataRootProbePath(dataDir, sessionId); + let fd: number | undefined; + try { + fd = openSync( + path, + fsConstants.O_RDONLY | (fsConstants.O_NOFOLLOW ?? 0) | fsConstants.O_NONBLOCK, + ); + const stat = fstatSync(fd); + const expectedUid = process.getuid?.(); + if (!stat.isFile() || stat.nlink !== 1 || stat.size <= 0 || stat.size > 8 * 1024 + || (stat.mode & 0o777) !== 0o600 + || (expectedUid !== undefined && stat.uid !== expectedUid)) return 'missing_or_unsafe'; + const body = Buffer.alloc(stat.size); + let offset = 0; + while (offset < body.length) { + const read = readSync(fd, body, offset, body.length - offset, null); + if (read <= 0) return 'missing_or_unsafe'; + offset += read; + } + const parsed = JSON.parse(body.toString('utf8')) as Record; + return parsed.domain === 'botmux.managed-origin-root-probe.v1' + && parsed.sessionId === sessionId + && parsed.dataDir === dataDir + ? 'host_accessible' + : 'missing_or_unsafe'; + } catch (err: any) { + // macOS Seatbelt returns EPERM for the profile deny. Do not accept EACCES: + // a confined child can manufacture that with chmod(000) in a forged root. + return err?.code === 'EPERM' ? 'sandbox_denied' : 'missing_or_unsafe'; + } finally { + if (fd !== undefined) { + try { closeSync(fd); } catch { /* best effort */ } + } + } +} + +export function ensureManagedOriginRootLocator( + osUserHomeDir: string, + sessionId: string, + dataDir: string, +): string { + if (!dataDir.startsWith('/')) throw new Error('managed origin data root must be absolute'); + const canonicalDataDir = realpathSync(dataDir); + const lexicalPath = managedOriginRootLocatorPath(osUserHomeDir, sessionId); + const lexicalParent = dirname(lexicalPath); + mkdirSync(lexicalParent, { recursive: true, mode: 0o700 }); + // Existing installations may symlink ~/.botmux. Resolve the trusted host + // parent once, validate the target directory, and write the leaf there; the + // CLI's lexical fixed-home path resolves to the same inode under Seatbelt. + const parent = realpathSync(lexicalParent); + const path = join(parent, basename(lexicalPath)); + const stat = lstatSync(parent); + const expectedUid = process.getuid?.(); + if (!stat.isDirectory() + || (expectedUid !== undefined && stat.uid !== expectedUid)) { + throw new Error(`managed origin locator parent is unsafe: ${parent}`); + } + replaceManagedOriginCapabilityFile(path, JSON.stringify({ + domain: 'botmux.managed-origin-root.v1', + sessionId, + dataDir: canonicalDataDir, + })); + chmodSync(path, 0o600); + return path; +} + +export function readManagedOriginRootLocator( + osUserHomeDir: string, + sessionId: string, +): ManagedOriginRootLocator | null { + const body = readManagedOriginAuthorityFile( + managedOriginRootLocatorPath(osUserHomeDir, sessionId), + 8 * 1024, + ); + if (!body) return null; + try { + const parsed = JSON.parse(body) as Record; + if (parsed.domain !== 'botmux.managed-origin-root.v1' + || parsed.sessionId !== sessionId + || typeof parsed.dataDir !== 'string' + || !parsed.dataDir.startsWith('/') + || parsed.dataDir.length > 4096 + || parsed.dataDir.includes('\0')) return null; + return { sessionId, dataDir: parsed.dataDir }; + } catch { + return null; + } +} + +/** Kernel-observable macOS isolation classifier. The Seatbelt profile denies + * this fixed, host-readable sentinel independent of env/argv/session ids. */ +export function managedOriginIsolationSentinelAccess( + homeDir: string, +): 'host_accessible' | 'sandbox_denied' | 'missing_or_unsafe' { + const path = managedOriginIsolationSentinelPath(homeDir); + let fd: number | undefined; + try { + fd = openSync( + path, + fsConstants.O_RDONLY | (fsConstants.O_NOFOLLOW ?? 0) | fsConstants.O_NONBLOCK, + ); + const stat = fstatSync(fd); + const expectedUid = process.getuid?.(); + if (!stat.isFile() || stat.nlink !== 1 || stat.size <= 0 || stat.size > 1024 + || (stat.mode & 0o777) !== 0o600 + || (expectedUid !== undefined && stat.uid !== expectedUid)) return 'missing_or_unsafe'; + const body = Buffer.alloc(stat.size); + let offset = 0; + while (offset < body.length) { + const read = readSync(fd, body, offset, body.length - offset, null); + if (read <= 0) return 'missing_or_unsafe'; + offset += read; + } + try { + const value = JSON.parse(body.toString('utf8')) as { domain?: unknown }; + return value.domain === 'botmux.read-isolation-sentinel.v1' + ? 'host_accessible' + : 'missing_or_unsafe'; + } catch { + return 'missing_or_unsafe'; + } + } catch (err: any) { + return err?.code === 'EACCES' || err?.code === 'EPERM' + ? 'sandbox_denied' + : 'missing_or_unsafe'; + } finally { + if (fd !== undefined) { + try { closeSync(fd); } catch { /* best effort */ } + } + } +} + +/** Kernel probe backed by an inode that legacy read-isolation profiles already + * denied: the dashboard HMAC secret. Only metadata is inspected; secret bytes + * are never read or returned. This closes the upgrade race where a stale + * pre-sentinel sandbox could replace a newly introduced probe path. */ +export function managedOriginLegacyIsolationProbeAccess( + osUserHomeDir: string, +): 'host_accessible' | 'sandbox_denied' | 'missing_or_unsafe' { + const path = join(osUserHomeDir, '.botmux', '.dashboard-secret'); + let fd: number | undefined; + try { + fd = openSync( + path, + fsConstants.O_RDONLY | (fsConstants.O_NOFOLLOW ?? 0) | fsConstants.O_NONBLOCK, + ); + const stat = fstatSync(fd); + const expectedUid = process.getuid?.(); + return stat.isFile() && stat.nlink === 1 + && (expectedUid === undefined || stat.uid === expectedUid) + ? 'host_accessible' + : 'missing_or_unsafe'; + } catch (err: any) { + return err?.code === 'EACCES' || err?.code === 'EPERM' + ? 'sandbox_denied' + : 'missing_or_unsafe'; + } finally { + if (fd !== undefined) { + try { closeSync(fd); } catch { /* best effort */ } + } + } +} + +/** Strict bounded reader for host-owned authority metadata. It never follows a + * leaf symlink and opens FIFOs/devices nonblocking before rejecting them by + * inode type, ownership, link count, mode, and size. */ +export function readManagedOriginAuthorityFile( + filePath: string, + maxBytes = 8 * 1024, +): string | null { + let fd: number | undefined; + try { + fd = openSync( + filePath, + fsConstants.O_RDONLY | (fsConstants.O_NOFOLLOW ?? 0) | fsConstants.O_NONBLOCK, + ); + const stat = fstatSync(fd); + const expectedUid = process.getuid?.(); + if (!stat.isFile() || stat.nlink !== 1 || stat.size <= 0 || stat.size > maxBytes + || (stat.mode & 0o777) !== 0o600 + || (expectedUid !== undefined && stat.uid !== expectedUid)) return null; + const body = Buffer.alloc(stat.size); + let offset = 0; + while (offset < body.length) { + const read = readSync(fd, body, offset, body.length - offset, null); + if (read <= 0) return null; + offset += read; + } + return body.toString('utf8'); + } catch { + return null; + } finally { + if (fd !== undefined) { + try { closeSync(fd); } catch { /* best effort */ } + } + } +} + +/** A durable, host-owned hint that this session was launched read-isolated. + * It is deliberately separate from the rotating capability so missing, + * corrupt, or revoked authority cannot silently downgrade `botmux send` to + * the ordinary direct path. The marker is only a fail-closed classification + * hint; the live daemon challenge remains the sole send authority. */ +export function hasManagedOriginIsolationMarker( + dataDir: string, + sessionId: string, + channelId: string, +): boolean { + const body = readManagedOriginAuthorityFile( + managedOriginIsolationMarkerPath(dataDir, sessionId, channelId), + 1024, + ); + if (!body) return false; + try { + const parsed = JSON.parse(body) as { + domain?: unknown; + sessionId?: unknown; + channelId?: unknown; + }; + return parsed.domain === 'botmux.read-isolation-origin.v1' + && parsed.sessionId === sessionId + && parsed.channelId === channelId; + } catch { + return false; + } +} + +/** Prepare the proof directory before Seatbelt canonicalizes its read carve. */ +export function ensureManagedOriginAttestationDirectory( + dataDir: string, + sessionId: string, + channelId: string, +): string { + const proofDir = managedOriginAttestationDirectory(dataDir, sessionId, channelId); + const parent = dirname(proofDir); + mkdirSync(parent, { recursive: true, mode: 0o700 }); + const expectedUid = process.getuid?.(); + const parentStat = lstatSync(parent); + if (!parentStat.isDirectory() || parentStat.isSymbolicLink()) { + throw new Error(`managed origin attestation parent is not a real directory: ${parent}`); + } + if (expectedUid !== undefined && parentStat.uid !== expectedUid) { + throw new Error(`managed origin attestation parent has the wrong owner: ${parent}`); + } + chmodSync(parent, 0o700); + try { + const existing = lstatSync(proofDir); + if (existing.isSymbolicLink() || !existing.isDirectory()) { + rmSync(proofDir, { recursive: true, force: true }); + } else if (expectedUid !== undefined && existing.uid !== expectedUid) { + throw new Error(`managed origin attestation directory has the wrong owner: ${proofDir}`); + } + } catch (err: any) { + if (err?.code !== 'ENOENT') throw err; + } + mkdirSync(proofDir, { recursive: true, mode: 0o700 }); + chmodSync(proofDir, 0o700); + const finalStat = lstatSync(proofDir); + if (!finalStat.isDirectory() || finalStat.isSymbolicLink() + || (expectedUid !== undefined && finalStat.uid !== expectedUid) + || (finalStat.mode & 0o777) !== 0o700) { + throw new Error(`managed origin attestation directory is unsafe: ${proofDir}`); + } + const markerPath = managedOriginIsolationMarkerPath(dataDir, sessionId, channelId); + const markerBody = JSON.stringify({ + domain: 'botmux.read-isolation-origin.v1', + sessionId, + channelId, + }); + if (!hasManagedOriginIsolationMarker(dataDir, sessionId, channelId)) { + replaceManagedOriginCapabilityFile(markerPath, markerBody); + chmodSync(markerPath, 0o600); + } + return proofDir; +} + +/** Bounded owner-startup cleanup. Never call this from the unauthenticated + * request path: a prefilled legacy directory must not turn every challenge + * into an unbounded synchronous daemon scan. */ +export function sweepManagedOriginAttestationProofs( + dataDir: string, + sessionId: string, + channelId: string, + maxEntries = 512, +): void { + const proofDir = ensureManagedOriginAttestationDirectory(dataDir, sessionId, channelId); + const expectedUid = process.getuid?.(); + const now = Date.now(); + const dir = opendirSync(proofDir); + let seen = 0; + try { + for (;;) { + const entry = dir.readSync(); + if (!entry) break; + if (++seen > maxEntries) { + throw new Error(`managed origin attestation directory exceeds ${maxEntries} entries`); + } + if (!/^[a-f0-9]{64}\.json$/.test(entry.name)) continue; + const candidate = join(proofDir, entry.name); + try { + const stat = lstatSync(candidate); + if (stat.isDirectory()) continue; + if (stat.isSymbolicLink() || !stat.isFile() + || (expectedUid !== undefined && stat.uid !== expectedUid) + || now - stat.mtimeMs > MANAGED_ORIGIN_STALE_PROOF_MS) { + unlinkSync(candidate); + } + } catch { /* raced cleanup or inaccessible leaf: fail closed per nonce */ } + } + } finally { + try { dir.closeSync(); } catch { /* best effort */ } + } +} + /** * Atomically replace a capability file without following an attacker-planted * destination symlink. The generic atomic writer intentionally follows @@ -50,6 +493,25 @@ export function replaceManagedOriginCapabilityFile(filePath: string, body: strin } } +/** Prepare a stable capability pathname without ever overwriting a successor + * generation's regular file. Unsafe leaf types are removed by directory-entry + * unlink (no follow); a regular leaf is left untouched for the daemon-owned + * current-worker publication path. */ +export function ensureManagedOriginCapabilityLeafSafe(filePath: string): void { + const parent = dirname(filePath); + mkdirSync(parent, { recursive: true, mode: 0o700 }); + const parentStat = lstatSync(parent); + if (!parentStat.isDirectory() || parentStat.isSymbolicLink()) { + throw new Error(`managed origin capability parent is not a real directory: ${parent}`); + } + try { + const leaf = lstatSync(filePath); + if (leaf.isSymbolicLink() || !leaf.isFile()) unlinkSync(filePath); + } catch (err: any) { + if (err?.code !== 'ENOENT') throw err; + } +} + /** * Read the current origin claim from the per-session sandbox relay (Linux) or * the exact Seatbelt carve-out (macOS). A file is only transport: the daemon @@ -60,21 +522,28 @@ export function readManagedOriginCapability( dataDir: string, sessionId: string | undefined, relayDir?: string, + channelId?: string, ): ManagedOriginCapabilityClaim | null { if (!sessionId) return null; const relay = !!relayDir; + if (!relay && !channelId) return null; const path = relay ? join(relayDir!, RELAY_ORIGIN_CAPABILITY_BASENAME) - : managedOriginCapabilityPath(dataDir, sessionId); + : managedOriginCapabilityPath(dataDir, sessionId, channelId!); try { - const parsed = JSON.parse(readFileSync(path, 'utf8')) as { + const body = readManagedOriginAuthorityFile(path, 8 * 1024); + if (!body) return null; + const parsed = JSON.parse(body) as { sessionId?: unknown; token?: unknown; capability?: unknown; turnId?: unknown; dispatchAttempt?: unknown; + ipcPort?: unknown; + channelId?: unknown; }; if (!relay && parsed.sessionId !== sessionId) return null; + if (!relay && parsed.channelId !== channelId) return null; const capability = typeof parsed.capability === 'string' ? parsed.capability : parsed.token; @@ -91,11 +560,18 @@ export function readManagedOriginCapability( && parsed.dispatchAttempt > 0 ? parsed.dispatchAttempt : undefined; + const ipcPort = typeof parsed.ipcPort === 'number' + && Number.isSafeInteger(parsed.ipcPort) + && parsed.ipcPort > 0 && parsed.ipcPort <= 65_535 + ? parsed.ipcPort + : undefined; return { sessionId, + ...(!relay && channelId ? { channelId } : {}), capability, ...(turnId ? { turnId } : {}), ...(dispatchAttempt !== undefined ? { dispatchAttempt } : {}), + ...(ipcPort !== undefined ? { ipcPort } : {}), }; } catch { return null; @@ -113,8 +589,9 @@ export function hasMatchingManagedOriginCapability( sessionId: string | undefined, expectedCapability: string | undefined, relayDir?: string, + channelId?: string, ): boolean { if (!expectedCapability) return false; - return readManagedOriginCapability(dataDir, sessionId, relayDir)?.capability + return readManagedOriginCapability(dataDir, sessionId, relayDir, channelId)?.capability === expectedCapability; } diff --git a/src/core/pending-repo-journal.ts b/src/core/pending-repo-journal.ts new file mode 100644 index 000000000..740f210c2 --- /dev/null +++ b/src/core/pending-repo-journal.ts @@ -0,0 +1,88 @@ +import type { DaemonSession } from './types.js'; +import type { PendingRepoSetup } from '../types.js'; +import * as sessionStore from '../services/session-store.js'; + +export function stagePendingRepoSetup( + ds: DaemonSession, + args: Pick & Partial>, +): void { + const prior = { + queued: ds.session.queued, + queuedPrompt: ds.session.queuedPrompt, + queuedCodexAppText: ds.session.queuedCodexAppText, + queuedCodexAppMessageContext: ds.session.queuedCodexAppMessageContext, + pendingRepoSetup: ds.session.pendingRepoSetup, + }; + const setup: PendingRepoSetup = { + mode: args.mode, + prompt: ds.pendingPrompt ?? '', + ...(ds.pendingRawInput ? { rawInput: ds.pendingRawInput } : {}), + ...(args.turnId ? { turnId: args.turnId } : {}), + ...(args.baseDir ? { baseDir: args.baseDir } : {}), + ...(ds.pendingCodexAppText !== undefined ? { codexAppText: ds.pendingCodexAppText } : {}), + ...(ds.pendingCodexAppApplicationContext + ? { codexAppApplicationContext: ds.pendingCodexAppApplicationContext } + : {}), + ...(ds.pendingCodexAppMessageContext + ? { codexAppMessageContext: ds.pendingCodexAppMessageContext } + : {}), + ...(ds.pendingAttachments?.length + ? { attachments: ds.pendingAttachments.map(attachment => ({ ...attachment })) } + : {}), + ...(ds.pendingMentions?.length + ? { mentions: ds.pendingMentions.map(mention => ({ ...mention })) } + : {}), + ...(ds.pendingSubstituteTrigger + ? { substituteTrigger: structuredClone(ds.pendingSubstituteTrigger) } + : {}), + ...(ds.pendingSender ? { sender: { ...ds.pendingSender } } : {}), + }; + ds.session.queued = true; + ds.session.queuedPrompt = setup.prompt; + ds.session.queuedCodexAppText = setup.codexAppText; + ds.session.queuedCodexAppMessageContext = setup.codexAppMessageContext; + ds.session.pendingRepoSetup = setup; + try { + sessionStore.updateSession(ds.session); + } catch (err) { + ds.session.queued = prior.queued; + ds.session.queuedPrompt = prior.queuedPrompt; + ds.session.queuedCodexAppText = prior.queuedCodexAppText; + ds.session.queuedCodexAppMessageContext = prior.queuedCodexAppMessageContext; + ds.session.pendingRepoSetup = prior.pendingRepoSetup; + throw err; + } +} + +export function persistPendingRepoCardMessageId(ds: DaemonSession, messageId: string): void { + const setup = ds.session.pendingRepoSetup; + if (!setup) return; + const prior = setup.repoCardMessageId; + setup.repoCardMessageId = messageId; + try { + sessionStore.updateSession(ds.session); + } catch (err) { + setup.repoCardMessageId = prior; + throw err; + } +} + +export function restorePendingRepoRuntime(ds: DaemonSession): boolean { + const setup = ds.session.pendingRepoSetup; + if (!setup || ds.session.queuedActivationPending) return false; + ds.pendingRepo = true; + ds.pendingPrompt = setup.prompt; + ds.pendingRawInput = setup.rawInput; + ds.pendingCodexAppText = setup.codexAppText; + ds.pendingCodexAppApplicationContext = setup.codexAppApplicationContext; + ds.pendingCodexAppMessageContext = setup.codexAppMessageContext; + ds.pendingAttachments = setup.attachments?.map(attachment => ({ ...attachment })); + ds.pendingMentions = setup.mentions?.map(mention => ({ ...mention })); + ds.pendingSubstituteTrigger = setup.substituteTrigger + ? structuredClone(setup.substituteTrigger) + : undefined; + ds.pendingSender = setup.sender ? { ...setup.sender } : undefined; + ds.repoCardMessageId = setup.repoCardMessageId; + ds.initialStartPending = false; + return true; +} diff --git a/src/core/persistent-backend.ts b/src/core/persistent-backend.ts index cb4eb3975..aea4acb0a 100644 --- a/src/core/persistent-backend.ts +++ b/src/core/persistent-backend.ts @@ -112,18 +112,20 @@ export function resolvePairedSpawnBackendType( /** * How a session's worker is torn down at daemon shutdown, branched on the * session's FROZEN backend (via getSessionPersistentBackendType), NOT live config: + * 'riff-drain-detach' — Riff only: three-phase non-cancelling write drain, + * durable lineage ACK, then worker exit. * 'detach' — persistent backend (tmux/herdr/zellij): SIGTERM the worker only, * leaving the multiplexer session alive for re-attach. * 'close' — non-persistent (frozen pty, or unresolvable legacy): killWorker. * Freezing here stops a live backendType edit from changing how a running session * tears down — e.g. detach-preserving a "herdr" session whose real pane is tmux. */ -export function shutdownBackendDisposition(ds: DaemonSession): 'detach' | 'close' { - // riff：远端任务独立于本地进程存活。daemon shutdown 走 'close' 会经 worker 的 - // destroySession() 取消远端任务——重启不该杀任务（血缘已持久化，重启后 - // follow-up 续上，agent 的 botmux send 照常送达）。detach = 仅 SIGTERM worker。 +export function shutdownBackendDisposition(ds: DaemonSession): 'riff-drain-detach' | 'detach' | 'close' { + // Riff 不能落进普通 detach 的直接 SIGTERM：create/follow-up 最长 10s 才返回 + // task id，而 worker SIGTERM 会立即 exit，丢掉唯一血缘。独立 disposition 迫使 + // daemon 先走 drain → durable ACK → commit 协议；类型检查防止未来回归。 const frozen = ds.initConfig?.backendType ?? ds.session.backendType; - if (frozen === 'riff') return 'detach'; + if (frozen === 'riff') return 'riff-drain-detach'; return getSessionPersistentBackendType(ds) ? 'detach' : 'close'; } diff --git a/src/core/process-start-identity.ts b/src/core/process-start-identity.ts new file mode 100644 index 000000000..161080755 --- /dev/null +++ b/src/core/process-start-identity.ts @@ -0,0 +1,52 @@ +import { execFileSync } from 'node:child_process'; +import { existsSync, readFileSync } from 'node:fs'; + +function systemPsBin(): string | undefined { + for (const candidate of ['/usr/bin/ps', '/bin/ps']) { + if (existsSync(candidate)) return candidate; + } + return undefined; +} + +/** Stable process-birth identity used by supervisor mutation authority. */ +export function readSupervisorProcessStartIdentity(pid: number): string | undefined { + if (!Number.isSafeInteger(pid) || pid <= 1) return undefined; + if (process.platform === 'linux') { + try { + const raw = readFileSync(`/proc/${pid}/stat`, 'utf8'); + const closeParen = raw.lastIndexOf(')'); + if (closeParen >= 0) { + const fields = raw.slice(closeParen + 2).trim().split(/\s+/); + if (fields[19]) return fields[19]; + } + } catch { /* exited or unreadable */ } + return undefined; + } + if (process.platform === 'win32') { + try { + const value = execFileSync('powershell.exe', [ + '-NoProfile', + '-NonInteractive', + '-Command', + `$p = Get-CimInstance Win32_Process -Filter \"ProcessId = ${pid}\"; ` + + 'if ($p) { $p.CreationDate.ToUniversalTime().Ticks }', + ], { + encoding: 'utf8', + timeout: 2_000, + stdio: ['ignore', 'pipe', 'ignore'], + }).trim(); + return value || undefined; + } catch { return undefined; } + } + const ps = systemPsBin(); + if (!ps) return undefined; + try { + const value = execFileSync(ps, ['-o', 'lstart=', '-p', String(pid)], { + encoding: 'utf8', + timeout: 2_000, + stdio: ['ignore', 'pipe', 'ignore'], + env: { PATH: '/usr/bin:/bin', LANG: 'C' }, + }).trim(); + return value || undefined; + } catch { return undefined; } +} diff --git a/src/core/raw-command-writer.ts b/src/core/raw-command-writer.ts new file mode 100644 index 000000000..a48c95e38 --- /dev/null +++ b/src/core/raw-command-writer.ts @@ -0,0 +1,75 @@ +/** Minimal write surface shared by PTY/session backends for literal slash + * commands. Historical backends return void on success; guarded backends + * return false when the target pane/process is already unavailable. */ +export interface RawCommandWriter { + write(data: string): void | boolean; + sendText?: (text: string) => void | boolean; + sendSpecialKeys?: (...keys: string[]) => void | boolean; +} + +export interface RawCommandWriteOptions { + coco?: boolean; + cocoThrottleMs?: number; + submitBeatMs?: number; + delay?: (ms: number) => Promise; +} + +/** Type one literal input line and report whether both the text and submit + * writes were accepted by the backend. `undefined` remains legacy success; + * explicit false is a proven drop and must never produce an activation ACK. */ +export async function writeRawCommandLine( + backend: RawCommandWriter, + content: string, + opts: RawCommandWriteOptions = {}, +): Promise { + const delay = opts.delay ?? (ms => new Promise(resolve => setTimeout(resolve, ms))); + const beatMs = opts.submitBeatMs ?? 200; + const sendText = backend.sendText?.bind(backend); + const sendSpecialKeys = backend.sendSpecialKeys?.bind(backend); + + if (sendText && sendSpecialKeys) { + if (opts.coco) { + const cmd = content.trim(); + const typed = cmd.includes(' ') ? cmd : `${cmd} `; + for (const ch of typed) { + if (sendText(ch) === false) return false; + await delay(opts.cocoThrottleMs ?? 40); + } + await delay(beatMs); + return sendSpecialKeys('Enter') !== false; + } + if (sendText(content) === false) return false; + await delay(beatMs); + return sendSpecialKeys('Enter') !== false; + } + + if (backend.write(content) === false) return false; + await delay(beatMs); + return backend.write('\r') !== false; +} + +export interface RawCommandDeliveryFinalizer { + accepted: boolean; + durableActivation: boolean; + acknowledgeActivation: boolean; + hasFollowUp: boolean; + onAccepted: () => void; + onFollowUp: () => void; + onActivationAck: () => void; + onDurableFailure: () => void; +} + +/** Apply the raw-input side effects at the acceptance boundary. A rejected + * text/Enter write can neither enqueue the follower nor ACK the durable head; + * it must retire the worker generation so daemon exit recovery keeps the exact + * journal routable. */ +export function finalizeRawCommandDelivery(args: RawCommandDeliveryFinalizer): boolean { + if (!args.accepted) { + if (args.durableActivation) args.onDurableFailure(); + return false; + } + args.onAccepted(); + if (args.hasFollowUp) args.onFollowUp(); + if (args.acknowledgeActivation) args.onActivationAck(); + return true; +} diff --git a/src/core/reply-target.ts b/src/core/reply-target.ts index c7051263f..2535d1d11 100644 --- a/src/core/reply-target.ts +++ b/src/core/reply-target.ts @@ -1,5 +1,5 @@ import type { DaemonSession } from './types.js'; -import type { Session } from '../types.js'; +import type { FrozenSessionReplyContext, Session } from '../types.js'; export type SessionReplyTarget = | { mode: 'plain'; chatId: string } @@ -114,6 +114,34 @@ export function beginReplyTargetTurn( nowIso = new Date().toISOString(), opts?: { quoteOnly?: boolean; substitute?: boolean }, ): void { + const exactTarget: SessionReplyTarget = ds.scope === 'chat' + ? replyRootId + ? opts?.quoteOnly + ? { mode: 'quote', rootMessageId: replyRootId } + : { mode: 'thread', rootMessageId: replyRootId } + : { mode: 'plain', chatId: ds.chatId } + : { mode: 'thread', rootMessageId: ds.session.rootMessageId }; + const exactContexts = { ...(ds.session.turnReplyContexts ?? {}) }; + // Re-insertion keeps the newest turn at the end for deterministic bounding. + delete exactContexts[turnId]; + exactContexts[turnId] = { + target: exactTarget, + ...(ds.session.quoteTargetId ? { quoteTargetId: ds.session.quoteTargetId } : {}), + ...(ds.session.quoteTargetSenderOpenId + ? { replyTargetSenderOpenId: ds.session.quoteTargetSenderOpenId } + : {}), + ...(ds.session.quoteTargetSenderIsBot !== undefined + ? { replyTargetSenderIsBot: ds.session.quoteTargetSenderIsBot } + : {}), + }; + const overflow = Object.keys(exactContexts).length - 256; + if (overflow > 0) { + for (const staleTurnId of Object.keys(exactContexts).slice(0, overflow)) { + delete exactContexts[staleTurnId]; + } + } + ds.session.turnReplyContexts = exactContexts; + if (ds.scope !== 'chat') return; if (replyRootId) { const aliases = { ...(ds.replyThreadAliases ?? ds.session.replyThreadAliases ?? {}) }; @@ -146,6 +174,16 @@ export function beginReplyTargetTurn( ds.session.currentReplyTarget = undefined; } +/** Resolve a turn's immutable inbound destination, falling back only for + * legacy/non-Lark turns that predate the bounded registry. */ +export function frozenReplyContextForTurn( + ds: Pick, + turnId?: string, +): FrozenSessionReplyContext { + const frozen = turnId ? ds.session.turnReplyContexts?.[turnId] : undefined; + return frozen ?? { target: resolveSessionReplyTarget(ds, turnId) }; +} + /** * Effective turnId for a daemon-side message. Callers that know their turn * (worker final_output, placeholder cards) pass it explicitly and the diff --git a/src/core/resource-monitor/runtime.ts b/src/core/resource-monitor/runtime.ts index b13bf68a2..6bb62a23a 100644 --- a/src/core/resource-monitor/runtime.ts +++ b/src/core/resource-monitor/runtime.ts @@ -23,7 +23,10 @@ export interface RuntimeBotInput { daemonStatus?: RuntimeDaemonStatus; } -const WORKING_STATUSES = new Set(['working', 'analyzing', 'active']); +// `stalled` means no observable progress, not completion: the CLI/model/tool +// may still be executing. Keep it in active runtime pressure rather than +// misclassifying it as idle, human-waiting, or unknown. +const WORKING_STATUSES = new Set(['working', 'analyzing', 'active', 'stalled']); const STARTING_STATUSES = new Set(['starting', 'queued']); const IDLE_STATUSES = new Set(['idle', 'dormant']); diff --git a/src/core/restart-report.ts b/src/core/restart-report.ts index e7c7a7ce7..aa8eef593 100644 --- a/src/core/restart-report.ts +++ b/src/core/restart-report.ts @@ -8,7 +8,9 @@ */ import { githubAuthHeaders, type GithubAuthResolveOptions } from './github-auth.js'; import type { RestartKind } from '../services/restart-intent-store.js'; -import { consumeRestartIntent } from '../services/restart-intent-store.js'; +import { + claimRestartIntentForReport, +} from '../services/restart-intent-store.js'; import { countActiveSessionsOnDisk } from '../services/session-store.js'; import { botmuxVersion } from '../utils/install-info.js'; import { t, localeForBot, type Locale } from '../i18n/index.js'; @@ -93,6 +95,10 @@ export interface RestartReportWiring { githubAuth?: GithubAuthResolveOptions; now?: number; log?: (msg: string) => void; + /** Test seam; production uses an unref-free ordinary async timer because the + * daemon itself remains alive. */ + wait?: (ms: number) => Promise; + preparedCommitWaitMs?: number; } /** @@ -103,8 +109,23 @@ export interface RestartReportWiring { */ export async function sendRestartReportIfPending(w: RestartReportWiring): Promise { const log = w.log ?? (() => {}); - const intent = consumeRestartIntent(w.now ?? Date.now()); - if (!intent) return; // no breadcrumb → crash/reboot → stay silent + const now = () => w.now ?? Date.now(); + const wait = w.wait ?? (ms => new Promise(resolve => setTimeout(resolve, ms))); + let claim = claimRestartIntentForReport(now()); + let remainingPreparedWaitMs = Math.max(0, w.preparedCommitWaitMs ?? 45_000); + const pollMs = 100; + // A primary daemon from a partial PM2 launch may reach this hook before the + // CLI has verified the configured fleet. One atomic claim distinguishes + // prepared/committed/absent; never infer state from two independently locked + // reads because commit can land between them. + while (claim.state === 'prepared' && remainingPreparedWaitMs > 0) { + const delayMs = Math.min(pollMs, remainingPreparedWaitMs); + await wait(delayMs); + remainingPreparedWaitMs -= delayMs; + claim = claimRestartIntentForReport(now()); + } + if (claim.state !== 'claimed') return; // no committed breadcrumb → stay silent + const intent = claim.intent; if (!w.ownerOpenId) { log('restart-report: no owner configured — skipping DM'); return; } const locale = localeForBot(w.primaryLarkAppId); diff --git a/src/core/riff-shutdown-detach.ts b/src/core/riff-shutdown-detach.ts new file mode 100644 index 000000000..4a20682e4 --- /dev/null +++ b/src/core/riff-shutdown-detach.ts @@ -0,0 +1,1049 @@ +import { randomUUID } from 'node:crypto'; +import type { ChildProcess } from 'node:child_process'; +import type { DaemonToWorker, WorkerToDaemon } from '../types.js'; +import * as sessionStore from '../services/session-store.js'; +import { logger } from '../utils/logger.js'; +import type { DaemonSession } from './types.js'; +import { + RIFF_ADMISSION_RESTORE_TIMEOUT_MS, + RIFF_SHUTDOWN_BATCH_PERSIST_TIMEOUT_MS, + RIFF_SHUTDOWN_DRAIN_TIMEOUT_MS, + RIFF_SHUTDOWN_INITIAL_SNAPSHOT_TIMEOUT_MS, +} from './shutdown-budgets.js'; + +type ShutdownPhase = 'prepare' | 'abort'; + +type ShutdownPhaseResult = { + ok: boolean; + taskId: string | null; + error?: string; +}; + +export type PreparedRiffShutdown = { + ok: true; + fence: 'prepared'; + requestId: string; + taskId: string | null; + /** Runtime lineage sampled before the worker fence. A workerless transaction + * must still own this exact value at persistence time; a live transaction may + * advance only to its drained `taskId`. */ + runtimeTaskIdAtPrepare: string | null; + /** Fresh durable lineage sampled before installing the fence. Phase 2 uses + * it as a lock-protected compare-and-set guard, while also accepting an + * already-idempotent target written by the ordinary task-id callback. */ + durableTaskIdAtPrepare: string | null; + durableOwnerAtPrepare: { + pid: number | null; + larkAppId: string | null; + backendType: string | null; + }; + /** Set only after the exact cross-process fresh read succeeds. Used when a + * prepared worker exits before an all-or-nothing rollback can reach it. */ + lineageVerified: boolean; + /** Exact generation fenced by `requestId`. Null means the active logical + * session was already workerless and only its runtime lineage needs an exact + * durable verification. */ + worker: ChildProcess | null; +}; + +export type RiffShutdownFailure = { + ok: false; + requestId?: string; + taskId: string | null; + error: string; + /** Phase-2 coordinator policy. Ownership ambiguity must stay fenced; a + * plain atomic-write I/O failure may restore the exact prepared worker. */ + rollbackDisposition?: 'abort_safe' | 'retain_fence'; +}; + +/** A prepare refusal that is proven to have happened before the worker/backend + * fence. It must never be sent an abort request. */ +export type UnfencedRiffShutdownRefusal = RiffShutdownFailure & { + fence: 'none'; +}; + +/** A prepare attempt whose exact worker may have installed its backend fence. + * Preparation deliberately does not restore admission inline; the fleet + * coordinator includes this handle in its one concurrent abort wave. */ +export type PossiblyFencedRiffShutdown = RiffShutdownFailure & { + fence: 'possible'; + requestId: string; + worker: ChildProcess; + expectedAbortTaskId?: string | null; +}; + +export type RiffShutdownPrepareResult = + | PreparedRiffShutdown + | UnfencedRiffShutdownRefusal + | PossiblyFencedRiffShutdown; + +export type RiffShutdownPrepareOptions = { + drainTimeoutMs?: number; + abortTimeoutMs?: number; + /** Absolute transaction deadline. A worker is never asked to fence unless + * phase-2 plus the configured admission-restore reserve remain after drain. */ + deadlineMs?: number; + now?: () => number; + /** One projection sampled by prepareRiffFleetForShutdown before any fence. */ + durableSnapshot?: sessionStore.ActiveRiffShutdownSnapshot; +}; + +export type RiffFleetPrepareEntry = { + ds: DaemonSession; + result: RiffShutdownPrepareResult; +}; + +export type FencedRiffShutdownParticipant = + | PreparedRiffShutdown + | PossiblyFencedRiffShutdown; + +export type UniqueDaemonShutdownSessions = + | { ok: true; sessions: DaemonSession[] } + | { ok: false; sessionId: string; error: string }; + +/** The active registry can retain multiple aliases to one exact runtime object + * after transfer/restore. Process that object once, but refuse two distinct + * objects claiming the same durable session id: there is no unique generation + * that shutdown can safely fence or retire. */ +export function collectUniqueDaemonShutdownSessions( + candidates: Iterable, +): UniqueDaemonShutdownSessions { + const seenObjects = new Set(); + const bySessionId = new Map(); + const sessions: DaemonSession[] = []; + for (const ds of candidates) { + if (seenObjects.has(ds)) continue; + seenObjects.add(ds); + const sessionId = ds.session.sessionId; + const existing = bySessionId.get(sessionId); + if (existing && existing !== ds) { + return { + ok: false, + sessionId, + error: `distinct daemon session generations share session id ${sessionId}`, + }; + } + bySessionId.set(sessionId, ds); + sessions.push(ds); + } + return { ok: true, sessions }; +} + +export type RiffShutdownDetachOutcome = + | { + ok: true; + requestId: string; + taskId: string | null; + disposition: 'lineage_persisted'; + worker?: ChildProcess; + } + | RiffShutdownFailure; + +function label(ds: DaemonSession): string { + return ds.session.sessionId.slice(0, 8); +} + +function workerHasExited(worker: ChildProcess): boolean { + return (worker.exitCode !== null && worker.exitCode !== undefined) + || (worker.signalCode !== null && worker.signalCode !== undefined); +} + +function send(worker: ChildProcess, message: DaemonToWorker): boolean { + try { + worker.send(message); + return true; + } catch { + return false; + } +} + +/** Attach the exact-worker response/exit listeners before publishing a phase + * request. Results from another generation/session/request are inert. */ +function requestPhase( + ds: DaemonSession, + worker: ChildProcess, + requestId: string, + phase: ShutdownPhase, + timeoutMs: number, +): Promise { + return new Promise(resolve => { + let settled = false; + const finish = (result: ShutdownPhaseResult): void => { + if (settled) return; + settled = true; + clearTimeout(timer); + worker.removeListener('message', onMessage); + worker.removeListener('exit', onExit); + resolve(result); + }; + const onMessage = (raw: unknown): void => { + const msg = raw as WorkerToDaemon; + if (msg?.type !== 'riff_shutdown_result' + || msg.requestId !== requestId + || msg.phase !== phase) return; + if (ds.worker !== worker) { + finish({ ok: false, taskId: msg.taskId, error: 'stale_worker_generation' }); + return; + } + finish({ + ok: msg.ok, + taskId: msg.taskId, + ...(msg.error ? { error: msg.error } : {}), + }); + }; + const onExit = (): void => finish({ + ok: false, + taskId: null, + error: `worker_exited_during_shutdown_${phase}`, + }); + const timer = setTimeout(() => finish({ + ok: false, + taskId: null, + error: `riff_shutdown_${phase}_timeout`, + }), timeoutMs); + timer.unref?.(); + worker.on('message', onMessage); + worker.once('exit', onExit); + const message: DaemonToWorker = phase === 'prepare' + ? { type: 'riff_shutdown_prepare', requestId } + : { type: 'riff_shutdown_abort', requestId }; + if (!send(worker, message)) { + finish({ ok: false, taskId: null, error: `riff_shutdown_${phase}_send_failed` }); + } + }); +} + +function clearWorkerOwnership(ds: DaemonSession, worker: ChildProcess): void { + if (ds.worker !== worker) return; + ds.worker = null; + ds.workerPort = null; + ds.workerToken = null; + ds.workerViewToken = null; + ds.managedTurnOrigin = undefined; + ds.riffShutdownState = undefined; +} + +function clearWorkerlessShutdownState(ds: DaemonSession, requestId: string): void { + if (ds.riffShutdownState?.requestId !== requestId) return; + if (ds.worker) return; + ds.riffShutdownState = undefined; +} + +/** Daemon-owned accepted input has not crossed worker IPC yet. The bot-wide + * mutation lease blocks new admissions, but these older buffers must also be + * empty before a worker can be detached. */ +function daemonInputBlocker(ds: DaemonSession): string | null { + const parts: string[] = []; + if (ds.initialStartPending) parts.push('initial_start=1'); + if (ds.pendingPrompt) parts.push('prompt=1'); + if (ds.pendingRawInput) parts.push('raw=1'); + if (ds.pendingFollowUpInput) parts.push('raw_followup=1'); + if ((ds.pendingFollowUps?.length ?? 0) > 0) { + parts.push(`followups=${ds.pendingFollowUps!.length}`); + } + if ((ds.pendingQueuedActivationFollowUps?.length ?? 0) > 0) { + parts.push(`activation_tail=${ds.pendingQueuedActivationFollowUps!.length}`); + } + if ((ds.session.queuedActivationTail?.length ?? 0) > 0) { + parts.push(`durable_activation_tail=${ds.session.queuedActivationTail!.length}`); + } + if (ds.session.queued) parts.push('queued=1'); + if (ds.session.queuedActivationPending) parts.push('activation_journal=1'); + return parts.length > 0 ? parts.join(',') : null; +} + +async function abortWorkerPreparation( + ds: DaemonSession, + worker: ChildProcess, + requestId: string, + timeoutMs: number, + lineageVerified = false, + exactTask?: { taskId: string | null }, +): Promise { + if (workerHasExited(worker)) { + if (ds.worker && ds.worker !== worker) { + return { + ok: false, + taskId: ds.riffShutdownState?.taskId ?? ds.session.riffParentTaskId ?? null, + error: 'new_worker_generation', + }; + } + if (lineageVerified) { + if (ds.worker === worker) clearWorkerOwnership(ds, worker); + else clearWorkerlessShutdownState(ds, requestId); + return { ok: true, taskId: ds.session.riffParentTaskId ?? null }; + } + // No backend remains to ACK admission restoration and the drained lineage + // was not durably recovered. Retain a daemon-side fence instead of claiming + // rollback success and permitting a stale-lineage replacement. + if (!ds.riffShutdownState || ds.riffShutdownState.requestId === requestId) { + ds.riffShutdownState = { + phase: 'prepared', + requestId, + taskId: ds.session.riffParentTaskId ?? null, + }; + } + return { + ok: false, + taskId: ds.session.riffParentTaskId ?? null, + error: 'worker_exited_before_admission_restore', + }; + } + if (ds.worker !== worker) { + return { + ok: false, + taskId: ds.riffShutdownState?.taskId ?? ds.session.riffParentTaskId ?? null, + error: 'new_worker_generation', + }; + } + const result = await requestPhase(ds, worker, requestId, 'abort', timeoutMs); + if (result.ok && exactTask && result.taskId !== exactTask.taskId) { + return { + ok: false, + taskId: result.taskId, + error: `abort_task_lineage_mismatch:expected=${exactTask.taskId ?? 'none'},` + + `actual=${result.taskId ?? 'none'}`, + }; + } + if (result.ok && ds.riffShutdownState?.requestId === requestId) { + ds.riffShutdownState = undefined; + } + return result; +} + +function unfencedRefusal( + ds: DaemonSession, + error: string, + requestId?: string, +): UnfencedRiffShutdownRefusal { + return { + ok: false, + fence: 'none', + ...(requestId ? { requestId } : {}), + taskId: ds.session.riffParentTaskId ?? null, + error, + }; +} + +/** Phase 1: fence one exact Riff generation and drain task-id materialization. + * Nothing exits or restores admission here. A failure after the prepare send + * returns an exact possibly-fenced handle so all peers can be restored in one + * concurrent fleet wave rather than serial drain+abort chains. */ +export async function prepareRiffSessionForShutdown( + ds: DaemonSession, + options: RiffShutdownPrepareOptions = {}, +): Promise { + const frozenBackend = ds.initConfig?.backendType ?? ds.session.backendType; + if (frozenBackend !== 'riff') { + return unfencedRefusal(ds, 'not_riff_backend'); + } + if (ds.riffCloseState || ds.riffShutdownState) { + return unfencedRefusal( + ds, + ds.riffCloseState ? 'explicit_close_in_progress' : 'shutdown_detach_in_progress', + ); + } + const daemonBlockerBeforePrepare = daemonInputBlocker(ds); + if (daemonBlockerBeforePrepare) { + return unfencedRefusal(ds, `daemon_inputs_not_drained:${daemonBlockerBeforePrepare}`); + } + + const runtimeTaskIdAtPrepare = ds.session.riffParentTaskId ?? null; + let durableSnapshot = options.durableSnapshot; + if (!durableSnapshot) { + try { + [durableSnapshot] = sessionStore.getActiveRiffShutdownSnapshotsBatch([ + ds.session.sessionId, + ]); + } catch (err) { + return unfencedRefusal( + ds, + `durable_session_read_failed:${err instanceof Error ? err.message : String(err)}`, + ); + } + } + if (!durableSnapshot || durableSnapshot.sessionId !== ds.session.sessionId) { + return unfencedRefusal(ds, 'durable_session_snapshot_mismatch'); + } + const durableTaskIdAtPrepare = durableSnapshot.taskId; + const runtimeOwner = { + pid: ds.session.pid ?? null, + larkAppId: ds.session.larkAppId ?? null, + backendType: ds.session.backendType ?? null, + }; + const durableOwnerAtPrepare = durableSnapshot.owner; + if (durableOwnerAtPrepare.pid !== runtimeOwner.pid + || durableOwnerAtPrepare.larkAppId !== runtimeOwner.larkAppId + || durableOwnerAtPrepare.backendType !== runtimeOwner.backendType) { + return unfencedRefusal( + ds, + `durable_session_owner_mismatch:current=${JSON.stringify(durableOwnerAtPrepare)},` + + `runtime=${JSON.stringify(runtimeOwner)}`, + ); + } + + const now = options.now ?? Date.now; + const abortReserveMs = options.abortTimeoutMs ?? RIFF_ADMISSION_RESTORE_TIMEOUT_MS; + let drainTimeoutMs = options.drainTimeoutMs ?? RIFF_SHUTDOWN_DRAIN_TIMEOUT_MS; + if (options.deadlineMs !== undefined) { + const availableDrainMs = options.deadlineMs + - now() + - RIFF_SHUTDOWN_BATCH_PERSIST_TIMEOUT_MS + - abortReserveMs; + if (availableDrainMs <= 0) { + return unfencedRefusal(ds, 'insufficient_abort_budget_before_fence'); + } + drainTimeoutMs = Math.min(drainTimeoutMs, availableDrainMs); + } + + const requestId = randomUUID(); + const worker = ds.worker; + if (!worker || workerHasExited(worker)) { + // Retire a previously observed dead handle before installing the + // workerless fence. Any later non-null ds.worker is then unambiguously a + // new generation and cannot be blessed by this transaction. + if (worker) clearWorkerOwnership(ds, worker); + // Workerless rows can still carry a newer runtime-only lineage after a + // failed ordinary riff_task_id save. Fence daemon admission now; phase 2 + // performs an exact owner/lineage CAS and fresh disk verification. + ds.riffShutdownState = { + phase: 'prepared', + requestId, + taskId: ds.session.riffParentTaskId ?? null, + }; + return { + ok: true, + fence: 'prepared', + requestId, + taskId: ds.session.riffParentTaskId ?? null, + runtimeTaskIdAtPrepare, + durableTaskIdAtPrepare, + durableOwnerAtPrepare, + lineageVerified: false, + worker: null, + }; + } + + ds.riffShutdownState = { phase: 'preparing', requestId }; + const prepared = await requestPhase( + ds, + worker, + requestId, + 'prepare', + drainTimeoutMs, + ); + if (!prepared.ok) { + // This exact response proves the worker refused before installing a backend + // fence. Every other failure has ambiguous fence state and requires a + // positive admission-restored ACK. + const unfencedWorkerRefusal = prepared.error === 'explicit_close_in_progress' + || prepared.error === 'not_riff_backend' + || prepared.error === 'riff_shutdown_prepare_send_failed' + || prepared.error?.startsWith('worker_inputs_not_drained:') === true; + if (unfencedWorkerRefusal && ds.riffShutdownState?.requestId === requestId) { + ds.riffShutdownState = undefined; + } + if (unfencedWorkerRefusal) { + return { + ok: false, + fence: 'none', + requestId, + taskId: prepared.taskId, + error: prepared.error ?? 'riff_shutdown_prepare_failed', + }; + } + return { + ok: false, + fence: 'possible', + requestId, + taskId: prepared.taskId, + error: prepared.error ?? 'riff_shutdown_prepare_failed', + worker, + }; + } + if (ds.worker !== worker) { + return { + ok: false, + fence: 'possible', + requestId, + taskId: prepared.taskId, + error: 'stale_worker_generation', + worker, + }; + } + ds.riffShutdownState = { phase: 'prepared', requestId, taskId: prepared.taskId }; + + // Worker events can release a queued-activation journal/tail independently + // of bot-turn admission. Re-sample after drain so commit cannot strand input. + const daemonBlockerAfterPrepare = daemonInputBlocker(ds); + if (daemonBlockerAfterPrepare) { + return { + ok: false, + fence: 'possible', + requestId, + taskId: prepared.taskId, + error: `daemon_inputs_not_drained:${daemonBlockerAfterPrepare}`, + worker, + expectedAbortTaskId: prepared.taskId, + }; + } + + return { + ok: true, + fence: 'prepared', + requestId, + taskId: prepared.taskId, + runtimeTaskIdAtPrepare, + durableTaskIdAtPrepare, + durableOwnerAtPrepare, + lineageVerified: false, + worker, + }; +} + +export type RiffFleetPrepareOptions = Omit< + RiffShutdownPrepareOptions, + 'durableSnapshot' +> & { + snapshotTimeoutMs?: number; +}; + +/** Take one fresh projection for the complete candidate set, then (and only + * then) publish prepare requests concurrently. */ +export async function prepareRiffFleetForShutdown( + candidates: readonly DaemonSession[], + options: RiffFleetPrepareOptions = {}, +): Promise { + if (candidates.length === 0) return []; + const now = options.now ?? Date.now; + const configuredSnapshotTimeout = options.snapshotTimeoutMs + ?? RIFF_SHUTDOWN_INITIAL_SNAPSHOT_TIMEOUT_MS; + const remaining = options.deadlineMs === undefined + ? configuredSnapshotTimeout + : Math.max(0, options.deadlineMs - now()); + if (remaining <= 0) { + return candidates.map(ds => ({ + ds, + result: unfencedRefusal(ds, 'shutdown_deadline_elapsed_before_initial_snapshot'), + })); + } + + let snapshots: sessionStore.ActiveRiffShutdownSnapshot[]; + try { + snapshots = sessionStore.getActiveRiffShutdownSnapshotsBatch( + candidates.map(ds => ds.session.sessionId), + { maxWaitMs: Math.min(configuredSnapshotTimeout, remaining) }, + ); + } catch (error) { + const message = `initial_riff_snapshot_failed:${error instanceof Error + ? error.message + : String(error)}`; + return candidates.map(ds => ({ ds, result: unfencedRefusal(ds, message) })); + } + + const snapshotsBySession = new Map(snapshots.map(snapshot => [snapshot.sessionId, snapshot])); + return Promise.all(candidates.map(async ds => ({ + ds, + result: await prepareRiffSessionForShutdown(ds, { + ...options, + durableSnapshot: snapshotsBySession.get(ds.session.sessionId), + }), + }))); +} + +function validatePreparedRiffShutdownForPersistence( + ds: DaemonSession, + prepared: PreparedRiffShutdown, +): { ok: true } | RiffShutdownFailure { + if (ds.riffShutdownState?.requestId !== prepared.requestId + || ds.riffShutdownState.phase !== 'prepared') { + return { + ok: false, + requestId: prepared.requestId, + taskId: prepared.taskId, + error: 'shutdown_prepare_ownership_lost', + rollbackDisposition: 'retain_fence', + }; + } + const daemonBlocker = daemonInputBlocker(ds); + if (daemonBlocker) { + return { + ok: false, + requestId: prepared.requestId, + taskId: prepared.taskId, + error: `daemon_inputs_not_drained:${daemonBlocker}`, + rollbackDisposition: 'abort_safe', + }; + } + + if (prepared.worker) { + if (ds.worker !== prepared.worker || workerHasExited(prepared.worker)) { + return { + ok: false, + requestId: prepared.requestId, + taskId: prepared.taskId, + error: 'stale_worker_generation', + rollbackDisposition: 'retain_fence', + }; + } + } else if (ds.worker) { + return { + ok: false, + requestId: prepared.requestId, + taskId: prepared.taskId, + error: 'new_worker_generation', + rollbackDisposition: 'retain_fence', + }; + } + const runtimeOwner = { + pid: ds.session.pid ?? null, + larkAppId: ds.session.larkAppId ?? null, + backendType: ds.session.backendType ?? null, + }; + if (runtimeOwner.pid !== prepared.durableOwnerAtPrepare.pid + || runtimeOwner.larkAppId !== prepared.durableOwnerAtPrepare.larkAppId + || runtimeOwner.backendType !== prepared.durableOwnerAtPrepare.backendType) { + return { + ok: false, + requestId: prepared.requestId, + taskId: prepared.taskId, + error: `runtime_owner_changed:${JSON.stringify(runtimeOwner)}`, + rollbackDisposition: 'retain_fence', + }; + } + const runtimeTaskId = ds.session.riffParentTaskId ?? null; + const runtimeLineageExpected = prepared.worker + ? runtimeTaskId === prepared.runtimeTaskIdAtPrepare || runtimeTaskId === prepared.taskId + : runtimeTaskId === prepared.runtimeTaskIdAtPrepare; + if (!runtimeLineageExpected) { + return { + ok: false, + requestId: prepared.requestId, + taskId: prepared.taskId, + error: `runtime_lineage_changed:${runtimeTaskId ?? 'none'}`, + rollbackDisposition: 'retain_fence', + }; + } + return { ok: true }; +} + +export type PreparedRiffFleetEntry = { + ds: DaemonSession; + result: PreparedRiffShutdown; +}; + +export type RiffFleetPersistenceResult = { ok: true } +| (RiffShutdownFailure & { + sessionIds: readonly string[]; + retainFencedSessionIds: readonly string[]; +}); + +/** Phase 2 fleet transaction: validate every runtime generation, then compare + * and publish all durable lineage rows with one lock and one rename. */ +export function persistPreparedRiffShutdownFleet( + entries: readonly PreparedRiffFleetEntry[], + options: { + persistTimeoutMs?: number; + deadlineMs?: number; + now?: () => number; + } = {}, +): RiffFleetPersistenceResult { + if (entries.length === 0) return { ok: true }; + const now = options.now ?? Date.now; + + const validationFailures = entries + .map(({ ds, result }) => ({ + sessionId: ds.session.sessionId, + failure: validatePreparedRiffShutdownForPersistence(ds, result), + })) + .filter((entry): entry is { + sessionId: string; + failure: RiffShutdownFailure; + } => !entry.failure.ok); + if (validationFailures.length > 0) { + const retainFencedSessionIds = validationFailures + .filter(({ failure }) => failure.rollbackDisposition === 'retain_fence') + .map(({ sessionId }) => sessionId); + return { + ok: false, + taskId: null, + error: `Riff fleet persistence preflight failed: ${validationFailures + .map(({ sessionId, failure }) => `${sessionId}:${failure.error}`) + .join(';')}`, + rollbackDisposition: retainFencedSessionIds.length > 0 ? 'retain_fence' : 'abort_safe', + sessionIds: validationFailures.map(({ sessionId }) => sessionId), + retainFencedSessionIds, + }; + } + + const configuredPersistTimeout = options.persistTimeoutMs + ?? RIFF_SHUTDOWN_BATCH_PERSIST_TIMEOUT_MS; + const remaining = options.deadlineMs === undefined + ? configuredPersistTimeout + : Math.max(0, options.deadlineMs - now()); + if (remaining <= 0) { + return { + ok: false, + taskId: null, + error: 'Riff fleet lineage batch failed (prewrite_io): shutdown deadline elapsed', + rollbackDisposition: 'abort_safe', + sessionIds: entries.map(({ ds }) => ds.session.sessionId), + retainFencedSessionIds: [], + }; + } + + try { + sessionStore.persistActiveRiffLineagesExactBatch(entries.map(({ ds, result }) => ({ + sessionId: ds.session.sessionId, + taskId: result.durableTaskIdAtPrepare, + owner: result.durableOwnerAtPrepare, + targetTaskId: result.taskId, + expectedCurrentTaskIds: [result.durableTaskIdAtPrepare, result.taskId], + })), { + maxWaitMs: Math.min(configuredPersistTimeout, remaining), + }); + } catch (error) { + const batchError = error instanceof sessionStore.RiffLineageBatchError ? error : undefined; + const stage = batchError?.stage ?? 'prewrite_io'; + const retainFencedSessionIds = stage === 'postrename_ambiguity' + ? entries.map(({ ds }) => ds.session.sessionId) + : stage === 'prewrite_ownership' + ? [...(batchError?.sessionIds ?? [])] + : []; + return { + ok: false, + taskId: null, + error: `Riff fleet lineage batch failed (${stage}): ${error instanceof Error + ? error.message + : String(error)}`, + rollbackDisposition: retainFencedSessionIds.length > 0 ? 'retain_fence' : 'abort_safe', + sessionIds: batchError?.sessionIds ?? entries.map(({ ds }) => ds.session.sessionId), + retainFencedSessionIds, + }; + } + + for (const { ds, result } of entries) { + ds.session.riffParentTaskId = result.taskId ?? undefined; + result.lineageVerified = true; + } + if (options.deadlineMs !== undefined && now() >= options.deadlineMs) { + const sessionIds = entries.map(({ ds }) => ds.session.sessionId); + return { + ok: false, + taskId: null, + error: 'shutdown_deadline_elapsed_after_batch_persist', + rollbackDisposition: 'retain_fence', + sessionIds, + retainFencedSessionIds: sessionIds, + }; + } + const lost = entries + .filter(({ ds, result }) => !isPreparedRiffSessionCurrent(ds, result)) + .map(({ ds }) => ds.session.sessionId); + if (lost.length > 0) { + return { + ok: false, + taskId: null, + error: `shutdown_prepare_ownership_lost_after_batch_persist:${lost.join(',')}`, + rollbackDisposition: 'retain_fence', + sessionIds: lost, + retainFencedSessionIds: lost, + }; + } + return { ok: true }; +} + +/** Single-session compatibility path. Fleet shutdown uses the batch function + * above; this API remains for explicit focused operations and older callers. */ +export function persistPreparedRiffShutdown( + ds: DaemonSession, + prepared: PreparedRiffShutdown, +): { ok: true } | RiffShutdownFailure { + const validation = validatePreparedRiffShutdownForPersistence(ds, prepared); + if (!validation.ok) return validation; + + try { + sessionStore.persistActiveRiffLineageExact(ds.session.sessionId, prepared.taskId, { + expectedCurrentTaskIds: [prepared.durableTaskIdAtPrepare, prepared.taskId], + expectedOwner: prepared.durableOwnerAtPrepare, + }); + } catch (err) { + return { + ok: false, + requestId: prepared.requestId, + taskId: prepared.taskId, + error: `lineage_persist_failed:${err instanceof Error ? err.message : String(err)}`, + rollbackDisposition: err instanceof sessionStore.RiffLineageOwnershipError + ? 'retain_fence' + : 'abort_safe', + }; + } + + let fresh: ReturnType; + try { + fresh = sessionStore.getSessionFresh(ds.session.sessionId); + } catch (err) { + return { + ok: false, + requestId: prepared.requestId, + taskId: prepared.taskId, + error: `fresh_lineage_verification_failed:${err instanceof Error ? err.message : String(err)}`, + rollbackDisposition: 'retain_fence', + }; + } + const freshTaskId = fresh?.riffParentTaskId ?? null; + const freshOwner = fresh + ? { + pid: fresh.pid ?? null, + larkAppId: fresh.larkAppId ?? null, + backendType: fresh.backendType ?? null, + } + : null; + const freshOwnerMatches = freshOwner !== null + && freshOwner.pid === prepared.durableOwnerAtPrepare.pid + && freshOwner.larkAppId === prepared.durableOwnerAtPrepare.larkAppId + && freshOwner.backendType === prepared.durableOwnerAtPrepare.backendType; + if (!fresh + || fresh.status !== 'active' + || freshTaskId !== prepared.taskId + || !freshOwnerMatches) { + return { + ok: false, + requestId: prepared.requestId, + taskId: prepared.taskId, + error: `fresh_lineage_verification_failed:status=${fresh?.status ?? 'missing'},` + + `task=${freshTaskId ?? 'none'},expected=${prepared.taskId ?? 'none'},` + + `owner=${JSON.stringify(freshOwner)},` + + `expectedOwner=${JSON.stringify(prepared.durableOwnerAtPrepare)}`, + rollbackDisposition: 'retain_fence', + }; + } + ds.session.riffParentTaskId = prepared.taskId ?? undefined; + prepared.lineageVerified = true; + + if (!isPreparedRiffSessionCurrent(ds, prepared)) { + return { + ok: false, + requestId: prepared.requestId, + taskId: prepared.taskId, + error: 'shutdown_prepare_ownership_lost_after_persist', + rollbackDisposition: 'retain_fence', + }; + } + return { ok: true }; +} + +export function isPreparedRiffSessionCurrent( + ds: DaemonSession, + prepared: PreparedRiffShutdown, +): boolean { + if (ds.riffShutdownState?.requestId !== prepared.requestId + || ds.riffShutdownState.phase !== 'prepared') return false; + if (prepared.worker) { + return ds.worker === prepared.worker && !workerHasExited(prepared.worker); + } + return ds.worker === null; +} + +/** After verified lineage persistence, an exact prepared worker that exited + * and was cleared by worker-pool can be safely rolled back locally. A new + * worker generation or a different fence remains ownership ambiguity. */ +export function canAbortVerifiedExitedRiffPreparation( + ds: DaemonSession, + prepared: PreparedRiffShutdown, +): boolean { + return prepared.lineageVerified + && !!prepared.worker + && workerHasExited(prepared.worker) + && ds.worker === null + && ds.riffShutdownState?.requestId === prepared.requestId; +} + +/** Roll back one prepared participant. State clears only after the exact worker + * ACKs admission restoration; timeout/late ACK remains deliberately fail-closed. */ +export async function abortPreparedRiffShutdown( + ds: DaemonSession, + prepared: PreparedRiffShutdown, + options: { abortTimeoutMs?: number } = {}, +): Promise { + if (ds.riffShutdownState?.requestId !== prepared.requestId) { + return { ok: false, taskId: prepared.taskId, error: 'shutdown_prepare_ownership_lost' }; + } + if (!prepared.worker) { + if (ds.worker) { + return { ok: false, taskId: prepared.taskId, error: 'new_worker_generation' }; + } + clearWorkerlessShutdownState(ds, prepared.requestId); + return { ok: true, taskId: prepared.taskId }; + } + if (workerHasExited(prepared.worker)) { + return abortWorkerPreparation( + ds, + prepared.worker, + prepared.requestId, + options.abortTimeoutMs ?? RIFF_ADMISSION_RESTORE_TIMEOUT_MS, + prepared.lineageVerified, + { taskId: prepared.taskId }, + ); + } + return abortWorkerPreparation( + ds, + prepared.worker, + prepared.requestId, + options.abortTimeoutMs ?? RIFF_ADMISSION_RESTORE_TIMEOUT_MS, + prepared.lineageVerified, + { taskId: prepared.taskId }, + ); +} + +async function abortPossiblyFencedRiffShutdown( + ds: DaemonSession, + participant: PossiblyFencedRiffShutdown, + timeoutMs: number, +): Promise { + if (ds.riffShutdownState?.requestId !== participant.requestId) { + return { + ok: false, + taskId: participant.taskId, + error: 'shutdown_prepare_ownership_lost', + }; + } + return abortWorkerPreparation( + ds, + participant.worker, + participant.requestId, + timeoutMs, + false, + Object.prototype.hasOwnProperty.call(participant, 'expectedAbortTaskId') + ? { taskId: participant.expectedAbortTaskId ?? null } + : undefined, + ); +} + +export type RiffFleetAbortEntry = { + ds: DaemonSession; + result: FencedRiffShutdownParticipant; +}; + +export type RiffFleetAbortResult = { + ds: DaemonSession; + participant: FencedRiffShutdownParticipant; + result: ShutdownPhaseResult; +}; + +/** Restore every exact prepared/possibly-fenced generation concurrently. No + * participant can consume another's timeout budget. */ +export async function abortRiffShutdownFleet( + entries: readonly RiffFleetAbortEntry[], + options: { + abortTimeoutMs?: number; + deadlineMs?: number; + now?: () => number; + } = {}, +): Promise { + const now = options.now ?? Date.now; + const configuredTimeout = options.abortTimeoutMs ?? RIFF_ADMISSION_RESTORE_TIMEOUT_MS; + const remaining = options.deadlineMs === undefined + ? configuredTimeout + : Math.max(0, options.deadlineMs - now()); + const timeoutMs = Math.min(configuredTimeout, remaining); + + return Promise.all(entries.map(async ({ ds, result: participant }) => { + if (timeoutMs <= 0) { + return { + ds, + participant, + result: { + ok: false, + taskId: participant.taskId, + error: 'shutdown_deadline_elapsed_before_abort', + }, + }; + } + const result = participant.ok + ? await abortPreparedRiffShutdown(ds, participant, { abortTimeoutMs: timeoutMs }) + : await abortPossiblyFencedRiffShutdown(ds, participant, timeoutMs); + return { ds, participant, result }; + })); +} + +/** Phase 3: synchronous, infallible-after-validation commit. The coordinator + * validates every participant first, then calls this without an intervening + * await, so one session can never refuse after a peer was detached. */ +export function commitPreparedRiffShutdown( + ds: DaemonSession, + prepared: PreparedRiffShutdown, +): boolean { + if (!isPreparedRiffSessionCurrent(ds, prepared)) return false; + if (!prepared.worker) { + clearWorkerlessShutdownState(ds, prepared.requestId); + return true; + } + if (!send(prepared.worker, { type: 'riff_shutdown_commit', requestId: prepared.requestId })) { + // Lineage is already durably fresh-verified. Direct retirement is safe even + // if the final IPC channel closed between validation and send. + try { prepared.worker.kill('SIGTERM'); } catch { /* already exited */ } + } + clearWorkerOwnership(ds, prepared.worker); + return true; +} + +/** Single-session compatibility wrapper. Daemon shutdown intentionally uses + * the explicit three-phase API above so a multi-Riff fleet cannot half-commit. */ +export async function detachRiffWorkerForShutdown( + ds: DaemonSession, + options: { drainTimeoutMs?: number; abortTimeoutMs?: number } = {}, +): Promise { + const prepared = await prepareRiffSessionForShutdown(ds, options); + if (!prepared.ok) { + if (prepared.fence === 'none') return prepared; + const [aborted] = await abortRiffShutdownFleet([{ ds, result: prepared }], options); + const abortSuffix = aborted.result.ok + ? '' + : `;admission_restore_failed:${aborted.result.error ?? 'unknown'}`; + logger.error( + `[${label(ds)}] Riff shutdown drain failed (${prepared.error}${abortSuffix}); ` + + (aborted.result.ok + ? 'worker retained with admission restored' + : 'worker retained and admission fence kept fail-closed'), + ); + return { + ok: false, + requestId: prepared.requestId, + taskId: prepared.taskId, + error: prepared.error + abortSuffix, + }; + } + const persisted = persistPreparedRiffShutdown(ds, prepared); + if (!persisted.ok) { + if (persisted.rollbackDisposition === 'retain_fence') return persisted; + const aborted = await abortPreparedRiffShutdown(ds, prepared, options); + return { + ok: false, + requestId: prepared.requestId, + taskId: prepared.taskId, + error: persisted.error + + (aborted.ok ? '' : `;admission_restore_failed:${aborted.error ?? 'unknown'}`), + }; + } + if (!commitPreparedRiffShutdown(ds, prepared)) { + return { + ok: false, + requestId: prepared.requestId, + taskId: prepared.taskId, + error: 'shutdown_prepare_ownership_lost_before_commit', + rollbackDisposition: 'retain_fence', + }; + } + logger.info( + `[${label(ds)}] Riff shutdown detach committed after durable lineage ` + + `${prepared.taskId ?? 'none'}`, + ); + return { + ok: true, + requestId: prepared.requestId, + taskId: prepared.taskId, + disposition: 'lineage_persisted', + ...(prepared.worker ? { worker: prepared.worker } : {}), + }; +} diff --git a/src/core/riff-worker-shutdown-readiness.ts b/src/core/riff-worker-shutdown-readiness.ts new file mode 100644 index 000000000..f7101dd19 --- /dev/null +++ b/src/core/riff-worker-shutdown-readiness.ts @@ -0,0 +1,31 @@ +export interface RiffWorkerShutdownInputSnapshot { + /** False while async init can still materialize an opening prompt later. */ + initPromptMaterialized: boolean; + /** A normal prompt has been removed from pendingMessages but has not yet + * crossed the adapter/backend write boundary. */ + isFlushing: boolean; + pendingMessages: number; + pendingRawInputs: number; + pendingSessionRename: boolean; + sessionRenameInFlight: boolean; + /** Raw command text -> Enter sequences that can still append a backend + * write after the shutdown fence is sampled. */ + commandLineWritesPending: number; +} + +/** Describe worker-owned input not proven to be inside RiffBackend.writeChain. */ +export function riffWorkerShutdownInputBlocker( + snapshot: RiffWorkerShutdownInputSnapshot, +): string | null { + const parts: string[] = []; + if (!snapshot.initPromptMaterialized) parts.push('init=materializing'); + if (snapshot.isFlushing) parts.push('flushing=1'); + if (snapshot.pendingMessages > 0) parts.push(`messages=${snapshot.pendingMessages}`); + if (snapshot.pendingRawInputs > 0) parts.push(`raw=${snapshot.pendingRawInputs}`); + if (snapshot.pendingSessionRename) parts.push('rename=1'); + if (snapshot.sessionRenameInFlight) parts.push('rename_inflight=1'); + if (snapshot.commandLineWritesPending > 0) { + parts.push(`command_writes=${snapshot.commandLineWritesPending}`); + } + return parts.length > 0 ? parts.join(',') : null; +} diff --git a/src/core/session-manager.ts b/src/core/session-manager.ts index 4e5acb225..83cd03109 100644 --- a/src/core/session-manager.ts +++ b/src/core/session-manager.ts @@ -12,7 +12,7 @@ import * as sessionStore from '../services/session-store.js'; import * as messageQueue from '../services/message-queue.js'; import { downloadMessageResource, listChatBotMembers, UserTokenMissingError } from '../im/lark/client.js'; import { logger } from '../utils/logger.js'; -import { forkWorker, sendWorkerInput, forkAdoptWorker, killStalePids, getCurrentCliVersion, restoreUsageLimitRuntimeState, setActiveSessionSafe, isRelayableRealSession, closeSession, getActiveSessionsRegistry, suspendWorker } from './worker-pool.js'; +import { forkWorker, sendWorkerInput, promoteQueuedActivationTail, forkAdoptWorker, killStalePids, getCurrentCliVersion, restoreUsageLimitRuntimeState, setActiveSessionSafe, isRelayableRealSession, closeSession, getActiveSessionsRegistry, suspendWorker, withActiveSessionKeyLock } from './worker-pool.js'; import { createCliAdapterSync } from '../adapters/cli/registry.js'; import { buildBotmuxShellHints } from '../adapters/cli/shared-hints.js'; import { @@ -39,10 +39,19 @@ import { validateZellijAdoptTarget } from './zellij-adopt-discovery.js'; import type { BackendType } from '../adapters/backend/types.js'; import type { CliTurnPayload, CodexAppAdditionalContextEntry, CodexAppTurnInput, LarkAttachment, LarkMention, ScheduledTask, SubstituteTrigger } from '../types.js'; import { addCodexAppContext } from '../utils/codex-app-context.js'; +import { hasUnsettledCodexAppDispatch } from '../utils/codex-app-dispatch-ledger.js'; +import { hasProtectedSessionMutationOwnership } from './session-mutation-guard.js'; import type { MessageResource } from '../im/lark/message-parser.js'; import type { ResolvedSender } from '../im/lark/identity-cache.js'; -import { activeSessionKey, sessionKey, sessionAnchorId, storedSessionAnchorId } from './types.js'; +import { + activeSessionKey, + sessionKey, + sessionAnchorId, + storedSessionAnchorId, + riffRetirementAdmissionPhase, +} from './types.js'; import type { DaemonSession } from './types.js'; +import { stagePendingRepoSetup, persistPendingRepoCardMessageId, restorePendingRepoRuntime } from './pending-repo-journal.js'; import { announceSessionRow, markSessionActivity, announcePendingRepoSession } from './session-activity.js'; import { scanMultipleProjects } from '../services/project-scanner.js'; import { buildRepoSelectCard } from '../im/lark/card-builder.js'; @@ -69,6 +78,114 @@ function sessionLastMessageAtMs(session: { createdAt?: string; lastMessageAt?: s return session.lastMessageAt ? (Date.parse(session.lastMessageAt) || sessionCreatedAtMs(session)) : sessionCreatedAtMs(session); } +async function resumeRestoredPendingRepoSetup( + ds: DaemonSession, + activeSessions: Map, +): Promise { + const setup = ds.session.pendingRepoSetup; + if (!setup || ds.session.queuedActivationPending || !ds.pendingRepo) return; + const anchor = sessionAnchorId(ds); + const notify = async (content: string): Promise => { + const { sendMessage, replyMessage } = await import('../im/lark/client.js'); + return ds.scope === 'thread' + ? replyMessage(ds.larkAppId, anchor, content, 'text', true) + : sendMessage(ds.larkAppId, ds.chatId, content, 'text'); + }; + + if (setup.mode === 'auto_worktree' && setup.baseDir) { + const { runAutoWorktreeCommit } = await import('../im/lark/card-handler.js'); + void runAutoWorktreeCommit({ + ds, + anchor, + larkAppId: ds.larkAppId, + baseDir: setup.baseDir, + title: ds.session.title, + prompt: setup.prompt, + operatorOpenId: ds.session.ownerOpenId, + activeSessions, + notify, + }).catch((err) => { + // Git/worktree recovery is deliberately detached. A failed publish or + // build may not reject daemon startup or erase this durable setup owner. + restorePendingRepoRuntime(ds); + logger.error( + `[${ds.session.sessionId.substring(0, 8)}] Pending auto-worktree recovery failed; ` + + `durable setup retained for retry: ${err instanceof Error ? err.message : String(err)}`, + ); + }); + logger.info(`[${ds.session.sessionId.substring(0, 8)}] Restarted durable pending auto-worktree setup`); + return; + } + + // Always publish a fresh picker after restart: a persisted message id proves + // identity, not that the Lark message still exists. Persist the replacement + // before best-effort withdrawal so exactly one card remains authoritative. + const oldCardMessageId = setup.repoCardMessageId; + const scanDirs = getProjectScanDirs(ds).filter(dir => existsSync(dir)); + const projects = scanDirs.length > 0 + ? scanMultipleProjects(scanDirs, 3, repoPickerScanOptions()) + : []; + if (projects.length > 0) { + const bot = getBot(ds.larkAppId); + const card = buildRepoSelectCard( + projects, + getSessionWorkingDir(ds), + anchor, + localeForBot(ds.larkAppId), + bot.config.worktreeMultiPicker, + ); + const { sendMessage, replyMessage, deleteMessage } = await import('../im/lark/client.js'); + const messageId = ds.scope === 'thread' + ? await replyMessage(ds.larkAppId, anchor, card, 'interactive', true) + : await sendMessage(ds.larkAppId, ds.chatId, card, 'interactive'); + ds.repoCardMessageId = messageId; + persistPendingRepoCardMessageId(ds, messageId); + if (oldCardMessageId && oldCardMessageId !== messageId) { + void deleteMessage(ds.larkAppId, oldCardMessageId).catch(() => {}); + } + announcePendingRepoSession(ds); + logger.info(`[${ds.session.sessionId.substring(0, 8)}] Re-posted durable pending repo picker`); + return; + } + + // The original pre-card scan may have crashed before learning there was no + // selectable repo. Resume the same no-project fallback without replacing N. + ds.pendingRepo = false; + ensureSessionWhiteboard(ds); + if (setup.rawInput) { + rememberLastCliInput(ds, setup.rawInput, setup.rawInput); + forkWorker(ds, '', { resume: false, turnId: setup.turnId }); + } else { + const bot = getBot(ds.larkAppId); + const availableBots = await getAvailableBots(ds.larkAppId, ds.chatId); + const input = buildNewTopicCliInput( + setup.prompt, + ds.session.sessionId, + ds.session.cliId ?? bot.config.cliId, + ds.session.cliPathOverride ?? bot.config.cliPathOverride, + ds.pendingAttachments, + ds.pendingMentions, + availableBots, + undefined, + { name: bot.botName, openId: bot.botOpenId }, + localeForBot(ds.larkAppId), + ds.pendingSender, + { + larkAppId: ds.larkAppId, + chatId: ds.chatId, + whiteboardId: ds.session.whiteboardId, + substituteTrigger: ds.pendingSubstituteTrigger, + codexAppText: ds.pendingCodexAppText, + codexAppApplicationContext: ds.pendingCodexAppApplicationContext, + codexAppMessageContext: ds.pendingCodexAppMessageContext, + }, + ); + rememberLastCliInput(ds, setup.prompt, input); + forkWorker(ds, input, { resume: false, turnId: setup.turnId }); + } + logger.info(`[${ds.session.sessionId.substring(0, 8)}] Resumed durable pending repo opening without selectable projects`); +} + function sameUsageLimit(a: DaemonSession['usageLimit'], b: DaemonSession['usageLimit']): boolean { if (!a && !b) return true; if (!a || !b) return false; @@ -101,6 +218,23 @@ async function closeActiveSessionIfCliMismatch(ds: DaemonSession): Promise void, batchSize: number = RECOVERY_FORK_BATCH_SIZE, delayMs: number = RECOVERY_FORK_DELAY_MS, + stillOwned: (ds: DaemonSession) => boolean = ds => ds.session.status === 'active', ): Promise { let spawnedInBatch = 0; for (const ds of sessions) { - if (ds.worker) continue; // already woken by a real message — don't clobber it - fork(ds); + // The batch delay is a lifecycle boundary: close/replace can remove this + // exact object while we sleep. Never resurrect a closed or orphaned ds. + if (ds.worker || ds.session.status !== 'active' || !stillOwned(ds)) continue; + try { + fork(ds); + } catch (err) { + // One malformed/stale pane or synchronous init-IPC failure must not + // abort recovery for every later durable owner. forkWorker compensates + // its own pre-init journal mutations; isolate this row and continue. + logger.error( + `[${ds.session.sessionId.substring(0, 8)}] Recovery fork failed; retained for later retry: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + continue; + } if (++spawnedInBatch >= batchSize) { spawnedInBatch = 0; if (delayMs > 0) await new Promise((r) => setTimeout(r, delayMs)); @@ -1197,6 +1353,7 @@ export async function restoreActiveSessions(activeSessions: Map 0, workingDir: session.workingDir, ownerOpenId: session.ownerOpenId, // Restore persisted streaming-card state — next screen_update will PATCH @@ -1374,15 +1621,50 @@ export async function restoreActiveSessions(activeSessions: Map 0 + && !promoteQueuedActivationTail(ds, { send: false })) { + throw new Error(`failed to promote durable activation tail for ${session.sessionId}`); + } const anchor = sessionAnchorId(ds); messageQueue.ensureQueue(anchor); if (ds.usageLimit) restoreUsageLimitRuntimeState(ds); // Same-key collision guard — see adopt-branch comment above. - await setActiveSessionSafe(activeSessions, activeSessionKey(ds), ds); + const registration = await setActiveSessionSafe(activeSessions, activeSessionKey(ds), ds); + if (!registration.accepted) { + if (registration.reason === 'both_pending' || registration.reason === 'cleanup_failed') { + logger.error( + `[${session.sessionId.substring(0, 8)}] Isolated active restore collision; ` + + `durable row/pane retained without aborting daemon startup: ${registration.reason === 'both_pending' + ? `two protected owners at ${activeSessionKey(ds)}` + : `cleanup failed for ${registration.cleanupSessionId}: ${registration.error}`}`, + ); + continue; + } + logger.warn(`[${session.sessionId.substring(0, 8)}] restore collision lost to unsettled session ${registration.keptSessionId.substring(0, 8)}`); + continue; + } announceSessionRow(ds); logger.debug(`Registered session ${session.sessionId} (scope: ${scope}, anchor: ${anchor})`); + } catch (err) { + // Restore is a per-row reconciliation job. A malformed legacy hybrid or + // one transient persistence failure must retain that row for inspection + // and retry without preventing every later healthy session from loading. + logger.error( + `[${session.sessionId.substring(0, 8)}] Isolated session restore failure; ` + + `durable row retained and daemon startup continuing: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + continue; + } } // Persistent backends: auto-fork workers for sessions whose backing session @@ -1410,13 +1692,32 @@ export async function restoreActiveSessions(activeSessions: Map forkWorker(ds, '', true)); + await staggeredRecoveryFork( + toReattach, + (ds) => { + const recoverExactNonCodex = ds.session.queuedActivationPending + && ds.session.cliId !== 'codex-app' + && ds.session.queuedActivationInput; + forkWorker( + ds, + recoverExactNonCodex || '', + recoverExactNonCodex + ? { + resume: ds.session.queuedActivationResume ?? ds.hasHistory, + turnId: ds.session.queuedActivationTurnId, + dispatchAttempt: ds.session.queuedActivationDispatchAttempt, + } + : true, + ); + }, + RECOVERY_FORK_BATCH_SIZE, + RECOVERY_FORK_DELAY_MS, + ds => activeSessions.get(activeSessionKey(ds)) === ds, + ); const hasPersistentBackend = [...activeSessions.values()].some(ds => !!getSessionPersistentBackendType(ds)); logger.info(`Restored ${active.length} session(s)${hasPersistentBackend ? '' : ', waiting for messages to resume'}`); @@ -1553,7 +1880,7 @@ export async function resumeSession( activeSessions: Map, ): Promise<{ ok: true; ds: DaemonSession } | { ok: false; error: 'not_found' | 'not_closed' | 'anchor_occupied' | 'adopt_unsupported' | 'vc_receiver_managed' | 'deferred_unmaterialized'; activeSessionId?: string }> { - const session = sessionStore.getSession(sessionId); + let session = sessionStore.getSession(sessionId); if (!session) return { ok: false, error: 'not_found' }; if (session.status !== 'closed') return { ok: false, error: 'not_closed' }; @@ -1590,6 +1917,22 @@ export async function resumeSession( const anchor = storedSessionAnchorId(session); const key = sessionKey(anchor, larkAppId); + return withActiveSessionKeyLock(activeSessions, key, async () => { + // The resume request may have waited behind a fresh creator. Re-read the + // durable row and keep the already-active creator as first owner. + const latest = sessionStore.getSession(sessionId); + if (!latest) return { ok: false as const, error: 'not_found' as const }; + if (latest.status !== 'closed') return { ok: false as const, error: 'not_closed' as const }; + if (latest.vcMeetingReceiver) return { ok: false as const, error: 'vc_receiver_managed' as const }; + if (latest.deferredScheduleRun + && !readDeferredTopicBinding(config.session.dataDir, latest.sessionId)) { + return { ok: false as const, error: 'deferred_unmaterialized' as const }; + } + if (latest.title?.startsWith('Adopt:') || latest.adoptedFrom) { + return { ok: false as const, error: 'adopt_unsupported' as const }; + } + session = latest; + // In-memory occupant check. A daemon-command scratch (e.g. an unconfirmed // `/relay` picker, a bare `/help`) parks a worker:null placeholder at this // anchor; daemon.ts creates one for ANY DAEMON_COMMAND in a session-less @@ -1605,7 +1948,12 @@ export async function resumeSession( // throwaway command container, so clobbering it would lose real intent. const existing = activeSessions.get(key); if (existing) { - if (isRelayableRealSession(existing) || existing.pendingRepo) { + if (hasProtectedSessionMutationOwnership(existing) + || isRelayableRealSession(existing) + || !!existing.session.riffParentTaskId + || existing.pendingRepo + || existing.initialStartPending + || existing.session.queued) { return { ok: false, error: 'anchor_occupied', activeSessionId: existing.session.sessionId }; } await closeSession(existing.session.sessionId); @@ -1621,9 +1969,10 @@ export async function resumeSession( // store-level conflict invisible to the Map check above. // // Same scratch carve-out applies on disk: a persisted scratch has neither - // `cliId` nor `lastCliInput` (those are only written once a real CLI ran). - // A real conflict (either marker present) still blocks; scratch-only - // conflicts get closed so they stop occupying the anchor on disk. + // `cliId` / `lastCliInput` nor a live Riff task id (those are only written + // once a real backend ran). A real conflict (any marker present) still + // blocks; scratch-only conflicts get closed so they stop occupying the + // anchor on disk. // // CAVEAT — this path canNOT honor the pendingRepo carve-out the in-memory // branch above applies: `pendingRepo` is a runtime DaemonSession flag that @@ -1639,11 +1988,19 @@ export async function resumeSession( const conflicts = sessionStore.listSessions().filter(s => s.sessionId !== sessionId && s.status === 'active' - && (s.larkAppId ?? '') === larkAppId + // Legacy active rows predate per-bot stamping. Conservatively attribute an + // unscoped row to this daemon, matching transfer/restore collision rules; + // otherwise a disk-only legacy anchor can be ghosted by the resumed row. + && (!s.larkAppId || s.larkAppId === larkAppId) && (s.scope === 'chat' ? 'chat' : 'thread') === scope && storedSessionAnchorId(s) === anchor, ); - const realConflict = conflicts.find(s => !!s.cliId || !!s.lastCliInput); + const realConflict = conflicts.find(s => + hasProtectedSessionMutationOwnership(s) + || !!s.cliId + || !!s.lastCliInput + || !!s.riffParentTaskId, + ); if (realConflict) { return { ok: false, error: 'anchor_occupied', activeSessionId: realConflict.sessionId }; } @@ -1651,12 +2008,25 @@ export async function resumeSession( await closeSession(scratch.sessionId); } - // Reactivate in store — clear closedAt so dashboard rows don't keep showing - // the stale close timestamp on the now-active session. - session.status = 'active'; - session.closedAt = undefined; - session.lastMessageAt = new Date().toISOString(); - sessionStore.updateSession(session); + // Scratch cleanup and disk scans can await. A creator that does not yet use + // the shared key lock still wins if it published meanwhile; never reactivate + // the closed row and then replace that fresh owner. + const lateExisting = activeSessions.get(key); + if (lateExisting) { + return { + ok: false, + error: 'anchor_occupied', + activeSessionId: lateExisting.session.sessionId, + }; + } + + // Reactivate and abandon any queued/setup ownership in one durable replace. + // Closed rows created by older releases may still contain a prepared head, + // tail, or repo picker; generic resume starts a new lifecycle and must never + // replay those abandoned inputs. + const reactivated = sessionStore.reactivateClosedSession(sessionId); + if (!reactivated.ok) return reactivated; + session = reactivated.session; const now = Date.now(); const ds: DaemonSession = { @@ -1688,12 +2058,13 @@ export async function resumeSession( }; messageQueue.ensureQueue(anchor); - // setActiveSessionSafe over a bare Map.set: the scratch-eviction above - // should already have freed `key`, but if any occupant remains it closes - // it rather than silently orphaning it (consistent with restore/transfer). - await setActiveSessionSafe(activeSessions, key, ds); + // Live creation is first-owner-wins. Restore-time setActiveSessionSafe may + // replace disposable historical collisions, but a user resume must never + // close a newly-created worker/reservation at this anchor. + activeSessions.set(key, ds); logger.info(`Resumed session ${sessionId.substring(0, 8)} (scope: ${scope}, anchor: ${anchor.substring(0, 12)})`); return { ok: true, ds }; + }); } // ─── Scheduled task execution ──────────────────────────────────────────────── @@ -1944,102 +2315,169 @@ export async function executeScheduledTask( refreshCliVersion(bot.config.cliId, bot.config.cliPathOverride); - // Silent fires flip the model's default from "post progress via botmux send" - // to "say nothing unless the alert condition is met". const firePrompt = silent ? `${buildSilentScheduleHint(task.name, localeForBot(larkAppId))}\n\n${task.prompt}` : task.prompt; - // Inject into a live session if one already exists at this anchor. - const existing = activeSessions.get(sessionKey(anchor, larkAppId)); - if (isContinuation && existing?.worker && !existing.worker.killed) { - markSessionActivity(existing); - try { - ensureSessionWhiteboard(existing); - if (sharedTopicRootId) { - beginReplyTargetTurn(existing, sharedTopicRootId, scheduledTurnId); - sessionStore.updateSession(existing.session); + const key = sessionKey(anchor, larkAppId); + return withActiveSessionKeyLock(activeSessions, key, async () => { + // Reuse the canonical owner only when it is an actual conversation. A + // worker:null entry is also used for three deliberately non-runnable + // states: first-turn setup, repo selection, and dashboard backlog. Waking + // any of those here would let the scheduled prompt overtake (or replace) + // the opening prompt that owns the reservation. + const existing = activeSessions.get(key); + if (existing) { + const reservedState = existing.pendingRepo + ? 'pending_repo' + : existing.initialStartPending + ? 'initial_start_pending' + : existing.worktreeCreating + ? 'worktree_creating' + : existing.session.queued + ? 'queued_backlog' + : undefined; + if (reservedState) { + throw new Error( + `Scheduled task ${task.id} found owner ${existing.session.sessionId} ` + + `in ${reservedState}; preserving its opening prompt`, + ); } - const input = buildFollowUpCliInput(firePrompt, existing.session.sessionId, { - isAdoptMode: false, - cliId: existing.session.cliId ?? bot.config.cliId, - cliPathOverride: existing.session.cliPathOverride ?? bot.config.cliPathOverride, - locale: localeForBot(larkAppId), - larkAppId, - chatId: task.chatId, - whiteboardId: existing.session.whiteboardId, - }); - rememberLastCliInput(existing, task.prompt, input); - if (silent) armSilentScheduledTurn(existing, scheduledTurnId); - if (!sendWorkerInput(existing, input, scheduledTurnId)) { - if (silent) disarmSilentScheduledTurn(existing, scheduledTurnId); - throw new Error('worker unavailable'); + const retirementPhase = riffRetirementAdmissionPhase(existing); + if (retirementPhase) { + throw new Error( + `Scheduled task ${task.id} found owner ${existing.session.sessionId} ` + + `in ${retirementPhase}; input was not accepted`, + ); + } + if (hasProtectedSessionMutationOwnership(existing)) { + throw new Error( + `Scheduled task ${task.id} found durable activation owner ` + + `${existing.session.sessionId}; preserving it for recovery`, + ); + } + + if (isRelayableRealSession(existing) || !!existing.session.riffParentTaskId) { + markSessionActivity(existing); + ensureSessionWhiteboard(existing); + if (sharedTopicRootId) { + beginReplyTargetTurn(existing, sharedTopicRootId, scheduledTurnId); + sessionStore.updateSession(existing.session); + } + const input = buildFollowUpCliInput(firePrompt, existing.session.sessionId, { + isAdoptMode: false, + cliId: existing.session.cliId ?? bot.config.cliId, + cliPathOverride: existing.session.cliPathOverride ?? bot.config.cliPathOverride, + locale: localeForBot(larkAppId), + larkAppId, + chatId: task.chatId, + whiteboardId: existing.session.whiteboardId, + }); + if (existing.worker && !existing.worker.killed) { + if (silent) armSilentScheduledTurn(existing, scheduledTurnId); + if (!sendWorkerInput(existing, input, scheduledTurnId)) { + if (silent) disarmSilentScheduledTurn(existing, scheduledTurnId); + throw new Error( + `Scheduled task ${task.id} worker refused input before acceptance`, + ); + } + rememberLastCliInput(existing, task.prompt, input); + } else { + rememberLastCliInput(existing, task.prompt, input); + if (silent) armSilentScheduledTurn(existing, scheduledTurnId); + try { + forkWorker(existing, input, { resume: existing.hasHistory, turnId: scheduledTurnId }); + } catch (err) { + if (silent) disarmSilentScheduledTurn(existing, scheduledTurnId); + throw err; + } + } + logger.info(`[scheduler] Task "${task.name}" routed to existing session ${existing.session.sessionId}`); + return; + } + + // Daemon-command scratches have no CLI history and no pending user + // intent. Retire the exact scratch before creating the scheduled + // conversation; never fork the scratch as if it were resumable. + const closeResult = await closeSession(existing.session.sessionId); + if (!closeResult.ok) { + throw new Error( + `Scheduled task ${task.id} could not retire owner ${existing.session.sessionId}: ` + + `${closeResult.error}; preserving its route for retry`, + ); + } + if (activeSessions.get(key) === existing) activeSessions.delete(key); + const successor = activeSessions.get(key); + if (successor) { + throw new Error( + `Scheduled task ${task.id} lost anchor ${key} to ${successor.session.sessionId} ` + + 'while retiring a scratch owner', + ); } - logger.info(`[scheduler] Task "${task.name}" injected into live session ${existing.session.sessionId}${silent ? ' (silent)' : ''}`); - return; - } catch (err: any) { - if (silent) disarmSilentScheduledTurn(existing, scheduledTurnId); - logger.warn(`[scheduler] Failed to inject into live session (${err.message}); spawning fresh worker`); } - } - // Spawn a fresh session bound to the chosen anchor. + // Spawn a fresh session bound to the chosen anchor. // Thread-scope: rootMessageId = anchor. Chat-scope: rootMessageId stores the - // chatId-as-seed for audit (sessionAnchorId() returns chatId via scope). If a - // formerly chat-scope task was redirected into a converted topic chat, promote - // the runtime session to thread-scope so follow-up replies stay in-thread. - const deferredFreshTopic = executionPosition === 'new-topic' && silent; - const runtimeScope: 'thread' | 'chat' = deferredFreshTopic - ? 'chat' - : scope === 'chat' && anchor !== task.chatId ? 'thread' : scope; - const session = sessionStore.createSession(task.chatId, anchor, `${t('schedule.title_prefix', undefined, localeForBot(larkAppId))} ${task.name}`, task.chatType === 'p2p' ? 'p2p' : 'group'); - const now = Date.now(); - session.larkAppId = larkAppId; - session.scope = runtimeScope; - if (deferredFreshTopic) { - session.deferredScheduleRun = { - taskId: task.id, - turnId: scheduledTurnId, - routingAnchor: anchor, - ...(task.topicTitle?.trim() ? { topicTitle: task.topicTitle.trim() } : {}), - createdAt: new Date(now).toISOString(), - }; - } - session.lastMessageAt = new Date(now).toISOString(); - sessionStore.updateSession(session); - messageQueue.ensureQueue(anchor); + // chatId-as-seed for audit (sessionAnchorId() returns chatId via scope). If a + // formerly chat-scope task was redirected into a converted topic chat, promote + // the runtime session to thread-scope so follow-up replies stay in-thread. + const deferredFreshTopic = executionPosition === 'new-topic' && silent; + const runtimeScope: 'thread' | 'chat' = deferredFreshTopic + ? 'chat' + : scope === 'chat' && anchor !== task.chatId ? 'thread' : scope; + const session = sessionStore.createSession(task.chatId, anchor, `${t('schedule.title_prefix', undefined, localeForBot(larkAppId))} ${task.name}`, task.chatType === 'p2p' ? 'p2p' : 'group'); + const now = Date.now(); + session.larkAppId = larkAppId; + session.scope = runtimeScope; + if (deferredFreshTopic) { + session.deferredScheduleRun = { + taskId: task.id, + turnId: scheduledTurnId, + routingAnchor: anchor, + ...(task.topicTitle?.trim() ? { topicTitle: task.topicTitle.trim() } : {}), + createdAt: new Date(now).toISOString(), + }; + } + session.lastMessageAt = new Date(now).toISOString(); + sessionStore.updateSession(session); + messageQueue.ensureQueue(anchor); - const ds: DaemonSession = { - session, - worker: null, - workerPort: null, - workerToken: null, - larkAppId, - chatId: task.chatId, - chatType: task.chatType === 'p2p' ? 'p2p' : 'group', - scope: runtimeScope, - spawnedAt: sessionCreatedAtMs(session), - cliVersion: getCurrentCliVersion(), - lastMessageAt: now, - hasHistory: isContinuation, - workingDir: task.workingDir, - }; - if (sharedTopicRootId) { - beginReplyTargetTurn(ds, sharedTopicRootId, scheduledTurnId); - sessionStore.updateSession(ds.session); - } - ensureSessionWhiteboard(ds); - const prompt = buildNewTopicCliInput(firePrompt, session.sessionId, bot.config.cliId, bot.config.cliPathOverride, undefined, undefined, undefined, undefined, { name: bot.botName, openId: bot.botOpenId }, localeForBot(larkAppId), undefined, { larkAppId, chatId: task.chatId, whiteboardId: ds.session.whiteboardId }); - activeSessions.set(sessionKey(anchor, larkAppId), ds); - rememberLastCliInput(ds, task.prompt, prompt); - if (silent) armSilentScheduledTurn(ds, scheduledTurnId); - try { - forkWorker(ds, prompt, scheduledTurnId); - } catch (err) { - if (silent) disarmSilentScheduledTurn(ds, scheduledTurnId); - throw err; - } + const ds: DaemonSession = { + session, + worker: null, + workerPort: null, + workerToken: null, + larkAppId, + chatId: task.chatId, + chatType: task.chatType === 'p2p' ? 'p2p' : 'group', + scope: runtimeScope, + spawnedAt: sessionCreatedAtMs(session), + cliVersion: getCurrentCliVersion(), + lastMessageAt: now, + hasHistory: isContinuation, + workingDir: task.workingDir, + initialStartPending: true, + pendingPrompt: firePrompt, + }; + if (sharedTopicRootId) { + beginReplyTargetTurn(ds, sharedTopicRootId, scheduledTurnId); + sessionStore.updateSession(ds.session); + } + ensureSessionWhiteboard(ds); + const prompt = buildNewTopicCliInput(firePrompt, session.sessionId, bot.config.cliId, bot.config.cliPathOverride, undefined, undefined, undefined, undefined, { name: bot.botName, openId: bot.botOpenId }, localeForBot(larkAppId), undefined, { larkAppId, chatId: task.chatId, whiteboardId: ds.session.whiteboardId }); + activeSessions.set(key, ds); + rememberLastCliInput(ds, task.prompt, prompt); + if (silent) armSilentScheduledTurn(ds, scheduledTurnId); + try { + forkWorker(ds, prompt, scheduledTurnId); + } catch (err) { + if (silent) disarmSilentScheduledTurn(ds, scheduledTurnId); + throw err; + } + ds.initialStartPending = false; + ds.pendingPrompt = undefined; - logger.info(`[scheduler] Task "${task.name}" spawned (session: ${session.sessionId}, scope: ${runtimeScope}, anchor: ${anchor}, continuation: ${isContinuation}${silent ? ', silent' : ''})`); + logger.info(`[scheduler] Task "${task.name}" spawned (session: ${session.sessionId}, scope: ${runtimeScope}, anchor: ${anchor}, continuation: ${isContinuation}${silent ? ', silent' : ''})`); + }); } // ─── Dashboard「创建会话」spawn / activate ─────────────────────────────────── @@ -2067,7 +2505,10 @@ function resolveDashboardSpawnWorkingDir(larkAppId: string, chatId: string): str * buildRepoSelectCard（含 worktree）。用户点卡片由 card-handler 的 pendingRepo 分支起 CLI。 * - 没钉也没项目 → 回退用 bot 默认 cwd 直接起。 * userContent 是已按角色包装好的首轮内容（lead 前言等），不论哪条路都原样带过去。 */ -async function forkOrShowRepoCard(ds: DaemonSession, userContent: string): Promise { +async function forkOrShowRepoCard( + ds: DaemonSession, + userContent: string, +): Promise<'forked' | 'pending_repo'> { const larkAppId = ds.larkAppId; const bot = getBot(larkAppId); const locale = localeForBot(larkAppId); @@ -2085,6 +2526,14 @@ async function forkOrShowRepoCard(ds: DaemonSession, userContent: string): Promi const baseDir = ds.workingDir; ds.pendingRepo = true; // router buffers concurrent msgs; commit clears it ds.pendingPrompt = userContent; // folded into the first turn by commitRepoSelection + stagePendingRepoSetup(ds, { + mode: 'auto_worktree', + baseDir, + turnId: ds.currentReplyTarget?.turnId ?? ds.session.rootMessageId, + }); + // Route visibility moves atomically from initial-start reservation to + // pendingRepo before the dynamic import can yield. + ds.initialStartPending = false; // (The pending dashboard row is announced inside runAutoWorktreeCommit so all // three spawn callers get it from one place — no publish needed here.) const { runAutoWorktreeCommit } = await import('../im/lark/card-handler.js'); @@ -2096,14 +2545,14 @@ async function forkOrShowRepoCard(ds: DaemonSession, userContent: string): Promi notify: (m) => sendMessage(larkAppId, ds.chatId, m), }); logger.info(`[createSession] session ${ds.session.sessionId.substring(0, 8)} → pending, building worktree off ${baseDir}`); - return; + return 'pending_repo'; } } const buildPrompt = () => buildNewTopicCliInput( userContent, ds.session.sessionId, bot.config.cliId, bot.config.cliPathOverride, - ds.pendingAttachments, undefined, undefined, undefined, - { name: bot.botName, openId: bot.botOpenId }, locale, undefined, + ds.pendingAttachments, ds.pendingMentions, undefined, ds.pendingFollowUps, + { name: bot.botName, openId: bot.botOpenId }, locale, ds.pendingSender, { larkAppId, chatId: ds.chatId, @@ -2111,6 +2560,8 @@ async function forkOrShowRepoCard(ds: DaemonSession, userContent: string): Promi codexAppText: ds.pendingCodexAppText, codexAppApplicationContext: ds.pendingCodexAppApplicationContext, codexAppMessageContext: ds.pendingCodexAppMessageContext, + codexAppFollowUps: ds.pendingCodexAppFollowUps, + codexAppFollowUpContexts: ds.pendingCodexAppFollowUpContexts, }, ); @@ -2119,24 +2570,56 @@ async function forkOrShowRepoCard(ds: DaemonSession, userContent: string): Promi const scanDirs = getProjectScanDirs(ds).filter(d => existsSync(d)); const projects = scanDirs.length > 0 ? scanMultipleProjects(scanDirs, 3, repoPickerScanOptions()) : []; if (projects.length > 0) { + const card = buildRepoSelectCard(projects, getSessionWorkingDir(ds), ds.chatId, locale, bot.config.worktreeMultiPicker); + const { sendMessage } = await import('../im/lark/client.js'); + ds.pendingRepo = true; + ds.pendingPrompt = userContent; try { - const card = buildRepoSelectCard(projects, getSessionWorkingDir(ds), ds.chatId, locale, bot.config.worktreeMultiPicker); - const { sendMessage } = await import('../im/lark/client.js'); - ds.pendingRepo = true; - ds.pendingPrompt = userContent; - ds.repoCardMessageId = await sendMessage(larkAppId, ds.chatId, card, 'interactive'); - announcePendingRepoSession(ds); - // 弹卡片这条路不经 forkWorker，session.spawned 不会自动发——手动 upsert 一条， - // 让 dashboard 显示这条「待选仓库」会话（in_progress 首次 spawn 走这里才会出现）。 - dashboardEventBus.publish({ type: 'session.spawned', body: { session: composeRowFromActive(ds) } }); - logger.info(`[createSession] repo select card posted for session ${ds.session.sessionId.substring(0, 8)} (${projects.length} projects)`); - return; + stagePendingRepoSetup(ds, { + mode: 'picker', + turnId: ds.currentReplyTarget?.turnId ?? ds.session.rootMessageId, + }); } catch (err) { - // 发卡失败：退回直接起，别把会话卡死在 pendingRepo。 + // Nothing external was published. The journal helper rolled its own + // mutation back, so direct fork remains a safe fallback. ds.pendingRepo = false; ds.pendingPrompt = undefined; ds.repoCardMessageId = undefined; - logger.warn(`[createSession] repo card failed (${(err as Error)?.message ?? err}); forking with default cwd`); + logger.warn(`[createSession] repo setup stage failed (${(err as Error)?.message ?? err}); forking with default cwd`); + } + if (ds.pendingRepo) { + let publishedCardId: string | undefined; + try { + publishedCardId = await sendMessage(larkAppId, ds.chatId, card, 'interactive'); + } catch (err) { + // No picker exists, so the already-durable opening may safely move + // into forkWorker's tokened activation journal. + ds.pendingRepo = false; + ds.repoCardMessageId = undefined; + logger.warn(`[createSession] repo card publish failed (${(err as Error)?.message ?? err}); forking with default cwd`); + } + if (publishedCardId) { + ds.repoCardMessageId = publishedCardId; + try { + persistPendingRepoCardMessageId(ds, publishedCardId); + } catch (err) { + // The card is already visible. Keep its runtime identity and the + // durable setup owner fail-closed; restart will publish and persist + // a fresh picker instead of starting the CLI behind this card. + logger.error( + `[createSession] repo card id persistence failed for ${ds.session.sessionId.substring(0, 8)}; ` + + `retaining pending picker ${publishedCardId}: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + } + ds.initialStartPending = false; + announcePendingRepoSession(ds); + // 弹卡片这条路不经 forkWorker，session.spawned 不会自动发——手动 upsert 一条， + // 让 dashboard 显示这条「待选仓库」会话（in_progress 首次 spawn 走这里才会出现）。 + dashboardEventBus.publish({ type: 'session.spawned', body: { session: composeRowFromActive(ds) } }); + logger.info(`[createSession] repo select card posted for session ${ds.session.sessionId.substring(0, 8)} (${projects.length} projects)`); + return 'pending_repo'; + } } } } @@ -2145,10 +2628,24 @@ async function forkOrShowRepoCard(ds: DaemonSession, userContent: string): Promi const prompt = buildPrompt(); rememberLastCliInput(ds, userContent, prompt); forkWorker(ds, prompt); - ds.pendingCodexAppText = undefined; - ds.pendingCodexAppApplicationContext = undefined; - ds.pendingCodexAppMessageContext = undefined; - ds.pendingAttachments = undefined; + // forkWorker pre-accept is synchronous. Keep the reservation and all input + // buffers intact if it throws; only expose the normal worker state after it + // has accepted the first turn. + if (!ds.session.queuedActivationPending) { + ds.initialStartPending = false; + ds.pendingPrompt = undefined; + ds.pendingCodexAppText = undefined; + ds.pendingCodexAppApplicationContext = undefined; + ds.pendingCodexAppMessageContext = undefined; + ds.pendingAttachments = undefined; + ds.pendingMentions = undefined; + ds.pendingSender = undefined; + ds.pendingFollowUps = undefined; + ds.pendingFollowUpTurnIds = undefined; + ds.pendingCodexAppFollowUps = undefined; + ds.pendingCodexAppFollowUpContexts = undefined; + } + return 'forked'; } export interface SpawnDashboardSessionArgs { @@ -2193,8 +2690,12 @@ export async function spawnDashboardSession( // chat-scope：锚点就是 chatId。先挡掉「同群同 bot 已有真会话」的撞键（会被 // Map.set 覆盖而泄漏 worker）。queued 占位 / 纯 scratch 不算冲突。 const anchor = chatId; - const existing = activeSessions.get(sessionKey(anchor, larkAppId)); - if (existing && (existing.worker || existing.session.queued || isRelayableRealSession(existing))) { + const key = sessionKey(anchor, larkAppId); + const existing = activeSessions.get(key); + if (existing && (existing.worker || existing.session.queued || existing.pendingRepo + || existing.initialStartPending || existing.worktreeCreating || isRelayableRealSession(existing) + || !!existing.session.riffParentTaskId + || hasProtectedSessionMutationOwnership(existing))) { return { ok: false, error: 'session_exists' }; } @@ -2230,65 +2731,133 @@ export async function spawnDashboardSession( const userContent = composeSpawnUserContent({ content, role, coworkers: args.coworkers, locale }); const codexAppMessageContext = composeSpawnCodexAppContext({ role, coworkers: args.coworkers, locale }); - const resolvedTitle = args.title || deriveSessionTitleFromContent(content); - const session = sessionStore.createSession(chatId, bannerMessageId ?? chatId, resolvedTitle, 'group'); - const now = Date.now(); - session.larkAppId = larkAppId; - session.scope = 'chat'; - session.ownerOpenId = args.ownerOpenId ?? getOwnerOpenId(larkAppId); - session.creatorOpenId = session.ownerOpenId; - if (args.ownerUnionId) session.ownerUnionId = args.ownerUnionId; - session.lastMessageAt = new Date(now).toISOString(); - if (args.attachments?.length) session.dashboardAttachments = args.attachments; - if (column === 'backlog') { - session.queued = true; - session.queuedPrompt = userContent; - session.queuedCodexAppText = content; - session.queuedCodexAppMessageContext = codexAppMessageContext; - session.queuedAttachments = args.attachments; - session.kanbanColumn = 'backlog'; - } + const registered = await withActiveSessionKeyLock(activeSessions, key, async () => { + const current = activeSessions.get(key); + if (current) { + const protectedOwner = current.worker || current.session.queued || current.pendingRepo + || current.initialStartPending || current.worktreeCreating + || isRelayableRealSession(current) + || !!current.session.riffParentTaskId + || hasProtectedSessionMutationOwnership(current); + if (protectedOwner) return undefined; + await closeSession(current.session.sessionId); + } - // 钉 workingDir：oncall 绑定（弹框填了目录）/ bot 默认。都没有 → undefined，激活/开跑时 - // 会弹 /repo 卡片让用户在群里选仓库（复用普通新话题逻辑）。 - const workingDir = resolveDashboardSpawnWorkingDir(larkAppId, chatId); - if (workingDir) session.workingDir = workingDir; - sessionStore.updateSession(session); - messageQueue.ensureQueue(anchor); + const resolvedTitle = args.title || deriveSessionTitleFromContent(content); + const session = sessionStore.createSession(chatId, bannerMessageId ?? chatId, resolvedTitle, 'group'); + const now = Date.now(); + session.larkAppId = larkAppId; + session.scope = 'chat'; + session.ownerOpenId = args.ownerOpenId ?? getOwnerOpenId(larkAppId); + session.creatorOpenId = session.ownerOpenId; + if (args.ownerUnionId) session.ownerUnionId = args.ownerUnionId; + session.lastMessageAt = new Date(now).toISOString(); + if (args.attachments?.length) session.dashboardAttachments = args.attachments; + if (column === 'backlog') { + session.queued = true; + session.queuedPrompt = userContent; + session.queuedCodexAppText = content; + session.queuedCodexAppMessageContext = codexAppMessageContext; + if (args.attachments?.length) session.queuedAttachments = args.attachments; + session.kanbanColumn = 'backlog'; + } - const ds: DaemonSession = { - session, - worker: null, - workerPort: null, - workerToken: null, - larkAppId, - chatId, - chatType: 'group', - scope: 'chat', - spawnedAt: sessionCreatedAtMs(session), - cliVersion: getCurrentCliVersion(), - lastMessageAt: now, - hasHistory: false, - workingDir, - ownerOpenId: session.ownerOpenId, - currentTurnTitle: resolvedTitle, - pendingCodexAppText: content, - pendingCodexAppMessageContext: codexAppMessageContext, - pendingAttachments: args.attachments, - }; - activeSessions.set(sessionKey(anchor, larkAppId), ds); + const workingDir = resolveDashboardSpawnWorkingDir(larkAppId, chatId); + if (workingDir) session.workingDir = workingDir; + sessionStore.updateSession(session); + messageQueue.ensureQueue(anchor); + + const ds: DaemonSession = { + session, + worker: null, + workerPort: null, + workerToken: null, + larkAppId, + chatId, + chatType: 'group', + scope: 'chat', + spawnedAt: sessionCreatedAtMs(session), + cliVersion: getCurrentCliVersion(), + lastMessageAt: now, + hasHistory: false, + workingDir, + ownerOpenId: session.ownerOpenId, + currentTurnTitle: resolvedTitle, + pendingCodexAppText: content, + pendingCodexAppMessageContext: codexAppMessageContext, + pendingAttachments: args.attachments, + }; + if (column !== 'backlog') { + ds.initialStartPending = true; + ds.pendingPrompt = userContent; + } + activeSessions.set(key, ds); + if (column === 'backlog') { + ds.pendingPrompt = userContent; + dashboardEventBus.publish({ type: 'session.spawned', body: { session: composeRowFromActive(ds) } }); + } else { + // Keep the key reservation through initial worker/repo-picker + // installation. Otherwise a concurrent creator can classify this brief + // worker:null interval as disposable scratch, replace it, and let this + // caller resume by forking an orphan. + try { + await forkOrShowRepoCard(ds, userContent); + } catch (err) { + const durableOwner = hasProtectedSessionMutationOwnership(ds.session); + const liveWorkerOwner = !!ds.worker && !ds.worker.killed; + if (durableOwner || liveWorkerOwner) { + // `forkOrShowRepoCard` can fail after the repo/setup journal was + // durably staged (for example picker publish fallback followed by a + // fork rejection). That journal is now the exact opening owner; never + // erase it in the generic spawn rollback. Rebuild volatile picker + // buffers where possible and leave this exact map row active/retryable. + if (!ds.session.queuedActivationPending && ds.session.pendingRepoSetup) { + restorePendingRepoRuntime(ds); + } + ds.initialStartPending = ds.session.queuedActivationPending === true + || (ds.session.queuedActivationTail?.length ?? 0) > 0; + // The create IPC reports failure, but this durable row is still the + // authoritative retry owner. Upsert it so dashboard clients do not + // see an invisible occupied chat until the next full hydrate. + announceSessionRow(ds); + logger.error( + `[createSession] opening failed after durable ownership transfer for ${session.sessionId}; ` + + `retaining exact active owner: ${err instanceof Error ? err.message : String(err)}`, + ); + return { error: err instanceof Error ? err.message : String(err) }; + } + // No durable or live owner accepted the opening. Roll back only our + // exact runtime claim and close only our row; a concurrently published + // successor must survive. + if (activeSessions.get(key) === ds) activeSessions.delete(key); + try { sessionStore.closeSession(session.sessionId); } + catch (closeErr) { + logger.error( + `[createSession] failed to close unaccepted session ${session.sessionId}: ` + + `${closeErr instanceof Error ? closeErr.message : String(closeErr)}`, + ); + } + return { + error: err instanceof Error ? err.message : String(err), + }; + } + } + return { ds, session }; + }); + if (!registered) return { ok: false, error: 'session_exists' }; + if ('error' in registered && typeof registered.error === 'string') { + return { ok: false, error: registered.error }; + } + const { ds, session } = registered; if (column === 'backlog') { // Parked：不起 CLI。手动广播 session.spawned，让 dashboard 立刻显示待办池卡片 // （forkWorker 才会自动发这个事件，parked 路径要自己发）。 - ds.pendingPrompt = userContent; - dashboardEventBus.publish({ type: 'session.spawned', body: { session: composeRowFromActive(ds) } }); logger.info(`[createSession] queued session ${session.sessionId.substring(0, 8)} (bot=${larkAppId}, chat=${chatId}, role=${role})`); return { ok: true, sessionId: session.sessionId }; } // in_progress：立即开跑或弹 /repo 卡片（没钉目录时）。userContent 已按角色包装好。 - await forkOrShowRepoCard(ds, userContent); logger.info(`[createSession] spawned session ${session.sessionId.substring(0, 8)} (bot=${larkAppId}, chat=${chatId}, role=${role}, pendingRepo=${!!ds.pendingRepo})`); return { ok: true, sessionId: session.sessionId }; } @@ -2301,33 +2870,103 @@ export async function activateQueuedSession(ds: DaemonSession): Promise<{ ok: bo return (ds.worker && !ds.worker.killed) ? { ok: true } : { ok: false, error: 'not_queued' }; } if (ds.worker && !ds.worker.killed) { + if (ds.session.queuedActivationPending + || (ds.session.queuedActivationTail?.length ?? 0) > 0 + || ds.session.pendingRepoSetup) { + // A contradictory live+queued snapshot can occur around recovery or a + // stale dashboard action. Never reinterpret the live child as proof that + // its durable activation/setup owner crossed the adapter ACK boundary. + logger.warn( + `[${ds.session.sessionId.substring(0, 8)}] Ignored queued activation cleanup while durable ownership remains`, + ); + return { ok: true }; + } // 不该发生（queued 一定 worker:null），但保险：清标记即可。 ds.session.queued = false; ds.session.queuedPrompt = undefined; ds.session.queuedCodexAppText = undefined; ds.session.queuedCodexAppMessageContext = undefined; ds.session.queuedAttachments = undefined; + ds.session.queuedActivationPending = undefined; + ds.session.queuedActivationToken = undefined; + ds.session.queuedActivationInput = undefined; + ds.session.queuedActivationTurnId = undefined; + ds.session.queuedActivationDispatchAttempt = undefined; + ds.session.queuedActivationResume = undefined; + ds.session.pendingRepoSetup = undefined; sessionStore.updateSession(ds.session); return { ok: true }; } - const content = ds.session.queuedPrompt ?? ds.pendingPrompt ?? ''; - // A parked dashboard task may have crossed a daemon restart. Restore the - // persisted clean-input sidecar before clearing the durable backlog fields; - // forkOrShowRepoCard will carry it through either immediate fork or /repo. - ds.pendingCodexAppText ??= ds.session.queuedCodexAppText; - ds.pendingCodexAppMessageContext ??= ds.session.queuedCodexAppMessageContext; - ds.pendingAttachments ??= ds.session.queuedAttachments; - ds.session.queued = false; - ds.session.queuedPrompt = undefined; - ds.session.queuedCodexAppText = undefined; - ds.session.queuedCodexAppMessageContext = undefined; - ds.session.queuedAttachments = undefined; - ds.pendingPrompt = undefined; - // 激活即视为开始：从待办池挪到进行中，让卡片归位。 - if (ds.session.kanbanColumn === 'backlog') ds.session.kanbanColumn = 'in_progress'; - sessionStore.updateSession(ds.session); - // 起会话或弹 /repo 卡片（没钉目录时）。content 已是包装好的首轮内容。 - await forkOrShowRepoCard(ds, content); - logger.info(`[createSession] activated queued session ${ds.session.sessionId.substring(0, 8)} (bot=${ds.larkAppId}, pendingRepo=${!!ds.pendingRepo})`); - return { ok: true }; + // Repo selection / auto-worktree already owns this activation attempt. Keep + // the durable queued payload as its crash-recovery journal and make repeated + // dashboard starts idempotent instead of posting a second picker/build. + if (ds.pendingRepo) return { ok: true }; + const activate = async (): Promise<{ ok: boolean; error?: string }> => { + if (!ds.session.queued) { + return (ds.worker && !ds.worker.killed) ? { ok: true } : { ok: false, error: 'not_queued' }; + } + const content = ds.session.queuedPrompt ?? ds.pendingPrompt ?? ''; + // Preserve the durable queued payload until fork or pendingRepo setup has + // succeeded. Ordinary inbound routing does not take the key lock, so the + // runtime reservation must also be visible throughout every await. + ds.initialStartPending = true; + ds.pendingPrompt = content; + ds.pendingCodexAppText ??= ds.session.queuedCodexAppText; + ds.pendingCodexAppMessageContext ??= ds.session.queuedCodexAppMessageContext; + ds.pendingAttachments ??= ds.session.queuedAttachments; + let outcome: 'forked' | 'pending_repo'; + try { + const exactRetry = ds.session.queuedActivationInput; + if (exactRetry) { + forkWorker(ds, exactRetry, { + resume: ds.session.queuedActivationResume ?? ds.hasHistory, + turnId: ds.session.queuedActivationTurnId, + dispatchAttempt: ds.session.queuedActivationDispatchAttempt, + }); + outcome = 'forked'; + } else { + outcome = await forkOrShowRepoCard(ds, content); + } + } catch (err) { + // queued remains authoritative and retryable. Undo only transient route + // state installed by this failed attempt; never discard the opening + // prompt/sidecar before a worker or repo picker accepted it. + ds.initialStartPending = false; + ds.pendingRepo = false; + ds.repoCardMessageId = undefined; + ds.pendingPrompt = content; + return { ok: false, error: (err as Error)?.message ?? 'start_failed' }; + } + + // Ownership has transferred to a worker or pending-repo flow. Nothing + // below this point may advertise a retryable activation failure. + if (outcome === 'forked') { + ds.session.queued = false; + ds.session.queuedAttachments = undefined; + } + // pending_repo deliberately retains queued + queuedPrompt: pendingPrompt + // is runtime-only, so clearing the durable copy here would lose the first + // turn if the daemon dies before repo selection/worktree commit forks. + if (ds.session.kanbanColumn === 'backlog') ds.session.kanbanColumn = 'in_progress'; + try { + sessionStore.updateSession(ds.session); + } catch (err) { + logger.error( + `[createSession] queued activation metadata persistence failed after ownership transfer: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + } + logger.info(`[createSession] activated queued session ${ds.session.sessionId.substring(0, 8)} (bot=${ds.larkAppId}, pendingRepo=${!!ds.pendingRepo})`); + return { ok: true }; + }; + + const registry = getActiveSessionsRegistry(); + if (!registry) return activate(); + const key = activeSessionKey(ds); + return withActiveSessionKeyLock(registry, key, async () => { + if (registry.get(key) !== ds || ds.session.status !== 'active') { + return { ok: false, error: 'session_not_active' }; + } + return activate(); + }); } diff --git a/src/core/session-marker.ts b/src/core/session-marker.ts index 32deba066..0e9989e5b 100644 --- a/src/core/session-marker.ts +++ b/src/core/session-marker.ts @@ -239,6 +239,7 @@ export function resolveSessionContext( dataDir: string, envSessionId: string | undefined, startPid: number = process.ppid, + originChannelId: string | undefined = process.env.BOTMUX_ORIGIN_CHANNEL_ID, ): AncestorSessionContext | null { const fromMarker = findAncestorSessionContext(dataDir, startPid); // A live process-tree marker is the primary source whenever it is visible. @@ -252,7 +253,7 @@ export function resolveSessionContext( // as daemon authority. It is consulted only when no live marker is visible, // so fields from two generations are never mixed. const protectedClaim = envSessionId - ? readManagedOriginCapability(dataDir, envSessionId) + ? readManagedOriginCapability(dataDir, envSessionId, undefined, originChannelId) : null; if (protectedClaim) { return { diff --git a/src/core/session-mutation-guard.ts b/src/core/session-mutation-guard.ts new file mode 100644 index 000000000..48b5bde30 --- /dev/null +++ b/src/core/session-mutation-guard.ts @@ -0,0 +1,61 @@ +import type { Session } from '../types.js'; +import { hasUnsettledCodexAppDispatch } from '../utils/codex-app-dispatch-ledger.js'; + +type ProtectedSessionMutationState = Pick< + Session, + | 'codexAppDispatchLedger' + | 'queued' + | 'queuedActivationPending' + | 'queuedActivationTail' + | 'pendingRepoSetup' +>; + +type ProtectedRuntimeMutationState = { + session: ProtectedSessionMutationState; + initialStartPending?: boolean; + riffCloseState?: unknown; + riffShutdownState?: unknown; +}; + +export type ProtectedSessionMutationReason = + | 'codex_app_dispatch' + | 'queued_todo' + | 'activation_head' + | 'activation_tail' + | 'repository_setup' + | 'initial_start' + | 'riff_close' + | 'riff_shutdown'; + +export function protectedSessionMutationReasons( + value: ProtectedRuntimeMutationState | ProtectedSessionMutationState, +): ProtectedSessionMutationReason[] { + const ds: ProtectedRuntimeMutationState | undefined = 'session' in value + ? value + : undefined; + const session: ProtectedSessionMutationState = ds + ? ds.session + : value as ProtectedSessionMutationState; + const reasons: ProtectedSessionMutationReason[] = []; + if (hasUnsettledCodexAppDispatch(session.codexAppDispatchLedger)) reasons.push('codex_app_dispatch'); + if (session.queued === true) reasons.push('queued_todo'); + if (session.queuedActivationPending === true) reasons.push('activation_head'); + if ((session.queuedActivationTail?.length ?? 0) > 0) reasons.push('activation_tail'); + if (session.pendingRepoSetup !== undefined) reasons.push('repository_setup'); + if (ds?.initialStartPending === true) reasons.push('initial_start'); + if (ds?.riffCloseState !== undefined) reasons.push('riff_close'); + if (ds?.riffShutdownState !== undefined) reasons.push('riff_shutdown'); + return reasons; +} + +/** + * True while a session owns work that an ordinary config/restart/switch + * mutation is not authorized to abandon. Explicit close remains the escape + * hatch. Keep this backend-neutral: the activation journal protects pty and + * Riff submissions just as the Codex App ledger protects accepted dispatches. + */ +export function hasProtectedSessionMutationOwnership( + value: ProtectedRuntimeMutationState | ProtectedSessionMutationState, +): boolean { + return protectedSessionMutationReasons(value).length > 0; +} diff --git a/src/core/shutdown-budgets.ts b/src/core/shutdown-budgets.ts new file mode 100644 index 000000000..cff6eae3a --- /dev/null +++ b/src/core/shutdown-budgets.ts @@ -0,0 +1,48 @@ +/** + * Graceful-shutdown budgets are shared with the CLI/PM2 supervisor. Keep the + * ordering monotonic: a Riff create/follow-up fetch is bounded at 10s, + * admission restoration after a refused prepare at 11s, and ordinary worker + * exit at 3s. Shutdown never cancels accepted Riff work as a fallback. + */ +export const RIFF_SHUTDOWN_DRAIN_TIMEOUT_MS = 12_000; +/** Admission restoration can wait for the same bounded 10s create/follow-up + * that prepare was draining. The daemon keeps its retirement fence throughout. */ +export const RIFF_ADMISSION_RESTORE_TIMEOUT_MS = 11_000; +/** Bounded acquisition of the bot-wide mutation lease. A timed-out waiter is + * removed and can never run after shutdown has already been refused. */ +export const BOT_TURN_MUTATION_SHUTDOWN_ACQUIRE_TIMEOUT_MS = 1_000; +/** Initial all-owner snapshot and phase-2 batch CAS each use one short lock. */ +export const RIFF_SHUTDOWN_INITIAL_SNAPSHOT_TIMEOUT_MS = 1_000; +export const RIFF_SHUTDOWN_BATCH_PERSIST_TIMEOUT_MS = 1_000; +/** Scheduling/logging slack inside the supervisor-visible daemon budget. */ +export const DAEMON_SHUTDOWN_OVERHEAD_MS = 2_000; +export const DAEMON_WORKER_EXIT_GRACE_MS = 3_000; +export const DAEMON_SHUTDOWN_MAX_MS = + BOT_TURN_MUTATION_SHUTDOWN_ACQUIRE_TIMEOUT_MS + + RIFF_SHUTDOWN_INITIAL_SNAPSHOT_TIMEOUT_MS + + RIFF_SHUTDOWN_DRAIN_TIMEOUT_MS + + RIFF_SHUTDOWN_BATCH_PERSIST_TIMEOUT_MS + + Math.max(RIFF_ADMISSION_RESTORE_TIMEOUT_MS, DAEMON_WORKER_EXIT_GRACE_MS) + + DAEMON_SHUTDOWN_OVERHEAD_MS; +export const PM2_DAEMON_KILL_TIMEOUT_MS = 29_000; +export const PM2_DAEMON_RESTART_DELAY_MS = 3_000; +/** A full restart-delay plus projection jitter. The fleet helper must observe + * this quiet window after every signalled generation exits. */ +export const FLEET_SUCCESSOR_SETTLE_MS = PM2_DAEMON_RESTART_DELAY_MS + 500; +export const FLEET_DAEMON_EXIT_WAIT_MS = 60_000; + +if (PM2_DAEMON_KILL_TIMEOUT_MS <= DAEMON_SHUTDOWN_MAX_MS) { + throw new Error('PM2 daemon kill timeout must exceed the complete daemon shutdown budget'); +} +if (DAEMON_SHUTDOWN_MAX_MS > 28_000) { + throw new Error('complete daemon shutdown budget must remain at or below 28 seconds'); +} +if (FLEET_DAEMON_EXIT_WAIT_MS <= PM2_DAEMON_KILL_TIMEOUT_MS) { + throw new Error('fleet restart wait must exceed the PM2 daemon kill timeout'); +} +if (FLEET_DAEMON_EXIT_WAIT_MS <= DAEMON_SHUTDOWN_MAX_MS + FLEET_SUCCESSOR_SETTLE_MS) { + throw new Error('fleet restart wait must cover daemon shutdown plus successor quiet window'); +} +if (FLEET_DAEMON_EXIT_WAIT_MS <= PM2_DAEMON_KILL_TIMEOUT_MS + FLEET_SUCCESSOR_SETTLE_MS) { + throw new Error('fleet restart wait must cover PM2 kill timeout plus successor quiet window'); +} diff --git a/src/core/supervisor-shutdown-ipc.ts b/src/core/supervisor-shutdown-ipc.ts new file mode 100644 index 000000000..d00dff066 --- /dev/null +++ b/src/core/supervisor-shutdown-ipc.ts @@ -0,0 +1,20 @@ +export const SUPERVISOR_SHUTDOWN_ROUTE = '/__supervisor-ipc/v1/shutdown'; + +export interface SupervisorShutdownIdentity { + larkAppId: string; + bootInstanceId: string; + processStartIdentity: string; +} + +export interface SupervisorShutdownRequest extends SupervisorShutdownIdentity {} + +export function isExactSupervisorShutdownRequest( + identity: SupervisorShutdownIdentity, + value: unknown, +): value is SupervisorShutdownRequest { + if (!value || typeof value !== 'object' || Array.isArray(value)) return false; + const record = value as Record; + return record.larkAppId === identity.larkAppId + && record.bootInstanceId === identity.bootInstanceId + && record.processStartIdentity === identity.processStartIdentity; +} diff --git a/src/core/supervisor-shutdown-protocol.ts b/src/core/supervisor-shutdown-protocol.ts new file mode 100644 index 000000000..625e88771 --- /dev/null +++ b/src/core/supervisor-shutdown-protocol.ts @@ -0,0 +1,16 @@ +/** + * Descriptor capability required before a supervisor may signal a live daemon. + * Bump this exact value whenever shutdown safety depends on a protocol that an + * already-running older daemon does not implement. + */ +export const SUPERVISOR_SHUTDOWN_PROTOCOL = 'riff-fleet-prepare-persist-commit-exit42-v2' as const; + +/** + * PM2 normalizes signal-only child exits to code 0 (`code || 0`) before it + * evaluates `stop_exit_codes`. Zero therefore cannot prove that the daemon + * completed the protocol above: SIGKILL/OOM may look identical. Only the + * successful end of daemon.shutdown() exits with this reserved non-zero code. + */ +export const DAEMON_GRACEFUL_EXIT_CODE = 42 as const; + +export type SupervisorShutdownProtocol = typeof SUPERVISOR_SHUTDOWN_PROTOCOL; diff --git a/src/core/trigger-session.ts b/src/core/trigger-session.ts index 8bf5204f7..61ccd9815 100644 --- a/src/core/trigger-session.ts +++ b/src/core/trigger-session.ts @@ -9,13 +9,22 @@ import { localeForBot, t } from '../i18n/index.js'; import { validateWorkingDir } from './working-dir.js'; import { buildFollowUpCliInput, buildNewTopicCliInput, ensureSessionWhiteboard, getAvailableBots, rememberLastCliInput } from './session-manager.js'; import { markSessionActivity } from './session-activity.js'; -import { forkWorker, getCurrentCliVersion, sendWorkerInput } from './worker-pool.js'; +import { + forkWorker, + getCurrentCliVersion, + hasQueuedActivationAdmissionGate, + sendWorkerInput, + withActiveSessionKeyLock, +} from './worker-pool.js'; import { botAutoWorktreeEnabled } from '../services/default-worktree.js'; import * as messageQueue from '../services/message-queue.js'; import type { DaemonSession } from './types.js'; -import { sessionKey } from './types.js'; +import { activeSessionKey, sessionKey, riffRetirementAdmissionPhase } from './types.js'; import type { TriggerRequest, TriggerResponse } from '../services/trigger-types.js'; import type { CliTurnPayload } from '../types.js'; +import { withBotTurnAdmission } from './bot-turn-mutation-gate.js'; +import { stagePendingRepoSetup } from './pending-repo-journal.js'; +import { hasProtectedSessionMutationOwnership } from './session-mutation-guard.js'; export interface TriggerSessionDeps { larkAppId: string; @@ -185,7 +194,18 @@ function waitForSessionFinalOutput( resolve({ ok: false, triggerId, errorCode: 'trigger_failed', error: err.message }); }, }); - dispatchTurn(); + try { + dispatchTurn(); + } catch (err) { + clearTimeout(timer); + ds.pendingWaitPromises?.delete(triggerId); + resolve({ + ok: false, + triggerId, + errorCode: 'trigger_failed', + error: err instanceof Error ? err.message : String(err), + }); + } }); } @@ -262,7 +282,7 @@ function buildExistingSessionContent( }); } -export async function triggerSessionTurn( +async function triggerSessionTurnAdmitted( req: TriggerRequest, deps: TriggerSessionDeps, internal?: TriggerSessionInternalOptions, @@ -373,94 +393,156 @@ export async function triggerSessionTurn( }; } - if (ds?.worker && !ds.worker.killed) { + const deliverToExisting = async (target: DaemonSession): Promise => { + const targetKey = activeSessionKey(target); + if (deps.activeSessions.get(targetKey) !== target || target.session.status !== 'active') { + return { + ok: false, + triggerId, + errorCode: 'session_not_found', + error: `active session ownership changed before dispatch: ${target.session.sessionId}`, + }; + } + const workerIsLive = !!target.worker && !target.worker.killed; + const retirementPhase = riffRetirementAdmissionPhase(target); + if (retirementPhase) { + return { + ok: false, + triggerId, + errorCode: 'trigger_failed', + error: `target session ${target.session.sessionId} is not accepting input (${retirementPhase})`, + }; + } + if (!workerIsLive && (target.pendingRepo || target.initialStartPending + || target.worktreeCreating || target.session.queued + || hasProtectedSessionMutationOwnership(target))) { + const state = target.pendingRepo + ? 'pending_repo' + : target.initialStartPending + ? 'initial_start_pending' + : target.worktreeCreating + ? 'worktree_creating' + : target.session.queued + ? 'queued_backlog' + : 'durable_owner'; + return { + ok: false, + triggerId, + errorCode: 'trigger_failed', + error: `target session ${target.session.sessionId} is not runnable (${state}); preserving its opening prompt`, + }; + } const content = buildExistingSessionContent( - ds, prompt, larkAppId, chatId, codexAppText, codexAppApplicationContext, codexAppMessageContext, + target, prompt, larkAppId, chatId, codexAppText, codexAppApplicationContext, codexAppMessageContext, ); - markSessionActivity(ds); - rememberInput(ds, prompt, content); + const queuedBehindActivation = workerIsLive + && hasQueuedActivationAdmissionGate(target); + const recordAcceptedInput = (): void => { + markSessionActivity(target); + rememberInput(target, prompt, content); + }; - if (req.options?.waitForFinalOutput) { - return waitForSessionFinalOutput( - ds, - triggerId, - req.options?.timeoutMs ?? 120_000, - (text) => ({ - ok: true, + if (workerIsLive) { + if (req.options?.waitForFinalOutput) { + return waitForSessionFinalOutput( + target, triggerId, - action: 'completed', - target: { kind: 'turn', sessionId: ds!.session.sessionId, chatId }, - output: { content: text }, - message: 'delivered to existing session and completed', - }), - () => { - const dispatchAttempt = prepareStableDispatch(ds!, false); - armFinalOutputSuppression(ds!, dispatchAttempt); - sendWorkerInput(ds!, content, triggerId, { - ...(dispatchAttempt !== undefined ? { dispatchAttempt } : {}), - }); - }, - ); - } - - if (req.options?.asyncReturnSessionId) { - beginAsyncTrigger(ds, triggerId); - const dispatchAttempt = prepareStableDispatch(ds, false); - armFinalOutputSuppression(ds, dispatchAttempt); - sendWorkerInput(ds, content, triggerId, { + req.options?.timeoutMs ?? 120_000, + (text) => ({ + ok: true, + triggerId, + action: 'completed', + target: { kind: 'turn', sessionId: target.session.sessionId, chatId }, + output: { content: text }, + message: 'delivered to existing session and completed', + }), + () => { + const dispatchAttempt = prepareStableDispatch(target, false); + armFinalOutputSuppression(target, dispatchAttempt); + const accepted = sendWorkerInput(target, content, triggerId, { + ...(dispatchAttempt !== undefined ? { dispatchAttempt } : {}), + }); + if (!accepted) throw new Error('worker refused trigger input before acceptance'); + recordAcceptedInput(); + }, + ); + } + + if (req.options?.asyncReturnSessionId) { + beginAsyncTrigger(target, triggerId); + const dispatchAttempt = prepareStableDispatch(target, false); + armFinalOutputSuppression(target, dispatchAttempt); + const accepted = sendWorkerInput(target, content, triggerId, { + ...(dispatchAttempt !== undefined ? { dispatchAttempt } : {}), + }); + if (!accepted) { + target.asyncTriggerResults?.delete(triggerId); + if (target.latestAsyncTriggerId === triggerId) target.latestAsyncTriggerId = undefined; + return { + ok: false, + triggerId, + errorCode: 'trigger_failed', + error: 'worker refused async trigger input before acceptance', + }; + } + recordAcceptedInput(); + return buildAsyncQueuedResponse( + triggerId, + target.session.sessionId, + chatId, + 'delivered to existing session; poll by sessionId or triggerId for final output', + ); + } + + const dispatchAttempt = prepareStableDispatch(target, false); + armFinalOutputSuppression(target, dispatchAttempt); + const accepted = sendWorkerInput(target, content, stableTurnId ? triggerId : undefined, { ...(dispatchAttempt !== undefined ? { dispatchAttempt } : {}), }); - return buildAsyncQueuedResponse( + if (!accepted) { + return { + ok: false, + triggerId, + errorCode: 'trigger_failed', + error: 'worker refused trigger input before acceptance', + }; + } + recordAcceptedInput(); + return { + ok: true, triggerId, - ds.session.sessionId, - chatId, - 'delivered to existing session; poll by sessionId or triggerId for final output', - ); + action: queuedBehindActivation ? 'queued' : 'delivered', + target: { kind: 'turn', sessionId: target.session.sessionId, chatId }, + message: queuedBehindActivation + ? 'durably queued behind the existing activation' + : 'delivered to existing session', + }; } - const dispatchAttempt = prepareStableDispatch(ds, false); - armFinalOutputSuppression(ds, dispatchAttempt); - sendWorkerInput(ds, content, stableTurnId ? triggerId : undefined, { - ...(dispatchAttempt !== undefined ? { dispatchAttempt } : {}), - }); - return { - ok: true, - triggerId, - action: 'delivered', - target: { kind: 'turn', sessionId: ds.session.sessionId, chatId }, - message: 'delivered to existing session', - }; - } - - // An explicit session target stays bound to that session even while its - // worker is dormant. The old rootMessageId-only condition accidentally fell - // through to createSession for chat-scope sessions, which is unsafe for a - // durable meeting receiver whose projection pins one receiverSessionId. - if (ds) { - const content = buildExistingSessionContent( - ds, prompt, larkAppId, chatId, codexAppText, codexAppApplicationContext, codexAppMessageContext, - ); - markSessionActivity(ds); - rememberInput(ds, prompt, content); + recordAcceptedInput(); + // An explicit session target stays bound to that session even while its + // worker is dormant. The old rootMessageId-only condition accidentally + // fell through to createSession for chat-scope sessions, which is unsafe + // for a durable meeting receiver whose projection pins one receiver id. if (req.options?.waitForFinalOutput) { return waitForSessionFinalOutput( - ds, + target, triggerId, req.options?.timeoutMs ?? 120_000, (text) => ({ ok: true, triggerId, action: 'completed', - target: { kind: 'turn', sessionId: ds!.session.sessionId, chatId }, + target: { kind: 'turn', sessionId: target.session.sessionId, chatId }, output: { content: text }, message: 'delivered to existing session and completed', }), () => { - const dispatchAttempt = prepareStableDispatch(ds!, true); - armFinalOutputSuppression(ds!, dispatchAttempt); - forkWorker(ds!, content, { - resume: ds!.hasHistory, + const dispatchAttempt = prepareStableDispatch(target, true); + armFinalOutputSuppression(target, dispatchAttempt); + forkWorker(target, content, { + resume: target.hasHistory, turnId: triggerId, ...(dispatchAttempt !== undefined ? { dispatchAttempt } : {}), }); @@ -469,26 +551,26 @@ export async function triggerSessionTurn( } if (req.options?.asyncReturnSessionId) { - beginAsyncTrigger(ds, triggerId); - const dispatchAttempt = prepareStableDispatch(ds, true); - armFinalOutputSuppression(ds, dispatchAttempt); - forkWorker(ds, content, { - resume: ds.hasHistory, + beginAsyncTrigger(target, triggerId); + const dispatchAttempt = prepareStableDispatch(target, true); + armFinalOutputSuppression(target, dispatchAttempt); + forkWorker(target, content, { + resume: target.hasHistory, turnId: triggerId, ...(dispatchAttempt !== undefined ? { dispatchAttempt } : {}), }); return buildAsyncQueuedResponse( triggerId, - ds.session.sessionId, + target.session.sessionId, chatId, 'delivered to existing session; poll by sessionId or triggerId for final output', ); } - const dispatchAttempt = prepareStableDispatch(ds, true); - armFinalOutputSuppression(ds, dispatchAttempt); - forkWorker(ds, content, { - resume: ds.hasHistory, + const dispatchAttempt = prepareStableDispatch(target, true); + armFinalOutputSuppression(target, dispatchAttempt); + forkWorker(target, content, { + resume: target.hasHistory, turnId: triggerId, ...(dispatchAttempt !== undefined ? { dispatchAttempt } : {}), }); @@ -496,10 +578,12 @@ export async function triggerSessionTurn( ok: true, triggerId, action: 'queued', - target: { kind: 'turn', sessionId: ds.session.sessionId, chatId }, + target: { kind: 'turn', sessionId: target.session.sessionId, chatId }, message: 'queued existing session turn', }; - } + }; + + if (ds) return deliverToExisting(ds); const wd = resolveWorkingDir(larkAppId, chatId); if (!wd.ok) { @@ -520,34 +604,6 @@ export async function triggerSessionTurn( scope = 'thread'; } - const session = sessionStore.createSession(chatId, anchor, triggerTitle(req), 'group'); - const now = Date.now(); - session.larkAppId = larkAppId; - session.scope = scope; - if (shouldOpenOwnTopic && topicMessage === null) session.externalTriggerTopicless = true; - session.lastMessageAt = new Date(now).toISOString(); - session.workingDir = wd.workingDir; - session.cliId = bot.config.cliId; - sessionStore.updateSession(session); - - messageQueue.ensureQueue(anchor); - - const newDs: DaemonSession = { - session, - worker: null, - workerPort: null, - workerToken: null, - larkAppId, - chatId, - chatType: 'group', - scope, - spawnedAt: Date.parse(session.createdAt) || now, - cliVersion: getCurrentCliVersion(), - lastMessageAt: now, - hasHistory: false, - workingDir: wd.workingDir, - }; - // 仅默认目录 + auto-worktree：chat 驱动的 webhook 开新会话且落在本 bot 自己的默认目录时，走 // pendingRepo 挂起 + 异步提交（登记挂起→关键路径外建 worktree→commitRepoSelection 提交+fork）， // detach 后立即返回 queued。规则：**仅普通 webhook 适用**——HTTP 应答模式（waitForFinalOutput / @@ -560,15 +616,81 @@ export async function triggerSessionTurn( && !stableTurnId && wd.fromBotDefault && botAutoWorktreeEnabled(larkAppId); - if (useAutoWt) { - // Register BEFORE the detached commit so its guard + the router's pendingRepo - // buffering both see the session. pendingRepo=true → no force-fork in the window. - newDs.pendingRepo = true; // router buffers concurrent events; commit clears it - newDs.pendingPrompt = prompt; // folded into the first turn by commitRepoSelection + + // New trigger sessions participate in the same first-owner lock as resume, + // dashboard creation, and scheduled creation. The earlier routing lookup + // necessarily precedes chat-membership/mode awaits, so it is only a hint; + // the owner must be re-read at the commit point. Publish a reservation + // before any post-registration await so resume can never pass its final + // check and then have this trigger overwrite it. + const key = sessionKey(anchor, larkAppId); + const claim = await withActiveSessionKeyLock(deps.activeSessions, key, () => { + const current = deps.activeSessions.get(key); + if (current) return { kind: 'existing' as const, ds: current }; + + const session = sessionStore.createSession(chatId, anchor, triggerTitle(req), 'group'); + const now = Date.now(); + session.larkAppId = larkAppId; + session.scope = scope; + if (shouldOpenOwnTopic && topicMessage === null) session.externalTriggerTopicless = true; + session.lastMessageAt = new Date(now).toISOString(); + session.workingDir = wd.workingDir; + session.cliId = bot.config.cliId; + sessionStore.updateSession(session); + messageQueue.ensureQueue(anchor); + + const newDs: DaemonSession = { + session, + worker: null, + workerPort: null, + workerToken: null, + larkAppId, + chatId, + chatType: 'group', + scope, + spawnedAt: Date.parse(session.createdAt) || now, + cliVersion: getCurrentCliVersion(), + lastMessageAt: now, + hasHistory: false, + workingDir: wd.workingDir, + }; + // Retain the complete opening input until a worker or repo workflow has + // synchronously accepted it. This is both the route reservation and the + // retry payload if a write-ahead hook/fork throws before acceptance. + newDs.pendingPrompt = prompt; newDs.pendingCodexAppText = codexAppText; newDs.pendingCodexAppApplicationContext = codexAppApplicationContext || undefined; newDs.pendingCodexAppMessageContext = codexAppMessageContext; - deps.activeSessions.set(sessionKey(anchor, larkAppId), newDs); + if (useAutoWt) { + newDs.pendingRepo = true; + try { + stagePendingRepoSetup(newDs, { + mode: 'auto_worktree', + baseDir: wd.workingDir, + turnId: triggerId, + }); + } catch (err) { + // The route was never published, but createSession already persisted + // an active row. Close only this unaccepted row so a staging fault + // cannot reappear as an unregistered owner after restart. + try { sessionStore.closeSession(session.sessionId); } + catch { /* keep the original admission error */ } + throw err; + } + } else { + newDs.initialStartPending = true; + } + deps.activeSessions.set(key, newDs); + return { kind: 'created' as const, ds: newDs }; + }); + + if (claim.kind === 'existing') return deliverToExisting(claim.ds); + const newDs = claim.ds; + const session = newDs.session; + + if (useAutoWt) { + // The key-lock claim registered pendingRepo before this dynamic import; + // repo commit and inbound routing therefore see the same reservation. const { runAutoWorktreeCommit } = await import('../im/lark/card-handler.js'); void runAutoWorktreeCommit({ ds: newDs, anchor, larkAppId, baseDir: wd.workingDir, title: triggerTitle(req), @@ -587,6 +709,20 @@ export async function triggerSessionTurn( } ensureSessionWhiteboard(newDs); + let availableBots: Awaited>; + try { + availableBots = await getAvailableBots(larkAppId, chatId); + } catch (err) { + // Prompt construction failed before any worker existed. Retire only the + // still-owned reservation so a retry is not blocked by a ghost active row. + await withActiveSessionKeyLock(deps.activeSessions, key, () => { + if (deps.activeSessions.get(key) === newDs && newDs.initialStartPending) { + deps.activeSessions.delete(key); + sessionStore.closeSession(session.sessionId); + } + }); + throw err; + } const promptInput = buildNewTopicCliInput( prompt, session.sessionId, @@ -594,7 +730,7 @@ export async function triggerSessionTurn( bot.config.cliPathOverride, undefined, undefined, - await getAvailableBots(larkAppId, chatId), + availableBots, undefined, { name: bot.botName, openId: bot.botOpenId }, localeForBot(larkAppId), @@ -608,12 +744,29 @@ export async function triggerSessionTurn( codexAppMessageContext, }, ); - // Register right before the fork branches (no await between here and forkWorker) - // so a concurrent inbound message can't observe this session worker-less and - // race a duplicate re-fork — the set-and-fork atomicity the original path had. - deps.activeSessions.set(sessionKey(anchor, larkAppId), newDs); + // No await from the ownership check through forkWorker. The reservation and + // opening buffers are released only after synchronous pre-accept succeeds. + if (deps.activeSessions.get(key) !== newDs + || newDs.session.status !== 'active' + || !newDs.initialStartPending) { + if (newDs.session.status !== 'closed') sessionStore.closeSession(session.sessionId); + return { + ok: false, + triggerId, + errorCode: 'trigger_failed', + error: 'new trigger session lost its first-owner reservation before startup', + }; + } rememberInput(newDs, prompt, promptInput); + const releaseInitialReservation = (): void => { + newDs.initialStartPending = false; + newDs.pendingPrompt = undefined; + newDs.pendingCodexAppText = undefined; + newDs.pendingCodexAppApplicationContext = undefined; + newDs.pendingCodexAppMessageContext = undefined; + }; + if (req.options?.waitForFinalOutput) { return waitForSessionFinalOutput( newDs, @@ -633,17 +786,19 @@ export async function triggerSessionTurn( forkWorker(newDs, promptInput, dispatchAttempt === undefined ? triggerId : { turnId: triggerId, dispatchAttempt }); + releaseInitialReservation(); }, ); } if (req.options?.asyncReturnSessionId) { - beginAsyncTrigger(newDs, triggerId); const dispatchAttempt = prepareStableDispatch(newDs, true); armFinalOutputSuppression(newDs, dispatchAttempt); forkWorker(newDs, promptInput, dispatchAttempt === undefined ? triggerId : { turnId: triggerId, dispatchAttempt }); + releaseInitialReservation(); + beginAsyncTrigger(newDs, triggerId); return buildAsyncQueuedResponse( triggerId, session.sessionId, @@ -658,8 +813,12 @@ export async function triggerSessionTurn( forkWorker(newDs, promptInput, dispatchAttempt === undefined ? triggerId : { turnId: triggerId, dispatchAttempt }); + releaseInitialReservation(); + } + else { + forkWorker(newDs, promptInput); + releaseInitialReservation(); } - else forkWorker(newDs, promptInput); return { ok: true, @@ -669,3 +828,14 @@ export async function triggerSessionTurn( message: 'queued new session turn', }; } + +export async function triggerSessionTurn( + req: TriggerRequest, + deps: TriggerSessionDeps, + internal?: TriggerSessionInternalOptions, +): Promise { + return withBotTurnAdmission( + deps.larkAppId, + () => triggerSessionTurnAdmitted(req, deps, internal), + ); +} diff --git a/src/core/types.ts b/src/core/types.ts index e77e04b6d..ca0c1dda9 100644 --- a/src/core/types.ts +++ b/src/core/types.ts @@ -1,6 +1,7 @@ import type { ChildProcess } from 'node:child_process'; import type { CodexAppTurnInput, + CliTurnPayload, Session, DaemonToWorker, LarkAttachment, @@ -77,6 +78,25 @@ export interface DaemonSession { * separate from worktreeCreating because plain select, skip, and /repo can * also await prompt context before the fork. */ pendingRepoCommitInFlight?: boolean; + /** A fresh live-route owner has been published but its opening input has not + * reached the first fork yet. Same-anchor turns must buffer into the opening + * input instead of reforking worker:null and overtaking it. In-memory only. */ + initialStartPending?: boolean; + /** Generation token for the handler that atomically reserved a worker:null + * refork. Later same-anchor handlers may prepare concurrently, but only this + * owner may cross the fork boundary; followers buffer behind its gate. */ + initialStartClaimToken?: string; + /** Number of activation-tail arrivals that reserved FIFO order before an + * asynchronous prompt/sender build and have not yet durably admitted or + * failed. An opening ACK must not clear the route while this is non-zero. */ + queuedActivationTailAdmissionsOutstanding?: number; + /** An opening ACK (or ordinary cold-start handoff) observed while an + * asynchronous tail admission was outstanding. The final settler replays + * this release so a late durable successor cannot be stranded. */ + queuedActivationTailReleasePending?: { acknowledgedToken?: string }; + /** Retry timer for an ordinary cold-start handoff whose durable promotion + * failed after all asynchronous admissions had settled. */ + queuedActivationTailReleaseRetryTimer?: ReturnType; repoCardMessageId?: string; // message_id of the repo selection card — for withdrawal /** * Repo-select card message ids already consumed by a successful pending→worker @@ -138,14 +158,35 @@ export interface DaemonSession { /** Exact turn for a same-caller pendingRawInput follow-up batch. Cleared on * mixed callers so the combined prompt fails closed instead of borrowing. */ pendingFollowUpTurnId?: string; + pendingFollowUpTurnIds?: string[]; // matching inbound ids; last id attributes a folded buffered batch pendingCodexAppFollowUps?: string[]; // matching raw user texts for clean Codex App materialization pendingCodexAppFollowUpContexts?: string[]; // matching metadata-only context; never duplicates the raw follow-up text + /** Arrival-time clean-input decisions matching pendingCodexAppFollowUps. + * Used only when a literal raw cold start must fold followers onto the same + * text→Enter IPC boundary. */ + pendingCodexAppFollowUpGateAccepted?: boolean[]; + /** Exact turns that arrived while a previously attempted queued activation + * was re-parked. They remain separate FIFO items behind the retained opening + * payload and advance only after worker acceptance. In-memory only. */ + pendingQueuedActivationFollowUps?: Array<{ + userPrompt: string; + cliInput: CliTurnPayload; + turnId: string; + dispatchAttempt?: number; + /** Legacy volatile entries already crossed the clean-input gate when they + * were staged. Migration must preserve that exact sidecar decision. */ + codexAppInputGateFrozen?: true; + }>; ownerOpenId?: string; // topic creator's open_id — receives write-enabled terminal link via DM streamCardId?: string; // message_id of the streaming card in group (PATCHed with live output) streamCardNonce?: string; // unique nonce for the current streaming card — embedded in button values to distinguish old vs current card streamCardPending?: boolean; // true when a new turn started, next screen_update creates a new card pendingLocalCliButtonRefresh?: boolean; // true when cli_session_id arrived while the streaming card POST was in flight pendingRiffUrlCardRefresh?: boolean; // true when riff_access_url arrived while the streaming card POST was in flight + /** A Riff task id announced while a queued activation journal is armed. + * Kept out of Session until the matching activation ACK can persist the new + * lineage and journal clear in one durable write. */ + pendingRiffActivationTaskId?: string; /** Set on sessions restored after a daemon restart: suppresses the automatic * card post/patch from the recovery re-fork so a restart stays silent in the * group (the owner gets a private DM summary instead). Cleared on the first @@ -181,6 +222,23 @@ export interface DaemonSession { * "Web终端" button opens the riff sandbox. In-memory only — re-sent by the * worker on each task. */ riffAccessUrl?: string; + /** In-memory Riff explicit-close transaction. `preparing` rejects racing + * turns while remote cancellation settles; `prepared` awaits the daemon's + * durable close before commit. `uncertain` means the exact worker vanished + * before admission restoration/final lineage was proven and therefore must + * remain fail-closed until an explicit reconciliation path is available. */ + riffCloseState?: { + phase: 'preparing' | 'prepared' | 'uncertain'; + requestId: string; + taskId?: string; + }; + /** In-memory graceful-shutdown Riff transaction. It rejects new turns and + * explicit close while the exact final lineage is drained/persisted. */ + riffShutdownState?: { + phase: 'preparing' | 'prepared'; + requestId: string; + taskId?: string | null; + }; usageLimit?: CliUsageLimitState; usageLimitRetryTimer?: NodeJS.Timeout; lastUserPrompt?: string; @@ -220,6 +278,8 @@ export interface DaemonSession { * (ask/relay) that cannot trust a long-lived CLI's spawn-time env. */ managedTurnOrigin?: { capability: string; + /** Unguessable Seatbelt pane/profile authority channel. */ + originChannelId?: string; turnId?: string; dispatchAttempt?: number; }; @@ -283,6 +343,15 @@ export interface DaemonSession { }; } +/** A non-null value means this Riff generation is deliberately rejecting new + * input until a close/shutdown commit or an ACKed admission restore. Callers + * must check this before recording a turn as accepted. */ +export function riffRetirementAdmissionPhase(ds: DaemonSession): string | null { + if (ds.riffShutdownState) return `shutdown-${ds.riffShutdownState.phase}`; + if (ds.riffCloseState) return `close-${ds.riffCloseState.phase}`; + return null; +} + /** Composite key for activeSessions — allows multiple bots to have independent * sessions anchored on the same id. The first arg is the **routing anchor**: * - thread-scope → rootMessageId diff --git a/src/core/worker-pool.ts b/src/core/worker-pool.ts index c6259aadf..0b9a13e68 100644 --- a/src/core/worker-pool.ts +++ b/src/core/worker-pool.ts @@ -18,11 +18,11 @@ import { config } from '../config.js'; import { readGlobalConfig } from '../global-config.js'; import * as sessionStore from '../services/session-store.js'; import { persistStreamCardState, rememberLastCliInput } from './session-manager.js'; -import { fallbackTurnId, isSubstituteTurn } from './reply-target.js'; +import { fallbackTurnId, frozenReplyContextForTurn, isSubstituteTurn } from './reply-target.js'; import { updateMessage, deleteMessage, sendEphemeralCard, sendUserMessage, addReaction, removeReaction, getMessageChatId, MessageWithdrawnError } from '../im/lark/client.js'; import { buildStreamingCard, buildPrivateSnapshotCard, buildSessionCard, buildTuiPromptCard, buildTuiPromptResolvedCard, buildRelayedFrozenCard, getCliDisplayName } from '../im/lark/card-builder.js'; import { loadFrozenCards, saveFrozenCards } from '../services/frozen-card-store.js'; -import { hashUrlForLog, cancelRiffTaskById } from '../adapters/backend/riff-backend.js'; +import { hashUrlForLog } from '../adapters/backend/riff-backend.js'; import { logger } from '../utils/logger.js'; import { createCliAdapterSync } from '../adapters/cli/registry.js'; import { botLocale, localeForBot, t as tr } from '../i18n/index.js'; @@ -36,6 +36,8 @@ import { TmuxBackend } from '../adapters/backend/tmux-backend.js'; import { HerdrBackend } from '../adapters/backend/herdr-backend.js'; import { sandboxEnabled } from '../adapters/backend/sandbox.js'; import { isSuspendableBackendType, getSessionPersistentBackendType, persistentSessionName, killPersistentSession, resolvePairedSpawnBackendType } from './persistent-backend.js'; +import { cleanupExplicitSessionBacking } from './explicit-session-backing-cleanup.js'; +import { withBotTurnMutation } from './bot-turn-mutation-gate.js'; import { getBot, getAllBots, loadBotConfigs, resolveBrandLabel } from '../bot-registry.js'; /** A random id minted once per daemon process (this lifetime). Stamped onto @@ -73,8 +75,31 @@ import type { CliId } from '../adapters/cli/types.js'; import { isStructuredBridgeAdoptCli } from '../services/structured-bridge-clis.js'; import { resolveEffectivePluginIds } from './plugins/effective.js'; import { ensureGatewayEntry } from './plugins/mcp/gateway-installer.js'; -import type { CliTurnPayload, CodexAppTurnInput, DaemonToWorker, WorkerToDaemon, Session, DisplayMode } from '../types.js'; -import { activeSessionKey, sessionKey, sessionAnchorId, isDocNativeSession, type DaemonSession } from './types.js'; +import type { + CliTurnPayload, + CodexAppDeliverySink, + CodexAppDispatchLedgerEntry, + CodexAppGenerationCommit, + CodexAppTurnInput, + FrozenSessionReplyTarget, + DaemonToWorker, + WorkerToDaemon, + Session, + DisplayMode, + QueuedActivationTailEntry, +} from '../types.js'; +import { + appendAcceptedCodexAppDispatch, + cancelCodexAppDispatch, + committedCodexAppSequence, + hasUnsettledCodexAppDispatch, + prepareCodexAppDispatch, + retryPreparedCodexAppDispatch, + retainFreshCodexAppGeneration, + settleCodexAppDispatch, +} from '../utils/codex-app-dispatch-ledger.js'; +import { activeSessionKey, sessionKey, sessionAnchorId, isDocNativeSession, riffRetirementAdmissionPhase, type DaemonSession } from './types.js'; +import { hasProtectedSessionMutationOwnership } from './session-mutation-guard.js'; import { DONE_REACTION_EMOJI_TYPE } from './pending-response.js'; import { buildTerminalUrl } from './terminal-url.js'; import { prependBotmuxBin } from './botmux-wrapper.js'; @@ -95,6 +120,11 @@ import { isSilentScheduledTurn } from './silent-schedule-turns.js'; import { writeDeferredTopicBinding } from './deferred-topic-binding.js'; import { deferWorkerSpawnDuringDeviceIsolation } from './device-isolation-activation.js'; import { acknowledgeSessionReady } from './session-ready-handshake.js'; +import { + managedOriginCapabilityPath, + replaceManagedOriginCapabilityFile, +} from './managed-origin-capability.js'; +import { RIFF_ADMISSION_RESTORE_TIMEOUT_MS } from './shutdown-budgets.js'; type WindowsForkOptions = ForkOptions & { windowsHide?: boolean }; @@ -135,6 +165,8 @@ export interface WorkerSessionReplyOptions { * share a visible chat anchor with ordinary sessions, so the anchor alone * cannot identify the transcript/lifecycle owner. */ sourceSessionId?: string; + /** Exact daemon-frozen destination for a durable turn. */ + replyTarget?: FrozenSessionReplyTarget; } export interface WorkerPoolCallbacks { @@ -148,8 +180,11 @@ export interface WorkerPoolCallbacks { ) => Promise; getSessionWorkingDir: (ds?: DaemonSession) => string; getActiveCount: () => number; - /** Close a stale session (message withdrawn, etc.) */ - closeSession: (ds: DaemonSession) => void; + /** Close a stale session (message withdrawn, etc.). `false` means the + * authoritative close failed and the active owner must remain retryable. + * `void` is retained for older embedders/tests that implement a synchronous + * best-effort close; the production daemon always returns an exact boolean. */ + closeSession: (ds: DaemonSession) => boolean | void | Promise; /** Re-check the per-bot resident-session cap after a process starts or an * over-cap busy session becomes idle. Optional for unit-test callers. */ enforceLiveSessionCap?: () => void; @@ -198,6 +233,19 @@ export interface WorkerPoolCallbacks { disposition: 'queued_removed' | 'cli_fenced'; }, ) => void; + /** Called only after a durable ledger mutation was persisted and its worker + * ACK was attempted. Runtime CLI-mismatch cleanup may now close the old + * generation without abandoning a FIFO entry. */ + onCodexAppLedgerDrained?: (ds: DaemonSession) => void | Promise; + /** The exact queued opening crossed the adapter submission boundary. The + * daemon may now release its runtime route reservation and flush follow-ups. */ + /** Return false only when a buffered follow-up was not accepted and should + * be retried. Once accepted, later presentation persistence is best-effort + * and must not request a delivery retry. */ + onQueuedActivationSubmitted?: ( + ds: DaemonSession, + activationToken: string, + ) => boolean | void | Promise; } let callbacks: WorkerPoolCallbacks | undefined; @@ -221,6 +269,18 @@ function requireCallbacks(): WorkerPoolCallbacks { // linear-scan by sessionId. let activeSessionsRegistry: Map | undefined; +type RiffWorkerCloseResult = { + ok: boolean; + taskId?: string; + error?: string; +}; + +const pendingRiffWorkerCloses = new Map void; +}>(); + export function setActiveSessionsRegistry(m: Map): void { activeSessionsRegistry = m; } @@ -1133,6 +1193,38 @@ function flushCardPatch(ds: DaemonSession): void { }); } +/** Root withdrawal is not an explicit abandon boundary. Preserve a Codex App + * owner (including its live worker) while any durable FIFO entry is unsettled; + * only ledger-empty sessions retain the historical stale-root auto-close. */ +async function closeWithdrawnSessionIfLedgerEmpty( + ds: DaemonSession, + context: string, +): Promise { + if (hasProtectedSessionMutationOwnership(ds)) { + logger.warn(`[${tag(ds)}] ${context}; preserving session because Codex App dispatch is unsettled`); + return false; + } + logger.warn(`[${tag(ds)}] ${context}; closing ledger-empty stale session`); + // The callback routes through the authoritative async closeSession helper. + // Do not pre-kill a Riff worker here: turning it worker-less before the + // prepare handshake would let an in-flight create materialize after the + // direct cancellation and orphan a credential-bearing remote task. + try { + const closed = await requireCallbacks().closeSession(ds); + if (closed === false) { + logger.warn(`[${tag(ds)}] ${context}; authoritative close failed, session retained for retry`); + return false; + } + return true; + } catch (err) { + logger.warn( + `[${tag(ds)}] ${context}; authoritative close threw, session retained for retry: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + return false; + } +} + // ─── Restart rate-limiting ────────────────────────────────────────────────── export const restartCounts = new Map(); @@ -1293,7 +1385,26 @@ export function ensureClaudeFolderTrust(workingDir: string, stateJsonPath: strin // ─── Kill worker ──────────────────────────────────────────────────────────── -export function killWorker(ds: DaemonSession): void { +export function killWorker( + ds: DaemonSession, + opts: { riffCloseCommitRequestId?: string } = {}, +): void { + const closeFrozenType = ds.initConfig?.backendType ?? ds.session.backendType; + if (ds.worker && !ds.worker.killed + && closeFrozenType === 'riff' + && !opts.riffCloseCommitRequestId) { + // A generic synchronous retirement cannot safely detach Riff. A task + // execute/follow-up may still be inside its HTTP await, so exiting the + // worker now can happen before adoptLateTask publishes and durably stores + // the new task id. Only explicit close's prepare/commit protocol may retire + // a live Riff owner; replacement-style callers must refuse or keep using + // the existing worker. + logger.error( + `[${tag(ds)}] Refused unprepared live Riff worker retirement; ` + + 'preserving worker and remote-task lineage', + ); + return; + } clearUsageLimitState(ds); ds.localProcessAttestation = undefined; // A managed-turn capability belongs to one concrete worker generation. @@ -1309,18 +1420,28 @@ export function killWorker(ds: DaemonSession): void { // session would leave an orphaned CLI running in tmux that still replies. // Destroy the backing session directly here so /close always terminates it. destroyOrphanedBackingSession(ds); + if (opts.riffCloseCommitRequestId + && ds.riffCloseState?.requestId === opts.riffCloseCommitRequestId) { + ds.riffCloseState = undefined; + } return; } try { - ds.worker.send({ type: 'close' } as DaemonToWorker); + if (opts.riffCloseCommitRequestId) { + ds.worker.send({ type: 'close_commit', requestId: opts.riffCloseCommitRequestId } as DaemonToWorker); + } else { + ds.worker.send({ type: 'close' } as DaemonToWorker); + } } catch { /* IPC already closed */ } + if (opts.riffCloseCommitRequestId + && ds.riffCloseState?.requestId === opts.riffCloseCommitRequestId) { + ds.riffCloseState = undefined; + } const w = ds.worker; - // riff：worker close 分支要有界 await 远端 task-cancel（destroySession 5s×2 重试， - // 外层 race 8s）。默认 2s SIGTERM backstop 会在取消发出前掐死进程，已关闭话题 - // 的远端任务照跑——冻结为 riff 的会话放宽到 24s（层级：destroy 20s < worker 22s - // < SIGTERM 24s < SIGKILL 29s；正常路径 worker 自行 exit，不会等满）。 - const closeFrozenType = ds.initConfig?.backendType ?? ds.session.backendType; - armWorkerKillBackstop(w, tag(ds), closeFrozenType === 'riff' ? 24_000 : WORKER_SIGTERM_BACKSTOP_MS); + // Explicit Riff close already completed its bounded remote prepare before + // reaching here. Generic live-Riff retirement returned above, so every path + // here is now authorized to exit on the ordinary backstop. + armWorkerKillBackstop(w, tag(ds)); ds.worker = null; ds.workerPort = null; ds.workerToken = null; @@ -1340,22 +1461,18 @@ export function killWorker(ds: DaemonSession): void { function destroyOrphanedBackingSession(ds: DaemonSession): void { if (ds.initConfig?.adoptMode || ds.adoptedFrom) return; reclaimParkedCrashDiagnostic(ds); - // riff：worker 已死时 /close 仍要取消持久化血缘指向的远端任务——否则已关闭 - // 话题的远端 agent 继续拿着注入凭证发消息。fire-and-forget（内部有界+重试）。 + // Riff cancellation is asynchronous and therefore cannot be made safe from + // this synchronous best-effort helper. The authoritative closeSession path + // awaits it BEFORE publishing the closed row and retains riffParentTaskId on + // failure. Never fire-and-forget here: doing so loses the only retry handle + // while a remote agent may still hold injected Lark credentials. const frozenType = ds.initConfig?.backendType ?? ds.session.backendType; if (frozenType === 'riff') { - const taskId = ds.session.riffParentTaskId; - if (taskId) { - try { - const riffCfg = getBot(ds.larkAppId).config.riff; - if (riffCfg?.baseUrl) { - void cancelRiffTaskById(riffCfg, taskId).then((ok) => { - if (ok) logger.info(`[${tag(ds)}] killWorker: orphan riff task ${taskId} cancelled`); - }); - } - } catch { /* bot deregistered — nothing to cancel with */ } - ds.session.riffParentTaskId = undefined; - sessionStore.updateSession(ds.session); + if (ds.session.riffParentTaskId) { + logger.warn( + `[${tag(ds)}] worker-less Riff teardown requires awaited explicit close; ` + + `retaining task ${ds.session.riffParentTaskId} for retry`, + ); } return; } @@ -1369,6 +1486,321 @@ function destroyOrphanedBackingSession(ds: DaemonSession): void { } } +type RiffClosePreparation = + | { ok: true; taskId?: string } + | { + ok: false; + error: + | 'riff_cancel_failed' + | 'riff_config_missing' + | 'riff_task_changed' + | 'riff_worker_close_failed' + | 'riff_row_inconsistent' + | 'riff_durable_close_failed' + | 'riff_close_reconciliation_required' + | 'riff_shutdown_fence_in_progress'; + retryable: true; + taskId?: string; + }; + +async function abortLiveRiffWorkerClose( + ds: DaemonSession, + requestId: string, + opts: { allowAbsentAfterProvenRestore?: boolean } = {}, +): Promise { + if (ds.riffCloseState?.requestId !== requestId) return false; + const worker = ds.worker; + if (!worker || worker.killed) { + if (opts.allowAbsentAfterProvenRestore) { + ds.riffCloseState = undefined; + return true; + } + ds.riffCloseState = { ...ds.riffCloseState, phase: 'uncertain' }; + return false; + } + + const restored = await new Promise<{ ok: boolean; error?: string }>(resolve => { + let settled = false; + const finish = (result: { ok: boolean; error?: string }): void => { + if (settled) return; + settled = true; + clearTimeout(timer); + worker.removeListener?.('message', onMessage); + worker.removeListener?.('exit', onExit); + resolve(result); + }; + const onMessage = (raw: unknown): void => { + const msg = raw as WorkerToDaemon; + if (msg?.type !== 'close_abort_result' || msg.requestId !== requestId) return; + if (ds.worker !== worker) { + finish({ ok: false, error: 'stale_worker_generation' }); + return; + } + finish({ ok: msg.ok, ...(msg.error ? { error: msg.error } : {}) }); + }; + // Losing the local backend is not proof that its write chain never created + // a late remote child, nor that an in-flight cancellation reached terminal. + // Only the matching close_abort_result may reopen admission. + const onExit = (): void => finish({ ok: false, error: 'worker_exited_before_close_abort_result' }); + const timer = setTimeout( + () => finish({ ok: false, error: 'close_abort_result_timeout' }), + RIFF_ADMISSION_RESTORE_TIMEOUT_MS, + ); + timer.unref?.(); + worker.on?.('message', onMessage); + worker.once?.('exit', onExit); + try { + worker.send({ type: 'close_abort', requestId } as DaemonToWorker); + } catch (err) { + finish({ ok: false, error: err instanceof Error ? err.message : String(err) }); + } + }); + + if (restored.ok && ds.riffCloseState?.requestId === requestId) { + ds.riffCloseState = undefined; + return true; + } + if (opts.allowAbsentAfterProvenRestore + && ds.riffCloseState?.requestId === requestId) { + // The matching close_result failure itself is a worker-side proof that + // abortDestroySession completed before the result was emitted. A redundant + // close_abort ACK is desirable but not required to preserve that proof. + ds.riffCloseState = undefined; + return true; + } + if (ds.riffCloseState?.requestId === requestId) { + ds.riffCloseState = { ...ds.riffCloseState, phase: 'uncertain' }; + } + logger.warn( + `[${tag(ds)}] Riff close abort was not acknowledged (${restored.error ?? 'unknown'}); ` + + 'retaining admission fence pending explicit lineage reconciliation', + ); + return false; +} + +async function prepareLiveRiffWorkerClose(ds: DaemonSession): Promise { + const worker = ds.worker; + if (!worker || worker.killed) return { ok: false, error: 'riff_worker_close_failed', retryable: true }; + if (ds.riffCloseState || ds.riffShutdownState) { + return { + ok: false, + error: 'riff_worker_close_failed', + retryable: true, + ...((ds.riffCloseState?.taskId ?? ds.riffShutdownState?.taskId) + ? { taskId: (ds.riffCloseState?.taskId ?? ds.riffShutdownState?.taskId)! } + : {}), + }; + } + const requestId = randomUUID(); + ds.riffCloseState = { + phase: 'preparing', + requestId, + ...(ds.session.riffParentTaskId ? { taskId: ds.session.riffParentTaskId } : {}), + }; + let matchedCloseResult = false; + const result = await new Promise((resolve) => { + let settled = false; + const finish = (value: RiffWorkerCloseResult) => { + if (settled) return; + settled = true; + clearTimeout(timer); + worker.removeListener?.('exit', onExit); + pendingRiffWorkerCloses.delete(requestId); + resolve(value); + }; + const onExit = () => finish({ ok: false, error: 'worker_exited_before_close_result' }); + const timer = setTimeout( + () => finish({ ok: false, error: 'worker_close_result_timeout' }), + 23_000, + ); + timer.unref?.(); + worker.once?.('exit', onExit); + pendingRiffWorkerCloses.set(requestId, { + sessionId: ds.session.sessionId, + worker, + resolve: value => { + matchedCloseResult = true; + finish(value); + }, + }); + try { + worker.send({ type: 'close', requestId } as DaemonToWorker); + } catch (err) { + finish({ ok: false, error: err instanceof Error ? err.message : String(err) }); + } + }); + + const taskId = result.taskId ?? ds.session.riffParentTaskId; + if (result.taskId) { + // Keep the newest lineage in memory even if persistence is temporarily + // unavailable. A failed save aborts close admission and the next retry + // attempts the same exact task id again. + ds.session.riffParentTaskId = result.taskId; + try { + sessionStore.updateSession(ds.session); + } catch (err) { + await abortLiveRiffWorkerClose(ds, requestId); + logger.error( + `[${tag(ds)}] Riff close lineage persistence failed; close aborted: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + return { + ok: false, + error: 'riff_durable_close_failed', + retryable: true, + taskId: result.taskId, + }; + } + } + + if (!result.ok) { + await abortLiveRiffWorkerClose(ds, requestId, { + // A matching close_result failure is emitted only after the worker has + // awaited RiffBackend.abortDestroySession(). If it exits immediately + // after that proof, the daemon may clear the fence without another ACK. + allowAbsentAfterProvenRestore: matchedCloseResult, + }); + logger.warn( + `[${tag(ds)}] Riff worker close prepare failed: ${result.error ?? 'unknown'}; ` + + `session remains active${taskId ? ` (task ${taskId})` : ''}`, + ); + return { + ok: false, + error: 'riff_worker_close_failed', + retryable: true, + ...(taskId ? { taskId } : {}), + }; + } + + ds.riffCloseState = { + phase: 'prepared', + requestId, + ...(taskId ? { taskId } : {}), + }; + logger.info(`[${tag(ds)}] Riff worker close prepared and remote task cancellation confirmed`); + return { ok: true, ...(taskId ? { taskId } : {}) }; +} + +/** Await remote cancellation for any Riff owner before its durable row is + * closed. A live worker uses a prepare/commit handshake so its closing fence + * drains and accounts for a task created by an in-flight request. Worker-less + * and disk-only rows cancel their persisted task through current bot config. */ +async function prepareRiffExplicitClose( + ds: DaemonSession | undefined, + stored: Session | undefined, +): Promise { + const session = ds?.session ?? stored; + if (!session) return { ok: true }; + const backendType = ds?.initConfig?.backendType ?? session.backendType; + const taskId = session.riffParentTaskId; + if (ds?.riffShutdownState) { + const fencedTaskId = Object.prototype.hasOwnProperty.call(ds.riffShutdownState, 'taskId') + ? ds.riffShutdownState.taskId + : taskId; + return { + ok: false, + error: 'riff_shutdown_fence_in_progress', + retryable: true, + ...(typeof fencedTaskId === 'string' && fencedTaskId + ? { taskId: fencedTaskId } + : {}), + }; + } + if (ds?.riffCloseState) { + return { + ok: false, + error: 'riff_close_reconciliation_required', + retryable: true, + ...(ds.riffCloseState.taskId ? { taskId: ds.riffCloseState.taskId } : {}), + }; + } + if (backendType !== 'riff') return { ok: true }; + if (ds?.initConfig?.adoptMode || ds?.adoptedFrom || session.adoptedFrom) return { ok: true }; + + if (ds?.worker && !ds.worker.killed) { + // Runtime ownership without the matching open durable row is already an + // inconsistent generation. Do not fence/exit the worker and then pretend + // an idempotent close succeeded; preserve it for explicit reconciliation. + if (!stored || stored.status !== 'active') { + return { + ok: false, + error: 'riff_row_inconsistent', + retryable: true, + ...(taskId ? { taskId } : {}), + }; + } + return prepareLiveRiffWorkerClose(ds); + } + + const durableBefore = sessionStore.getSession(session.sessionId); + if (ds && durableBefore + && ds.session.riffParentTaskId !== durableBefore.riffParentTaskId) { + const authoritativeTaskId = durableBefore.riffParentTaskId ?? ds.session.riffParentTaskId; + logger.warn( + `[${tag(ds)}] explicit close refused before cancellation: runtime/durable Riff lineage differs ` + + `(${ds.session.riffParentTaskId ?? 'none'}/${durableBefore.riffParentTaskId ?? 'none'})`, + ); + return { + ok: false, + error: 'riff_task_changed', + retryable: true, + ...(authoritativeTaskId ? { taskId: authoritativeTaskId } : {}), + }; + } + + if (!taskId) return { ok: true }; + + let riffConfig; + const larkAppId = ds?.larkAppId ?? session.larkAppId; + try { if (larkAppId) riffConfig = getBot(larkAppId).config.riff; } catch { /* bot removed */ } + const cleanup = await cleanupExplicitSessionBacking({ + sessionId: session.sessionId, + backendType, + riffParentTaskId: taskId, + riffConfig, + }); + const label = ds ? tag(ds) : session.sessionId.slice(0, 8); + if (!cleanup.ok) { + logger.warn( + `[${label}] explicit close refused: ${cleanup.kind}; ` + + `Riff task ${taskId} remains active and retryable`, + ); + return { + ok: false, + error: cleanup.kind === 'riff_config_missing' ? 'riff_config_missing' : 'riff_cancel_failed', + retryable: true, + taskId, + }; + } + if (cleanup.kind !== 'cancelled_riff') return { ok: true }; + + // Worker-less cancellation still verifies the exact durable lineage after + // the provider await before erasing its retry handle. + const latest = sessionStore.getSession(session.sessionId); + const latestTaskId = latest?.riffParentTaskId; + const runtimeTaskId = ds?.session.riffParentTaskId; + if (!latest || latest.status !== 'active' || latestTaskId !== cleanup.taskId + || (ds && runtimeTaskId !== cleanup.taskId)) { + logger.warn( + `[${label}] explicit close cancelled stale Riff task ${cleanup.taskId}, ` + + `but runtime/durable lineage or status changed to ` + + `${runtimeTaskId ?? 'none'}/${latestTaskId ?? 'none'}/${latest?.status ?? 'missing'}; retry required`, + ); + return { + ok: false, + error: 'riff_task_changed', + retryable: true, + taskId: latestTaskId ?? runtimeTaskId ?? cleanup.taskId, + }; + } + + // Do not erase lineage during prepare. The authoritative close operation + // clears it atomically with status='closed'; if that durable commit throws, + // the active row keeps this exact cancelled task as its retry/resume anchor. + logger.info(`[${label}] Riff task ${cleanup.taskId} cancellation prepared for explicit close`); + return { ok: true, taskId: cleanup.taskId }; +} + /** * Reclaim a session's parked crash-diagnostic shell (`bmx-diag-`) and its * captured `.ansi` file. The worker normally tears these down itself (killCli / @@ -1384,6 +1816,10 @@ function reclaimParkedCrashDiagnostic(ds: DaemonSession): void { } export function suspendWorker(ds: DaemonSession, reason = 'suspended_idle'): boolean { + if (hasProtectedSessionMutationOwnership(ds)) { + logger.warn(`[${tag(ds)}] Refused worker suspend (${reason}) while Codex App dispatch ownership is non-empty`); + return false; + } if (!ds.worker || ds.worker.killed) { // There is no live generation that can still own this capability. ds.managedTurnOrigin = undefined; @@ -1470,10 +1906,34 @@ function armWorkerKillBackstop(w: ChildProcess, label: string, sigtermMs: number * Calling this on an unknown sessionId, an already-closed session, or a session * whose worker died asynchronously must still resolve with `{ ok: true }`. */ +export type CloseSessionResult = + | { ok: true; alreadyClosed: boolean } + | { + ok: false; + alreadyClosed: false; + error: + | 'riff_cancel_failed' + | 'riff_config_missing' + | 'riff_task_changed' + | 'riff_worker_close_failed' + | 'riff_row_inconsistent' + | 'riff_durable_close_failed' + | 'riff_close_reconciliation_required' + | 'riff_shutdown_fence_in_progress'; + retryable: true; + taskId?: string; + }; + export async function closeSession( sessionId: string, -): Promise<{ ok: true; alreadyClosed: boolean }> { +): Promise { const ds = findActiveBySessionId(sessionId); + const stored = sessionStore.getSession(sessionId); + const wasOpen = !!stored && stored.status !== 'closed'; + const isOwnedRiffClose = !ds?.initConfig?.adoptMode + && !ds?.adoptedFrom + && !stored?.adoptedFrom + && (ds?.initConfig?.backendType ?? ds?.session.backendType ?? stored?.backendType) === 'riff'; let killedLive = false; // 会话关闭即可回收其崩溃重启计数；否则每个曾崩溃过的 session 会在 daemon // 生命周期内永久占位（restartCounts 此前无任何 delete）。 @@ -1482,40 +1942,51 @@ export async function closeSession( // Usage ledger: flush the final delta before the worker goes away (a // crash/limited turn may never have reached an idle edge). recordUsageForDaemonSession(ds); - killWorker(ds); - // 文档入口清理：会话关闭即删除其绑定。只有旧 - // /subscribe-lark-doc 记录需要调飞书逐文件退订 API； - // /watch-comment 仅依赖应用级评论事件，删本地监听表即可。 - try { - const anchor = sessionAnchorId(ds); - const subs = listDocSubscriptionsForSession(config.session.dataDir, ds.larkAppId, anchor); - for (const sub of subs) { - if (sub.managedBy !== 'watch-comment') { - await unsubscribeDocFile(ds.larkAppId, { fileToken: sub.fileToken, fileType: sub.fileType }); - } - removeDocSubscription(config.session.dataDir, ds.larkAppId, sub.fileToken); - } - if (subs.length) logger.info(`[doc-comment] session ${sessionId.slice(0, 8)} closed → removed ${subs.length} doc binding(s)`); - } catch (err: any) { - logger.warn(`[doc-comment] cleanup on close failed for ${sessionId.slice(0, 8)}: ${err?.message ?? err}`); - } - activeSessionsRegistry?.delete(activeSessionKey(ds)); - killedLive = true; - if (!ds.exitEventEmitted) { - ds.exitEventEmitted = true; - dashboardEventBus.publish({ - type: 'session.exited', - body: { sessionId, reason: 'dashboard_close' }, - }); - emitSessionLifecycleHook(ds, 'session.exit', { reason: 'dashboard_close' }); - } } - // Persistence path — load → mark closed → save (delegated to sessionStore). - const stored = sessionStore.getSession(sessionId); - const wasOpen = !!stored && stored.status !== 'closed'; + // A Riff task is a remote credential-bearing process, not a local pane that + // can be synchronously killed after the row is closed. Await its bounded + // cancellation first. Live workers use an explicit result handshake that + // fences/drains late task creation. On failure the active row, route, worker + // and any known task id remain authoritative so close can be retried. + const prepared = await prepareRiffExplicitClose(ds, stored); + if (!prepared.ok) { + return { ...prepared, alreadyClosed: false }; + } + const preparedRiffRequestId = ds?.riffCloseState?.phase === 'prepared' + ? ds.riffCloseState.requestId + : undefined; + + // Explicit abandon is durable before any worker/provider cleanup. A process + // crash or a hanging document unsubscribe must not leave an active row with + // its accepted FIFO after the routing owner has been removed. if (wasOpen) { - sessionStore.closeSession(sessionId); + try { + // SessionStore clears Riff lineage in this same atomic file replace and + // restores it on failure. Only after this returns may the worker receive + // close_commit. + if (isOwnedRiffClose) { + sessionStore.closeSession(sessionId, { clearRiffParentTaskId: true }); + } else { + sessionStore.closeSession(sessionId); + } + } catch (err) { + if (!isOwnedRiffClose) throw err; + if (ds && preparedRiffRequestId) { + await abortLiveRiffWorkerClose(ds, preparedRiffRequestId); + } + logger.error( + `[${sessionId.slice(0, 8)}] Durable session close failed after Riff prepare; ` + + `worker admission restored: ${err instanceof Error ? err.message : String(err)}`, + ); + return { + ok: false, + alreadyClosed: false, + error: 'riff_durable_close_failed', + retryable: true, + ...(prepared.taskId ? { taskId: prepared.taskId } : {}), + }; + } const after = sessionStore.getSession(sessionId); dashboardEventBus.publish({ type: 'session.update', @@ -1530,6 +2001,62 @@ export async function closeSession( }); } + if (ds) { + killWorker(ds, { + ...(preparedRiffRequestId ? { riffCloseCommitRequestId: preparedRiffRequestId } : {}), + }); + // Detach the routing slot before the first async cleanup below. Callers + // such as the read-isolation two-phase sweep intentionally start every + // close without awaiting it; leaving the old ds discoverable until a doc + // unsubscribe completed would let a new turn refork the killed generation + // and then be erased by this stale close continuation. + // A transfer/restore bug can leave the exact ds object under more than one + // alias. Remove every identity match, but never delete a same-key successor + // that has already replaced it. + if (activeSessionsRegistry) { + for (const [registeredKey, candidate] of activeSessionsRegistry) { + if (candidate === ds) activeSessionsRegistry.delete(registeredKey); + } + } + killedLive = true; + if (!ds.exitEventEmitted) { + ds.exitEventEmitted = true; + dashboardEventBus.publish({ + type: 'session.exited', + body: { sessionId, reason: 'dashboard_close' }, + }); + emitSessionLifecycleHook(ds, 'session.exit', { reason: 'dashboard_close' }); + } + + // Remove local routing bindings synchronously with the authoritative + // close. If provider unsubscribe hangs, poll/WS delivery must still be + // unable to auto-create a replacement session from a stale binding. + const anchor = sessionAnchorId(ds); + const subs = listDocSubscriptionsForSession(config.session.dataDir, ds.larkAppId, anchor); + for (const sub of subs) { + removeDocSubscription(config.session.dataDir, ds.larkAppId, sub.fileToken); + } + if (subs.length) { + logger.info(`[doc-comment] session ${sessionId.slice(0, 8)} closed → removed ${subs.length} local doc binding(s)`); + } + + // Provider cleanup is deliberately detached from the authoritative close. + // A slow/hung unsubscribe must not retain a bot-wide mutation gate after + // the durable row is closed, FIFO cleared, worker killed, route detached, + // and local delivery binding removed. + void (async () => { + try { + for (const sub of subs) { + if (sub.managedBy !== 'watch-comment') { + await unsubscribeDocFile(ds.larkAppId, { fileToken: sub.fileToken, fileType: sub.fileType }); + } + } + } catch (err: any) { + logger.warn(`[doc-comment] cleanup on close failed for ${sessionId.slice(0, 8)}: ${err?.message ?? err}`); + } + })(); + } + // alreadyClosed = nothing happened on either path. const alreadyClosed = !killedLive && !wasOpen; return { ok: true, alreadyClosed }; @@ -1555,20 +2082,219 @@ export async function closeSession( * * Setting the same `ds` at its own key is a no-op (no close). */ +export type SetActiveSessionResult = + | { accepted: true; closedSessionId?: string } + | { + accepted: false; + reason: 'kept_pending_owner'; + keptSessionId: string; + closedIncomingSessionId: string; + } + | { + accepted: false; + reason: 'both_pending'; + keptSessionId: string; + preservedIncomingSessionId: string; + } + | { + accepted: false; + reason: 'cleanup_failed'; + keptSessionId: string; + preservedIncomingSessionId: string; + cleanupSessionId: string; + error: string; + taskId?: string; + }; + +const activeSessionKeyLocks = new WeakMap< + Map, + Map> +>(); + +/** Serialize asynchronous ownership decisions for one registry key. Ordinary + * bot turn admissions are intentionally concurrent, so a read/await/set helper + * is not a CAS unless every contender shares this lock. */ +export async function withActiveSessionKeyLock( + map: Map, + key: string, + action: () => Promise | T, +): Promise { + let locks = activeSessionKeyLocks.get(map); + if (!locks) { + locks = new Map(); + activeSessionKeyLocks.set(map, locks); + } + const previous = locks.get(key) ?? Promise.resolve(); + let release!: () => void; + const hold = new Promise(resolve => { release = resolve; }); + const tail = previous.catch(() => { /* predecessor errors do not poison the lock */ }).then(() => hold); + locks.set(key, tail); + await previous.catch(() => { /* predecessor already reported its own error */ }); + try { + return await action(); + } finally { + release(); + if (locks.get(key) === tail) locks.delete(key); + } +} + +async function closeUnregisteredCollisionLoser( + map: Map, + loser: DaemonSession, +): Promise<{ ok: true } | { ok: false; error: string; taskId?: string }> { + // Operate on exact object identity in the caller's registry. Looking up by + // session id through the daemon-global registry can conflate a non-global + // restore map (or a stale duplicate object) with an unrelated canonical + // owner and close the wrong generation. + const sameIdDifferentOwner = [...map.values()].some(candidate => + candidate !== loser && candidate.session.sessionId === loser.session.sessionId) + || (!!activeSessionsRegistry && activeSessionsRegistry !== map + && [...activeSessionsRegistry.values()].some(candidate => + candidate !== loser && candidate.session.sessionId === loser.session.sessionId)); + if (sameIdDifferentOwner) { + logger.error( + `[setActiveSessionSafe] refusing persistence close for ambiguous session id ` + + `${loser.session.sessionId}; preserving every owner`, + ); + return { ok: false, error: 'ambiguous_session_id' }; + } + + const stored = sessionStore.getSession(loser.session.sessionId); + const isOwnedRiffLoser = !loser.initConfig?.adoptMode + && !loser.adoptedFrom + && !stored?.adoptedFrom + && (loser.initConfig?.backendType ?? loser.session.backendType ?? stored?.backendType) === 'riff'; + const prepared = await prepareRiffExplicitClose(loser, stored); + if (!prepared.ok) { + return { + ok: false, + error: prepared.error, + ...(prepared.taskId ? { taskId: prepared.taskId } : {}), + }; + } + const commitRequestId = loser.riffCloseState?.phase === 'prepared' + ? loser.riffCloseState.requestId + : undefined; + + try { + if (stored?.status !== 'closed') { + if (isOwnedRiffLoser) { + sessionStore.closeSession(loser.session.sessionId, { clearRiffParentTaskId: true }); + } else { + sessionStore.closeSession(loser.session.sessionId); + } + } + } catch (err) { + if (commitRequestId) await abortLiveRiffWorkerClose(loser, commitRequestId); + logger.error( + `[setActiveSessionSafe] durable loser close failed for ${loser.session.sessionId}: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + return { + ok: false, + error: 'riff_durable_close_failed', + ...(prepared.taskId ? { taskId: prepared.taskId } : {}), + }; + } + + killWorker(loser, { + ...(commitRequestId ? { riffCloseCommitRequestId: commitRequestId } : {}), + }); + for (const [registeredKey, candidate] of map) { + if (candidate === loser) map.delete(registeredKey); + } + return { ok: true }; +} + export async function setActiveSessionSafe( map: Map, key: string, ds: DaemonSession, -): Promise { - const prev = map.get(key); - if (prev && prev !== ds) { - logger.warn( - `[setActiveSessionSafe] key already occupied by ${prev.session.sessionId.substring(0, 8)} ` + - `(worker=${prev.worker ? 'live' : 'null'}); closing it before set`, +): Promise { + const canonicalKey = activeSessionKey(ds); + if (canonicalKey !== key) { + throw new Error( + `refusing noncanonical active-session registration: requested=${key} canonical=${canonicalKey}`, ); - await closeSession(prev.session.sessionId); } - map.set(key, ds); + return withActiveSessionKeyLock(map, key, async () => { + const prev = map.get(key); + if (prev && prev !== ds) { + if (prev.session.sessionId === ds.session.sessionId) { + logger.error( + `[setActiveSessionSafe] refusing collision between distinct owners of session id ` + + `${prev.session.sessionId}; preserving both objects and the durable row`, + ); + return { + accepted: false, + reason: 'cleanup_failed', + keptSessionId: prev.session.sessionId, + preservedIncomingSessionId: ds.session.sessionId, + cleanupSessionId: prev.session.sessionId, + error: 'ambiguous_session_id', + }; + } + const prevPending = hasProtectedSessionMutationOwnership(prev); + const incomingPending = hasProtectedSessionMutationOwnership(ds); + if (prevPending && incomingPending) { + logger.error( + `[setActiveSessionSafe] refusing collision between two unsettled Codex App owners ` + + `${prev.session.sessionId} and ${ds.session.sessionId}; preserving both rows/panes`, + ); + return { + accepted: false, + reason: 'both_pending', + keptSessionId: prev.session.sessionId, + preservedIncomingSessionId: ds.session.sessionId, + }; + } + if (prevPending) { + logger.warn( + `[setActiveSessionSafe] keeping unsettled owner ${prev.session.sessionId.substring(0, 8)}; ` + + `closing ledger-empty incoming ${ds.session.sessionId.substring(0, 8)}`, + ); + const cleanup = await closeUnregisteredCollisionLoser(map, ds); + if (!cleanup.ok) { + return { + accepted: false, + reason: 'cleanup_failed', + keptSessionId: prev.session.sessionId, + preservedIncomingSessionId: ds.session.sessionId, + cleanupSessionId: ds.session.sessionId, + error: cleanup.error, + ...(cleanup.taskId ? { taskId: cleanup.taskId } : {}), + }; + } + return { + accepted: false, + reason: 'kept_pending_owner', + keptSessionId: prev.session.sessionId, + closedIncomingSessionId: ds.session.sessionId, + }; + } + logger.warn( + `[setActiveSessionSafe] key already occupied by ${prev.session.sessionId.substring(0, 8)} ` + + `(worker=${prev.worker ? 'live' : 'null'}); closing it before set`, + ); + const cleanup = await closeUnregisteredCollisionLoser(map, prev); + if (!cleanup.ok) { + return { + accepted: false, + reason: 'cleanup_failed', + keptSessionId: prev.session.sessionId, + preservedIncomingSessionId: ds.session.sessionId, + cleanupSessionId: prev.session.sessionId, + error: cleanup.error, + ...(cleanup.taskId ? { taskId: cleanup.taskId } : {}), + }; + } + } + map.set(key, ds); + return { + accepted: true, + ...(prev && prev !== ds ? { closedSessionId: prev.session.sessionId } : {}), + }; + }); } // ─── Session transfer (cross-chat relay) ──────────────────────────────────── @@ -1641,6 +2367,8 @@ export async function transferSession( forkWorkerImpl?: typeof forkWorker; /** @internal Override for tests — mirror of forkWorkerImpl for killWorker. */ killWorkerImpl?: typeof killWorker; + /** @internal Recursive marker: the bot-wide transfer mutation is held. */ + mutationHeld?: boolean; }, ): Promise<{ ok: true } | { ok: false; error: string }> { // Depth defense — unreachable per TS narrowing above, but guards against @@ -1648,9 +2376,38 @@ export async function transferSession( if ((targetChatType as string) !== 'group' && (targetChatType as string) !== 'p2p') { return { ok: false, error: 'target_chat_type_unsupported' }; } + const initial = findActiveBySessionId(sessionId); + if (!initial) return { ok: false, error: 'session_not_active' }; + if (!opts?.mutationHeld) { + // Transfer rewrites both the source and target routing identities and may + // await scratch cleanup. Drain every admitted turn for this bot first so + // no source prompt can become durable mid-transfer and no target creator + // can slip into the cleanup window. + return withBotTurnMutation(initial.larkAppId, () => transferSession( + sessionId, + targetChatId, + targetRootMessageId, + targetChatType, + targetScope, + { ...opts, mutationHeld: true }, + )); + } const ds = findActiveBySessionId(sessionId); - if (!ds) return { ok: false, error: 'session_not_active' }; + if (!ds || ds.session.status !== 'active') return { ok: false, error: 'session_not_active' }; if (ds.session.vcMeetingReceiver) return { ok: false, error: 'vc_receiver_not_relayable' }; + if ((ds.initConfig?.backendType ?? ds.session.backendType) === 'riff' + || ds.session.cliId === 'riff') { + // Riff follow-ups reuse the original remote sandbox and send only + // parentTaskId + prompt. Its BOTMUX_CHAT_ID/root credentials therefore + // remain bound to the source route; rewriting only the local session row + // would make the agent continue replying into the old chat. Preserve the + // source owner and fail before any scratch cleanup, worker retirement, M1, + // registry mutation, or durable route update. + return { ok: false, error: 'riff_not_relayable' }; + } + if (hasProtectedSessionMutationOwnership(ds)) { + return { ok: false, error: 'codex_app_dispatch_pending' }; + } // Anchor-based identity. A thread-scope session in the SAME chat (different // root) is a legitimate cross-topic move, so we refuse only when the target // anchor equals the source anchor (relaying a session onto itself). Replaces @@ -1707,22 +2464,56 @@ export async function transferSession( // Anchor-based: chat-scope anchors on chatId, thread-scope on rootMessageId. // Only a session at the target anchor collides — same-chat other-topic // sessions have a different anchor and are fine (enables同群话题间搬运). + const targetKey = sessionKey(targetAnchor, ds.larkAppId); + const sourceKey = activeSessionKey(ds); const scratchesToClose: string[] = []; if (activeSessionsRegistry) { for (const existing of activeSessionsRegistry.values()) { if (existing === ds) continue; if (existing.larkAppId !== ds.larkAppId) continue; - if (sessionAnchorId(existing) !== targetAnchor) continue; - if (!existing.worker) { - scratchesToClose.push(existing.session.sessionId); - continue; + // VC receivers intentionally share visible output routes while occupying + // an immutable `vc-receiver:` registry key. Only an owner of + // the ordinary target key can collide with this transfer. + if (activeSessionKey(existing) !== targetKey) continue; + if (isRelayableRealSession(existing) + || existing.pendingRepo + || existing.session.queued + || hasProtectedSessionMutationOwnership(existing)) { + return { ok: false, error: 'target_chat_has_session' }; } - return { ok: false, error: 'target_chat_has_session' }; + // The only disposable occupant is a proven command scratch: no worker, + // no persisted CLI markers, no pending picker/backlog, and no ledger. + scratchesToClose.push(existing.session.sessionId); } } for (const sid of scratchesToClose) { await closeSession(sid); } + // A partially hydrated daemon can have an active target row absent from the + // runtime map. Durable ownership makes that row a real conflict even when it + // has worker:null and no legacy CLI markers; only proven ledger-empty + // scratch rows may be retired. + const runtimeIds = new Set(activeSessionsRegistry + ? [...activeSessionsRegistry.values()].map(item => item.session.sessionId) + : []); + const persistedTargetConflicts = sessionStore.listSessions().filter(item => + item.sessionId !== ds.session.sessionId + && item.status === 'active' + && (!item.larkAppId || item.larkAppId === ds.larkAppId) + && !item.vcMeetingReceiver + && (item.scope === 'chat' ? item.chatId : item.rootMessageId) === targetAnchor + && !runtimeIds.has(item.sessionId), + ); + if (persistedTargetConflicts.some(item => + hasProtectedSessionMutationOwnership(item) + || !!item.cliId + || !!item.lastCliInput + || item.queued === true)) { + return { ok: false, error: 'target_chat_has_session' }; + } + for (const scratch of persistedTargetConflicts) { + await closeSession(scratch.sessionId); + } const fkw = opts?.forkWorkerImpl ?? forkWorker; const kw = opts?.killWorkerImpl ?? killWorker; @@ -1730,36 +2521,40 @@ export async function transferSession( const tagPrefix = sessionId.substring(0, 8); const oldAnchor = sessionAnchorId(ds); const oldChatId = ds.chatId; - - // Freeze the source-chat streaming card BEFORE we kill the worker (and - // before we clear streamCardId below). The live card's action buttons - // (close / toggle / get write link) carry `session_id` in their value, so - // clicks AFTER relay still reach the now-relocated session — closing it, - // toggling its display mode, etc. — with feedback landing on the NEW - // card in the target chat. PATCH the source-chat card to an inert - // snapshot so the user sees clearly it's historical, and so the buttons - // are gone. Best-effort: on PATCH failure (card withdrawn, expired) we - // log and continue; the relay itself must not depend on this. - if (ds.streamCardId && ds.streamCardId !== CARD_POSTING_SENTINEL) { - try { - const cliId = (ds.session.cliId as CliId | undefined) - ?? (() => { try { return getBot(ds.larkAppId).config.cliId; } catch { return undefined; } })(); - const frozenJson = buildRelayedFrozenCard( - ds.currentTurnTitle || ds.session.title || '', - cliId, - ds.currentImageKey, - localeForBot(ds.larkAppId), - ); - await updateMessage(ds.larkAppId, ds.streamCardId, frozenJson); - } catch (err) { - logger.warn(`[${tagPrefix}] freeze source-chat card failed: ${err instanceof Error ? err.message : err}`); - } + const oldStreamCardId = ds.streamCardId; + const oldCurrentImageKey = ds.currentImageKey; + + // Scratch/store cleanup above awaited. Re-resolve BOTH identities before + // the final synchronous move. A close/replace from a detached lifecycle + // path must win without letting this stale transfer kill/fork its successor. + if (ds.session.status !== 'active' + || findActiveBySessionId(sessionId) !== ds + || activeSessionsRegistry?.get(sourceKey) !== ds) { + return { ok: false, error: 'session_not_active' }; + } + // No await is allowed between this final reservation check and the map set + // below. A target session that arrived during the earlier scratch cleanup + // wins; the source is still live and untouched at this point. + const lateTarget = activeSessionsRegistry?.get(targetKey); + if (lateTarget && lateTarget !== ds) { + return { ok: false, error: 'target_chat_has_session' }; + } + // Scratch/store cleanup above awaited. A fresh source turn may have been + // admitted during that window; recheck in the same synchronous section as + // the final target claim so transfer never kills newly durable work. + if (hasProtectedSessionMutationOwnership(ds)) { + return { ok: false, error: 'codex_app_dispatch_pending' }; + } + if (ds.worker && !ds.worker.killed + && ds.lastScreenStatus !== 'idle' + && ds.lastScreenStatus !== 'limited') { + return { ok: false, error: 'worker_busy' }; } // Detach worker — TmuxBackend.kill() does NOT destroy the tmux session, so // the CLI process and its rolling jsonl continue running. kw(ds); - activeSessionsRegistry?.delete(sessionKey(oldAnchor, ds.larkAppId)); + activeSessionsRegistry?.delete(sourceKey); // Rewrite routing fields per the requested target scope. // chat-scope: routes by chatId; `targetRootMessageId` (e.g. an M1 id) is @@ -1792,7 +2587,36 @@ export async function transferSession( const newAnchor = sessionAnchorId(ds); if (activeSessionsRegistry) { - await setActiveSessionSafe(activeSessionsRegistry, sessionKey(newAnchor, ds.larkAppId), ds); + // targetKey was checked immediately before the synchronous kill/rewrite; + // direct publication avoids an awaited registration/rollback window where + // a stale transfer could overwrite a newly-created source successor. + activeSessionsRegistry.set(sessionKey(newAnchor, ds.larkAppId), ds); + } + + // Re-attach while the routing move is still one synchronous ownership + // commit. The source-card PATCH below awaits Lark and a detached close may + // legitimately win during that wait; deferring this fork until afterward + // would resurrect the stale transferred object after its route was removed. + fkw(ds, '', /*resume*/true); + + // Routing/store ownership is now committed. Only now freeze the old card: + // if a target arrived during an earlier await we returned with the source + // fully untouched, rather than irreversibly disabling its live controls. + // The old identifiers were captured before the target fields were cleared. + if (oldStreamCardId && oldStreamCardId !== CARD_POSTING_SENTINEL) { + try { + const cliId = (ds.session.cliId as CliId | undefined) + ?? (() => { try { return getBot(ds.larkAppId).config.cliId; } catch { return undefined; } })(); + const frozenJson = buildRelayedFrozenCard( + ds.currentTurnTitle || ds.session.title || '', + cliId, + oldCurrentImageKey, + localeForBot(ds.larkAppId), + ); + await updateMessage(ds.larkAppId, oldStreamCardId, frozenJson); + } catch (err) { + logger.warn(`[${tagPrefix}] freeze source-chat card failed: ${err instanceof Error ? err.message : err}`); + } } dashboardEventBus.publish({ @@ -1808,10 +2632,6 @@ export async function transferSession( }, }); - // forkWorker with resume=true — TmuxBackend.spawn detects the surviving - // `bmx-` session and re-attaches instead of creating a new one. - fkw(ds, '', /*resume*/true); - logger.info( `[${tagPrefix}] transferred ${oldChatId} → ${targetChatId} ` + `(anchor ${oldAnchor.substring(0, 8)} → ${newAnchor.substring(0, 8)})`, @@ -1830,18 +2650,374 @@ function resolvesToHome(p: string): boolean { catch { return p === homedir(); } } +export function codexAppCleanInputAcceptedForSession(ds: DaemonSession): boolean { + try { + const botCfg = getBot(ds.larkAppId).config; + const effectiveCliId = ds.session.cliId ?? botCfg.cliId; + return effectiveCliId === 'codex-app' + && botCfg.codexAppCleanInput === true + && !ds.adoptedFrom; + } catch { + // Admission metadata is optional for every other CLI. If bot config is + // temporarily unavailable, fail closed instead of leaking a clean sidecar. + return false; + } +} + +function cloneFrozenCodexAppInput( + input: CodexAppTurnInput | undefined, + turnId?: string, +): CodexAppTurnInput | undefined { + if (!input) return undefined; + const cloned = structuredClone(input); + if (turnId && !cloned.clientUserMessageId) cloned.clientUserMessageId = turnId; + return cloned; +} + function codexAppInputForSession( ds: DaemonSession, input: CodexAppTurnInput | undefined, turnId?: string, ): CodexAppTurnInput | undefined { if (!input) return undefined; + if (!codexAppCleanInputAcceptedForSession(ds)) return undefined; + return cloneFrozenCodexAppInput(input, turnId); +} + +function codexAppDeliverySinkForTurn( + ds: DaemonSession, + turnId: string, + dispatchAttempt: number | undefined, +): CodexAppDeliverySink { + const armedThrough = ds.suppressedFinalOutputTurns?.get(turnId); + if (dispatchAttempt !== undefined + && armedThrough !== undefined + && dispatchAttempt <= armedThrough) return 'suppressed'; + if (ds.session.docCommentTargets?.[turnId] || ds.docCommentTurns?.has(turnId)) { + return 'doc_comment'; + } + if (ds.pendingWaitPromises?.has(turnId)) return 'http_wait'; + if (ds.asyncTriggerResults?.has(turnId)) return 'http_async'; + return 'lark'; +} + +/** A recovered transient/non-IM sink has no safe provider to replay into. + * Treat it as consumed so the runner can advance, but never fall through to a + * Lark card. Doc-comment replay also fails closed: chunk posting has no durable + * per-chunk effect journal yet, so blind crash replay could duplicate comments. */ +function codexAppDeliveryMustFailClosed( + ds: DaemonSession, + entry: CodexAppDispatchLedgerEntry, +): boolean { + const sink = entry.deliverySink + ?? (ds.session.docCommentTargets?.[entry.turnId] + ? 'doc_comment' + : ds.chatId.startsWith('http_wait_') + ? 'http_wait' + : ds.chatId.startsWith('http_async_') + ? 'http_async' + : 'lark'); + if (sink === 'suppressed') return true; + if (sink === 'doc_comment') return !ds.docCommentTurns?.has(entry.turnId); + if (sink === 'http_wait') return !ds.pendingWaitPromises?.has(entry.turnId); + if (sink === 'http_async') return !ds.asyncTriggerResults?.has(entry.turnId); + return false; +} + +function acceptCodexAppDispatch( + ds: DaemonSession, + payload: { + content: string; + codexAppInput?: CodexAppTurnInput; + replyTurnId?: string; + replyTarget?: FrozenSessionReplyTarget; + quoteTargetId?: string; + replyTargetSenderOpenId?: string; + replyTargetSenderIsBot?: boolean; + queuedActivationToken?: string; + }, + turnId: string | undefined, + dispatchAttempt: number | undefined, + vcMeetingImTurnOrigin: ReturnType, + opts: { persist?: boolean } = {}, +): string | undefined { const botCfg = getBot(ds.larkAppId).config; const effectiveCliId = ds.session.cliId ?? botCfg.cliId; - if (effectiveCliId !== 'codex-app' || botCfg.codexAppCleanInput !== true || ds.adoptedFrom) return undefined; - return turnId && !input.clientUserMessageId - ? { ...input, clientUserMessageId: turnId } - : input; + if (effectiveCliId !== 'codex-app' || !turnId) return undefined; + const dispatchId = randomUUID(); + const priorLedger = ds.session.codexAppDispatchLedger; + ds.session.codexAppDispatchLedger = appendAcceptedCodexAppDispatch( + ds.session.codexAppDispatchLedger ?? [], + { + dispatchId, + turnId, + ...(payload.queuedActivationToken + ? { queuedActivationToken: payload.queuedActivationToken } + : {}), + ...(payload.replyTurnId ? { replyTurnId: payload.replyTurnId } : {}), + ...(payload.replyTarget ? { replyTarget: payload.replyTarget } : {}), + ...(payload.quoteTargetId ? { quoteTargetId: payload.quoteTargetId } : {}), + ...(payload.replyTargetSenderOpenId + ? { replyTargetSenderOpenId: payload.replyTargetSenderOpenId } + : {}), + ...(payload.replyTargetSenderIsBot !== undefined + ? { replyTargetSenderIsBot: payload.replyTargetSenderIsBot } + : {}), + deliverySink: codexAppDeliverySinkForTurn(ds, turnId, dispatchAttempt), + ...(dispatchAttempt !== undefined ? { dispatchAttempt } : {}), + content: payload.content, + ...(payload.codexAppInput ? { codexAppInput: payload.codexAppInput } : {}), + ...(vcMeetingImTurnOrigin ? { vcMeetingImTurnOrigin } : {}), + }, + ); + if (opts.persist !== false) { + try { + sessionStore.updateSession(ds.session); + } catch (err) { + ds.session.codexAppDispatchLedger = priorLedger; + throw err; + } + } + return dispatchId; +} + +function rollbackAcceptedCodexAppDispatch( + ds: DaemonSession, + dispatchId: string | undefined, + turnId: string | undefined, + dispatchAttempt: number | undefined, +): void { + if (!dispatchId || !turnId) return; + const cancelled = cancelCodexAppDispatch(ds.session.codexAppDispatchLedger ?? [], { + dispatchId, + turnId, + ...(dispatchAttempt !== undefined ? { dispatchAttempt } : {}), + }); + if (!cancelled.ok) return; + const priorLedger = ds.session.codexAppDispatchLedger; + ds.session.codexAppDispatchLedger = cancelled.ledger; + try { + sessionStore.updateSession(ds.session); + if (!hasUnsettledCodexAppDispatch(ds.session.codexAppDispatchLedger)) { + void Promise.resolve(callbacks?.onCodexAppLedgerDrained?.(ds)).catch(err => { + logger.error( + `[${ds.session.sessionId.slice(0, 8)}] post-rollback ledger-drain cleanup failed: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + }); + } + } catch { + ds.session.codexAppDispatchLedger = priorLedger; + } +} + +type QueuedWorkerForkSnapshot = { + queued: true; + queuedPrompt: Session['queuedPrompt']; + queuedCodexAppText: Session['queuedCodexAppText']; + queuedCodexAppMessageContext: Session['queuedCodexAppMessageContext']; + queuedActivationPending: Session['queuedActivationPending']; + queuedActivationToken: Session['queuedActivationToken']; + queuedActivationInput: Session['queuedActivationInput']; + queuedActivationTurnId: Session['queuedActivationTurnId']; + queuedActivationDispatchAttempt: Session['queuedActivationDispatchAttempt']; + queuedActivationResume: Session['queuedActivationResume']; + initialStartPending: DaemonSession['initialStartPending']; +}; + +type QueuedActivationJournalSnapshot = Pick< + Session, + | 'queuedActivationPending' + | 'queuedActivationToken' + | 'queuedActivationInput' + | 'queuedActivationTurnId' + | 'queuedActivationDispatchAttempt' + | 'queuedActivationResume' + | 'queuedPrompt' + | 'queuedCodexAppText' + | 'queuedCodexAppMessageContext' + | 'pendingRepoSetup' +>; + +function snapshotQueuedActivationJournal(session: Session): QueuedActivationJournalSnapshot { + return { + queuedActivationPending: session.queuedActivationPending, + queuedActivationToken: session.queuedActivationToken, + queuedActivationInput: session.queuedActivationInput, + queuedActivationTurnId: session.queuedActivationTurnId, + queuedActivationDispatchAttempt: session.queuedActivationDispatchAttempt, + queuedActivationResume: session.queuedActivationResume, + queuedPrompt: session.queuedPrompt, + queuedCodexAppText: session.queuedCodexAppText, + queuedCodexAppMessageContext: session.queuedCodexAppMessageContext, + pendingRepoSetup: session.pendingRepoSetup + ? structuredClone(session.pendingRepoSetup) + : undefined, + }; +} + +function restoreQueuedActivationJournal( + session: Session, + snapshot: QueuedActivationJournalSnapshot, +): void { + Object.assign(session, snapshot); +} + +function clearQueuedActivationJournal(session: Session): void { + session.queuedActivationPending = undefined; + session.queuedActivationToken = undefined; + session.queuedActivationInput = undefined; + session.queuedActivationTurnId = undefined; + session.queuedActivationDispatchAttempt = undefined; + session.queuedActivationResume = undefined; + session.queuedPrompt = undefined; + session.queuedCodexAppText = undefined; + session.queuedCodexAppMessageContext = undefined; + session.pendingRepoSetup = undefined; +} + +/** A worker disappeared before the exact activation ACK. Re-park non-Codex + * work from the retained exact input; ACK loss may duplicate, but N can never + * be silently replaced by a later inbound turn in the same daemon lifetime. */ +function reparkUnsubmittedQueuedActivation(ds: DaemonSession, reason: string): boolean { + if (!ds.session.queuedActivationPending + || !ds.session.queuedActivationToken + || ds.session.cliId === 'codex-app') return false; + ds.session.queued = true; + ds.session.queuedActivationPending = undefined; + ds.session.queuedActivationToken = undefined; + ds.initialStartPending = false; + ds.initialStartClaimToken = undefined; + ds.pendingRiffActivationTaskId = undefined; + ds.pendingPrompt ??= ds.session.queuedPrompt; + try { + sessionStore.updateSession(ds.session); + } catch (err) { + // Memory remains explicitly parked while the old durable marker still + // provides restart recovery on disk. + logger.error( + `[${tag(ds)}] Failed to persist queued activation re-park after ${reason}: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + } + logger.warn(`[${tag(ds)}] Re-parked unacknowledged queued activation after ${reason}`); + return true; +} + +/** The opening activation was ACKed, but one of the turns held behind its + * runtime reservation was not accepted before that worker exited. Promote the + * exact FIFO head into a new durable queued activation so the next inbound or + * Dashboard activation reforks it before every remaining tail item. */ +function reparkQueuedActivationFollowUpTail(ds: DaemonSession, reason: string): boolean { + if (!ds.initialStartPending || ds.session.queuedActivationPending) return false; + const next = ds.pendingQueuedActivationFollowUps?.[0]; + if (!next) return false; + const matchingCodexEntries = ds.session.cliId === 'codex-app' + ? (ds.session.codexAppDispatchLedger ?? []).filter(entry => + (entry.state === 'accepted' || entry.state === 'prepared') + && entry.turnId === next.turnId + && entry.dispatchAttempt === next.dispatchAttempt) + : []; + if (matchingCodexEntries.length > 1) { + logger.error( + `[${tag(ds)}] Cannot re-park queued follow-up after ${reason}: ` + + `${matchingCodexEntries.length} Codex entries match turn ${next.turnId}`, + ); + return false; + } + const retainedCodexEntry = matchingCodexEntries[0]; + const retainedCodexToken = retainedCodexEntry + ? (retainedCodexEntry.queuedActivationToken ?? randomUUID()) + : undefined; + // A failed daemon→worker IPC normally rolls its newly accepted Codex entry + // back. If that rollback persistence itself failed, the durable FIFO remains + // authoritative: recover it through a tokened ACK journal instead of + // creating an invalid queued+unsettled hybrid. + ds.session.queued = !retainedCodexEntry; + ds.session.queuedPrompt = next.cliInput.content; + ds.session.queuedCodexAppText = next.cliInput.codexAppInput?.text; + ds.session.queuedCodexAppMessageContext = undefined; + ds.session.queuedActivationInput = next.cliInput; + ds.session.queuedActivationTurnId = next.turnId; + ds.session.queuedActivationDispatchAttempt = next.dispatchAttempt; + ds.session.queuedActivationPending = retainedCodexEntry ? true : undefined; + ds.session.queuedActivationToken = retainedCodexToken; + if (retainedCodexEntry && retainedCodexToken) { + retainedCodexEntry.queuedActivationToken = retainedCodexToken; + } + ds.pendingQueuedActivationFollowUps!.shift(); + if (ds.pendingQueuedActivationFollowUps!.length === 0) { + ds.pendingQueuedActivationFollowUps = undefined; + } + ds.pendingPrompt = next.cliInput.content; + ds.initialStartPending = false; + ds.initialStartClaimToken = undefined; + ds.pendingRiffActivationTaskId = undefined; + try { + sessionStore.updateSession(ds.session); + } catch (err) { + // Keep the exact head parked in memory. The caller has already fenced the + // dead worker, so a later inbound can safely retry this owner even if the + // durable projection is temporarily unavailable. + logger.error( + `[${tag(ds)}] Failed to persist queued follow-up re-park after ${reason}: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + } + logger.warn(`[${tag(ds)}] Re-parked unaccepted queued activation follow-up after ${reason}`); + return true; +} + +export const __testOnly_reparkQueuedActivationFollowUpTail = reparkQueuedActivationFollowUpTail; + +type AcceptedWorkerForkDispatch = { + dispatchId: string; + turnId: string; + dispatchAttempt?: number; +}; + +/** Compensate only the durable mutations made before a worker accepts its init + * IPC. This is one synchronous call stack, so removing the exact dispatch that + * this fork appended cannot race a later worker transition or damage the FIFO + * that existed before the attempt. Persist the queued payload and ledger in one + * write so a daemon crash cannot observe only half of the compensation. */ +function rollbackWorkerForkPreInit( + ds: DaemonSession, + queuedSnapshot: QueuedWorkerForkSnapshot | undefined, + acceptedDispatch: AcceptedWorkerForkDispatch | undefined, +): void { + const exactActivationInput = ds.session.queuedActivationInput; + const exactActivationTurnId = ds.session.queuedActivationTurnId; + const exactActivationDispatchAttempt = ds.session.queuedActivationDispatchAttempt; + const exactActivationResume = ds.session.queuedActivationResume; + let changed = false; + if (acceptedDispatch) { + const cancelled = cancelCodexAppDispatch( + ds.session.codexAppDispatchLedger ?? [], + acceptedDispatch, + ); + if (!cancelled.ok) { + throw new Error(`failed to cancel pre-init Codex App dispatch: ${cancelled.error}`); + } + ds.session.codexAppDispatchLedger = cancelled.ledger; + changed = true; + } + if (queuedSnapshot) { + ds.session.queued = queuedSnapshot.queued; + restoreQueuedActivationJournal(ds.session, queuedSnapshot); + // The activation may have folded a triggering group reply into the final + // init payload. Retain that exact retry body even though the in-flight + // marker/token are rolled back with the fenced child. + ds.session.queuedActivationPending = undefined; + ds.session.queuedActivationToken = undefined; + ds.session.queuedActivationInput = exactActivationInput; + ds.session.queuedActivationTurnId = exactActivationTurnId; + ds.session.queuedActivationDispatchAttempt = exactActivationDispatchAttempt; + ds.session.queuedActivationResume = exactActivationResume; + ds.initialStartPending = queuedSnapshot.initialStartPending; + changed = true; + } + if (changed) sessionStore.updateSession(ds.session); } /** Send one normal (non-raw) worker turn while applying the per-bot Codex App @@ -1855,23 +3031,300 @@ export function sendWorkerInput( dispatchAttempt?: number; } = {}, ): boolean { + const riffRetirementPhase = riffRetirementAdmissionPhase(ds); + if (riffRetirementPhase) { + logger.warn( + `[${tag(ds)}] Rejected turn ${turnId ?? '?'} while Riff retirement fence is ${riffRetirementPhase}`, + ); + void callbacks?.sessionReply( + sessionAnchorId(ds), + tr('worker.riff_close_in_progress', undefined, localeForBot(ds.larkAppId)), + 'text', + ds.larkAppId, + turnId, + ).catch(err => { + logger.warn( + `[${tag(ds)}] Failed to notify rejected Riff close-race turn: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + }); + return false; + } if (!ds.worker || ds.worker.killed) return false; const normalized = typeof payload === 'string' ? { content: payload } : payload; - const codexAppInput = codexAppInputForSession(ds, normalized.codexAppInput, turnId); - const vcMeetingImTurnOrigin = resolveVcMeetingImTurnOrigin(ds.session, turnId); - ds.worker.send({ - type: 'message', - content: normalized.content, + const effectiveCliId = ds.session.cliId ?? getBot(ds.larkAppId).config.cliId; + const effectiveTurnId = turnId ?? (effectiveCliId === 'codex-app' + ? `codex-app-dispatch-${randomUUID()}` + : undefined); + const replyTurnId = turnId ? undefined : ds.currentReplyTarget?.turnId; + const routingTurnId = turnId ?? replyTurnId; + const replyContext = frozenReplyContextForTurn(ds, routingTurnId); + const vcMeetingImTurnOrigin = resolveVcMeetingImTurnOrigin(ds.session, routingTurnId); + if (hasQueuedActivationAdmissionGate(ds)) { + const queuedTurnId = effectiveTurnId + ?? routingTurnId + ?? `queued-activation-followup-${randomUUID()}`; + try { + admitQueuedActivationTail(ds, { + userPrompt: normalized.content, + cliInput: { + content: normalized.content, + ...(normalized.codexAppInput + ? { codexAppInput: normalized.codexAppInput } + : {}), + }, + turnId: queuedTurnId, + ...(opts.dispatchAttempt !== undefined + ? { dispatchAttempt: opts.dispatchAttempt } + : {}), + }); + logger.info( + `[${tag(ds)}] Staged turn ${queuedTurnId} behind queued activation ACK`, + ); + return true; + } catch (err) { + logger.error( + `[${tag(ds)}] Failed to durably stage turn ${queuedTurnId} behind activation: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + return false; + } + } + const codexAppInput = codexAppInputForSession(ds, normalized.codexAppInput, effectiveTurnId); + const codexAppDispatchId = acceptCodexAppDispatch( + ds, + { + content: normalized.content, + ...(codexAppInput ? { codexAppInput } : {}), + ...(replyTurnId ? { replyTurnId } : {}), + replyTarget: replyContext.target, + ...(replyContext.quoteTargetId ? { quoteTargetId: replyContext.quoteTargetId } : {}), + ...(replyContext.replyTargetSenderOpenId + ? { replyTargetSenderOpenId: replyContext.replyTargetSenderOpenId } + : {}), + ...(replyContext.replyTargetSenderIsBot !== undefined + ? { replyTargetSenderIsBot: replyContext.replyTargetSenderIsBot } + : {}), + }, + effectiveTurnId, + opts.dispatchAttempt, + vcMeetingImTurnOrigin, + ); + try { + ds.worker.send({ + type: 'message', + content: normalized.content, + ...(codexAppInput ? { codexAppInput } : {}), + ...(effectiveTurnId ? { turnId: effectiveTurnId } : {}), + ...(replyTurnId ? { replyTurnId } : {}), + ...(opts.dispatchAttempt !== undefined ? { dispatchAttempt: opts.dispatchAttempt } : {}), + ...(codexAppDispatchId ? { codexAppDispatchId } : {}), + ...(vcMeetingImTurnOrigin + ? { vcMeetingImTurnOrigin } + : {}), + } as DaemonToWorker); + } catch { + rollbackAcceptedCodexAppDispatch(ds, codexAppDispatchId, effectiveTurnId, opts.dispatchAttempt); + return false; + } + return true; +} + +/** Promote the oldest durable activation successor into a fresh tokened + * journal and (for Codex App) its single accepted-ledger owner in one store + * update, then hand it to the current worker. The tail is never shifted merely + * because ChildProcess.send returned: the promoted journal survives until the + * adapter ACK carrying this token arrives. */ +export function promoteQueuedActivationTail( + ds: DaemonSession, + opts: { send?: boolean } = {}, +): boolean { + if (ds.session.queuedActivationPending) return true; + if (opts.send !== false + && (!ds.worker || ds.worker.killed || ds.riffCloseState || ds.riffShutdownState)) return false; + const ordered = [...(ds.session.queuedActivationTail ?? [])] + .sort((a, b) => a.order - b.order || a.id.localeCompare(b.id)); + const head = ordered[0]; + if (!head) return false; + + const priorJournal = snapshotQueuedActivationJournal(ds.session); + const priorTail = ds.session.queuedActivationTail?.map(entry => ({ + ...entry, + cliInput: { + ...entry.cliInput, + ...(entry.cliInput.codexAppInput + ? { codexAppInput: structuredClone(entry.cliInput.codexAppInput) } + : {}), + }, + })); + const priorLedger = ds.session.codexAppDispatchLedger?.map(entry => ({ ...entry })); + const priorPendingPrompt = ds.pendingPrompt; + const token = randomUUID(); + const replyContext = frozenReplyContextForTurn(ds, head.turnId); + // The tail entry crossed its admission boundary with the clean-input gate + // already frozen. Do not re-run that immediate bot-config gate at promotion: + // a true→false toggle between N+1 admission and N's ACK would otherwise + // discard N+1's exact sidecar/hidden context. Only fill the deterministic id + // for legacy entries persisted before admission began stamping it. + const codexAppInput = cloneFrozenCodexAppInput( + head.cliInput.codexAppInput, + head.turnId, + ); + const exactInput: CliTurnPayload = { + content: head.cliInput.content, ...(codexAppInput ? { codexAppInput } : {}), - ...(turnId ? { turnId } : {}), - ...(opts.dispatchAttempt !== undefined ? { dispatchAttempt: opts.dispatchAttempt } : {}), - ...(vcMeetingImTurnOrigin - ? { vcMeetingImTurnOrigin } - : {}), - } as DaemonToWorker); + }; + const vcMeetingImTurnOrigin = resolveVcMeetingImTurnOrigin(ds.session, head.turnId); + + ds.session.queued = false; + ds.session.queuedActivationPending = true; + ds.session.queuedActivationToken = token; + ds.session.queuedActivationInput = exactInput; + ds.session.queuedActivationTurnId = head.turnId; + ds.session.queuedActivationDispatchAttempt = head.dispatchAttempt; + ds.session.queuedActivationResume = ds.hasHistory; + ds.session.queuedActivationTail = ordered.slice(1); + if (ds.session.queuedActivationTail.length === 0) { + ds.session.queuedActivationTail = undefined; + } + ds.pendingPrompt = exactInput.content; + ds.initialStartPending = true; + ds.pendingRiffActivationTaskId = undefined; + + let codexAppDispatchId: string | undefined; + try { + codexAppDispatchId = acceptCodexAppDispatch( + ds, + { + content: exactInput.content, + ...(codexAppInput ? { codexAppInput } : {}), + queuedActivationToken: token, + replyTarget: replyContext.target, + ...(replyContext.quoteTargetId ? { quoteTargetId: replyContext.quoteTargetId } : {}), + ...(replyContext.replyTargetSenderOpenId + ? { replyTargetSenderOpenId: replyContext.replyTargetSenderOpenId } + : {}), + ...(replyContext.replyTargetSenderIsBot !== undefined + ? { replyTargetSenderIsBot: replyContext.replyTargetSenderIsBot } + : {}), + }, + head.turnId, + head.dispatchAttempt, + vcMeetingImTurnOrigin, + { persist: false }, + ); + sessionStore.updateSession(ds.session); + } catch (err) { + restoreQueuedActivationJournal(ds.session, priorJournal); + ds.session.queuedActivationTail = priorTail; + ds.session.codexAppDispatchLedger = priorLedger; + ds.pendingPrompt = priorPendingPrompt; + logger.error( + `[${tag(ds)}] Failed to atomically promote queued activation tail ${head.id}: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + return false; + } + + if (opts.send === false) return true; + try { + ds.worker!.send({ + type: 'message', + content: exactInput.content, + ...(codexAppInput ? { codexAppInput } : {}), + turnId: head.turnId, + ...(head.dispatchAttempt !== undefined + ? { dispatchAttempt: head.dispatchAttempt } + : {}), + ...(codexAppDispatchId ? { codexAppDispatchId } : {}), + queuedActivationToken: token, + ...(vcMeetingImTurnOrigin ? { vcMeetingImTurnOrigin } : {}), + } as DaemonToWorker); + } catch (err) { + // Durable ownership already moved to the journal (and Codex ledger). Never + // append another owner on retry; fence this IPC generation and let recovery + // replay the exact tokened head. + logger.error( + `[${tag(ds)}] Worker IPC rejected promoted activation ${head.id}; retaining journal: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + try { ds.worker!.kill(); } catch { /* exit/error path will fence runtime */ } + } return true; } +export type QueuedActivationTailReservation = Pick & { + /** Clean-input decision captured synchronously with FIFO order, before any + * caller-specific async prompt construction. This field is runtime-only. */ + codexAppInputAccepted?: boolean; +}; + +/** Reserve arrival order synchronously, before any caller-specific prompt + * rendering may await. Gaps are harmless; reusing an order after a failed + * durable admission would not be. */ +export function reserveQueuedActivationTailAdmission( + ds: DaemonSession, +): QueuedActivationTailReservation { + const order = (ds.session.queuedActivationTailNextOrder ?? 0) + 1; + ds.session.queuedActivationTailNextOrder = order; + return { + id: randomUUID(), + order, + codexAppInputAccepted: codexAppCleanInputAcceptedForSession(ds), + }; +} + +/** Admit one exact successor behind a queued activation. The response boundary + * is the session-store write: callers must not report acceptance or use live + * worker IPC unless this returns. `reservation` lets routes reserve FIFO order + * before asynchronous prompt construction without duplicating persistence + * logic. */ +export function admitQueuedActivationTail( + ds: DaemonSession, + entry: Omit, + reservation: QueuedActivationTailReservation = reserveQueuedActivationTailAdmission(ds), + opts: { codexAppInputGateFrozen?: boolean } = {}, +): QueuedActivationTailEntry { + const acceptedCodexAppInput = opts.codexAppInputGateFrozen === true + ? cloneFrozenCodexAppInput(entry.cliInput.codexAppInput, entry.turnId) + : reservation.codexAppInputAccepted === true + ? cloneFrozenCodexAppInput(entry.cliInput.codexAppInput, entry.turnId) + : undefined; + const admitted: QueuedActivationTailEntry = { + id: reservation.id, + order: reservation.order, + ...entry, + cliInput: { + content: entry.cliInput.content, + ...(acceptedCodexAppInput ? { codexAppInput: acceptedCodexAppInput } : {}), + }, + }; + const priorTail = ds.session.queuedActivationTail; + const next = [...(priorTail ?? [])]; + if (!next.some(candidate => candidate.id === admitted.id)) next.push(admitted); + next.sort((a, b) => a.order - b.order || a.id.localeCompare(b.id)); + ds.session.queuedActivationTail = next; + try { + sessionStore.updateSession(ds.session); + } catch (err) { + ds.session.queuedActivationTail = priorTail; + throw err; + } + return admitted; +} + +/** True while a live worker's opening activation still owns submission order. + * Every ingress that sees this state must use admitQueuedActivationTail rather + * than ordinary worker IPC. */ +export function hasQueuedActivationAdmissionGate(ds: DaemonSession): boolean { + return ds.session.queuedActivationPending === true + || (ds.session.queuedActivationTail?.length ?? 0) > 0 + || (ds.queuedActivationTailAdmissionsOutstanding ?? 0) > 0 + || ds.queuedActivationTailReleasePending !== undefined + || (ds.initialStartPending === true + && ds.session.queuedActivationInput !== undefined); +} + export function forkWorker( ds: DaemonSession, promptInput: string | CliTurnPayload, @@ -1879,6 +3332,9 @@ export function forkWorker( resume?: boolean; turnId?: string; dispatchAttempt?: number; + /** The payload is an exact retained activation journal. Do not re-read the + * live clean-input feature flag on retry/replay. */ + codexAppInputGateFrozen?: boolean; } = false, ): void { // Device enrollment briefly freezes every daemon before the one-way host @@ -1894,45 +3350,161 @@ export function forkWorker( logger.info(`[${tag(ds)}] worker spawn deferred during device credential activation`); return; } + const retirementPhase = riffRetirementAdmissionPhase(ds); + if (retirementPhase) { + // This must precede queued-journal, cwd, sandbox, dispatch-ledger, and child + // mutations. A workerless retained shutdown fence owns the exact pending + // payload just as strongly as a still-live fenced worker generation. + logger.warn( + `[${tag(ds)}] Refused worker fork while Riff retirement fence is ${retirementPhase}`, + ); + throw new Error(`riff_retirement_fence:${retirementPhase}`); + } + const promptPayload = typeof promptInput === 'string' ? { content: promptInput } : promptInput; + const prompt = promptPayload.content; + let resume = false; + let initTurnId: string | undefined; + let initDispatchAttempt: number | undefined; + let initCodexAppInputGateFrozen = promptInput === ds.session.queuedActivationInput; + if (typeof resumeOrTurnId === 'string') { + initTurnId = resumeOrTurnId; + } else if (typeof resumeOrTurnId === 'object' && resumeOrTurnId !== null) { + resume = resumeOrTurnId.resume === true; + initTurnId = resumeOrTurnId.turnId; + initDispatchAttempt = resumeOrTurnId.dispatchAttempt; + initCodexAppInputGateFrozen ||= resumeOrTurnId.codexAppInputGateFrozen === true; + } else { + resume = resumeOrTurnId; + } + if (ds.session.queuedActivationPending && ds.session.queuedActivationResume !== undefined) { + resume = ds.session.queuedActivationResume; + } + + const liveFrozenBackendType = ds.initConfig?.backendType ?? ds.session.backendType; + if (ds.worker && !ds.worker.killed && liveFrozenBackendType === 'riff') { + // Riff has no safe synchronous detach boundary: task-execute/follow-up may + // be awaiting its task id. Preserve that exact worker generation so its + // onTaskId IPC can durably advance lineage. A real prompt is just another + // ordinary Riff follow-up; an empty restore/refork is a no-op. + if (prompt.length === 0) { + logger.warn(`[${tag(ds)}] Refused empty Riff worker refork; existing generation retained`); + return; + } + const routed = sendWorkerInput(ds, promptPayload, initTurnId, { + ...(initDispatchAttempt !== undefined ? { dispatchAttempt: initDispatchAttempt } : {}), + }); + logger[routed ? 'info' : 'warn']( + `[${tag(ds)}] ${routed ? 'Routed' : 'Failed to route'} double-fork prompt through existing Riff worker`, + ); + return; + } + + // Central double-fork guard. A live worker with durable ownership must never + // be killed and replaced: empty reforks are rejected, while a real follow-up + // is appended through the existing worker's ordinary durable FIFO. Keep this + // before queued/cwd/sandbox mutations so a rejected refork is side-effect free. + if (ds.worker && !ds.worker.killed + && hasProtectedSessionMutationOwnership(ds)) { + if (prompt.length === 0) { + logger.warn(`[${tag(ds)}] Refused empty worker refork while durable activation ownership is non-empty`); + return; + } + if (hasQueuedActivationAdmissionGate(ds)) { + const turnId = initTurnId + ?? ds.currentReplyTarget?.turnId + ?? `queued-activation-followup-${randomUUID()}`; + admitQueuedActivationTail(ds, { + userPrompt: prompt, + cliInput: { + content: prompt, + ...(promptPayload.codexAppInput + ? { codexAppInput: promptPayload.codexAppInput } + : {}), + }, + turnId, + ...(initDispatchAttempt !== undefined + ? { dispatchAttempt: initDispatchAttempt } + : {}), + }); + logger.info( + `[${tag(ds)}] Staged double-fork prompt behind queued activation ACK ` + + `(turn=${turnId})`, + ); + return; + } + const routed = sendWorkerInput(ds, promptPayload, initTurnId, { + ...(initDispatchAttempt !== undefined ? { dispatchAttempt: initDispatchAttempt } : {}), + }); + logger[routed ? 'info' : 'warn']( + `[${tag(ds)}] ${routed ? 'Routed' : 'Failed to route'} double-fork prompt through existing durable owner`, + ); + return; + } + const cb = requireCallbacks(); const bot = getBot(ds.larkAppId); const botCfg = bot.config; - const promptPayload = typeof promptInput === 'string' ? { content: promptInput } : promptInput; - const prompt = promptPayload.content; - // 不变式：一旦真正起 CLI，会话就不再是「待办池(queued)」parked 态。无论由哪条 - // 路径触发（激活按钮 / 拖到进行中 / 群里来消息抢先起会话），都在此清掉 queued - // 标记并落盘——否则重启后会被当 parked 恢复成 hasHistory:false 而丢掉真历史。 - if (ds.session.queued) { + // A bare /repo placeholder (and a non-Codex empty group-join setup) owns no + // model turn. Starting its CLI with an empty prompt must not mint a queued + // activation token: the worker has nothing to submit and could never ACK it. + // Real raw openings and clean Codex sidecars remain tokened. + if (ds.session.queued + && ds.session.pendingRepoSetup + && prompt.length === 0 + && !promptPayload.codexAppInput + && !ds.pendingRawInput + && (ds.session.queuedActivationTail?.length ?? 0) === 0) { + const priorJournal = snapshotQueuedActivationJournal(ds.session); ds.session.queued = false; - ds.session.queuedPrompt = undefined; - ds.session.queuedCodexAppText = undefined; - ds.session.queuedCodexAppMessageContext = undefined; - sessionStore.updateSession(ds.session); + clearQueuedActivationJournal(ds.session); + try { + sessionStore.updateSession(ds.session); + } catch (err) { + ds.session.queued = true; + restoreQueuedActivationJournal(ds.session, priorJournal); + throw err; + } } - // worker.js lives in the same directory as daemon.js (src/) - const workerPath = join(__dirname, '..', 'worker.js'); + const queuedForkSnapshot: QueuedWorkerForkSnapshot | undefined = ds.session.queued + ? { + queued: true, + queuedPrompt: ds.session.queuedPrompt, + queuedCodexAppText: ds.session.queuedCodexAppText, + queuedCodexAppMessageContext: ds.session.queuedCodexAppMessageContext, + queuedActivationPending: ds.session.queuedActivationPending, + queuedActivationToken: ds.session.queuedActivationToken, + queuedActivationInput: ds.session.queuedActivationInput, + queuedActivationTurnId: ds.session.queuedActivationTurnId, + queuedActivationDispatchAttempt: ds.session.queuedActivationDispatchAttempt, + queuedActivationResume: ds.session.queuedActivationResume, + initialStartPending: ds.initialStartPending, + } + : undefined; + let acceptedForkDispatch: AcceptedWorkerForkDispatch | undefined; + let spawnedWorker: ChildProcess | undefined; + let worker!: ChildProcess; + let startupState!: WorkerStartupState; + let initMsg!: DaemonToWorker; + let agentCfg!: ReturnType; const t = tag(ds); ds.localProcessAttestation = undefined; - - let resume = false; - let initTurnId: string | undefined; - let initDispatchAttempt: number | undefined; - if (typeof resumeOrTurnId === 'string') { - initTurnId = resumeOrTurnId; - } else if (typeof resumeOrTurnId === 'object' && resumeOrTurnId !== null) { - resume = resumeOrTurnId.resume === true; - initTurnId = resumeOrTurnId.turnId; - initDispatchAttempt = resumeOrTurnId.dispatchAttempt; - } else { - resume = resumeOrTurnId; - } + try { + // worker.js lives in the same directory as daemon.js (src/) + const workerPath = join(__dirname, '..', 'worker.js'); // Per-turn authority is never inferred from mutable session state. Human // message routes pass their accepted Lark message id explicitly; restore, // scheduler, card retry and other system starts stay unattributed. Falling // back to currentReplyTarget here would let a later system prompt reuse an // older human turn after a worker replacement. - const initAttributionTurnId = initTurnId; + let initAttributionTurnId = initTurnId + ?? (queuedForkSnapshot ? ds.session.pendingRepoSetup?.turnId : undefined); + // Reply routing is frozen only from the same explicitly accepted turn. A + // system prompt must not borrow an older mutable currentReplyTarget. + const initReplyTurnId = initTurnId; + const initReplyContext = prompt.length > 0 + ? frozenReplyContextForTurn(ds, initReplyTurnId) + : undefined; // A fork() whose cwd no longer exists emits an unhandled 'error' (spawn // ENOENT) that crashes the WHOLE daemon (→ pm2 crash-loop). Fall back to @@ -1992,8 +3564,17 @@ export function forkWorker( // Guard against double-fork: if a worker is already running, kill it first if (ds.worker && !ds.worker.killed) { + const existingBackendType = ds.initConfig?.backendType ?? ds.session.backendType; + if (existingBackendType === 'riff') { + // Defense in depth for any future mutation inserted between the early + // Riff guard and this legacy replacement block. + logger.error(`[${t}] Refused unsafe live Riff replacement at fork boundary`); + return; + } logger.warn(`[${t}] Worker already running (pid: ${ds.worker.pid}), killing before re-fork`); - try { ds.worker.send({ type: 'close' } as DaemonToWorker); } catch { /* ignore */ } + try { + ds.worker.send({ type: 'close' } as DaemonToWorker); + } catch { /* ignore */ } try { ds.worker.kill(); } catch { /* ignore */ } ds.worker = null; ds.workerPort = null; @@ -2016,7 +3597,10 @@ export function forkWorker( // real bmx-; without this, bmx-diag- + its .ansi file would leak. if (!ds.initConfig?.adoptMode && !ds.adoptedFrom) reclaimParkedCrashDiagnostic(ds); - const agentCfg = sessionAgentConfig(ds, botCfg); + agentCfg = sessionAgentConfig(ds, botCfg); + if (!initTurnId && prompt.length > 0 && agentCfg.cliId === 'codex-app') { + initAttributionTurnId = `codex-app-dispatch-${randomUUID()}`; + } ensureCliEnv(agentCfg.cliId, agentCfg.cliPathOverride); // Claude Code blocks on the interactive folder-trust dialog the first time // it runs in an untrusted workingDir; pre-accept it so the spawn doesn't hang. @@ -2025,6 +3609,17 @@ export function forkWorker( // (`~/.claude.json` for claude, `.claude-runtime/.claude.json` for seed). const familyAdapter = createCliAdapterSync(agentCfg.cliId, agentCfg.cliPathOverride); if (familyAdapter.claudeStateJsonPath) ensureClaudeFolderTrust(cwd, familyAdapter.claudeStateJsonPath); + const resolvedBackendType = resolvePairedSpawnBackendType( + agentCfg.cliId, + ds.session.backendType, + botCfg.backendType, + config.daemon.backendType, + ); + if (ds.session.cliId !== agentCfg.cliId || ds.session.backendType !== resolvedBackendType) { + ds.session.cliId = agentCfg.cliId; + ds.session.backendType = resolvedBackendType; + sessionStore.updateSession(ds.session); + } // Prepend ~/.botmux/bin to PATH so CLIs can call `botmux send` etc. // The wrapper script there is written by the daemon at startup. @@ -2032,7 +3627,105 @@ export function forkWorker( const pathWithBotmux = prependBotmuxBin(botmuxBinDir, process.env.PATH); const forkEnv = workerForkEnv(process.env); - const worker = fork(workerPath, [], { + // Dequeue only after every earlier launch-preparation write has finished. + // The exact input remains journaled until the worker confirms its adapter + // submission boundary; daemon send/ready are intentionally too early. + const recoveredActivationLedgerEntry = ds.session.queuedActivationPending + ? [...(ds.session.codexAppDispatchLedger ?? [])] + .reverse() + .find(entry => entry.state === 'accepted' || entry.state === 'prepared') + : undefined; + const queuedActivationToken = queuedForkSnapshot + ? randomUUID() + : ds.session.queuedActivationPending + ? (ds.session.queuedActivationToken + ?? recoveredActivationLedgerEntry?.queuedActivationToken + ?? randomUUID()) + : undefined; + if (queuedForkSnapshot) { + ds.session.queued = false; + ds.session.queuedActivationPending = true; + ds.session.queuedActivationToken = queuedActivationToken; + ds.session.queuedActivationResume = resume; + ds.initialStartPending = true; + ds.pendingRiffActivationTaskId = undefined; + } else if (ds.session.queuedActivationPending && queuedActivationToken) { + // Migrate journals written before the token/ACK protocol. The queued + // activation is the newest unsettled FIFO entry; stamp the same token onto + // both journal and entry before the recovery worker can submit it. + let migrated = ds.session.queuedActivationToken !== queuedActivationToken; + ds.session.queuedActivationToken = queuedActivationToken; + ds.initialStartPending = true; + ds.pendingRiffActivationTaskId = undefined; + if (recoveredActivationLedgerEntry + && recoveredActivationLedgerEntry.queuedActivationToken !== queuedActivationToken) { + recoveredActivationLedgerEntry.queuedActivationToken = queuedActivationToken; + migrated = true; + } + if (migrated) sessionStore.updateSession(ds.session); + } + + // Snapshot the prior durable FIFO before accepting this fork's new prompt. + // A pure reattach receives the full snapshot; a refork carrying N+1 restores + // old N first and reserves N+1 through the normal worker write path. + const codexAppRecoveredDispatches = agentCfg.cliId === 'codex-app' + ? (ds.session.codexAppDispatchLedger ?? []).map(entry => ({ ...entry })) + : []; + const promptCodexAppInput = initCodexAppInputGateFrozen + ? cloneFrozenCodexAppInput(promptPayload.codexAppInput, initAttributionTurnId) + : codexAppInputForSession(ds, promptPayload.codexAppInput, initAttributionTurnId); + if (queuedForkSnapshot) { + ds.session.queuedActivationInput = { + content: prompt, + ...(promptCodexAppInput ? { codexAppInput: promptCodexAppInput } : {}), + }; + ds.session.queuedActivationTurnId = initAttributionTurnId; + ds.session.queuedActivationDispatchAttempt = initDispatchAttempt; + } + const initVcMeetingImTurnOrigin = resolveVcMeetingImTurnOrigin( + ds.session, + initTurnId ?? initReplyTurnId, + ); + const codexAppDispatchId = prompt.length > 0 + ? acceptCodexAppDispatch( + ds, + { + content: prompt, + ...(promptCodexAppInput ? { codexAppInput: promptCodexAppInput } : {}), + ...(queuedActivationToken ? { queuedActivationToken } : {}), + ...(initReplyTurnId ? { replyTurnId: initReplyTurnId } : {}), + ...(initReplyContext ? { + replyTarget: initReplyContext.target, + ...(initReplyContext.quoteTargetId + ? { quoteTargetId: initReplyContext.quoteTargetId } + : {}), + ...(initReplyContext.replyTargetSenderOpenId + ? { replyTargetSenderOpenId: initReplyContext.replyTargetSenderOpenId } + : {}), + ...(initReplyContext.replyTargetSenderIsBot !== undefined + ? { replyTargetSenderIsBot: initReplyContext.replyTargetSenderIsBot } + : {}), + } : {}), + }, + initAttributionTurnId, + initDispatchAttempt, + initVcMeetingImTurnOrigin, + ) + : undefined; + if (codexAppDispatchId && initAttributionTurnId) { + acceptedForkDispatch = { + dispatchId: codexAppDispatchId, + turnId: initAttributionTurnId, + ...(initDispatchAttempt !== undefined + ? { dispatchAttempt: initDispatchAttempt } + : {}), + }; + } + if (queuedForkSnapshot && !codexAppDispatchId) { + sessionStore.updateSession(ds.session); + } + + worker = fork(workerPath, [], { windowsHide: true, stdio: ['ignore', 'pipe', 'pipe', 'ipc'], cwd, @@ -2047,7 +3740,8 @@ export function forkWorker( LARK_APP_SECRET: botCfg.larkAppSecret, }, } as WindowsForkOptions); - const startupState: WorkerStartupState = { + spawnedWorker = worker; + startupState = { ready: false, failureNotified: false, initTurnId: initAttributionTurnId, initDispatchAttempt, @@ -2059,6 +3753,33 @@ export function forkWorker( worker.on('error', (err) => { const reason = (err as Error)?.message ?? String(err); logger.error(`[${t}] Worker fork error: ${reason}`); + if (ds.worker === worker) { + const retainExactRetirementGeneration = ds.riffShutdownState !== undefined + || ds.riffCloseState !== undefined; + if (ds.session.queuedActivationPending) { + // Retain every backend's exact tokened head. Non-Codex/Riff recovery is + // at-least-once but must preserve its frozen resume mode; converting a + // historical successor back into backlog (`queued`) loses context. + ds.initialStartPending = false; + ds.initialStartClaimToken = undefined; + ds.pendingRiffActivationTaskId = undefined; + } else { + reparkQueuedActivationFollowUpTail(ds, 'worker error during activation follow-up handoff'); + } + if (!retainExactRetirementGeneration) { + ds.worker = null; + ds.workerPort = null; + ds.workerToken = null; + ds.workerViewToken = null; + ds.managedTurnOrigin = undefined; + ds.riffCloseState = undefined; + } + // The Riff shutdown/close coordinator owns these fences. An exact worker + // can fail after backend prepare but before abort/persistence is verified; + // retain its exact ChildProcess pointer until exit or commit. Clearing it + // on `error` before exitCode changes loses the only generation handle. + try { worker.kill(); } catch { /* best-effort failed-child fence */ } + } if (startupState.failureNotified) return; startupState.failureNotified = true; emitSessionLifecycleHook(ds, 'session.requires_attention', { @@ -2108,12 +3829,7 @@ export function forkWorker( }); // Send init config — use per-bot settings - const promptCodexAppInput = codexAppInputForSession( - ds, - promptPayload.codexAppInput, - initAttributionTurnId, - ); - const initMsg: DaemonToWorker = { + initMsg = { type: 'init', sessionId: ds.session.sessionId, chatId: ds.chatId, @@ -2156,13 +3872,14 @@ export function forkWorker( // the real persistent pane (the stamp is written below; restore reads it via // getSessionPersistentBackendType). A brand-new session (no stamp) resolves // from live config, so a dashboard backend switch only affects NEW sessions. - backendType: resolvePairedSpawnBackendType(agentCfg.cliId, ds.session.backendType, botCfg.backendType, config.daemon.backendType), + backendType: resolvedBackendType, backendConfig: botCfg.riff, riffParentTaskId: ds.session.riffParentTaskId, riffRepoDirs: ds.session.riffRepoDirs, deferredScheduleRun: ds.session.deferredScheduleRun, prompt, ...(promptCodexAppInput ? { promptCodexAppInput } : {}), + ...(queuedActivationToken ? { queuedActivationToken } : {}), resume, cliSessionId: ds.session.cliSessionId, ownerOpenId: ds.ownerOpenId, @@ -2174,67 +3891,104 @@ export function forkWorker( botOpenId: bot.botOpenId, locale: botLocale(botCfg), turnId: initAttributionTurnId, + ...(initReplyTurnId ? { replyTurnId: initReplyTurnId } : {}), dispatchAttempt: initDispatchAttempt, - vcMeetingImTurnOrigin: resolveVcMeetingImTurnOrigin( - ds.session, - initAttributionTurnId, - ), + ...(codexAppDispatchId ? { codexAppDispatchId } : {}), + ...(codexAppRecoveredDispatches.length > 0 + ? { codexAppRecoveredDispatches } + : {}), + ...((ds.session.codexAppGenerationCommits?.length ?? 0) > 0 + ? { codexAppGenerationCommits: ds.session.codexAppGenerationCommits } + : {}), + vcMeetingImTurnOrigin: initVcMeetingImTurnOrigin, pluginBindings: botCfg.plugins, skillPolicy: botCfg.skills, }; - worker.send(initMsg); - ds.initConfig = initMsg; - - // Stamp cliId on the persisted session so the dashboard can show a CLI badge - // even after the session is closed. Do this before installing worker handlers: - // a fast worker can emit `ready` immediately after init, and card rendering - // must see the session-level CLI identity rather than the bot default. - if (ds.session.cliId !== agentCfg.cliId) { - ds.session.cliId = agentCfg.cliId; - sessionStore.updateSession(ds.session); - } - - // Stamp the resolved backend on the persisted session. Since PTY退役, the - // worker no longer silently downgrades an unavailable backend (it hard-gates - // instead), so the requested backend here IS the effective one for any - // session that actually runs. Restore reads this back (see - // getSessionPersistentBackendType) so an upgraded daemon doesn't re-derive a - // session's backend from the now-always-tmux default and misclassify a legacy - // PTY session as a tmux zombie. - if (ds.session.backendType !== initMsg.backendType) { - ds.session.backendType = initMsg.backendType; - sessionStore.updateSession(ds.session); - } - - // Use shared handler for IPC messages and exit + // Install ownership and every IPC handler before init can produce a fast + // submission ACK. If send throws synchronously, the pre-init catch below + // fences this child and rolls the durable journal/FIFO back. setupWorkerHandlers(ds, worker, startupState); - ds.worker = worker; ds.spawnedAt = Date.now(); ds.cliVersion = currentCliVersion; - sessionStore.updateSessionPid(ds.session.sessionId, worker.pid ?? null); + worker.send(initMsg); + // A later child 'error' event is ambiguous: init may already be executing, + // so only synchronous failures before this point are safe to compensate. + spawnedWorker = undefined; + } catch (err) { + if (ds.worker === spawnedWorker) { + ds.worker = null; + ds.workerPort = null; + ds.workerToken = null; + ds.workerViewToken = null; + } + if (spawnedWorker) { + try { spawnedWorker.kill(); } catch { /* best-effort pre-init child fence */ } + } + try { + rollbackWorkerForkPreInit(ds, queuedForkSnapshot, acceptedForkDispatch); + } catch (rollbackErr) { + logger.error( + `[${tag(ds)}] Worker pre-init failure could not durably restore queued/FIFO state: ` + + `${rollbackErr instanceof Error ? rollbackErr.message : String(rollbackErr)}`, + ); + throw new AggregateError( + [err, rollbackErr], + `Worker pre-init failed and durable rollback failed for ${ds.session.sessionId}`, + ); + } + throw err; + } + ds.initConfig = initMsg; + try { + sessionStore.updateSessionPid(ds.session.sessionId, worker.pid ?? null); + } catch (err) { + // Init may already be executing. PID bookkeeping failure must not turn an + // accepted activation into a retryable error or orphan the attached child. + logger.error(`[${t}] Failed to persist attached worker pid: ${err instanceof Error ? err.message : String(err)}`); + } logger.info(`[${t}] Worker forked (pid: ${worker.pid}, active: ${cb.getActiveCount()})`); // Reset the exit-emit flag for the freshly spawned worker so a subsequent // exit publishes again (the previous lifecycle's flag would otherwise mask it). ds.exitEventEmitted = false; // Notify dashboard SSE subscribers a new session is live. - dashboardEventBus.publish({ - type: 'session.spawned', - body: { session: composeRowFromActive(ds) }, - }); - cb.enforceLiveSessionCap?.(); - emitSessionLifecycleHook(ds, 'session.start', { - reason: resume ? 'resume' : 'worker_spawn', - pid: worker.pid ?? null, - }); + try { + dashboardEventBus.publish({ + type: 'session.spawned', + body: { session: composeRowFromActive(ds) }, + }); + } catch (err) { + logger.error(`[${t}] Failed to publish attached worker state: ${err instanceof Error ? err.message : String(err)}`); + } + try { + cb.enforceLiveSessionCap?.(); + } catch (err) { + logger.error(`[${t}] Failed to enforce live-session cap after attach: ${err instanceof Error ? err.message : String(err)}`); + } + try { + emitSessionLifecycleHook(ds, 'session.start', { + reason: resume ? 'resume' : 'worker_spawn', + pid: worker.pid ?? null, + }); + } catch (err) { + logger.error(`[${t}] Failed to emit attached worker lifecycle hook: ${err instanceof Error ? err.message : String(err)}`); + } // Usage ledger: fresh spawns anchor the baseline so pre-existing transcript // history is never billed. Restores reconcile instead — an in-flight turn // may have completed inside tmux while the daemon was down, and that work // was submitted by botmux (anchoring would swallow it). - if (resume) reconcileUsageForDaemonSession(ds); - else anchorUsageForDaemonSession(ds); - recordOwnershipForDaemonSession(ds); + try { + if (resume) reconcileUsageForDaemonSession(ds); + else anchorUsageForDaemonSession(ds); + } catch (err) { + logger.error(`[${t}] Failed to initialize attached worker usage state: ${err instanceof Error ? err.message : String(err)}`); + } + try { + recordOwnershipForDaemonSession(ds); + } catch (err) { + logger.error(`[${t}] Failed to record attached worker ownership: ${err instanceof Error ? err.message : String(err)}`); + } } // ─── Shared worker IPC handler ────────────────────────────────────────────── @@ -2385,6 +4139,125 @@ function setupWorkerHandlers( }; break; } + case 'queued_activation_submitted': { + if (ds.worker !== worker || msg.sessionId !== ds.session.sessionId) break; + if (!ds.session.queuedActivationPending + || !ds.session.queuedActivationToken + || ds.session.queuedActivationToken !== msg.activationToken) { + logger.warn(`[${t}] Ignored stale queued activation ACK ${msg.activationToken.substring(0, 8)}`); + break; + } + const releaseReservation = async (attempt: number): Promise => { + if (ds.worker !== worker || !ds.initialStartPending) return; + try { + if (cb.onQueuedActivationSubmitted) { + const released = await cb.onQueuedActivationSubmitted(ds, msg.activationToken); + if (released === false && ds.worker === worker && ds.initialStartPending) { + const delayMs = Math.min(100 * (2 ** Math.min(attempt, 6)), 5_000); + logger.warn( + `[${t}] Queued activation follow-up was not accepted (attempt ${attempt + 1}); ` + + `retrying in ${delayMs}ms`, + ); + const timer = setTimeout(() => { void releaseReservation(attempt + 1); }, delayMs); + timer.unref?.(); + } + } else { + ds.initialStartPending = false; + ds.initialStartClaimToken = undefined; + } + } catch (err) { + logger.error( + `[${t}] Queued activation follow-up release failed unexpectedly; ` + + `delivery was not retried because acceptance is unknown: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + } + }; + + const frozenBackendType = ds.initConfig?.backendType ?? ds.session.backendType; + const frozenRiff = frozenBackendType === 'riff'; + if (frozenRiff) { + if (effectiveCliId !== 'riff' + || typeof msg.riffTaskId !== 'string' + || msg.riffTaskId.trim().length === 0) { + logger.error( + `[${t}] Rejected Riff queued activation ACK without a valid frozen Riff task id`, + ); + break; + } + if (ds.pendingRiffActivationTaskId !== msg.riffTaskId) { + logger.error( + `[${t}] Rejected Riff queued activation ACK for unmatched task ${msg.riffTaskId} ` + + `(announced=${ds.pendingRiffActivationTaskId ?? 'none'})`, + ); + break; + } + + const journal = snapshotQueuedActivationJournal(ds.session); + const priorRiffParentTaskId = ds.session.riffParentTaskId; + const persistRiffAck = async (attempt: number): Promise => { + if (ds.worker !== worker + || !ds.session.queuedActivationPending + || ds.session.queuedActivationToken !== msg.activationToken + || ds.pendingRiffActivationTaskId !== msg.riffTaskId) return; + // This is the sole durable activation-lineage edge: the new Riff + // child and removal of replayable N are committed together. + ds.session.riffParentTaskId = msg.riffTaskId; + clearQueuedActivationJournal(ds.session); + try { + sessionStore.updateSession(ds.session); + } catch (err) { + restoreQueuedActivationJournal(ds.session, journal); + ds.session.riffParentTaskId = priorRiffParentTaskId; + const delayMs = Math.min(100 * (2 ** Math.min(attempt, 6)), 5_000); + logger.error( + `[${t}] Failed to persist atomic Riff activation ACK; retained journal and lineage ` + + `for retry in ${delayMs}ms: ${err instanceof Error ? err.message : String(err)}`, + ); + const timer = setTimeout(() => { void persistRiffAck(attempt + 1); }, delayMs); + timer.unref?.(); + return; + } + ds.pendingRiffActivationTaskId = undefined; + ds.hasHistory = true; + await releaseReservation(0); + }; + await persistRiffAck(0); + break; + } + if (msg.riffTaskId !== undefined) { + logger.error(`[${t}] Rejected queued activation ACK carrying Riff lineage for non-Riff backend`); + break; + } + + const journal = snapshotQueuedActivationJournal(ds.session); + const persistActivationAck = async (attempt: number): Promise => { + if (ds.worker !== worker + || !ds.session.queuedActivationPending + || ds.session.queuedActivationToken !== msg.activationToken) return; + clearQueuedActivationJournal(ds.session); + try { + sessionStore.updateSession(ds.session); + } catch (err) { + restoreQueuedActivationJournal(ds.session, journal); + const delayMs = Math.min(100 * (2 ** Math.min(attempt, 6)), 5_000); + logger.error( + `[${t}] Failed to persist queued activation ACK; retained journal and ` + + `will retry in ${delayMs}ms: ${err instanceof Error ? err.message : String(err)}`, + ); + const timer = setTimeout(() => { void persistActivationAck(attempt + 1); }, delayMs); + timer.unref?.(); + return; + } + // Crossing the adapter boundary means this session now owns CLI + // history even if the worker dies before it can publish a session id. + ds.hasHistory = true; + await releaseReservation(0); + }; + await persistActivationAck(0); + break; + } + case 'ready': { startupState.ready = true; ds.workerPort = msg.port; @@ -2570,9 +4443,7 @@ function setupWorkerHandlers( flushPendingRiffUrlPatch(ds); } catch (err) { if (err instanceof MessageWithdrawnError) { - logger.warn(`[${t}] Root message withdrawn, closing stale session`); - killWorker(ds); - cb.closeSession(ds); + await closeWithdrawnSessionIfLedgerEmpty(ds, 'Root message withdrawn while creating worker-ready card'); break; } logger.warn(`[${t}] Failed to send streaming card, falling back to static card: ${err}`); @@ -2616,9 +4487,7 @@ function setupWorkerHandlers( } } catch (fallbackErr) { if (fallbackErr instanceof MessageWithdrawnError) { - logger.warn(`[${t}] Root message withdrawn, closing stale session`); - killWorker(ds); - cb.closeSession(ds); + await closeWithdrawnSessionIfLedgerEmpty(ds, 'Root message withdrawn while creating fallback worker-ready card'); break; } throw fallbackErr; @@ -2659,10 +4528,15 @@ function setupWorkerHandlers( ds.worker.send({ type: 'raw_input', content: rawInput, - ...(rawTurnId ? { turnId: rawTurnId } : {}), + ...((ds.session.queuedActivationTurnId ?? rawTurnId) + ? { turnId: ds.session.queuedActivationTurnId ?? rawTurnId } + : {}), followUpContent: followUp?.cliInput, ...(followUp?.turnId ? { followUpTurnId: followUp.turnId } : {}), ...(followUpCodexAppInput ? { followUpCodexAppInput } : {}), + ...(ds.session.queuedActivationToken + ? { queuedActivationToken: ds.session.queuedActivationToken } + : {}), } as DaemonToWorker); logger.info(`[${t}] Sent pending raw input after prompt_ready: ${rawInput.substring(0, 80)}${followUp ? ` (+follow-up ${followUp.cliInput.length} chars)` : ''}`); if (followUp) rememberLastCliInput(ds, followUp.userPrompt, { @@ -2812,11 +4686,9 @@ function setupWorkerHandlers( flushPendingLocalCliOpenReadinessPatch(ds); flushPendingRiffUrlPatch(ds); }) - .catch(err => { + .catch(async err => { if (err instanceof MessageWithdrawnError) { - logger.warn(`[${t}] Root message withdrawn, closing stale session`); - killWorker(ds); - cb.closeSession(ds); + await closeWithdrawnSessionIfLedgerEmpty(ds, 'Root message withdrawn while creating streaming card'); return; } logger.debug(`[${t}] Failed to create streaming card: ${err}`); @@ -3016,6 +4888,21 @@ function setupWorkerHandlers( break; } + if ((ds.initConfig?.backendType ?? ds.session.backendType) === 'riff' + || effectiveCliId === 'riff') { + // Riff does not have a local CLI crash/replacement contract. If a + // future backend change starts emitting claude_exit, never feed it + // into the local auto-restart/crash-loop retirement machinery: that + // can exit before a late task id is persisted. Preserve the durable + // row + lineage for an explicit close or operator reconciliation. + logger.error( + `[${t}] Unexpected Riff backend exit; refusing automatic generation replacement ` + + `(task=${ds.session.riffParentTaskId ?? 'pending'})`, + ); + ds.lastScreenStatus = 'idle'; + break; + } + // Rate-limit auto-restart to prevent crash loops const key = ds.session.sessionId; const rc = restartCounts.get(key) ?? { count: 0, lastAt: 0 }; @@ -3077,8 +4964,7 @@ function setupWorkerHandlers( await scopedReply(parts.join('\n\n'), 'text', undefined); } catch (replyErr) { if (replyErr instanceof MessageWithdrawnError) { - logger.warn(`[${t}] Root message withdrawn, closing stale session`); - cb.closeSession(ds); + await closeWithdrawnSessionIfLedgerEmpty(ds, 'Root message withdrawn while sending crash diagnostic'); } } } @@ -3088,7 +4974,7 @@ function setupWorkerHandlers( // Auto-restart CLI within the same worker if (ds.worker && !ds.worker.killed) { logger.info(`[${t}] Auto-restarting ${getCliDisplayName(effectiveCliId)}...`); - ds.worker.send({ type: 'restart' } as DaemonToWorker); + ds.worker.send({ type: 'restart', reason: 'cli_crash' } as DaemonToWorker); } break; } @@ -3123,11 +5009,26 @@ function setupWorkerHandlers( case 'riff_task_id': { if (ds.worker !== worker) break; + if (ds.session.queuedActivationPending) { + // Do not durably publish the new child beside a still-replayable + // activation journal. The immediately-following tokened ACK commits + // both facts atomically; null/failure preserves the prior lineage. + ds.pendingRiffActivationTaskId = msg.taskId ?? undefined; + logger.info( + `[${t}] Staged Riff activation lineage ${msg.taskId ?? 'null'} pending tokened ACK`, + ); + break; + } if (msg.taskId === null) { // follow-up 血缘断裂：清掉持久化锚点，否则 daemon 重启会复活已判坏的 parent。 if (ds.session.riffParentTaskId) { + const priorTaskId = ds.session.riffParentTaskId; ds.session.riffParentTaskId = undefined; - sessionStore.updateSession(ds.session); + try { sessionStore.updateSession(ds.session); } + catch (err) { + ds.session.riffParentTaskId = priorTaskId; + logger.error(`[${t}] Failed to clear Riff lineage: ${err instanceof Error ? err.message : String(err)}`); + } } break; } @@ -3137,7 +5038,31 @@ function setupWorkerHandlers( // next message continues the riff conversation in the warm sandbox // instead of cold-booting a context-less fresh task (4-5 min). ds.session.riffParentTaskId = msg.taskId; - sessionStore.updateSession(ds.session); + try { sessionStore.updateSession(ds.session); } + catch (err) { + // Retain the newest runtime lineage. A close_result or later task-id + // event retries persistence; reverting here would make a follow-up + // target the stale parent while the worker owns the new child. + logger.error(`[${t}] Failed to persist Riff lineage ${msg.taskId}: ${err instanceof Error ? err.message : String(err)}`); + } + break; + } + + case 'close_result': { + const pending = pendingRiffWorkerCloses.get(msg.requestId); + if (!pending + || pending.worker !== worker + || pending.sessionId !== ds.session.sessionId + || ds.worker !== worker) { + logger.warn(`[${t}] Ignored stale/unmatched Riff close result ${msg.requestId}`); + break; + } + pendingRiffWorkerCloses.delete(msg.requestId); + pending.resolve({ + ok: msg.ok, + ...(msg.taskId ? { taskId: msg.taskId } : {}), + ...(msg.error ? { error: msg.error } : {}), + }); break; } @@ -3283,8 +5208,47 @@ function setupWorkerHandlers( logger.warn(`[${t}] Dropped managed_turn_origin with mismatched sessionId`); break; } + // macOS uses one stable per-session pathname visible inside Seatbelt. + // Only the daemon handler for the CURRENT ChildProcess generation may + // replace it. Stale workers can still emit IPC, but the identity guard + // above drops them before filesystem mutation, so they cannot overwrite + // a successor capability (or unlink it during teardown). + if (process.platform === 'darwin' && msg.originChannelId) { + if (!/^[a-f0-9]{64}$/.test(msg.originChannelId)) { + ds.managedTurnOrigin = undefined; + logger.error(`[${t}] Refused managed origin publication with an invalid pane channel`); + break; + } + try { + const ipcPort = Number(process.env.BOTMUX_DAEMON_IPC_PORT); + replaceManagedOriginCapabilityFile( + managedOriginCapabilityPath( + config.session.dataDir, + msg.sessionId, + msg.originChannelId, + ), + JSON.stringify({ + sessionId: msg.sessionId, + channelId: msg.originChannelId, + capability: msg.capability, + ...(Number.isSafeInteger(ipcPort) && ipcPort > 0 && ipcPort <= 65_535 + ? { ipcPort } + : {}), + ...(msg.turnId ? { turnId: msg.turnId } : {}), + ...(msg.dispatchAttempt !== undefined + ? { dispatchAttempt: msg.dispatchAttempt } + : {}), + }), + ); + } catch (err) { + ds.managedTurnOrigin = undefined; + logger.error(`[${t}] Failed to publish daemon-owned managed origin capability: ${err instanceof Error ? err.message : String(err)}`); + break; + } + } ds.managedTurnOrigin = { capability: msg.capability, + ...(msg.originChannelId ? { originChannelId: msg.originChannelId } : {}), ...(msg.turnId ? { turnId: msg.turnId } : {}), ...(msg.dispatchAttempt !== undefined ? { dispatchAttempt: msg.dispatchAttempt } @@ -3310,6 +5274,12 @@ function setupWorkerHandlers( logger.warn(`[${t}] Ignored stale managed turn origin revoke after capability rotation`); break; } + if (msg.originChannelId + && ds.managedTurnOrigin?.originChannelId + && ds.managedTurnOrigin.originChannelId !== msg.originChannelId) { + logger.warn(`[${t}] Ignored managed_turn_origin_revoked for a different pane channel`); + break; + } if (!msg.capability && ds.managedTurnOrigin && (ds.managedTurnOrigin.turnId !== msg.turnId || ds.managedTurnOrigin.dispatchAttempt !== msg.dispatchAttempt)) { @@ -3320,7 +5290,235 @@ function setupWorkerHandlers( break; } + case 'codex_app_dispatch_transition': { + const acknowledge = (ok: boolean, error?: string): void => { + try { + worker.send({ + type: 'codex_app_dispatch_persisted', + requestId: msg.requestId, + ok, + ...(error ? { error } : {}), + } as DaemonToWorker); + } catch { /* worker exit makes the prepared/final state replayable */ } + }; + if (ds.worker !== worker) { + acknowledge(false, 'stale_worker_generation'); + break; + } + if (msg.sessionId !== ds.session.sessionId) { + acknowledge(false, 'session_mismatch'); + break; + } + const ledger = ds.session.codexAppDispatchLedger ?? []; + let next: { ok: true; ledger: CodexAppDispatchLedgerEntry[] } | { ok: false; error: string }; + if (msg.operation === 'submit') { + if (msg.entries.length !== 1) { + acknowledge(false, 'submit_requires_one_entry'); + break; + } + next = prepareCodexAppDispatch(ledger, msg.entries[0]); + } else if (msg.operation === 'retry') { + if (msg.entries.length !== 1) { + acknowledge(false, 'retry_requires_one_entry'); + break; + } + next = retryPreparedCodexAppDispatch(ledger, msg.entries[0]); + } else { + if (msg.entries.length !== 1) { + acknowledge(false, 'cancel_requires_one_entry'); + break; + } + next = cancelCodexAppDispatch(ledger, msg.entries[0]); + } + if (!next.ok) { + acknowledge(false, next.error); + break; + } + const priorLedger = ds.session.codexAppDispatchLedger; + ds.session.codexAppDispatchLedger = next.ledger; + let persisted = false; + try { + sessionStore.updateSession(ds.session); + persisted = true; + acknowledge(true); + } catch (err: any) { + ds.session.codexAppDispatchLedger = priorLedger; + acknowledge(false, err?.message ?? 'session_store_write_failed'); + } + if (persisted && !hasUnsettledCodexAppDispatch(ds.session.codexAppDispatchLedger)) { + try { await cb.onCodexAppLedgerDrained?.(ds); } + catch (err) { logger.error(`[${t}] post-drain cleanup failed: ${err instanceof Error ? err.message : String(err)}`); } + } + break; + } + + case 'codex_app_generation_active': { + if (ds.worker !== worker || msg.sessionId !== ds.session.sessionId) break; + if (!msg.fresh) break; + const priorCommits = ds.session.codexAppGenerationCommits; + ds.session.codexAppGenerationCommits = retainFreshCodexAppGeneration( + ds.session.codexAppGenerationCommits ?? [], + msg.generation, + ); + try { + sessionStore.updateSession(ds.session); + } catch (err) { + ds.session.codexAppGenerationCommits = priorCommits; + logger.error(`[${t}] Failed to persist fresh Codex App generation: ${err instanceof Error ? err.message : String(err)}`); + } + break; + } + case 'final_output': { + if (msg.codexAppSettlement) { + const settlement = msg.codexAppSettlement; + const acknowledge = (ok: boolean, error?: string): void => { + try { + worker.send({ + type: 'codex_app_dispatch_persisted', + requestId: settlement.requestId, + ok, + ...(error ? { error } : {}), + } as DaemonToWorker); + } catch { /* runner keeps the final unacknowledged for replacement */ } + }; + if (ds.worker !== worker) { + acknowledge(false, 'stale_worker_generation'); + break; + } + if (msg.sessionId !== ds.session.sessionId) { + acknowledge(false, 'session_mismatch'); + break; + } + if (committedCodexAppSequence( + ds.session.codexAppGenerationCommits ?? [], + settlement.generation, + settlement.seq, + )) { + acknowledge(true); + if (!hasUnsettledCodexAppDispatch(ds.session.codexAppDispatchLedger)) { + try { await cb.onCodexAppLedgerDrained?.(ds); } + catch (err) { logger.error(`[${t}] post-drain cleanup failed: ${err instanceof Error ? err.message : String(err)}`); } + } + break; + } + const identity = { + dispatchId: settlement.dispatchId, + turnId: msg.turnId, + ...(msg.dispatchAttempt !== undefined + ? { dispatchAttempt: msg.dispatchAttempt } + : {}), + }; + const preview = settleCodexAppDispatch( + ds.session.codexAppDispatchLedger ?? [], + ds.session.codexAppGenerationCommits ?? [], + identity, + settlement.generation, + settlement.seq, + ); + if (!preview.ok) { + acknowledge(false, preview.error); + break; + } + const key = `${ds.session.sessionId}:${settlement.generation}:${settlement.seq}`; + let inFlight = codexAppFinalSettlementInFlight.get(key); + if (!inFlight) { + inFlight = (async () => { + const unavailableSinkFailClosed = codexAppDeliveryMustFailClosed( + ds, + preview.settledEntry, + ); + const deliverySuppressed = msg.suppressDelivery === true + || managedFinalOutputSuppressed(msg.turnId, msg.dispatchAttempt) + || unavailableSinkFailClosed; + if (unavailableSinkFailClosed + && preview.settledEntry.deliverySink !== 'suppressed' + && msg.suppressDelivery !== true) { + logger.warn( + `[${t}] Codex App recovery suppressed unavailable ` + + `${preview.settledEntry.deliverySink ?? 'legacy non-Lark'} sink ` + + `(turn ${msg.turnId.substring(0, 8)})`, + ); + } + const alreadyDelivered = ds.lastBridgeEmittedUuid === finalOutputDedupeKey(ds, msg); + const owned = deliverySuppressed || !msg.content.trim() || alreadyDelivered + ? true + : await new Promise(resolve => { + deliverFinalOutput( + ds, + msg, + t, + 0, + resolve, + () => ds.worker === worker + && ds.session.sessionId === msg.sessionId, + preview.settledEntry.replyTarget, + ); + }); + if (!owned) return false; + + // Re-read after the asynchronous external delivery. A concurrent + // replacement may already have committed this signed sequence; + // that is idempotent success, never a second FIFO pop. + if (committedCodexAppSequence( + ds.session.codexAppGenerationCommits ?? [], + settlement.generation, + settlement.seq, + )) return true; + const committed = settleCodexAppDispatch( + ds.session.codexAppDispatchLedger ?? [], + ds.session.codexAppGenerationCommits ?? [], + identity, + settlement.generation, + settlement.seq, + ); + if (!committed.ok) return false; + if (msg.dispatchAttempt !== undefined) { + try { + // Durable receivers must not depend on the worker surviving + // the daemon ACK long enough to emit a later terminal IPC. + // Persist the exact completed attempt first; the worker's + // ordered duplicate terminal remains idempotent. + await cb.onTurnTerminal?.(ds, { + type: 'turn_terminal', + sessionId: ds.session.sessionId, + turnId: msg.turnId, + dispatchAttempt: msg.dispatchAttempt, + status: 'completed', + }, { workerGeneration }); + } catch (err) { + logger.error(`[${t}] Failed to persist Codex App settlement terminal: ${err instanceof Error ? err.message : String(err)}`); + return false; + } + } + const priorLedger = ds.session.codexAppDispatchLedger; + const priorCommits = ds.session.codexAppGenerationCommits; + ds.session.codexAppDispatchLedger = committed.ledger; + ds.session.codexAppGenerationCommits = committed.commits; + try { + // One atomic sessions-file replacement owns both the exact FIFO + // pop and cumulative runner ACK boundary. Only after this write + // may the worker acknowledge final-end to the runner. + sessionStore.updateSession(ds.session); + return true; + } catch (err) { + ds.session.codexAppDispatchLedger = priorLedger; + ds.session.codexAppGenerationCommits = priorCommits; + logger.error(`[${t}] Failed to persist Codex App final settlement: ${err instanceof Error ? err.message : String(err)}`); + return false; + } + })().finally(() => codexAppFinalSettlementInFlight.delete(key)); + codexAppFinalSettlementInFlight.set(key, inFlight); + } + const persisted = await inFlight; + acknowledge(persisted, persisted ? undefined : 'final_settlement_failed'); + if (persisted && !hasUnsettledCodexAppDispatch(ds.session.codexAppDispatchLedger)) { + try { await cb.onCodexAppLedgerDrained?.(ds); } + catch (err) { logger.error(`[${t}] post-drain cleanup failed: ${err instanceof Error ? err.message : String(err)}`); } + } + break; + } + // Adopt-bridge: worker harvested the assistant turn from Claude Code's // transcript JSONL and forwarded it to us. Dedup with a session-scoped // key so a re-drain can't re-send the same answer or cross-suppress @@ -3343,7 +5541,15 @@ function setupWorkerHandlers( // Worker pops the turn off its queue right after emit, so it will // NOT re-send this payload on its own. Daemon owns retry on // transient Lark failures. - deliverFinalOutput(ds, msg, t, 0); + deliverFinalOutput( + ds, + msg, + t, + 0, + undefined, + () => ds.worker === worker + && (!msg.sessionId || ds.session.sessionId === msg.sessionId), + ); break; } @@ -3399,9 +5605,26 @@ function setupWorkerHandlers( // A stale takeover worker never clears the replacement — during takeover the // old worker's exit fires AFTER the new worker has been assigned. if (ds.worker === worker) { + if (ds.session.queuedActivationPending) { + // Journal ownership is backend-independent. The next worker replays + // this exact head with queuedActivationResume before durable tail N+1. + ds.initialStartPending = false; + ds.initialStartClaimToken = undefined; + ds.pendingRiffActivationTaskId = undefined; + } else { + reparkQueuedActivationFollowUpTail(ds, 'worker exit during activation follow-up handoff'); + } ds.worker = null; ds.workerPort = null; + ds.workerToken = null; + ds.workerViewToken = null; ds.managedTurnOrigin = undefined; + if (ds.riffCloseState) { + ds.riffCloseState = { ...ds.riffCloseState, phase: 'uncertain' }; + } + // Do not clear riffShutdownState here. The exact worker may have fenced + // its backend before exiting, and only the shutdown coordinator knows + // whether final lineage or admission restoration was verified. } try { const notified = cb.onWorkerExit?.(ds, { @@ -3439,6 +5662,7 @@ function setupWorkerHandlers( // ─── Bridge final-output delivery (with retry) ────────────────────────────── const FINAL_OUTPUT_RETRY_BACKOFF_MS = [0, 5000, 15000]; // immediate, +5s, +15s +const codexAppFinalSettlementInFlight = new Map>(); function finalOutputDedupeKey(ds: DaemonSession, msg: Extract): string { return `${msg.sessionId ?? ds.session.sessionId}:${msg.lastUuid || msg.turnId}`; @@ -3522,9 +5746,9 @@ async function finishTurnReactions(ds: DaemonSession): Promise { } } -/** Deliver a bridge `final_output` to Lark. The worker emits each turn - * exactly once (it pops the turn off its queue at emit time), so the - * daemon owns retries on transient failures. After 3 attempts we log +/** Deliver a bridge `final_output` to Lark. The current worker generation pops + * the turn at emit time, while replacement recovery may replay it with the + * same provider key; the daemon owns bounded transient retries. After 3 attempts we log * and give up — the user's answer is lost; better than leaking memory * via an unbounded retry loop. */ function deliverFinalOutput( @@ -3532,7 +5756,14 @@ function deliverFinalOutput( msg: Extract, t: string, attempt: number, + onComplete?: (owned: boolean) => void, + isStillOwned: () => boolean = () => true, + frozenReplyTarget?: FrozenSessionReplyTarget, ): void { + if (!isStillOwned()) { + onComplete?.(false); + return; + } const managedReceiver = !!ds.session.vcMeetingReceiver; // Wait Mode / HTTP Sync Override: // If this turn is being waited for by an HTTP webhook request, intercept the @@ -3544,6 +5775,7 @@ function deliverFinalOutput( waitPromise.resolve(msg.content); ds.lastBridgeEmittedUuid = finalOutputDedupeKey(ds, msg); logger.info(`[${t}] Intercepted final_output for Wait Mode HTTP request (turn ${msg.turnId.substring(0, 8)})`); + onComplete?.(true); return; } @@ -3554,6 +5786,7 @@ function deliverFinalOutput( asyncResult.completedAt = Date.now(); ds.lastBridgeEmittedUuid = finalOutputDedupeKey(ds, msg); logger.info(`[${t}] Captured final_output for Async HTTP request (turn ${msg.turnId.substring(0, 8)})`); + onComplete?.(true); return; } const cb = requireCallbacks(); @@ -3574,11 +5807,17 @@ function deliverFinalOutput( : opts, ); setTimeout(async () => { + if (!isStillOwned()) { + logger.info(`[${t}] Bridge final_output abandoned — worker/session ownership changed`); + onComplete?.(false); + return; + } // Guard: if the user closed the session (or it was torn down for any // other reason) between attempts, don't post a stale final answer to // a closed thread. if (ds.session.status === 'closed') { logger.info(`[${t}] Bridge final_output abandoned — session closed (turn ${msg.turnId.substring(0, 8)})`); + onComplete?.(true); return; } try { @@ -3606,6 +5845,7 @@ function deliverFinalOutput( } ds.lastBridgeEmittedUuid = finalOutputDedupeKey(ds, msg); logger.info(`[${t}] doc-comment final_output → posted ${chunks.length} comment(s) on file=${docTurn.fileToken.slice(0, 12)} (turn ${msg.turnId.substring(0, 8)})`); + onComplete?.(true); return; } @@ -3639,6 +5879,7 @@ function deliverFinalOutput( `[${t}] VC final_output lost current membership authority ` + `(${managedDecision.errorCode}) turn=${msg.turnId.substring(0, 8)}`, ); + onComplete?.(true); return; } const revalidateManagedSend = (): void => { @@ -3720,12 +5961,25 @@ function deliverFinalOutput( }, proposedOutput) : undefined; const preparedListenerReply = preparedImReply ?? preparedDeliveryReply; + // Codex App final settlement is delivery-before-commit. If the daemon + // dies after Lark accepts the reply but before the FIFO/sequence commit, + // the replacement must retry with the same provider key. Keep the key + // dispatch-stable (not generation/seq-stable), so the user-visible Lark + // message is idempotent across crash reconciliation. The existing + // outbound-hook contract remains unchanged; it is a separate best-effort + // side effect and is not covered by the provider UUID. + const codexAppSettlementReply = msg.codexAppSettlement + ? { + uuid: `ca_${msg.codexAppSettlement.dispatchId}`.slice(0, 50), + } + : undefined; if (preparedListenerReply?.kind === 'conflict') { ds.lastBridgeEmittedUuid = finalOutputDedupeKey(ds, msg); logger.error( `[${t}] VC listener fallback suppressed (${preparedListenerReply.reason}) ` + `turn=${msg.turnId.substring(0, 8)}: ${preparedListenerReply.detail}`, ); + onComplete?.(true); return; } const canonicalOutput = preparedListenerReply?.canonicalOutput ?? proposedOutput; @@ -3764,6 +6018,7 @@ function deliverFinalOutput( `[${t}] VC listener fallback replayed existing provider result ` + `(turn ${msg.turnId.substring(0, 8)})`, ); + onComplete?.(true); return; } @@ -3771,21 +6026,28 @@ function deliverFinalOutput( // place. message.patch is silent (no Feishu notification / unread), which // used to swallow the answer; a brand-new message always pings. revalidateManagedSend(); + if (!isStillOwned()) { + onComplete?.(false); + return; + } + const deliveryReplyOptions = preparedListenerReply + ? { + uuid: preparedListenerReply.providerKey, + quoteMessageId: canonicalOutput.quoteTargetId, + beforeQuoteFallback: revalidateManagedSend, + // Managed output has one audited external effect (the Lark + // provider call). Never fan meeting content out to user hooks, + // including the first attempt and crash reconciliation replay. + suppressHook: true, + } + : codexAppSettlementReply; const messageId = await scopedReply( canonicalOutput.content, canonicalOutput.msgType, - msg.turnId, - preparedListenerReply - ? { - uuid: preparedListenerReply.providerKey, - quoteMessageId: canonicalOutput.quoteTargetId, - beforeQuoteFallback: revalidateManagedSend, - // Managed output has one audited external effect (the Lark - // provider call). Never fan meeting content out to user hooks, - // including the first attempt and crash reconciliation replay. - suppressHook: true, - } - : undefined, + msg.replyTurnId ?? msg.turnId, + frozenReplyTarget && !managedReceiver + ? { ...deliveryReplyOptions, replyTarget: frozenReplyTarget } + : deliveryReplyOptions, ); recordPrimaryOutput(messageId); if (preparedListenerReply?.kind === 'send' || preparedListenerReply?.kind === 'succeeded') { @@ -3793,13 +6055,21 @@ function deliverFinalOutput( } ds.lastBridgeEmittedUuid = finalOutputDedupeKey(ds, msg); logger.info(`[${t}] Bridge final_output forwarded (turn ${msg.turnId.substring(0, 8)}, ${msg.content.length} chars, kind=${msg.kind ?? 'bridge'}, attempt ${attempt + 1})`); + onComplete?.(true); } catch (err: any) { if (err instanceof MessageWithdrawnError) { - // Root message gone — no point retrying. Mark as emitted so any - // duplicate IPC is correctly deduped, and tear the session down. - ds.lastBridgeEmittedUuid = finalOutputDedupeKey(ds, msg); - logger.warn(`[${t}] Root message withdrawn while forwarding final_output, closing session`); - cb.closeSession(ds); + // Withdrawal is permanent for this target, but it is not explicit + // abandon. Keep an unsettled FIFO owner recoverable; only ledger-empty + // sessions may commit the dedupe marker and auto-close. + if (await closeWithdrawnSessionIfLedgerEmpty( + ds, + 'Root message withdrawn while forwarding final_output', + )) { + ds.lastBridgeEmittedUuid = finalOutputDedupeKey(ds, msg); + onComplete?.(true); + } else { + onComplete?.(false); + } return; } const next = attempt + 1; @@ -3807,10 +6077,11 @@ function deliverFinalOutput( logger.error(`[${t}] Bridge final_output gave up after ${next} attempts (turn ${msg.turnId.substring(0, 8)}): ${err.message}`); // Don't commit the dedup marker — leave room for any future // retransmit (e.g. daemon restart that re-fires the IPC). + onComplete?.(false); return; } logger.warn(`[${t}] Bridge final_output attempt ${next} failed (${err.message}); retrying in ${FINAL_OUTPUT_RETRY_BACKOFF_MS[next]}ms`); - deliverFinalOutput(ds, msg, t, next); + deliverFinalOutput(ds, msg, t, next, onComplete, isStillOwned, frozenReplyTarget); } }, FINAL_OUTPUT_RETRY_BACKOFF_MS[attempt] ?? 0); } @@ -4192,10 +6463,19 @@ function cleanupPersistentBackendSessions(backendType: 'tmux' | 'herdr', activeS let lastCliId: string | undefined; try { lastCliId = readFileSync(cliIdFile, 'utf-8').trim(); } catch { /* first run */ } const currentCliId = config.daemon.cliId; + const unsettledNames = new Set( + activeSessions_ + .filter(hasProtectedSessionMutationOwnership) + .map(session => backend.sessionName(session.sessionId)), + ); if (!multiBot && lastCliId && lastCliId !== currentCliId) { - logger.info(`CLI_ID changed (${lastCliId} → ${currentCliId}), killing all ${backendType} sessions`); + logger.info(`CLI_ID changed (${lastCliId} → ${currentCliId}), retiring ledger-empty ${backendType} sessions`); for (const name of backend.listBotmuxSessions()) { + if (unsettledNames.has(name)) { + logger.warn(`Preserving ${backendType} ${name}: unsettled Codex App dispatch must reconcile first`); + continue; + } backend.killSession(name); } } else { @@ -4218,6 +6498,10 @@ function cleanupPersistentBackendSessions(backendType: 'tmux' | 'herdr', activeS try { botCliId = getBot(session.larkAppId).config.cliId; } catch { continue; } if (botCliId && sessionCliId !== botCliId) { const name = backend.sessionName(session.sessionId); + if (hasProtectedSessionMutationOwnership(session)) { + logger.warn(`Preserving CLI-mismatched ${backendType} ${name}: durable activation ownership must reconcile first`); + continue; + } logger.info(`CLI mismatch for ${session.sessionId.substring(0, 8)} (session=${sessionCliId}, bot=${botCliId}), killing ${backendType} ${name}`); backend.killSession(name); } diff --git a/src/daemon.ts b/src/daemon.ts index 8c25c4121..7b791498e 100644 --- a/src/daemon.ts +++ b/src/daemon.ts @@ -18,7 +18,7 @@ import { import { repoPickerScanOptions } from './global-config.js'; import { buildDashboardUrls } from './core/dashboard-url.js'; import { resolveBotmuxDataDir } from './core/data-dir.js'; -import { writeHeartbeat } from './core/daemon-heartbeat.js'; +import { isMaintenanceBusyStatus, writeHeartbeat } from './core/daemon-heartbeat.js'; import { botmuxWrapperFiles } from './core/botmux-wrapper.js'; import { startMaintenance, stopMaintenance } from './core/maintenance.js'; import { @@ -27,6 +27,12 @@ import { stopCliRuntimeUpdateMonitor, } from './core/cli-runtime-update.js'; import { sendRestartReportIfPending } from './core/restart-report.js'; +import { + DAEMON_GRACEFUL_EXIT_CODE, + SUPERVISOR_SHUTDOWN_PROTOCOL, + type SupervisorShutdownProtocol, +} from './core/supervisor-shutdown-protocol.js'; +import { readSupervisorProcessStartIdentity } from './core/process-start-identity.js'; import { statSync } from 'node:fs'; import { addReaction, getChatMode, getMessageChatId, listChatMemberOpenIds, MessageWithdrawnError, replyMessage, resolveAllowedUsersWithMap, sendMessage, sendUserMessage, updateMessage } from './im/lark/client.js'; import { resolveGroupJoinPrompt, waitForAllowedUserInChat } from './core/auto-start.js'; @@ -62,10 +68,16 @@ import { bindResourcesToMessage, composeForwardFollowupContent, mergeMessageMent import { buildQuoteHint } from './im/lark/quote-hint.js'; import { logger } from './utils/logger.js'; import { withFileLock } from './utils/file-lock.js'; +import { + hasUnsettledCodexAppDispatch, + retireCodexAppDispatchAfterBackingMissing, + validateCodexAppManagedSendOrigin, +} from './utils/codex-app-dispatch-ledger.js'; +import { hasProtectedSessionMutationOwnership } from './core/session-mutation-guard.js'; import { delay } from './utils/timing.js'; import { BoundedMap } from './utils/bounded-map.js'; import { checkAllowedChatGroupsConfig } from './services/allowed-chat-groups.js'; -import type { Session, VcMeetingImTurnOrigin } from './types.js'; +import type { CliTurnPayload, Session, VcMeetingImTurnOrigin } from './types.js'; import { ensureCjkFontsInstalled } from './utils/font-installer.js'; import { scrubTmuxServerGlobalEnv } from './setup/ensure-tmux.js'; import { invalidWorkingDirs } from './utils/working-dir.js'; @@ -73,7 +85,14 @@ import { validateWorkingDir } from './core/working-dir.js'; import type { DaemonToWorker, LarkMessage } from './types.js'; export type { DaemonSession } from './core/types.js'; import type { DaemonSession } from './core/types.js'; -import { activeSessionKey, sessionKey, sessionAnchorId, storedSessionAnchorId } from './core/types.js'; +import { + activeSessionKey, + sessionKey, + sessionAnchorId, + storedSessionAnchorId, + riffRetirementAdmissionPhase, +} from './core/types.js'; +import { stagePendingRepoSetup, persistPendingRepoCardMessageId } from './core/pending-repo-journal.js'; import { buildTerminalUrl, setTerminalProxyPort, setTerminalExternalPort } from './core/terminal-url.js'; import { startTerminalProxy, type TerminalProxyHandle } from './core/terminal-proxy.js'; import type { CliId } from './adapters/cli/types.js'; @@ -89,6 +108,12 @@ import { setActiveSessionsRegistry, forkWorker, sendWorkerInput, + promoteQueuedActivationTail, + admitQueuedActivationTail, + reserveQueuedActivationTailAdmission, + type QueuedActivationTailReservation, + codexAppCleanInputAcceptedForSession, + hasQueuedActivationAdmissionGate, killWorker, reapOrphanWorkers, scheduleCardPatch, @@ -101,10 +126,27 @@ import { sweepGlobalBotmuxSkills, writableTerminalLinkFor, findActiveBySessionId, + withActiveSessionKeyLock, getDaemonBootId, type WorkerSessionReplyOptions, } from './core/worker-pool.js'; -import { ipcRoute, isTrustedHostIpcRequest, jsonRes, readJsonBody, setBotName, setLarkAppId, startIpcServer, setBotRenamer, setBotAvatarChanger } from './core/dashboard-ipc-server.js'; +import { + abortRiffShutdownFleet, + canAbortVerifiedExitedRiffPreparation, + collectUniqueDaemonShutdownSessions, + commitPreparedRiffShutdown, + isPreparedRiffSessionCurrent, + persistPreparedRiffShutdownFleet, + prepareRiffFleetForShutdown, + type FencedRiffShutdownParticipant, + type PreparedRiffShutdown, +} from './core/riff-shutdown-detach.js'; +import { + BOT_TURN_MUTATION_SHUTDOWN_ACQUIRE_TIMEOUT_MS, + DAEMON_SHUTDOWN_MAX_MS, + DAEMON_WORKER_EXIT_GRACE_MS, +} from './core/shutdown-budgets.js'; +import { ipcRoute, isTrustedHostIpcRequest, jsonRes, readJsonBody, setBotName, setLarkAppId, startIpcServer, setBotRenamer, setBotAvatarChanger, setSupervisorShutdownHandler } from './core/dashboard-ipc-server.js'; import { setDeviceIsolationDaemonIdentity } from './core/device-isolation-daemon.js'; import { cancelSessionReadyAck, @@ -143,8 +185,15 @@ import { rememberLastCliInput, ensureTerminalWorkerPort, ensureSessionWhiteboard, + closeCliMismatchedSessionsForBot, } from './core/session-manager.js'; import { triggerSessionTurn } from './core/trigger-session.js'; +import { + runDetachedBotTurnMutation, + tryWithBotTurnMutation, + withBotTurnAdmission, + withBotTurnMutation, +} from './core/bot-turn-mutation-gate.js'; import { applyQueuedCodexAppLegacyFallback, mergeQueuedCodexAppTurn } from './core/session-create.js'; import { findOnlineDaemon, listOnlineDaemons } from './utils/daemon-discovery.js'; import { beginReplyTargetTurn, fallbackTurnId, isSubstituteTurn, resolveSessionReplyTarget, syncReplyTargetState } from './core/reply-target.js'; @@ -154,7 +203,7 @@ import { sweepOrphanSandboxes } from './adapters/backend/sandbox.js'; import { TmuxBackend } from './adapters/backend/tmux-backend.js'; import { HerdrBackend } from './adapters/backend/herdr-backend.js'; import { ZellijBackend } from './adapters/backend/zellij-backend.js'; -import { sweepIdleWorkers, DEFAULT_MAX_LIVE_WORKERS } from './core/idle-worker-sweeper.js'; +import { sweepIdleWorkersAfterTurnDrain, DEFAULT_MAX_LIVE_WORKERS } from './core/idle-worker-sweeper.js'; import { getSessionPersistentBackendType, killPersistentSession, @@ -248,7 +297,7 @@ let selfV3BootInstanceId: string | undefined; let selfDaemonLarkAppId: string | undefined; let vcMeetingTerminalReconciler: VcMeetingTerminalReconciler | undefined; import { isBotMentioned, probeBotOpenId, startLarkEventDispatcher, markForwardFollowupsSessionsReady, writeBotInfoFile, canOperate, evaluateTalk, grantCommandRestriction, isKnownPeerBot, checkRequiredScopes, type RoutingContext, type TalkEvaluation, type DocCommentContext, type EventHandlers } from './im/lark/event-dispatcher.js'; -import { getDocSubscription, listAllDocSubscriptions, listDocSubscriptionsForSession, removeDocSubscription, setDocCommentPollCursor, type DocSubscription } from './services/doc-subs-store.js'; +import { getDocSubscription, listAllDocSubscriptions, listDocSubscriptionsForSession, putDocSubscription, removeDocSubscription, setDocCommentPollCursor, type DocSubscription } from './services/doc-subs-store.js'; import { BOT_REPLY_SENTINEL, subscribeDocFile, unsubscribeDocFile, addCommentReaction, hasBotSentinel, isBotAuthoredReply, listDocComments } from './im/lark/doc-comment.js'; import { learnFromMentions, resolveSender, flushIdentityCacheSync } from './im/lark/identity-cache.js'; import { normalizeBrand } from './im/lark/lark-hosts.js'; @@ -474,7 +523,11 @@ let vcMeetingReceiverRecoveryReady = false; let vcMeetingReceiverRecoverySchedulingComplete = false; const vcMeetingReceiverRecoveryPending = new Set(); const vcMeetingReceiverRecoveryTimers = new Map>(); -const vcMeetingReceiverRecoveryScopes = new Map(); +type VcMeetingBootRecoveryScope = VcMeetingDeliveryScope & { + turnId: string; + dispatchAttempt: number; +}; +const vcMeetingReceiverRecoveryScopes = new Map(); // Once the 10s ACK timeout fires, recovery has committed to kill + orphan // teardown. A delayed receiver_reset_ready from the old worker generation may // no longer cancel phase 2 or make the receiver ready early. @@ -501,13 +554,80 @@ function clearVcMeetingReceiverRecoveryPending(key: string): void { function addVcMeetingReceiverRecoveryPending( key: string, - scope: VcMeetingDeliveryScope, + scope: VcMeetingBootRecoveryScope, ): void { vcMeetingReceiverRecoveryPending.add(key); vcMeetingReceiverRecoveryScopes.set(key, scope); refreshVcMeetingReceiverRecoveryReady(); } +let testOnlyVcMeetingBootDispatchRetirement: + | ((sessionId: string, turnId: string, dispatchAttempt: number) => boolean) + | undefined; + +/** + * Retire the exact daemon-side Codex App ownership record after the owned CLI + * backing has been proved absent. This is intentionally stronger than the + * ordinary worker `cancel` transition: a dead generation may have accepted a + * successor behind this VC attempt, so recovery must remove only the fenced + * identity without requiring the old worker to prove FIFO state. + * + * A missing entry is idempotent success (the live worker may already have + * durably cancelled it before exiting). Multiple matches are corruption and + * remain fail-closed. The in-memory mutation is rolled back if the session + * store write fails, so callers can keep the delivery fence armed and retry. + */ +function retireVcMeetingCodexAppDispatchAfterBackingMissing( + sessionId: string, + turnId: string, + dispatchAttempt: number, +): boolean { + if (testOnlyVcMeetingBootDispatchRetirement) { + return testOnlyVcMeetingBootDispatchRetirement(sessionId, turnId, dispatchAttempt); + } + const session = findActiveBySessionId(sessionId)?.session ?? sessionStore.getSession(sessionId); + if (!session) return true; + const ledger = session.codexAppDispatchLedger ?? []; + const retired = retireCodexAppDispatchAfterBackingMissing(ledger, turnId, dispatchAttempt); + if (!retired.ok) { + logger.error( + `[vc-delivery] exact Codex App dispatch retirement failed (${retired.error}) ` + + `session=${sessionId} turn=${turnId.slice(0, 12)} attempt=${dispatchAttempt}`, + ); + return false; + } + if (retired.ledger.length === ledger.length) return true; + const priorLedger = session.codexAppDispatchLedger; + session.codexAppDispatchLedger = retired.ledger; + try { + sessionStore.updateSession(session); + const cleanupAppId = session.larkAppId; + if (cleanupAppId && !hasUnsettledCodexAppDispatch(session.codexAppDispatchLedger)) { + // Backing-missing retirement bypasses the normal worker ACK path, but it + // still creates the same durable empty-ledger boundary. Run mismatch + // cleanup only after persistence; failure is logged and never rolls back + // an already-authoritative retirement. + void runDetachedBotTurnMutation(cleanupAppId, async () => { + await closeCliMismatchedSessionsForBot(cleanupAppId); + }).catch(err => { + logger.error( + `[vc-delivery] post-retirement ledger-drain cleanup failed session=${sessionId}: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + }); + } + return true; + } catch (err) { + session.codexAppDispatchLedger = priorLedger; + logger.error( + `[vc-delivery] failed to persist exact Codex App dispatch retirement ` + + `session=${sessionId} turn=${turnId.slice(0, 12)} attempt=${dispatchAttempt}: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + return false; + } +} + function isVcMeetingBootRecoveryBlocked(request: VcMeetingDeliveryRequest): boolean { // The brief pre-enumeration interval is the only global barrier. Once boot // has enumerated stale receipts, one broken receiver must not stall unrelated @@ -583,9 +703,20 @@ function scheduleVcMeetingBootRecoveryProbe( if (!vcMeetingReceiverRecoveryPending.has(key)) return; vcMeetingReceiverRecoveryTimers.delete(key); if (vcMeetingBootBackingMissing(sessionId, true)) { - logger.error(`[vc-delivery] boot receiver force-fenced session=${sessionId}`); - clearVcMeetingReceiverRecoveryPending(key); - return; + const scope = vcMeetingReceiverRecoveryScopes.get(key); + if (scope && retireVcMeetingCodexAppDispatchAfterBackingMissing( + sessionId, + scope.turnId, + scope.dispatchAttempt, + )) { + logger.error(`[vc-delivery] boot receiver force-fenced session=${sessionId}`); + clearVcMeetingReceiverRecoveryPending(key); + return; + } + logger.error( + `[vc-delivery] boot receiver backing missing but dispatch retirement unproven; ` + + `reprobe scheduled session=${sessionId}`, + ); } logger.error(`[vc-delivery] boot receiver teardown unproven; reprobe scheduled session=${sessionId}`); scheduleVcMeetingBootRecoveryProbe(key, sessionId, VC_MEETING_RUNTIME_EXPIRY_REPROBE_MS); @@ -620,8 +751,19 @@ function acknowledgeVcMeetingReceiverRecovery(key: string): boolean { const sessionId = vcMeetingReceiverRecoveryScopes.get(key)?.receiverSessionId; if (!sessionId) return false; if (vcMeetingBootBackingMissing(sessionId, false)) { - clearVcMeetingReceiverRecoveryPending(key); - return true; + const scope = vcMeetingReceiverRecoveryScopes.get(key); + if (scope && retireVcMeetingCodexAppDispatchAfterBackingMissing( + sessionId, + scope.turnId, + scope.dispatchAttempt, + )) { + clearVcMeetingReceiverRecoveryPending(key); + return true; + } + logger.error( + `[vc-delivery] boot receiver ACK has missing backing but dispatch retirement is unproven ` + + `session=${sessionId}`, + ); } logger.error(`[vc-delivery] boot receiver ACK lacked backing teardown proof session=${sessionId}`); escalateVcMeetingBootRecovery(key, sessionId); @@ -653,6 +795,8 @@ export const __testOnly_vcMeetingReceiverRecovery = { meetingId: scope.meetingId ?? 'meeting_test', memberId: scope.memberId ?? 'member_test', memberEpoch: scope.memberEpoch ?? 1, + turnId, + dispatchAttempt, }); armVcMeetingReceiverRecoveryTimeout(key, sessionId); return key; @@ -678,6 +822,14 @@ export const __testOnly_vcMeetingReceiverRecovery = { }, setBackingMissingProbe(probe: (sessionId: string, destroy: boolean) => boolean): void { testOnlyVcMeetingBootBackingMissing = probe; + // Test recovery lifecycles must not load or mutate the developer's real + // session store merely because they replaced the backing probe. + testOnlyVcMeetingBootDispatchRetirement ??= () => true; + }, + setDispatchRetirement( + retire: (sessionId: string, turnId: string, dispatchAttempt: number) => boolean, + ): void { + testOnlyVcMeetingBootDispatchRetirement = retire; }, reset(): void { for (const timer of vcMeetingReceiverRecoveryTimers.values()) clearTimeout(timer); @@ -688,6 +840,7 @@ export const __testOnly_vcMeetingReceiverRecovery = { vcMeetingReceiverRecoveryReady = false; vcMeetingReceiverRecoverySchedulingComplete = false; testOnlyVcMeetingBootBackingMissing = undefined; + testOnlyVcMeetingBootDispatchRetirement = undefined; }, }; @@ -714,6 +867,7 @@ type VcMeetingRuntimeLeaseRecoveryDeps = { backendAvailable: (backendType: PersistentBackendType) => boolean; killPersistent: (backendType: PersistentBackendType, sessionName: string) => void; probePersistent: (backendType: PersistentBackendType, sessionName: string) => 'exists' | 'missing' | 'unknown'; + retireDispatch: (sessionId: string, turnId: string, dispatchAttempt: number) => boolean; warn: (message: string) => void; error: (message: string) => void; }; @@ -812,8 +966,24 @@ function createVcMeetingRuntimeLeaseRecovery(deps: VcMeetingRuntimeLeaseRecovery // PTY has no surviving backing process once its worker has crossed the // SIGKILL backstop. Persistent backends unlock only after every applicable - // deterministic name probes authoritatively missing. + // deterministic name probes authoritatively missing. The backing proof is + // not itself enough: atomically retire this exact old daemon-ledger attempt + // before admitting hub replay, or accepted/prepared N can be restored next + // to replayed N+1 by a replacement worker. if (allMissing) { + if (!deps.retireDispatch( + fence.receiverSessionId, + fence.deliveryKey, + fence.dispatchAttempt, + )) { + deps.error( + `[vc-delivery] runtime lease backing missing but exact dispatch retirement failed; ` + + `delivery remains gated session=${fence.receiverSessionId} ` + + `attempt=${fence.dispatchAttempt}`, + ); + scheduleReprobe(fence, `${reason} dispatch retirement`); + return false; + } clearFence(fence); deps.warn( `[vc-delivery] runtime lease fence cleared after ${reason} ` @@ -1060,6 +1230,7 @@ const vcMeetingRuntimeLeaseRecovery = createVcMeetingRuntimeLeaseRecovery({ backendAvailable: vcMeetingPersistentBackendAvailable, killPersistent: killPersistentSession, probePersistent: probePersistentSession, + retireDispatch: retireVcMeetingCodexAppDispatchAfterBackingMissing, warn: message => logger.warn(message), error: message => logger.error(message), }); @@ -2593,6 +2764,29 @@ async function sessionReply( return sendWithHookPolicy(chatId, content, msgType, opts.uuid); } } + if (opts?.replyTarget?.mode === 'thread') { + return replyWithHookPolicy( + opts.replyTarget.rootMessageId, + content, + msgType, + true, + opts.uuid, + ); + } + if (opts?.replyTarget?.mode === 'plain') { + if (opts.replyTarget.chatId !== chatId) { + throw new Error('frozen reply target does not match the session chat'); + } + // Preserve the existing regular-group → topic conversion safety without + // consulting the mutable per-turn target that may now belong to turn B. + if (ds?.scope === 'chat' && ds.session.rootMessageId) { + const mode = await getChatMode(appId, chatId, { forceRefresh: true }); + if (mode === 'topic') { + return replyWithHookPolicy(ds.session.rootMessageId, content, msgType, true, opts.uuid); + } + } + return sendWithHookPolicy(chatId, content, msgType, opts.uuid); + } if (ds?.scope === 'chat') { const fresh = readSessionFreshFromDisk(ds.session.sessionId, ds.larkAppId); if (fresh) syncReplyTargetState(ds, fresh); @@ -2622,6 +2816,12 @@ async function sessionReply( } // Thread-scope (or unknown / legacy): reply in thread. + if (opts?.replyTarget?.mode === 'plain') { + throw new Error('plain frozen reply target is invalid for a thread-scoped session'); + } + if (opts?.replyTarget?.mode === 'thread') { + return replyWithHookPolicy(opts.replyTarget.rootMessageId, content, msgType, true, opts.uuid); + } return replyWithHookPolicy(anchor, content, msgType, true, opts?.uuid); } @@ -2846,11 +3046,16 @@ interface DaemonDescriptor { botIndex: number; ipcPort: number; pid: number; + /** Kernel/OS process-birth identity; prevents fresh stale PID reuse. */ + processStartIdentity: string; startedAt: number; /** Public, random audience that changes on every daemon process start. */ bootInstanceId: string; /** Full-envelope Workflow mutation protocol supported by this process. */ workflowIpcProtocol: 'v1'; + /** Exact supervisor protocol this in-memory daemon will execute on signal. + * Absent until the SIGTERM/SIGINT handlers and all captured state are ready. */ + supervisorShutdownProtocol?: SupervisorShutdownProtocol; lastHeartbeat: number; /** * Resolved open_ids from this bot's allowedUsers config (post-email @@ -3403,6 +3608,30 @@ async function prewarmDocCommentSession(ds: DaemonSession, sub: DocSubscription) const turnId = `doc-watch-${Date.now()}-${sub.fileToken.slice(0, 8)}`; const sender = sub.ownerOpenId ? await resolveSender(ds.larkAppId, sub.ownerOpenId, 'user') : undefined; + const retirementPhase = riffRetirementAdmissionPhase(ds); + if (retirementPhase) { + logger.warn( + `[${tag(ds)}] Doc-watch warmup refused before acceptance during ${retirementPhase}`, + ); + await sessionReply( + sessionAnchorId(ds), + tr('worker.riff_close_in_progress', undefined, loc), + 'text', + ds.larkAppId, + turnId, + ); + return; + } + + if ((!ds.worker || ds.worker.killed) + && (ds.pendingRepo + || ds.worktreeCreating + || hasProtectedSessionMutationOwnership(ds))) { + throw new Error( + `doc-watch warmup deferred: session ${ds.session.sessionId} has durable opening ownership`, + ); + } + beginNewTurn(ds, title); ds.lastMessageAt = Date.now(); ds.session.lastMessageAt = new Date(ds.lastMessageAt).toISOString(); @@ -3426,7 +3655,9 @@ async function prewarmDocCommentSession(ds: DaemonSession, sub: DocSubscription) }); rememberLastCliInput(ds, promptContent, cliInput); sessionStore.updateSession(ds.session); - sendWorkerInput(ds, cliInput, turnId); + if (!sendWorkerInput(ds, cliInput, turnId)) { + throw new Error('doc-watch warmup worker input was not accepted'); + } markSessionActivity(ds); } else { ensureSessionWhiteboard(ds); @@ -3446,6 +3677,8 @@ async function prewarmDocCommentSession(ds: DaemonSession, sub: DocSubscription) logger.info(`[${tag(ds)}] doc-comment watch prewarm injected file=${sub.fileToken.slice(0, 12)}`); } +export const __testOnly_prewarmDocCommentSession = prewarmDocCommentSession; + // Dependencies passed to command-handler const commandDeps: CommandHandlerDeps = { activeSessions, @@ -4211,6 +4444,20 @@ ipcRoute('POST', '/api/attention', async (req, res) => { if (!verified.ok) { return jsonRes(res, 403, { ok: false, error: verified.error }); } + if (!ds?.managedTurnOrigin + || typeof raw.originTurnId !== 'string' + || raw.originTurnId !== ds.managedTurnOrigin.turnId + || claimedAttempt !== ds.managedTurnOrigin.dispatchAttempt) { + return jsonRes(res, 403, { ok: false, error: 'origin_identity_mismatch' }); + } + const codexDecision = validateCodexAppManagedSendOrigin( + ds.session.codexAppDispatchLedger, + ds.managedTurnOrigin, + ds.initConfig?.cliId === 'codex-app' || ds.session.cliId === 'codex-app', + ); + if (!codexDecision.ok) { + return jsonRes(res, 409, { ok: false, error: 'origin_not_sendable' }); + } } if (!ds) return jsonRes(res, 404, { ok: false, error: 'session_not_found' }); @@ -4359,6 +4606,14 @@ ipcRoute('POST', '/api/hooks/emit', async (req, res) => { if (!verified.ok) { return jsonRes(res, 403, { ok: false, error: verified.error }); } + const codexDecision = validateCodexAppManagedSendOrigin( + ds!.session.codexAppDispatchLedger, + ds!.managedTurnOrigin!, + ds!.initConfig?.cliId === 'codex-app' || ds!.session.cliId === 'codex-app', + ); + if (!codexDecision.ok) { + return jsonRes(res, 409, { ok: false, error: 'origin_not_sendable' }); + } // Do not let a valid capability for session A forge session B's identity // inside the hook payload. Hook-specific fields remain caller supplied; // routing/identity fields come only from the authenticated daemon session. @@ -14054,6 +14309,119 @@ export const __testOnly_resolvePinnedWorkingDir = resolvePinnedWorkingDir; export const __testOnly_handleNewTopic = (data: any, ctx: RoutingContext): Promise => handleNewTopic(data, ctx); export const __testOnly_handleThreadReply = (data: any, ctx: RoutingContext): Promise => handleThreadReply(data, ctx); +type NewDaemonSessionClaim = + | { accepted: true; key: string; owner: DaemonSession } + | { + accepted: false; + key: string; + owner: DaemonSession; + reason: 'existing_owner'; + closedIncomingSessionId: string; + } + | { + accepted: false; + key: string; + owner: DaemonSession; + reason: 'both_pending' | 'incoming_pending'; + preservedIncomingSessionId: string; + }; + +/** + * Claim the canonical routing slot for a newly-created daemon session. + * + * Restore/transfer intentionally use setActiveSessionSafe's replacement + * policy to clean historical collisions. Live creation is different: once a + * caller has won a slot, a concurrent creator must never close that winner + * while the first caller is still preparing/forking it. First owner wins; + * ledger-empty losers are closed before the caller can observe rejection. + * Any incoming owner with an unsettled durable turn is preserved and fails + * closed because choosing either generation would risk dropping accepted + * work. + */ +async function claimNewDaemonSession( + map: Map, + incoming: DaemonSession, +): Promise { + const key = activeSessionKey(incoming); + return withActiveSessionKeyLock(map, key, async () => { + const incumbent = map.get(key); + if (!incumbent || incumbent === incoming) { + map.set(key, incoming); + return { accepted: true as const, key, owner: incoming }; + } + + const incumbentPending = hasProtectedSessionMutationOwnership(incumbent); + // A newly-created loser's transient initialStartPending is still owned by + // this caller and can be rerouted to the incumbent after the empty row is + // closed. Only durable session-level ownership makes the loser itself + // irreplaceable here. + const incomingPending = hasProtectedSessionMutationOwnership(incoming.session); + if (incomingPending) { + const reason = incumbentPending ? 'both_pending' as const : 'incoming_pending' as const; + logger.error( + `[session-registration] refusing ${reason} collision key=${key} ` + + `incumbent=${incumbent.session.sessionId} incoming=${incoming.session.sessionId}; ` + + 'preserving both rows and failing closed', + ); + return { + accepted: false as const, + key, + owner: incumbent, + reason, + preservedIncomingSessionId: incoming.session.sessionId, + }; + } + + // All live creation paths reach this helper before a worker/worktree is + // started, so persistence close is the complete loser cleanup. Do it while + // holding the key lock: rejection never exposes an active ghost row. + sessionStore.closeSession(incoming.session.sessionId); + logger.warn( + `[session-registration] canonical owner ${incumbent.session.sessionId.substring(0, 8)} ` + + `kept for key=${key}; closed incoming ${incoming.session.sessionId.substring(0, 8)}`, + ); + return { + accepted: false as const, + key, + owner: incumbent, + reason: 'existing_owner' as const, + closedIncomingSessionId: incoming.session.sessionId, + }; + }); +} + +export const __testOnly_claimNewDaemonSession = ( + map: Map, + incoming: DaemonSession, +): Promise => claimNewDaemonSession(map, incoming); + +/** Persist pending-repo N immediately after the caller wins the canonical + * runtime slot and before any card/worktree/fork publication. Staging before + * the claim creates a durable same-key loser that can poison restart with two + * protected owners. A failed write unpublishes the not-yet-durably-admitted + * runtime owner and fails loudly. This remains inside the documented residual + * WS-claim → durable-admission gap; no redelivery guarantee is asserted here. */ +function stageClaimedPendingRepoSetup( + map: Map, + ds: DaemonSession, + args: Parameters[1], +): void { + try { + stagePendingRepoSetup(ds, args); + } catch (err) { + const key = activeSessionKey(ds); + if (map.get(key) === ds) map.delete(key); + try { sessionStore.closeSession(ds.session.sessionId); } + catch (closeErr) { + logger.error( + `[session-registration] setup staging and loser cleanup both failed for ` + + `${ds.session.sessionId}: ${closeErr instanceof Error ? closeErr.message : String(closeErr)}`, + ); + } + throw err; + } +} + /** * 该新会话是否要走「仅默认目录 + 自动建 worktree」：pinned dir 来自本 bot 自己的 * defaultWorkingDir (layer 3) 且开关打开。为真时，spawn 路径不再同步 fork，而是把会话登记 @@ -14084,6 +14452,351 @@ function startAutoWorktreePending(ds: DaemonSession, args: { logger.info(`[${tag(ds)}] auto-worktree → pending, building worktree off ${args.baseDir}`); } +type AvailableBot = Awaited>[number]; + +/** Materialize the opening input at the final synchronous boundary before + * fork. All potentially-slow preparation must happen before this call; the + * pendingFollowUps snapshot and fork then cannot be interleaved by a later + * same-anchor handler. */ +function buildReservedInitialInput( + ds: DaemonSession, + availableBots: AvailableBot[], +) { + const selfBot = getBot(ds.larkAppId); + const botCfg = selfBot.config; + return buildNewTopicCliInput( + ds.pendingPrompt ?? '', + ds.session.sessionId, + ds.session.cliId ?? botCfg.cliId, + ds.session.cliPathOverride ?? botCfg.cliPathOverride, + ds.pendingAttachments, + ds.pendingMentions, + availableBots, + ds.pendingFollowUps, + { name: selfBot.botName, openId: selfBot.botOpenId }, + localeForBot(ds.larkAppId), + ds.pendingSender, + { + larkAppId: ds.larkAppId, + chatId: ds.chatId, + whiteboardId: ds.session.whiteboardId, + substituteTrigger: ds.pendingSubstituteTrigger, + codexAppText: ds.pendingCodexAppText, + codexAppApplicationContext: ds.pendingCodexAppApplicationContext, + codexAppMessageContext: ds.pendingCodexAppMessageContext, + codexAppFollowUps: ds.pendingCodexAppFollowUps, + codexAppFollowUpContexts: ds.pendingCodexAppFollowUpContexts, + }, + ); +} + +function clearInitialStartBuffers(ds: DaemonSession): void { + ds.pendingPrompt = undefined; + ds.pendingCodexAppText = undefined; + ds.pendingCodexAppApplicationContext = undefined; + ds.pendingCodexAppMessageContext = undefined; + ds.pendingAttachments = undefined; + ds.pendingMentions = undefined; + ds.pendingSubstituteTrigger = undefined; + ds.pendingSender = undefined; + ds.pendingFollowUps = undefined; + ds.pendingFollowUpTurnIds = undefined; + ds.pendingCodexAppFollowUps = undefined; + ds.pendingCodexAppFollowUpContexts = undefined; + ds.pendingCodexAppFollowUpGateAccepted = undefined; +} + +function clearInitialStartClaim(ds: DaemonSession, token?: string): boolean { + if (token !== undefined && ds.initialStartClaimToken !== token) return false; + ds.initialStartClaimToken = undefined; + ds.initialStartPending = false; + if ((ds.queuedActivationTailAdmissionsOutstanding ?? 0) === 0) { + ds.queuedActivationTailReleasePending = undefined; + if (ds.queuedActivationTailReleaseRetryTimer) { + clearTimeout(ds.queuedActivationTailReleaseRetryTimer); + ds.queuedActivationTailReleaseRetryTimer = undefined; + } + } + return true; +} + +/** Fence an arrival before any sender/prompt await. Besides reserving durable + * FIFO order, keep the opening route owned until this reservation either lands + * in the durable tail or fails. */ +function reserveAsyncQueuedActivationTailAdmission( + ds: DaemonSession, +): QueuedActivationTailReservation { + const reservation = reserveQueuedActivationTailAdmission(ds); + ds.queuedActivationTailAdmissionsOutstanding = + (ds.queuedActivationTailAdmissionsOutstanding ?? 0) + 1; + return reservation; +} + +function scheduleQueuedActivationTailReleaseRetry( + ds: DaemonSession, + acknowledgedToken?: string, +): void { + if (!ds.initialStartPending || ds.queuedActivationTailReleaseRetryTimer) return; + const timer = setTimeout(() => { + if (ds.queuedActivationTailReleaseRetryTimer !== timer) return; + ds.queuedActivationTailReleaseRetryTimer = undefined; + if (!ds.initialStartPending) return; + if (!releaseQueuedActivationReservation(ds, acknowledgedToken)) { + scheduleQueuedActivationTailReleaseRetry(ds, acknowledgedToken); + } + }, 100); + timer.unref?.(); + ds.queuedActivationTailReleaseRetryTimer = timer; +} + +/** Complete one async reservation. If the predecessor ACK arrived during its + * await, the final settler performs the deferred handoff immediately. */ +function settleAsyncQueuedActivationTailAdmission(ds: DaemonSession): void { + const outstanding = Math.max( + 0, + (ds.queuedActivationTailAdmissionsOutstanding ?? 0) - 1, + ); + ds.queuedActivationTailAdmissionsOutstanding = outstanding || undefined; + if (outstanding > 0 || !ds.queuedActivationTailReleasePending) return; + const pending = ds.queuedActivationTailReleasePending; + ds.queuedActivationTailReleasePending = undefined; + if (!releaseQueuedActivationReservation(ds, pending.acknowledgedToken)) { + scheduleQueuedActivationTailReleaseRetry(ds, pending.acknowledgedToken); + } +} + +export const __testOnly_reserveAsyncQueuedActivationTailAdmission = + reserveAsyncQueuedActivationTailAdmission; +export const __testOnly_settleAsyncQueuedActivationTailAdmission = + settleAsyncQueuedActivationTailAdmission; + +/** Release a queued activation's runtime route reservation only after the + * worker ACKs actual adapter submission. Turns that arrived meanwhile are sent + * as one ordered follow-up, never allowed to overtake the opening item. */ +function releaseQueuedActivationReservation(ds: DaemonSession, acknowledgedToken?: string): boolean { + if ((ds.queuedActivationTailAdmissionsOutstanding ?? 0) > 0) { + // The worker may ACK while N+1 is still awaiting sender/resource prompt + // construction. Keep the route gate and replay this release when the last + // reserved arrival either persists or fails. + ds.queuedActivationTailReleasePending = acknowledgedToken === undefined + ? {} + : { acknowledgedToken }; + return false; + } + ds.queuedActivationTailReleasePending = undefined; + // A prior callback retry may observe the successor already promoted. That is + // success for the acknowledged predecessor; never append/send the same tail + // head a second time while its fresh token owns the journal. + if (ds.session.queuedActivationPending) { + return acknowledgedToken !== undefined + && ds.session.queuedActivationToken !== acknowledgedToken; + } + const buffered = [...(ds.pendingFollowUps ?? [])]; + if (buffered.length > 0) { + const bot = getBot(ds.larkAppId); + const cliId = ds.session.cliId ?? bot.config.cliId; + const rawCodexText = (ds.pendingCodexAppFollowUps ?? []).join('\n\n'); + const followUp = buildFollowUpCliInput( + buffered.join('\n\n'), + ds.session.sessionId, + { + cliId, + cliPathOverride: ds.session.cliPathOverride ?? bot.config.cliPathOverride, + locale: localeForBot(ds.larkAppId), + larkAppId: ds.larkAppId, + chatId: ds.chatId, + whiteboardId: ds.session.whiteboardId, + codexAppText: rawCodexText || buffered.join('\n\n'), + codexAppApplicationContext: ds.pendingCodexAppApplicationContext, + codexAppMessageContext: (ds.pendingCodexAppFollowUpContexts ?? []) + .filter(Boolean) + .join('\n\n'), + }, + ); + const bufferedTurnIds = ds.pendingFollowUpTurnIds ?? []; + const turnId = bufferedTurnIds[bufferedTurnIds.length - 1] + ?? ds.currentReplyTarget?.turnId + ?? `queued-activation-followup-${randomUUID()}`; + const reservation = reserveQueuedActivationTailAdmission(ds); + const bufferedGateDecisions = ds.pendingCodexAppFollowUpGateAccepted ?? []; + // Runtime buffers are rolling-upgrade compatibility only. If they carry an + // arrival-time decision, preserve it; otherwise fail closed rather than + // sampling a later ON setting and inventing a sidecar at ACK time. + reservation.codexAppInputAccepted = bufferedGateDecisions.length > 0 + && bufferedGateDecisions.every(Boolean); + try { + admitQueuedActivationTail(ds, { + userPrompt: rawCodexText || buffered.join('\n\n'), + cliInput: followUp, + turnId, + }, reservation); + } catch (err) { + logger.error( + `[${ds.session.sessionId.slice(0, 8)}] Failed to persist buffered activation successor: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + return false; + } + // Ownership has moved into the durable exact FIFO. Clear the runtime source + // only after its store write succeeds. + ds.pendingFollowUps = undefined; + ds.pendingFollowUpTurnIds = undefined; + ds.pendingCodexAppFollowUps = undefined; + ds.pendingCodexAppFollowUpContexts = undefined; + ds.pendingCodexAppFollowUpGateAccepted = undefined; + ds.pendingCodexAppApplicationContext = undefined; + } + + // Migrate pre-durable runtime tails (and tests/rolling-upgrade state) before + // advancing. Persistence happens before the volatile cursor is cleared. + for (const staged of ds.pendingQueuedActivationFollowUps ?? []) { + const reservation = reserveQueuedActivationTailAdmission(ds); + try { + admitQueuedActivationTail(ds, { + userPrompt: staged.userPrompt, + cliInput: staged.cliInput, + turnId: staged.turnId, + dispatchAttempt: staged.dispatchAttempt, + }, reservation, { codexAppInputGateFrozen: true }); + } catch (err) { + logger.error( + `[${ds.session.sessionId.slice(0, 8)}] Failed to migrate activation successor: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + return false; + } + } + ds.pendingQueuedActivationFollowUps = undefined; + + if ((ds.session.queuedActivationTail?.length ?? 0) === 0) { + clearInitialStartBuffers(ds); + clearInitialStartClaim(ds); + return true; + } + const head = [...ds.session.queuedActivationTail!] + .sort((a, b) => a.order - b.order || a.id.localeCompare(b.id))[0]!; + const workerUnavailable = !ds.worker + || ds.worker.killed + || !!ds.riffCloseState + || !!ds.riffShutdownState; + if (!promoteQueuedActivationTail(ds, workerUnavailable ? { send: false } : {})) return false; + try { + rememberLastCliInput(ds, head.userPrompt, head.cliInput, { + codexAppInputAccepted: !!head.cliInput.codexAppInput, + }); + } catch (err) { + logger.error( + `[${ds.session.sessionId.slice(0, 8)}] Failed to persist promoted activation metadata: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + } + clearInitialStartBuffers(ds); + if (workerUnavailable) { + // N's child can exit after its ACK while N+1 is still being prepared. Park + // the late exact head in the durable journal, but release the runtime claim + // so the next inbound turn can refork that head before accepting N+2. + clearInitialStartClaim(ds); + } + return true; +} + +export const __testOnly_releaseQueuedActivationReservation = releaseQueuedActivationReservation; + +/** Exact production callback passed to initWorkerPool. Keep the boolean return: + * false means the worker did not accept the staged FIFO head and the pool must + * retry instead of silently dropping the activation reservation. */ +function onQueuedActivationSubmitted(ds: DaemonSession, activationToken?: string): boolean { + return releaseQueuedActivationReservation(ds, activationToken); +} + +export const __testOnly_onQueuedActivationSubmitted = onQueuedActivationSubmitted; + +/** Fork an ordinary opening turn and release its route reservation. There is + * deliberately no await between the final buffered-input snapshot, fork, and + * release: a later handler either buffers before this block or observes the + * live worker after it. */ +function forkReservedInitialSession(ds: DaemonSession, availableBots: AvailableBot[]): void { + const userPrompt = ds.pendingPrompt ?? ''; + const input = buildReservedInitialInput(ds, availableBots); + rememberLastCliInput(ds, userPrompt, input); + const turnId = ds.pendingTurnId ?? ds.session.pendingRepoSetup?.turnId; + forkWorker(ds, input, turnId ? { turnId } : false); + ds.pendingTurnId = undefined; + // A no-project pendingRepo fallback enters forkWorker with `session.queued` + // still set. forkWorker materializes the exact opening as a tokened + // activation journal before returning; keep the route gate until the + // adapter-level ACK so a live-worker follower cannot bypass its FIFO tail. + const waitsForQueuedAck = ds.session.queuedActivationPending === true; + ds.initialStartPending = waitsForQueuedAck + || (ds.session.queuedActivationTail?.length ?? 0) > 0 + || (ds.queuedActivationTailAdmissionsOutstanding ?? 0) > 0; + clearInitialStartBuffers(ds); + if (!waitsForQueuedAck && ds.initialStartPending) { + // A follower may have reserved FIFO order while the opening prompt was + // still being built. Hand off now, or defer until its reservation settles. + void releaseQueuedActivationReservation(ds); + } +} + +/** Raw slash-command cold starts type the raw command only after prompt_ready. + * Buffer later same-anchor messages into the ordered follow-up payload carried + * on that same raw_input IPC; a separate worker message could overtake the + * command's text→Enter beat. */ +function forkReservedInitialRawSession(ds: DaemonSession, availableBots: AvailableBot[]): void { + const hasBufferedInput = + (ds.pendingPrompt?.trim().length ?? 0) > 0 + || (ds.pendingAttachments?.length ?? 0) > 0 + || (ds.pendingFollowUps?.length ?? 0) > 0; + if (hasBufferedInput) { + ensureSessionWhiteboard(ds); + const input = buildReservedInitialInput(ds, availableBots); + const botCfg = getBot(ds.larkAppId).config; + const bufferedGateDecisions = ds.pendingCodexAppFollowUpGateAccepted ?? []; + const bufferedCodexAppInputAccepted = bufferedGateDecisions.length > 0 + ? bufferedGateDecisions.every(Boolean) + : codexAppCleanInputAcceptedForSession(ds); + ds.pendingFollowUpInput = { + userPrompt: ds.pendingCodexAppText !== undefined || ds.pendingCodexAppFollowUps + ? [ds.pendingCodexAppText ?? '', ...(ds.pendingCodexAppFollowUps ?? [])].filter(Boolean).join('\n\n') + : ds.pendingPrompt || ds.pendingFollowUps?.join('\n\n') || '', + cliInput: input.content, + ...((ds.session.cliId === 'codex-app' || botCfg.cliId === 'codex-app') + && bufferedCodexAppInputAccepted + && input.codexAppInput + ? { codexAppInput: input.codexAppInput } + : {}), + codexAppInputGateFrozen: true, + }; + } + const rawInput = ds.pendingRawInput ?? ''; + rememberLastCliInput(ds, rawInput, rawInput); + // A non-owner turn can reserve/admit a durable successor while this literal + // raw opening is still preparing. Arm the same tokened activation journal + // used by pending-repo raw starts so that successor is released only after + // both the raw text and Enter have crossed the adapter boundary. + const armRawActivationAck = !ds.session.queuedActivationPending + && ((ds.session.queuedActivationTail?.length ?? 0) > 0 + || (ds.queuedActivationTailAdmissionsOutstanding ?? 0) > 0); + if (armRawActivationAck) ds.session.queued = true; + try { + forkWorker(ds, '', false); + } catch (err) { + // forkWorker's pre-init compensation restores its queued snapshot. This + // synthetic marker belongs only to the raw ACK fence, not the Dashboard + // backlog, so remove it while retaining pendingRawInput for a real retry. + if (armRawActivationAck && ds.session.queued === true) { + ds.session.queued = false; + sessionStore.updateSession(ds.session); + } + throw err; + } + // Raw text and Enter are a single adapter-submission boundary. When this + // cold start came through the durable pendingRepo fallback, retain the gate + // through that ACK so a follower cannot be sent between those two beats. + ds.initialStartPending = ds.session.queuedActivationPending === true; + clearInitialStartBuffers(ds); +} + async function replyInvalidWorkingDirs( anchor: string, larkAppId: string, @@ -14097,7 +14810,8 @@ async function replyInvalidWorkingDirs( if (invalid.length === 0) return false; ds.pendingRepo = false; - activeSessions.delete(sessionKey(anchor, larkAppId)); + const key = activeSessionKey(ds); + if (activeSessions.get(key) === ds) activeSessions.delete(key); sessionStore.closeSession(ds.session.sessionId); const msg = tr('cmd.repo.working_dir_not_exist', { dirs: invalid.map(d => `\`${d}\``).join(', '), @@ -14133,10 +14847,13 @@ async function startInitialPassthroughSession(args: { ownerOpenId: string | undefined; ownerUnionId: string | undefined; creatorOpenId: string | undefined; + /** Re-enter the existing-session route if another creator won the slot. */ + routeToCanonicalOwner: () => Promise; }): Promise { const { larkAppId, chatId, chatType, scope, anchor, messageId, replyRootId, parsed, commandContent, senderOpenId, senderUnionId, memberUnionId, ownerOpenId, ownerUnionId, creatorOpenId, + routeToCanonicalOwner, } = args; if (!await enforceMessageQuotaForCliInput(larkAppId, chatId, senderOpenId, messageId, anchor, senderUnionId, memberUnionId, chatType)) { return; @@ -14171,8 +14888,6 @@ async function startInitialPassthroughSession(args: { session.lastMessageAt = new Date(now).toISOString(); session.scope = scope; sessionStore.updateSession(session); - messageQueue.ensureQueue(anchor); - messageQueue.appendMessage(anchor, { ...parsed, content: commandContent }); const { pinnedWorkingDir, oncallEntry, inheritedFrom, pinnedFromBotDefault } = await resolvePinnedWorkingDir({ scope, anchor, chatId, chatType, larkAppId }); // Auto-worktree: register PENDING (router buffers, no force-fork) and build the @@ -14191,6 +14906,7 @@ async function startInitialPassthroughSession(args: { cliVersion: cliVersionCache.get(botCfg.cliId)?.version ?? 'unknown', lastMessageAt: now, hasHistory: false, + initialStartPending: true, pendingRepo: !pinnedWorkingDir || autoWt, pendingPrompt: '', pendingRawInput: commandContent, @@ -14206,10 +14922,27 @@ async function startInitialPassthroughSession(args: { } beginReplyTargetTurn(ds, replyRootId, messageId); sessionStore.updateSession(ds.session); - activeSessions.set(sessionKey(anchor, larkAppId), ds); + const registration = await claimNewDaemonSession(activeSessions, ds); + if (!registration.accepted) { + if (registration.reason === 'existing_owner' + && activeSessions.get(registration.key) === registration.owner) { + await routeToCanonicalOwner(); + } + return; + } + if (ds.pendingRepo) { + stageClaimedPendingRepoSetup(activeSessions, ds, { + mode: autoWt ? 'auto_worktree' : 'picker', + ...(autoWt && pinnedWorkingDir ? { baseDir: pinnedWorkingDir } : {}), + turnId: messageId, + }); + } + messageQueue.ensureQueue(anchor); + messageQueue.appendMessage(anchor, { ...parsed, content: commandContent }); if (pinnedWorkingDir && autoWt) { if (await replyInvalidWorkingDirs(anchor, larkAppId, ds)) return; + ds.initialStartPending = false; // pendingRepo/worktree now owns buffering // 挂起态提交：worktree 建好后经 runAutoWorktreeCommit → commitRepoSelection 拉起 // pendingRawInput 冷启动会话。detach → 立即返回，不阻塞本条消息处理。 startAutoWorktreePending(ds, { anchor, baseDir: pinnedWorkingDir, title: session.title, prompt: commandContent, operatorOpenId: ownerOpenId }); @@ -14218,10 +14951,8 @@ async function startInitialPassthroughSession(args: { if (pinnedWorkingDir) { if (await replyInvalidWorkingDirs(anchor, larkAppId, ds)) return; - rememberLastCliInput(ds, commandContent, commandContent); - // Raw passthrough gets its human turn only at the literal PTY write - // boundary (prompt_ready -> raw_input), never during CLI startup. - forkWorker(ds, '', false); + const availableBots = await getAvailableBots(larkAppId, chatId); + forkReservedInitialRawSession(ds, availableBots); const reason = oncallEntry ? `oncall-bound chat ${chatId}` : inheritedFrom @@ -14235,17 +14966,19 @@ async function startInitialPassthroughSession(args: { const scanDirs = getProjectScanDirs(ds).filter(d => existsSync(d)); const projects = scanDirs.length > 0 ? scanMultipleProjects(scanDirs, 3, repoPickerScanOptions()) : []; if (projects.length > 0) { + ds.initialStartPending = false; // pendingRepo/card now owns buffering lastRepoScan.set(chatId, projects); const cardJson = buildRepoSelectCard(projects, getSessionWorkingDir(ds), anchor, localeForBot(larkAppId), getBot(larkAppId).config.worktreeMultiPicker); ds.repoCardMessageId = await sessionReply(anchor, cardJson, 'interactive', larkAppId); + persistPendingRepoCardMessageId(ds, ds.repoCardMessageId); announcePendingRepoSession(ds); logger.info(`[${tag(ds)}] Waiting for repo selection before initial raw passthrough (${projects.length} projects)`); return; } ds.pendingRepo = false; - rememberLastCliInput(ds, commandContent, commandContent); - forkWorker(ds, '', false); + const availableBots = await getAvailableBots(larkAppId, chatId); + forkReservedInitialRawSession(ds, availableBots); logger.info(`[${tag(ds)}] No projects to select, queued initial raw passthrough ${commandContent.substring(0, 40)}`); } @@ -14271,6 +15004,13 @@ function mergeVcMeetingApplicationContext( } async function handleNewTopic(data: any, ctx: RoutingContext): Promise { + return withBotTurnAdmission( + ctx.larkAppId, + () => handleNewTopicAdmitted(data, ctx), + ); +} + +async function handleNewTopicAdmitted(data: any, ctx: RoutingContext): Promise { const { chatId, messageId, chatType, larkAppId, replyRootId, substituteTrigger } = ctx; // scope/anchor are mutable here: `/t` / `/topic` may flip a 普通群 chat-scope // routing into thread-scope so the bot's first reply seeds a Lark thread. @@ -14479,6 +15219,11 @@ async function handleNewTopic(data: any, ctx: RoutingContext): Promise { ownerOpenId: senderOpenId, ownerUnionId: senderUnionId, creatorOpenId: senderOpenId, + routeToCanonicalOwner: () => handleThreadReplyAdmitted(data, { + ...ctx, + scope, + anchor, + }), }); return; } @@ -14554,7 +15299,7 @@ async function handleNewTopic(data: any, ctx: RoutingContext): Promise { cmdPending = { pendingRepo: true, pendingPrompt: '', workingDir: pinnedWorkingDir }; } sessionStore.updateSession(session); - activeSessions.set(sessionKey(anchor, larkAppId), { + const cmdDs: DaemonSession = { session, worker: null, workerPort: null, @@ -14569,7 +15314,16 @@ async function handleNewTopic(data: any, ctx: RoutingContext): Promise { hasHistory: false, ownerOpenId: senderOpenId, ...cmdPending, - }); + }; + const registration = await claimNewDaemonSession(activeSessions, cmdDs); + if (!registration.accepted) { + if (registration.reason !== 'existing_owner') return; + } else if (cmdDs.pendingRepo) { + stageClaimedPendingRepoSetup(activeSessions, cmdDs, { + mode: 'picker', + turnId: messageId, + }); + } // Pass mention-stripped content so /command argument parsing works. await handleCommand(cmd, anchor, { ...parsed, content: commandContent }, commandDeps, larkAppId); return; @@ -14646,8 +15400,6 @@ async function handleNewTopic(data: any, ctx: RoutingContext): Promise { session.lastMessageAt = new Date(now).toISOString(); session.scope = scope; sessionStore.updateSession(session); - messageQueue.ensureQueue(anchor); - messageQueue.appendMessage(anchor, parsed); // Control card compensates for the card-less (avatar-style) chat-scope // substitute session. Topic-group substitute sessions (#475) are thread-scope @@ -14670,6 +15422,7 @@ async function handleNewTopic(data: any, ctx: RoutingContext): Promise { cliVersion: cliVersionCache.get(botCfg.cliId)?.version ?? 'unknown', lastMessageAt: now, hasHistory: false, + initialStartPending: true, pendingRepo: !pinnedWorkingDir || autoWt, pendingPrompt: promptContent, pendingTurnId: messageId, @@ -14694,13 +15447,30 @@ async function handleNewTopic(data: any, ctx: RoutingContext): Promise { : 'thread'; beginReplyTargetTurn(ds, replyRootId, messageId, new Date().toISOString(), { quoteOnly: substituteReplyMode === 'quote', substitute: !!substituteTrigger }); sessionStore.updateSession(ds.session); - activeSessions.set(sessionKey(anchor, larkAppId), ds); + const registration = await claimNewDaemonSession(activeSessions, ds); + if (!registration.accepted) { + if (registration.reason === 'existing_owner' + && activeSessions.get(registration.key) === registration.owner) { + await handleThreadReplyAdmitted(data, { ...ctx, scope, anchor }); + } + return; + } + if (ds.pendingRepo) { + stageClaimedPendingRepoSetup(activeSessions, ds, { + mode: autoWt ? 'auto_worktree' : 'picker', + ...(autoWt && pinnedWorkingDir ? { baseDir: pinnedWorkingDir } : {}), + turnId: messageId, + }); + } + messageQueue.ensureQueue(anchor); + messageQueue.appendMessage(anchor, parsed); // Auto-worktree: session is registered PENDING; build the worktree off the // critical path, then commitRepoSelection pins it + forks (folding in any // messages buffered during creation). detach → return immediately. if (pinnedWorkingDir && autoWt) { if (await replyInvalidWorkingDirs(anchor, larkAppId, ds)) return; + ds.initialStartPending = false; // pendingRepo/worktree now owns buffering startAutoWorktreePending(ds, { anchor, baseDir: pinnedWorkingDir, title: session.title, prompt: promptContent, operatorOpenId: senderOpenId }); return; } @@ -14708,13 +15478,10 @@ async function handleNewTopic(data: any, ctx: RoutingContext): Promise { // Pinned (oncall binding or inherited from sibling bot): spawn CLI immediately. if (pinnedWorkingDir) { if (await replyInvalidWorkingDirs(anchor, larkAppId, ds)) return; - const selfBot = getBot(larkAppId); ensureSessionWhiteboard(ds); - const prompt = buildNewTopicCliInput(promptContent, session.sessionId, botCfg.cliId, botCfg.cliPathOverride, attachments, parsed.mentions, await getAvailableBots(larkAppId, chatId), undefined, { name: selfBot.botName, openId: selfBot.botOpenId }, localeForBot(larkAppId), newTopicSender, { larkAppId, chatId, whiteboardId: ds.session.whiteboardId, substituteTrigger, codexAppText: codexAppVisibleText, codexAppApplicationContext, codexAppMessageContext }); + const availableBots = await getAvailableBots(larkAppId, chatId); await noteTurnReceived(ds, messageId, content, newTopicSender, messageId, substituteTrigger ? SUBSTITUTE_RECEIVED_REACTION_EMOJI_TYPE : undefined); - rememberLastCliInput(ds, promptContent, prompt); - forkWorker(ds, prompt, { turnId: messageId }); - ds.pendingTurnId = undefined; + forkReservedInitialSession(ds, availableBots); const reason = oncallEntry ? `oncall-bound chat ${chatId}` : inheritedFrom @@ -14732,22 +15499,21 @@ async function handleNewTopic(data: any, ctx: RoutingContext): Promise { projects = scanMultipleProjects(scanDirs, 3, repoPickerScanOptions()); } if (projects.length > 0) { + ds.initialStartPending = false; // pendingRepo/card now owns buffering lastRepoScan.set(chatId, projects); const currentCwd = getSessionWorkingDir(ds); const cardJson = buildRepoSelectCard(projects, currentCwd, anchor, localeForBot(larkAppId), getBot(larkAppId).config.worktreeMultiPicker); ds.repoCardMessageId = await sessionReply(anchor, cardJson, 'interactive', larkAppId); + persistPendingRepoCardMessageId(ds, ds.repoCardMessageId); announcePendingRepoSession(ds); logger.info(`[${tag(ds)}] Waiting for repo selection (${projects.length} projects)`); } else { // No projects found — skip repo selection, spawn directly ds.pendingRepo = false; - const selfBot = getBot(larkAppId); ensureSessionWhiteboard(ds); - const prompt = buildNewTopicCliInput(promptContent, session.sessionId, botCfg.cliId, botCfg.cliPathOverride, attachments, parsed.mentions, await getAvailableBots(larkAppId, chatId), undefined, { name: selfBot.botName, openId: selfBot.botOpenId }, localeForBot(larkAppId), newTopicSender, { larkAppId, chatId, whiteboardId: ds.session.whiteboardId, substituteTrigger, codexAppText: codexAppVisibleText, codexAppApplicationContext, codexAppMessageContext }); + const availableBots = await getAvailableBots(larkAppId, chatId); await noteTurnReceived(ds, messageId, content, newTopicSender, messageId, substituteTrigger ? SUBSTITUTE_RECEIVED_REACTION_EMOJI_TYPE : undefined); - rememberLastCliInput(ds, promptContent, prompt); - forkWorker(ds, prompt, { turnId: messageId }); - ds.pendingTurnId = undefined; + forkReservedInitialSession(ds, availableBots); logger.info(`Session ${session.sessionId} ready (no projects to select), total active: ${getActiveCount()}`); } } @@ -14906,6 +15672,7 @@ async function handleBotAdded(chatId: string, operatorOpenId: string | undefined cliVersion: cliVersionCache.get(botCfg.cliId)?.version ?? 'unknown', lastMessageAt: now, hasHistory: false, + initialStartPending: true, pendingRepo: !pinnedWorkingDir || autoWt, pendingPrompt: promptBody, pendingCodexAppText: botCfg.cliId === 'codex-app' ? codexAppText : undefined, @@ -14913,22 +15680,28 @@ async function handleBotAdded(chatId: string, operatorOpenId: string | undefined currentTurnTitle: title, workingDir: pinnedWorkingDir, }; - activeSessions.set(dsKey, ds); + const registration = await claimNewDaemonSession(activeSessions, ds); + if (!registration.accepted) { + if (activeSessions.get(registration.key) === registration.owner) { + groupJoinAnchorByChat.set(chatLiveKey, registration.key); + } + return; + } + if (ds.pendingRepo) { + stageClaimedPendingRepoSetup(activeSessions, ds, { + mode: autoWt ? 'auto_worktree' : 'picker', + ...(autoWt && pinnedWorkingDir ? { baseDir: pinnedWorkingDir } : {}), + turnId: anchor, + }); + } // Register the anchor so a later duplicate bot.added for this chat is deduped // even in 话题群 (where dsKey is the seed id, not chatId). groupJoinAnchorByChat.set(chatLiveKey, dsKey); - const selfBot = getBot(larkAppId); - const buildPrompt = async () => buildNewTopicCliInput( - promptBody, session.sessionId, botCfg.cliId, botCfg.cliPathOverride, - undefined, undefined, await getAvailableBots(larkAppId, chatId), undefined, - { name: selfBot.botName, openId: selfBot.botOpenId }, localeForBot(larkAppId), undefined, - { larkAppId, chatId, whiteboardId: ds.session.whiteboardId, codexAppText }, - ); - // Auto-worktree: register PENDING, build worktree off-path, commit+fork later. if (pinnedWorkingDir && autoWt) { if (await replyInvalidWorkingDirs(anchor, larkAppId, ds)) return; + ds.initialStartPending = false; // pendingRepo/worktree now owns buffering startAutoWorktreePending(ds, { anchor, baseDir: pinnedWorkingDir, title, prompt: promptBody, operatorOpenId }); return; } @@ -14937,10 +15710,9 @@ async function handleBotAdded(chatId: string, operatorOpenId: string | undefined if (pinnedWorkingDir) { if (await replyInvalidWorkingDirs(anchor, larkAppId, ds)) return; ensureSessionWhiteboard(ds); - const prompt = await buildPrompt(); + const availableBots = await getAvailableBots(larkAppId, chatId); await noteTurnReceived(ds, anchor, promptBody); - rememberLastCliInput(ds, promptBody, prompt); - forkWorker(ds, prompt); + forkReservedInitialSession(ds, availableBots); logger.info(`[auto-start:入群] ${chatId.substring(0, 12)} 自动开工（${mode}/${scope}），workingDir=${pinnedWorkingDir}`); return; } @@ -14950,18 +15722,19 @@ async function handleBotAdded(chatId: string, operatorOpenId: string | undefined const scanDirs = getProjectScanDirs(ds).filter(d => existsSync(d)); const projects = scanDirs.length > 0 ? scanMultipleProjects(scanDirs, 3, repoPickerScanOptions()) : []; if (projects.length > 0) { + ds.initialStartPending = false; // pendingRepo/card now owns buffering lastRepoScan.set(chatId, projects); const cardJson = buildRepoSelectCard(projects, getSessionWorkingDir(ds), anchor, localeForBot(larkAppId), getBot(larkAppId).config.worktreeMultiPicker); ds.repoCardMessageId = await sessionReply(anchor, cardJson, 'interactive', larkAppId); + persistPendingRepoCardMessageId(ds, ds.repoCardMessageId); announcePendingRepoSession(ds); logger.info(`[auto-start:入群] ${chatId.substring(0, 12)} 无默认目录，弹 repo 选择卡（${projects.length} 个项目）`); } else { ds.pendingRepo = false; ensureSessionWhiteboard(ds); - const prompt = await buildPrompt(); + const availableBots = await getAvailableBots(larkAppId, chatId); await noteTurnReceived(ds, anchor, promptBody); - rememberLastCliInput(ds, promptBody, prompt); - forkWorker(ds, prompt); + forkReservedInitialSession(ds, availableBots); logger.info(`[auto-start:入群] ${chatId.substring(0, 12)} 无默认目录且无可选项目，直接开工`); } } finally { @@ -15002,6 +15775,22 @@ function lookupForeignBotName(senderOpenId: string, larkAppId: string): string { } async function handleThreadReply(data: any, ctx: RoutingContext): Promise { + // Admission is bot-wide but intentionally concurrent. Add a narrower FIFO + // before entering it so two same-anchor deliveries can never invert while + // quota/resource/sender preparation awaits. Synthetic keys share the same + // lock registry without occupying or mutating an active-session slot. + const deliveryKey = `\u0000thread-delivery:${ctx.larkAppId}:${ctx.scope}:${ctx.anchor}`; + return withActiveSessionKeyLock( + activeSessions, + deliveryKey, + () => withBotTurnAdmission( + ctx.larkAppId, + () => handleThreadReplyAdmitted(data, ctx), + ), + ); +} + +async function handleThreadReplyAdmitted(data: any, ctx: RoutingContext): Promise { const { chatId: ctxChatId, chatType: ctxChatType, scope, anchor, larkAppId, replyRootId, substituteTrigger } = ctx; await resolveNonsupportMessage(data, larkAppId); const { parsed, resources } = parseEventMessage(data); @@ -15239,6 +16028,11 @@ async function handleThreadReply(data: any, ctx: RoutingContext): Promise ownerOpenId: isForeignBot ? undefined : threadSenderOpenId, ownerUnionId: isForeignBot ? undefined : data?.sender?.sender_id?.union_id, creatorOpenId: threadSenderOpenId, + routeToCanonicalOwner: () => handleThreadReplyAdmitted(data, { + ...ctx, + scope, + anchor, + }), }); return; } @@ -15251,6 +16045,32 @@ async function handleThreadReply(data: any, ctx: RoutingContext): Promise // 收紧到与 daemon 命令同档；这会同时改变真人 oncall 成员的现有行为，应单独评估。 const ds = existingDs; if (ds?.worker && !ds.worker.killed) { + const retirementPhase = riffRetirementAdmissionPhase(ds); + if (retirementPhase) { + sessionReply( + anchor, + tr('worker.riff_close_in_progress', undefined, localeForBot(larkAppId)), + 'text', + larkAppId, + ); + logger.warn( + `[${anchor.substring(0, 12)}] Refused passthrough ${cmd} during ${retirementPhase}`, + ); + return; + } + if (hasQueuedActivationAdmissionGate(ds)) { + sessionReply( + anchor, + tr('daemon.cmd_activation_pending', { cmd }, localeForBot(larkAppId)), + 'text', + larkAppId, + ); + logger.warn( + `[${anchor.substring(0, 12)}] Refused passthrough ${cmd} before acceptance ` + + 'while queued activation owns raw submission order', + ); + return; + } // Passthrough commands bypass the normal message-forwarding block // below, so bind this accepted Lark turn here before the worker rotates // its marker at the actual PTY write boundary. @@ -15335,7 +16155,7 @@ async function handleThreadReply(data: any, ctx: RoutingContext): Promise cmdPending = { pendingRepo: true, pendingPrompt: '', workingDir: pinnedWorkingDir }; } sessionStore.updateSession(session); - activeSessions.set(sessionKey(anchor, larkAppId), { + const cmdDs: DaemonSession = { session, worker: null, workerPort: null, @@ -15350,7 +16170,16 @@ async function handleThreadReply(data: any, ctx: RoutingContext): Promise hasHistory: false, ownerOpenId: threadSenderOpenId, ...cmdPending, - }); + }; + const registration = await claimNewDaemonSession(activeSessions, cmdDs); + if (!registration.accepted) { + if (registration.reason !== 'existing_owner') return; + } else if (cmdDs.pendingRepo) { + stageClaimedPendingRepoSetup(activeSessions, cmdDs, { + mode: 'picker', + turnId: parsed.messageId, + }); + } } // Pass mention-stripped content so /command argument parsing works. // chatId lets session-less handlers (e.g. /group) reach the chat roster. @@ -15414,6 +16243,38 @@ async function handleThreadReply(data: any, ctx: RoutingContext): Promise } } + // Reserve an existing worker:null owner before ANY post-routing await. + // withBotTurnAdmission permits concurrent turns, so two + // same-anchor handlers can both resolve the same cold owner and otherwise + // race through resource/sender preparation into two forks. The token makes + // one handler the opening owner; every other handler observes the gate and + // buffers. Queued activations retain the token through adapter ACK, while an + // ordinary refork releases it immediately after its buffered tail is handed + // to the worker. + let initialStartClaimToken: string | undefined; + let initialStartClaimWaitsForQueuedAck = false; + let retainInitialStartClaim = false; + const tryAcquireInitialStartClaim = (): void => { + if (initialStartClaimToken + || !ds + || (ds.worker && !ds.worker.killed) + || ds.pendingRepo + || ds.initialStartPending === true) return; + initialStartClaimToken = randomUUID(); + ds.initialStartClaimToken = initialStartClaimToken; + ds.initialStartPending = true; + initialStartClaimWaitsForQueuedAck = ds.session.queued === true + || ds.session.queuedActivationPending === true + || ds.session.queuedActivationInput !== undefined; + }; + tryAcquireInitialStartClaim(); + const ownsInitialStartClaim = (): boolean => !!( + ds + && initialStartClaimToken + && ds.initialStartClaimToken === initialStartClaimToken + ); + + try { const quotaSenderOpenId = threadSenderOpenId; if (!await enforceMessageQuotaForCliInput(larkAppId, ctxChatId ?? data?.message?.chat_id, quotaSenderOpenId, parsed.messageId, anchor, threadTeamTrustUnionId, threadSenderUnionId, ctxChatType)) { return; @@ -15466,8 +16327,57 @@ async function handleThreadReply(data: any, ctx: RoutingContext): Promise sessionStore.updateSession(ds.session); } - // If waiting for repo selection, buffer the message and remind user - if (ds?.pendingRepo || ds?.pendingRepoCommitInFlight) { + // The first owner may have failed while this handler was awaiting resource + // preparation. Re-check once at the final routing boundary so the oldest + // surviving follower atomically takes over instead of proceeding unclaimed. + tryAcquireInitialStartClaim(); + + // A published worker:null owner is not necessarily ready to refork: its + // opening turn may still be preparing. Buffer both repo-pending and + // initial-start-pending messages into that opening turn so a later prompt + // cannot overtake it. + const initialStartPending = ds?.initialStartPending === true && !ownsInitialStartClaim(); + if (ds && initialStartPending && !ds.pendingRepo) { + const tailReservation = reserveAsyncQueuedActivationTailAdmission(ds); + try { + const botCfg = getBot(ds.larkAppId).config; + const followUp = buildFollowUpCliInput(promptContent, ds.session.sessionId, { + attachments, + mentions: parsed.mentions, + isAdoptMode: false, + cliId: ds.session.cliId ?? botCfg.cliId, + cliPathOverride: ds.session.cliPathOverride ?? botCfg.cliPathOverride, + sender: await getThreadSender(), + larkAppId, + chatId: ds.session.chatId, + whiteboardId: ds.session.whiteboardId, + substituteTrigger, + codexAppText: parsed.content, + codexAppApplicationContext, + codexAppMessageContext, + }); + admitQueuedActivationTail(ds, { + userPrompt: promptContent, + cliInput: followUp, + turnId: parsed.messageId, + }, tailReservation); + } finally { + settleAsyncQueuedActivationTailAdmission(ds); + } + logger.info( + `[${tag(ds)}] buffered same-anchor turn ${parsed.messageId.substring(0, 12)} ` + + 'behind queued activation submission ACK', + ); + return; + } + if (ds?.pendingRepo || initialStartPending) { + const durableTailReservation = ds.pendingRepo + ? reserveAsyncQueuedActivationTailAdmission(ds) + : undefined; + const bufferedCodexAppInputAccepted = initialStartPending && !ds.pendingRepo + ? codexAppCleanInputAcceptedForSession(ds) + : undefined; + try { // Enrich content with attachment hints and mention metadata (same as normal send) const codexAppFollowUpContextParts: string[] = []; if (codexAppMessageContext) codexAppFollowUpContextParts.push(codexAppMessageContext); @@ -15514,6 +16424,65 @@ async function handleThreadReply(data: any, ctx: RoutingContext): Promise codexAppFollowUpContextParts.unshift(followUpSenderBlock); } } + if (ds.pendingRepo) { + const hasOpening = (ds.pendingPrompt?.trim().length ?? 0) > 0 + || (ds.pendingAttachments?.length ?? 0) > 0 + || !!ds.pendingRawInput; + ds.session.queued = true; + if (!hasOpening) { + // A bare /repo placeholder has no N yet. This inbound becomes the + // durable opening instead of an impossible successor to an empty ACK. + ds.pendingPrompt = promptContent; + ds.pendingTurnId = parsed.messageId; + ds.pendingCodexAppText = parsed.content; + ds.pendingCodexAppApplicationContext = codexAppApplicationContext; + ds.pendingCodexAppMessageContext = codexAppMessageContext; + ds.pendingAttachments = attachments.length > 0 ? attachments : undefined; + ds.pendingMentions = parsed.mentions; + ds.pendingSender = followUpSender; + const existingSetup = ds.session.pendingRepoSetup; + stagePendingRepoSetup(ds, { + mode: existingSetup?.mode ?? 'picker', + ...(existingSetup?.baseDir ? { baseDir: existingSetup.baseDir } : {}), + turnId: parsed.messageId, + }); + } else { + const botCfg = getBot(ds.larkAppId).config; + const exactFollowUp = buildFollowUpCliInput(promptContent, ds.session.sessionId, { + attachments, + mentions: parsed.mentions, + isAdoptMode: false, + cliId: ds.session.cliId ?? botCfg.cliId, + cliPathOverride: ds.session.cliPathOverride ?? botCfg.cliPathOverride, + sender: followUpSender, + larkAppId, + chatId: ds.session.chatId, + whiteboardId: ds.session.whiteboardId, + substituteTrigger, + codexAppText: parsed.content, + codexAppApplicationContext, + codexAppMessageContext, + }); + ds.session.queuedPrompt ??= ds.pendingPrompt; + ds.session.queuedCodexAppText ??= ds.pendingCodexAppText; + ds.session.queuedCodexAppMessageContext ??= ds.pendingCodexAppMessageContext; + admitQueuedActivationTail(ds, { + userPrompt: promptContent, + cliInput: exactFollowUp, + turnId: parsed.messageId, + }, durableTailReservation!); + } + const pendingReplyKey = ds.worktreeCreating + ? 'daemon.worktree_building_wait' + : 'daemon.choose_repo_first'; + await sessionReply(anchor, tr(pendingReplyKey, undefined, localeForBot(larkAppId)), 'text', larkAppId); + return; + } + + // Initial-start compatibility path after repo selection: only the durable + // queued-activation gate belongs here. pendingRepoCommitInFlight protects a + // second repo selection, but must not divert ordinary turns away from the + // already-forked worker into source buffers that will never be consumed. const sameInitialCaller = !!followUpSender?.openId && followUpSender.openId === ds.pendingSender?.openId; if (ds.pendingRawInput) { @@ -15544,16 +16513,29 @@ async function handleThreadReply(data: any, ctx: RoutingContext): Promise } if (!ds.pendingFollowUps) ds.pendingFollowUps = []; ds.pendingFollowUps.push(enriched); + if (!ds.pendingFollowUpTurnIds) ds.pendingFollowUpTurnIds = []; + ds.pendingFollowUpTurnIds.push(parsed.messageId); if (!ds.pendingCodexAppFollowUps) ds.pendingCodexAppFollowUps = []; ds.pendingCodexAppFollowUps.push(parsed.content); if (!ds.pendingCodexAppFollowUpContexts) ds.pendingCodexAppFollowUpContexts = []; ds.pendingCodexAppFollowUpContexts.push(codexAppFollowUpContextParts.join('\n\n')); + if (!ds.pendingCodexAppFollowUpGateAccepted) { + ds.pendingCodexAppFollowUpGateAccepted = []; + } + ds.pendingCodexAppFollowUpGateAccepted.push(bufferedCodexAppInputAccepted === true); if (codexAppApplicationContext) { ds.pendingCodexAppApplicationContext = mergeVcMeetingApplicationContext( ds.pendingCodexAppApplicationContext, codexAppApplicationContext, ); } + if (initialStartPending && !ds.pendingRepo) { + logger.info( + `[${tag(ds)}] buffered same-anchor turn ${parsed.messageId.substring(0, 12)} ` + + 'behind reserved initial start', + ); + return; + } // Auto-worktree pending (worktreeCreating) has no repo card to point at — the // message IS buffered (folded on commit), so just say "hold on, building worktree" // instead of the misleading "pick a repo from the card above". @@ -15562,19 +16544,19 @@ async function handleThreadReply(data: any, ctx: RoutingContext): Promise : 'daemon.choose_repo_first'; await sessionReply(anchor, tr(pendingReplyKey, undefined, localeForBot(larkAppId)), 'text', larkAppId); return; + } finally { + if (durableTailReservation) settleAsyncQueuedActivationTailAdmission(ds); + } } - // Route to file queue (keyed by anchor: rootMessageId for thread, chatId for chat) - messageQueue.ensureQueue(anchor); - messageQueue.appendMessage(anchor, parsed); - if (!ds) { // No active session at this anchor — auto-create. This branch is mostly a // safety net; the dispatcher routes here only when isSessionOwner() returns // true, but races (between check and execution, or session-closed events) // can land us here. if (activeSessions.has(sessionKey(anchor, larkAppId))) { - logger.info(`[${larkAppId}] Session already exists for ${scope}-scope ${anchor}, skipping auto-create`); + logger.info(`[${larkAppId}] Session already exists for ${scope}-scope ${anchor}, routing to canonical owner`); + await handleThreadReplyAdmitted(data, ctx); return; } @@ -15646,6 +16628,7 @@ async function handleThreadReply(data: any, ctx: RoutingContext): Promise cliVersion: cliVersionCache.get(botCfg.cliId)?.version ?? 'unknown', lastMessageAt: now, hasHistory: false, + initialStartPending: true, pendingRepo: !pinnedWorkingDir || autoWt, pendingPrompt: promptContent, pendingTurnId: parsed.messageId, @@ -15670,11 +16653,30 @@ async function handleThreadReply(data: any, ctx: RoutingContext): Promise : 'thread'; beginReplyTargetTurn(newDs, replyRootId, parsed.messageId, new Date().toISOString(), { quoteOnly: substituteReplyMode === 'quote', substitute: !!substituteTrigger }); sessionStore.updateSession(newDs.session); - activeSessions.set(sessionKey(anchor, larkAppId), newDs); + const registration = await claimNewDaemonSession(activeSessions, newDs); + if (!registration.accepted) { + if (registration.reason === 'existing_owner' + && activeSessions.get(registration.key) === registration.owner) { + await handleThreadReplyAdmitted(data, ctx); + } + return; + } + if (newDs.pendingRepo) { + stageClaimedPendingRepoSetup(activeSessions, newDs, { + mode: autoWt ? 'auto_worktree' : 'picker', + ...(autoWt && pinnedWorkingDir ? { baseDir: pinnedWorkingDir } : {}), + turnId: parsed.messageId, + }); + } + // Route to file queue only after ownership is committed. A rejected + // creator re-enters the canonical route above, which appends exactly once. + messageQueue.ensureQueue(anchor); + messageQueue.appendMessage(anchor, parsed); // Auto-worktree: register PENDING, build worktree off-path, commit+fork later. if (pinnedWorkingDir && autoWt) { if (await replyInvalidWorkingDirs(anchor, larkAppId, newDs)) return; + newDs.initialStartPending = false; // pendingRepo/worktree now owns buffering startAutoWorktreePending(newDs, { anchor, baseDir: pinnedWorkingDir, title: parsed.content.substring(0, 50), prompt: promptContent, operatorOpenId: ownerOpenId }); return; } @@ -15683,13 +16685,10 @@ async function handleThreadReply(data: any, ctx: RoutingContext): Promise // spawn CLI immediately, skip repo selection. if (pinnedWorkingDir) { if (await replyInvalidWorkingDirs(anchor, larkAppId, newDs)) return; - const selfBot = getBot(larkAppId); ensureSessionWhiteboard(newDs); - const prompt = buildNewTopicCliInput(promptContent, session.sessionId, botCfg.cliId, botCfg.cliPathOverride, attachments, parsed.mentions, await getAvailableBots(larkAppId, autoCreateChatId), undefined, { name: selfBot.botName, openId: selfBot.botOpenId }, localeForBot(larkAppId), autoCreateSender, { larkAppId, chatId: autoCreateChatId, whiteboardId: newDs.session.whiteboardId, substituteTrigger, codexAppText: parsed.content, codexAppApplicationContext, codexAppMessageContext }); + const availableBots = await getAvailableBots(larkAppId, autoCreateChatId); await noteTurnReceived(newDs, parsed.messageId, parsed.content, autoCreateSender, parsed.messageId, substituteTrigger ? SUBSTITUTE_RECEIVED_REACTION_EMOJI_TYPE : undefined); - rememberLastCliInput(newDs, promptContent, prompt); - forkWorker(newDs, prompt, { turnId: parsed.messageId }); - newDs.pendingTurnId = undefined; + forkReservedInitialSession(newDs, availableBots); const reason = oncallEntry ? `oncall-bound chat ${autoCreateChatId}` : inheritedFrom @@ -15707,27 +16706,49 @@ async function handleThreadReply(data: any, ctx: RoutingContext): Promise projects = scanMultipleProjects(scanDirs2, 3, repoPickerScanOptions()); } if (projects.length > 0) { + newDs.initialStartPending = false; // pendingRepo/card now owns buffering lastRepoScan.set(autoCreateChatId, projects); const currentCwd = getSessionWorkingDir(newDs); const cardJson = buildRepoSelectCard(projects, currentCwd, anchor, localeForBot(larkAppId), getBot(larkAppId).config.worktreeMultiPicker); newDs.repoCardMessageId = await sessionReply(anchor, cardJson, 'interactive', larkAppId); + persistPendingRepoCardMessageId(newDs, newDs.repoCardMessageId); announcePendingRepoSession(newDs); logger.info(`[${tag(newDs)}] Waiting for repo selection (${projects.length} projects)`); } else { // No projects found — skip repo selection, spawn directly newDs.pendingRepo = false; - const selfBot = getBot(larkAppId); ensureSessionWhiteboard(newDs); - const prompt = buildNewTopicCliInput(promptContent, session.sessionId, botCfg.cliId, botCfg.cliPathOverride, attachments, parsed.mentions, await getAvailableBots(larkAppId, autoCreateChatId), undefined, { name: selfBot.botName, openId: selfBot.botOpenId }, localeForBot(larkAppId), autoCreateSender, { larkAppId, chatId: autoCreateChatId, whiteboardId: newDs.session.whiteboardId, substituteTrigger, codexAppText: parsed.content, codexAppApplicationContext, codexAppMessageContext }); + const availableBots = await getAvailableBots(larkAppId, autoCreateChatId); await noteTurnReceived(newDs, parsed.messageId, parsed.content, autoCreateSender, parsed.messageId, substituteTrigger ? SUBSTITUTE_RECEIVED_REACTION_EMOJI_TYPE : undefined); - rememberLastCliInput(newDs, promptContent, prompt); - forkWorker(newDs, prompt, { turnId: parsed.messageId }); - newDs.pendingTurnId = undefined; + forkReservedInitialSession(newDs, availableBots); } return; } + // A close/shutdown abort whose restoration ACK timed out deliberately keeps + // this fence beyond the mutation that initiated it. Refuse visibly BEFORE + // appending the message queue, adding an accepted reaction, or persisting it + // as lastCliInput; sendWorkerInput(false) must never look like acceptance. + const retirementPhase = riffRetirementAdmissionPhase(ds); + if (retirementPhase) { + logger.warn( + `[${tag(ds)}] Rejected inbound ${parsed.messageId} before acceptance during ${retirementPhase}`, + ); + await sessionReply( + anchor, + tr('worker.riff_close_in_progress', undefined, localeForBot(larkAppId)), + 'text', + larkAppId, + parsed.messageId, + ); + return; + } + + // Existing-owner route: append once after all pending-repo early returns. + messageQueue.ensureQueue(anchor); + messageQueue.appendMessage(anchor, parsed); + // Send message to worker via IPC if (ds.worker && !ds.worker.killed) { const dsBotCfgForMsg = getBot(ds.larkAppId).config; @@ -15767,7 +16788,10 @@ async function handleThreadReply(data: any, ctx: RoutingContext): Promise beginNewTurn(ds, parsed.content); await noteTurnReceived(ds, parsed.messageId, parsed.content, await getThreadSender(), parsed.messageId, substituteTrigger ? SUBSTITUTE_RECEIVED_REACTION_EMOJI_TYPE : undefined); rememberLastCliInput(ds, promptContent, cliInput); - sendWorkerInput(ds, cliInput, parsed.messageId); + if (!sendWorkerInput(ds, cliInput, parsed.messageId)) { + logger.warn(`[${tag(ds)}] Inbound ${parsed.messageId} was not accepted by the live worker`); + return; + } } else { // Worker not running — re-fork with resume. This is a NEW turn, so drop // any restored streaming-card reference; worker_ready will POST a fresh @@ -15813,31 +16837,124 @@ async function handleThreadReply(data: any, ctx: RoutingContext): Promise // bridge session whose worker died would gain a whiteboard binding (and a // block in its refork prompt) that its live turns never had. if (!ds.adoptedFrom) ensureSessionWhiteboard(ds); + const stageCurrentBehindQueuedActivation = async (): Promise => { + const tailReservation = reserveAsyncQueuedActivationTailAdmission(ds); + try { + const currentFollowUp = buildFollowUpCliInput(promptContent, ds.session.sessionId, { + attachments, + mentions: parsed.mentions, + isAdoptMode: false, + cliId: ds.session.cliId ?? dsBotCfgForFork.cliId, + cliPathOverride: ds.session.cliPathOverride ?? dsBotCfgForFork.cliPathOverride, + sender: await getThreadSender(), + larkAppId, + chatId: ds.session.chatId, + whiteboardId: ds.session.whiteboardId, + substituteTrigger, + codexAppText: parsed.content, + codexAppApplicationContext, + codexAppMessageContext, + }); + admitQueuedActivationTail(ds, { + userPrompt: promptContent, + cliInput: currentFollowUp, + turnId: parsed.messageId, + }, tailReservation); + } finally { + settleAsyncQueuedActivationTailAdmission(ds); + } + ds.initialStartPending = true; + await noteTurnReceived( + ds, + parsed.messageId, + parsed.content, + await getThreadSender(), + parsed.messageId, + substituteTrigger ? SUBSTITUTE_RECEIVED_REACTION_EMOJI_TYPE : undefined, + ); + }; + // Codex App may retain an ACK journal/FIFO head after its worker dies. A + // new inbound turn wakes a pure recovery fork first; it stays in the exact + // runtime tail until that recovered head produces its submission ACK. + if (ds.session.queuedActivationPending) { + await stageCurrentBehindQueuedActivation(); + try { + const recoverThroughCodexLedger = (ds.session.cliId ?? dsBotCfgForFork.cliId) === 'codex-app'; + forkWorker( + ds, + recoverThroughCodexLedger ? '' : (ds.session.queuedActivationInput ?? ''), + { + resume: ds.session.queuedActivationResume ?? ds.hasHistory, + turnId: ds.session.queuedActivationTurnId, + dispatchAttempt: ds.session.queuedActivationDispatchAttempt, + }, + ); + if (ownsInitialStartClaim()) retainInitialStartClaim = true; + } catch (err) { + ds.initialStartPending = false; + throw err; + } + return; + } + // A worker that died before submitting a queued activation leaves the + // exact opening payload parked in the durable journal. Wake that opening + // unchanged and stage this new inbound turn behind it as a separate FIFO + // item; rebuilding `queuedPrompt + current` here would lose any reply that + // had already been folded into the retained activation payload. + const retainedQueuedActivation = ds.session.queued === true + ? ds.session.queuedActivationInput + : undefined; + if (retainedQueuedActivation) { + await stageCurrentBehindQueuedActivation(); + try { + forkWorker(ds, retainedQueuedActivation, { + resume: ds.hasHistory, + turnId: ds.session.queuedActivationTurnId, + dispatchAttempt: ds.session.queuedActivationDispatchAttempt, + }); + if (ownsInitialStartClaim()) retainInitialStartClaim = true; + } catch (err) { + // Keep the staged tail for a later activation attempt, but release the + // route gate so another inbound turn can trigger that attempt. + ds.initialStartPending = false; + throw err; + } + return; + } // 待办池(queued)会话：CLI 从没起过，暂存的任务内容(queuedPrompt，已按角色包装好) // 必须当首轮发出去——否则群里来的这第一条消息会顶替掉它、把用户分配的任务丢掉。 // 把暂存任务前置、用户这条消息拼在后面，一并作为首轮。forkWorker 随后清 queued。 const queuedDashboardTurn = !!(ds.session.queued && ds.session.queuedPrompt); - const reforkContent = queuedDashboardTurn + // A restored pending-repo/backlog owner may already have durable N+1... + // behind its unopened N. Appending this inbound to N would overtake those + // persisted entries. Keep N exact and append the current turn after the + // existing tail before starting the activation journal. + const queuedHasDurableTail = queuedDashboardTurn + && (ds.session.queuedActivationTail?.length ?? 0) > 0; + if (queuedHasDurableTail) await stageCurrentBehindQueuedActivation(); + const reforkContent = queuedDashboardTurn && !queuedHasDurableTail ? `${ds.session.queuedPrompt}\n\n${promptContent}` - : promptContent; + : queuedHasDurableTail + ? ds.session.queuedPrompt! + : promptContent; const queuedCodexAppText = ds.session.queuedCodexAppText ?? ds.pendingCodexAppText; const reforkCodexApp = mergeQueuedCodexAppTurn({ queued: queuedDashboardTurn, queuedText: queuedCodexAppText, queuedMessageContext: ds.session.queuedCodexAppMessageContext ?? ds.pendingCodexAppMessageContext, - currentText: parsed.content, - currentMessageContext: codexAppMessageContext, + currentText: queuedHasDurableTail ? '' : parsed.content, + currentMessageContext: queuedHasDurableTail ? undefined : codexAppMessageContext, }); const builtReforkInput = buildReforkCliInput(ds, reforkContent, { - attachments, - mentions: parsed.mentions, + attachments: queuedHasDurableTail ? undefined : attachments, + mentions: queuedHasDurableTail ? undefined : parsed.mentions, cliId: ds.session.cliId ?? dsBotCfgForFork.cliId, cliPathOverride: ds.session.cliPathOverride ?? dsBotCfgForFork.cliPathOverride, selfMention: { name: selfBot.botName, openId: selfBot.botOpenId }, - sender: await getThreadSender(), - substituteTrigger, + sender: queuedHasDurableTail ? undefined : await getThreadSender(), + substituteTrigger: queuedHasDurableTail ? undefined : substituteTrigger, codexAppText: reforkCodexApp.text, - codexAppApplicationContext, + codexAppApplicationContext: queuedHasDurableTail ? undefined : codexAppApplicationContext, codexAppMessageContext: reforkCodexApp.messageContext, }); const wrappedInput = applyQueuedCodexAppLegacyFallback(builtReforkInput, { @@ -15851,13 +16968,35 @@ async function handleThreadReply(data: any, ctx: RoutingContext): Promise // contain the reply and would silently discard the original task. logger.warn(`[${tag(ds)}] Legacy queued dashboard task has no clean-input text; using the full legacy activation prompt`); } - await noteTurnReceived(ds, parsed.messageId, parsed.content, await getThreadSender(), parsed.messageId, substituteTrigger ? SUBSTITUTE_RECEIVED_REACTION_EMOJI_TYPE : undefined); - rememberLastCliInput(ds, promptContent, wrappedInput); + if (!queuedHasDurableTail) { + await noteTurnReceived(ds, parsed.messageId, parsed.content, await getThreadSender(), parsed.messageId, substituteTrigger ? SUBSTITUTE_RECEIVED_REACTION_EMOJI_TYPE : undefined); + } + rememberLastCliInput(ds, queuedHasDurableTail ? ds.session.queuedPrompt! : promptContent, wrappedInput); sessionStore.updateSession(ds.session); forkWorker(ds, wrappedInput, { resume: ds.hasHistory, - turnId: parsed.messageId, + turnId: queuedHasDurableTail + ? (ds.session.queuedActivationTurnId ?? `queued-opening:${ds.session.sessionId}`) + : parsed.messageId, }); + if (ownsInitialStartClaim()) { + if (initialStartClaimWaitsForQueuedAck) { + retainInitialStartClaim = true; + } else { + // Ordinary cold reforks have no adapter-submission ACK. The child owns + // the opening init after forkWorker returns, so hand any concurrently + // buffered followers to its IPC queue now in exact arrival order. + retainInitialStartClaim = !releaseQueuedActivationReservation(ds); + } + } + } + } finally { + // Pre-fork quota/resource/persistence failures and any non-queued route + // that returned without accepting a worker must not strand the route gate. + // Fence by token so an older handler can never clear a newer generation. + if (ownsInitialStartClaim() && !retainInitialStartClaim) { + clearInitialStartClaim(ds!, initialStartClaimToken); + } } } @@ -15885,7 +17024,11 @@ async function autoCreateDocSession(sub: DocSubscription, larkAppId: string, ctx const title = `[Doc] ${sub.fileToken.slice(0, 8)}: ${ctx.text.slice(0, 40)}`; const virtualChatId = `doc:${sub.fileToken}`; - const virtualAnchor = sub.sessionAnchor; + // A native doc session is chat-scoped, therefore its canonical routing + // anchor is its virtual chat id. Persist that same identity back into the + // subscription; using the historical IM anchor here made close/restore and + // comment lookup disagree about which Map key owned the session. + const virtualAnchor = virtualChatId; const now = Date.now(); const session = sessionStore.createSession(virtualChatId, virtualAnchor, title, 'group'); @@ -15939,7 +17082,21 @@ async function autoCreateDocSession(sub: DocSubscription, larkAppId: string, ctx (ds.session.docCommentTargets ??= {})[turnId] = docTarget; try { sessionStore.updateSession(ds.session); } catch { /* best-effort */ } - activeSessions.set(sessionKey(virtualAnchor, larkAppId), ds); + const key = activeSessionKey(ds); + if (key !== sessionKey(virtualAnchor, larkAppId)) { + throw new Error(`doc session canonical key mismatch: ${key}`); + } + if (activeSessions.has(key)) { + throw new Error(`doc session canonical key already owned: ${key}`); + } + activeSessions.set(key, ds); + Object.assign(sub, { + sessionAnchor: virtualAnchor, + sessionId: session.sessionId, + scope: 'chat' as const, + chatId: virtualChatId, + }); + putDocSubscription(config.session.dataDir, larkAppId, sub); // 不在这里 forkWorker —— handleDocComment 会统一处理（它会检查 worker 状态、 // 加 reaction、设 docCommentTurns、然后 fork 或 send）。这里只建好 session 骨架。 @@ -15959,30 +17116,136 @@ async function autoCreateDocSession(sub: DocSubscription, larkAppId: string, ctx * 走 resume 重 fork）；已 /close 的会话其订阅在关闭时已退订，这里查不到 ds 即跳过。 */ async function handleDocComment(ctx: DocCommentContext): Promise { - const { larkAppId, sub, commentId, text } = ctx; - const turnId = ctx.replyId || commentId; - const claimKey = `${larkAppId}:${sub.fileToken}:${turnId}`; + return withBotTurnAdmission( + ctx.larkAppId, + () => handleDocCommentAdmitted(ctx), + ); +} + +async function handleDocCommentAdmitted(ctx: DocCommentContext): Promise { + const turnId = ctx.replyId || ctx.commentId; + const claimKey = `${ctx.larkAppId}:${ctx.sub.fileToken}:${turnId}`; if (handledDocCommentTurns.has(claimKey)) { - logger.info(`[doc-comment] duplicate turn skipped file=${sub.fileToken.slice(0, 12)} turn=${turnId.slice(0, 12)}`); + logger.info(`[doc-comment] duplicate turn skipped file=${ctx.sub.fileToken.slice(0, 12)} turn=${turnId.slice(0, 12)}`); return true; // 已处理过，算成功（让 poller 推进游标） } - const loc = localeForBot(larkAppId); + const existing = docCommentTurnsInFlight.get(claimKey); + if (existing) { + logger.info(`[doc-comment] duplicate turn awaiting in-flight result file=${ctx.sub.fileToken.slice(0, 12)} turn=${turnId.slice(0, 12)}`); + return existing; + } - let ds: DaemonSession | undefined | null = activeSessions.get(sessionKey(sub.sessionAnchor, larkAppId)); - if (!ds) { - // 无活跃 session → 自动为该文档创建一个（用虚拟 anchor = doc:{fileToken}） - logger.info(`[doc-comment] no active session for anchor=${sub.sessionAnchor.slice(0, 12)}; auto-creating for file=${sub.fileToken.slice(0, 12)}`); - ds = await autoCreateDocSession(sub, larkAppId, ctx); - if (!ds) { - // auto-create 失败：不设 claim（允许后续重试），返回 false 让 poller 不推进游标 - logger.warn(`[doc-comment] auto-create session failed for file=${sub.fileToken.slice(0, 12)}; will retry comment ${commentId.slice(0, 12)}`); - return false; + // Defer the body by one microtask so the Promise is published before any + // async lookup can yield. WS and poll deliveries of the same comment then + // observe one authoritative success/failure result. + const delivery = Promise.resolve().then(async () => { + // Different comments for one document are also ordered through initial + // worker creation. Without this full-delivery lane, both can resolve the + // same worker:null owner, await reactions, and each fork a generation. + const deliveryKey = `\u0000doc-delivery:${ctx.larkAppId}:${ctx.sub.fileToken}`; + const delivered = await withActiveSessionKeyLock( + activeSessions, + deliveryKey, + () => deliverDocCommentTurn(ctx, claimKey), + ); + if (delivered) handledDocCommentTurns.set(claimKey, Date.now()); + return delivered; + }); + docCommentTurnsInFlight.set(claimKey, delivery); + try { + return await delivery; + } finally { + if (docCommentTurnsInFlight.get(claimKey) === delivery) { + docCommentTurnsInFlight.delete(claimKey); } } - // ds 确认有效后才设 claim——auto-create 失败时不占坑，允许重试。 - handledDocCommentTurns.set(claimKey, Date.now()); +} +async function deliverDocCommentTurn(ctx: DocCommentContext, claimKey: string): Promise { + const { larkAppId, commentId, text } = ctx; + let sub = ctx.sub; + const turnId = ctx.replyId || commentId; + const loc = localeForBot(larkAppId); + + // The document identity, not a possibly-stale subscription anchor, is the + // creation lock. Otherwise one caller can lock an old IM anchor while a + // second locks doc:, and both can create a native session. + const docKey = sessionKey(`doc:${sub.fileToken}`, larkAppId); try { + const resolved = await withActiveSessionKeyLock(activeSessions, docKey, async () => { + sub = getDocSubscription(config.session.dataDir, larkAppId, sub.fileToken) ?? sub; + let current = sub.sessionId + ? [...activeSessions.values()].find(candidate => + candidate.larkAppId === larkAppId + && candidate.session.sessionId === sub.sessionId) + : undefined; + current ??= activeSessions.get(sessionKey(sub.sessionAnchor, larkAppId)); + // A legacy native doc session may still be stored under the subscription's + // old IM anchor. Prefer an already-canonical owner if one exists, otherwise + // migrate the exact object synchronously while holding the doc key lock. + if (current?.scope === 'chat' && current.chatId === `doc:${sub.fileToken}`) { + current = activeSessions.get(docKey) ?? current; + if (activeSessions.get(docKey) !== current) { + for (const [registeredKey, candidate] of activeSessions) { + if (candidate === current) activeSessions.delete(registeredKey); + } + activeSessions.set(docKey, current); + } + } + current ??= activeSessions.get(docKey); + if (!current) { + logger.info(`[doc-comment] no active session for anchor=${sub.sessionAnchor.slice(0, 12)}; auto-creating for file=${sub.fileToken.slice(0, 12)}`); + current = await autoCreateDocSession(sub, larkAppId, ctx) ?? undefined; + if (!current) return { failed: true as const }; + } + + const canonicalAnchor = sessionAnchorId(current); + if (sub.sessionAnchor !== canonicalAnchor + || sub.sessionId !== current.session.sessionId + || sub.scope !== current.scope + || sub.chatId !== current.chatId) { + sub = { + ...sub, + sessionAnchor: canonicalAnchor, + sessionId: current.session.sessionId, + scope: current.scope, + chatId: current.chatId, + }; + putDocSubscription(config.session.dataDir, larkAppId, sub); + } + return { ds: current }; + }); + if ('failed' in resolved) { + logger.warn(`[doc-comment] auto-create session failed for file=${sub.fileToken.slice(0, 12)}; will retry comment ${commentId.slice(0, 12)}`); + return false; + } + const ds = resolved.ds; + + const retirementPhase = riffRetirementAdmissionPhase(ds); + if (retirementPhase) { + // Keep the provider cursor unchanged. Poll delivery is the durable retry + // owner for document comments, so the same exact comment is attempted + // again after admission is restored instead of being ACKed and dropped. + logger.warn( + `[doc-comment] turn ${turnId.slice(0, 12)} deferred before acceptance during ${retirementPhase}`, + ); + return false; + } + + if ((!ds.worker || ds.worker.killed) + && (ds.pendingRepo + || ds.worktreeCreating + || hasProtectedSessionMutationOwnership(ds))) { + // The provider cursor is the retry owner. A dormant setup/activation must + // not be replaced by a comment refork; returning false leaves this exact + // comment available after the opening owner reconciles. + logger.warn( + `[doc-comment] turn ${turnId.slice(0, 12)} deferred before acceptance ` + + `while session ${ds.session.sessionId.slice(0, 8)} has durable opening ownership`, + ); + return false; + } + // 给用户的回复加 "Typing" reaction，让评论者知道 bot 正在处理。 const userReplyId = ctx.replyId; let reactionId: string | undefined; @@ -16044,7 +17307,10 @@ async function handleDocComment(ctx: DocCommentContext): Promise { sessionStore.updateSession(ds.session); // 先落盘，botmux send 子进程才读得到落点 await noteTurnReceived(ds, commentId, text, sender, turnId); rememberLastCliInput(ds, promptContent, cliInput); - sendWorkerInput(ds, cliInput, turnId); + if (!sendWorkerInput(ds, cliInput, turnId)) { + logger.warn(`[${tag(ds)}] doc-comment turn was not accepted; provider retry retained`); + return false; + } logger.info(`[${tag(ds)}] doc-comment turn injected (turn ${turnId.slice(0, 8)})`); } else { // Worker 挂起 / 已退出 —— resume 重 fork（与 handleThreadReply 同路）。 @@ -16078,15 +17344,61 @@ async function handleDocComment(ctx: DocCommentContext): Promise { } return true; } catch (err) { - // 投递失败：清理 claim 允许重试，返回 false 让 poller 不推进游标。 - handledDocCommentTurns.delete(claimKey); - logger.warn(`[doc-comment] delivery failed, claim released for retry file=${sub.fileToken.slice(0, 12)} turn=${turnId.slice(0, 12)} err=${err instanceof Error ? err.message : String(err)}`); + // No completed claim exists yet. Return false to keep the poll cursor in + // place; every concurrent duplicate shares this same failure via the + // single-flight Promise and a later delivery can retry. + logger.warn(`[doc-comment] delivery failed, retry remains eligible file=${sub.fileToken.slice(0, 12)} turn=${turnId.slice(0, 12)} claim=${claimKey} err=${err instanceof Error ? err.message : String(err)}`); return false; } } /** 同一条评论可能同时被长连接通知和 --all 轮询看到；daemon 内统一去重。 */ const handledDocCommentTurns = new BoundedMap(5_000); +const docCommentTurnsInFlight = new Map>(); + +export const __testOnly_handleDocComment = (ctx: DocCommentContext): Promise => handleDocComment(ctx); +export function __testOnly_resetDocCommentClaims(): void { + handledDocCommentTurns.clear(); + docCommentTurnsInFlight.clear(); +} + +/** Preserve accepted/pending work when a normal group is converted to topic + * mode. The converted event is routed to a new thread key, so keeping the old + * chat owner does not block the new session; it only keeps the durable owner + * addressable for completion or explicit close. */ +function handleChatModeConverted(chatId: string, larkAppId: string): boolean { + const key = sessionKey(chatId, larkAppId); + const owner = activeSessions.get(key); + if (!owner) { + logger.info(`[chat-mode-converted] ${chatId.substring(0, 12)} evicted=false; no chat-scope owner`); + return false; + } + // Conversion preprocessing can run outside bot-turn admission. Never evict + // a Riff owner between shutdown's registry snapshots merely because its + // in-memory fence has not been installed yet. Converted turns route by their + // new thread key, so retaining the old chat key does not block them. + const frozenBackendType = owner.initConfig?.backendType ?? owner.session.backendType; + const ownsRiff = frozenBackendType === 'riff' + || owner.session.backendType === 'riff' + || String(owner.session.cliId ?? '') === 'riff' + || String(owner.initConfig?.cliId ?? '') === 'riff' + || owner.session.riffParentTaskId !== undefined; + const pending = ownsRiff + || hasProtectedSessionMutationOwnership(owner) + || owner.pendingRepo === true; + if (pending) { + logger.warn( + `[chat-mode-converted] ${chatId.substring(0, 12)} preserved pending owner ` + + `${owner.session.sessionId.substring(0, 8)}; converted turns route by thread key`, + ); + return false; + } + const evicted = activeSessions.get(key) === owner && activeSessions.delete(key); + logger.info(`[chat-mode-converted] ${chatId.substring(0, 12)} evicted=${evicted}; worker (if any) keeps running until /close`); + return evicted; +} + +export const __testOnly_handleChatModeConverted = handleChatModeConverted; let docCommentPollRunning = false; @@ -16419,6 +17731,10 @@ export async function startDaemon(botIndex?: number): Promise { // agent-facing, live-origin-gated endpoints. Internal control endpoints use // a separate daemon-to-daemon credential and never trust this port marker. process.env.BOTMUX_DAEMON_IPC_PORT = String(ipcPort); + const daemonProcessStartIdentity = readSupervisorProcessStartIdentity(process.pid); + if (!daemonProcessStartIdentity) { + throw new Error('cannot bind daemon descriptor to a process-start identity'); + } const desc: DaemonDescriptor = { larkAppId: cfg.larkAppId, botName: cfg.displayName ?? cfg.larkAppId, @@ -16426,6 +17742,7 @@ export async function startDaemon(botIndex?: number): Promise { botIndex: idx, ipcPort, pid: process.pid, + processStartIdentity: daemonProcessStartIdentity, startedAt: Date.now(), bootInstanceId: generateWorkflowDaemonBootInstanceId(), workflowIpcProtocol: 'v1', @@ -16488,29 +17805,71 @@ export async function startDaemon(botIndex?: number): Promise { // One cap implementation shared by event-driven checks (process start / idle // edge) and the 60s safety-net timer below. Each daemon owns exactly one // bot's activeSessions map, so the configured limit is per bot. + let liveSessionCapSweepPending = false; const enforceLiveSessionCap = (source: 'session_change' | 'periodic'): void => { - const maxLiveWorkers = getBot(cfg.larkAppId).config.maxLiveWorkers; - const suspended = sweepIdleWorkers(activeSessions, { maxLiveWorkers }); - if (suspended.length > 0) { - logger.info( - `[idle-worker-sweeper] suspended ${suspended.length} session(s) over per-bot cap ` - + `${maxLiveWorkers ?? DEFAULT_MAX_LIVE_WORKERS} source=${source}`, - ); - } + // Coalesce spawn/idle/timer bursts. The detached mutation evaluates the + // current map and current config only after every admitted turn has drained, + // so one pending sweep subsumes all triggers that arrive before it runs. + if (liveSessionCapSweepPending) return; + liveSessionCapSweepPending = true; + void (async () => { + try { + const maxLiveWorkers = getBot(cfg.larkAppId).config.maxLiveWorkers; + const suspended = await sweepIdleWorkersAfterTurnDrain( + cfg.larkAppId, + activeSessions, + { maxLiveWorkers }, + ); + if (suspended.length > 0) { + logger.info( + `[idle-worker-sweeper] suspended ${suspended.length} session(s) over per-bot cap ` + + `${maxLiveWorkers ?? DEFAULT_MAX_LIVE_WORKERS} source=${source}`, + ); + } + } catch (err) { + logger.warn(`[idle-worker-sweeper] cap enforcement failed: ${err instanceof Error ? err.message : String(err)}`); + } finally { + liveSessionCapSweepPending = false; + } + })(); }; // Initialise worker pool with daemon callbacks initWorkerPool({ sessionReply, getSessionWorkingDir, getActiveCount, - closeSession(ds: DaemonSession) { + closeSession(ds: DaemonSession): Promise { // Route through the dashboard-aware helper so session.exited / session.update // events fire for withdrawn-message / crash / adopt-exit teardown paths too, // matching the dashboard-driven close. - void closeSessionHelper(ds.session.sessionId).catch(() => { /* idempotent */ }); - logger.info(`[${ds.session.sessionId.substring(0, 8)}] Session auto-closed (message withdrawn)`); + const sessionId = ds.session.sessionId; + return runDetachedBotTurnMutation(ds.larkAppId, async () => { + const current = findActiveBySessionId(sessionId); + if (current !== ds) return false; + if (hasProtectedSessionMutationOwnership(current)) { + logger.info(`[${sessionId.substring(0, 8)}] Withdraw auto-close deferred: Codex App FIFO became unsettled`); + return false; + } + const result = await closeSessionHelper(sessionId); + if (!result.ok) { + logger.warn( + `[${sessionId.substring(0, 8)}] Withdraw auto-close failed (${result.error}); ` + + `session retained${result.taskId ? ` task=${result.taskId}` : ''}`, + ); + return false; + } + logger.info(`[${sessionId.substring(0, 8)}] Session auto-closed (message withdrawn)`); + return true; + }).catch((err) => { + logger.warn( + `[${sessionId.substring(0, 8)}] Withdraw auto-close mutation failed; session retained: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + return false; + }); }, enforceLiveSessionCap: () => enforceLiveSessionCap('session_change'), + onQueuedActivationSubmitted, onTurnTerminal(ds, terminal, context) { const enqueued = vcMeetingTerminalReconciler?.enqueue(terminal, context); if (terminal.dispatchAttempt !== undefined && enqueued?.accepted) { @@ -16566,6 +17925,11 @@ export async function startDaemon(botIndex?: number): Promise { onDurableExpiryReady(_ds, context) { vcMeetingRuntimeLeaseRecovery.acknowledge(context); }, + async onCodexAppLedgerDrained(ds) { + await withBotTurnMutation(ds.larkAppId, async () => { + await closeCliMismatchedSessionsForBot(ds.larkAppId); + }); + }, }); // Expose the activeSessions Map (owned by daemon) to worker-pool readers, // so dashboard IPC and other consumers can list/lookup live sessions. @@ -16595,10 +17959,13 @@ export async function startDaemon(botIndex?: number): Promise { loadOrCreateDashboardSecret( join(homedir(), '.botmux', '.dashboard-secret'), ); + let markIpcReady!: () => void; + const ipcReady = new Promise((resolve) => { markIpcReady = resolve; }); const ipcHandle = await startIpcServer({ port: ipcPort, host: '127.0.0.1', authRequired: true, + ready: ipcReady, }); // startIpcServer probes upward on EADDRINUSE (e.g. a second botmux instance on // this host already holds ipcBasePort+idx), so the bound port may differ from @@ -16608,6 +17975,20 @@ export async function startDaemon(botIndex?: number): Promise { process.env.BOTMUX_DAEMON_IPC_PORT = String(ipcHandle.port); logger.info(`[dashboard-ipc] listening on 127.0.0.1:${ipcHandle.port} (bot ${idx})`); + // Publish daemon ownership immediately after IPC binds, then perform the + // first session-file load under SessionStore's cross-process lock. An + // offline CLI either observes this descriptor and delegates, or it already + // holds the file lock; in the latter case this load waits and sees its atomic + // mutation. Never load a stale cache in an unadvertised startup window. + desc.lastHeartbeat = Date.now(); + writeDaemonDescriptor(desc); + sessionStore.listSessions(); + const descriptorHeartbeat = setInterval(() => { + desc.lastHeartbeat = Date.now(); + try { writeDaemonDescriptor(desc); } catch { /* best effort */ } + }, 30_000); + if (typeof descriptorHeartbeat.unref === 'function') descriptorHeartbeat.unref(); + // Single reverse-proxy port that fronts every session's web terminal under // /s/{sessionId}, so dev-machine users forward one port (proxyBasePort+idx) // instead of one per topic. Bound on the public host so `ssh -L` can reach it. @@ -16626,10 +18007,25 @@ export async function startDaemon(botIndex?: number): Promise { // Quiet-restart leaves sessions registered but worker-less until messaged. // Wake the worker on terminal access so links open without a manual ping. ensureWorkerPort: async (sessionId) => { + let owner: DaemonSession | undefined; for (const ds of activeSessions.values()) { - if (ds.session.sessionId === sessionId) return ensureTerminalWorkerPort(ds); + if (ds.session.sessionId === sessionId) { + owner = ds; + break; + } } - return undefined; + if (!owner) return undefined; + return withBotTurnAdmission(owner.larkAppId, () => { + // The gate may have held this request behind a mutation/close. Resolve + // the canonical record again before a lazy fork; never wake a stale ds + // captured before the wait. + for (const current of activeSessions.values()) { + if (current.session.sessionId === sessionId) { + return ensureTerminalWorkerPort(current); + } + } + return undefined; + }); }, }); // Only mark the proxy live after a successful bind — buildTerminalUrl then @@ -16650,17 +18046,6 @@ export async function startDaemon(botIndex?: number): Promise { logger.info(`[terminal-proxy] terminal links advertise external port ${config.web.externalPort + idx} (WEB_EXTERNAL_PORT ${config.web.externalPort} + bot ${idx})`); } - // Now that the IPC port is actually listening, publish the descriptor so - // the dashboard can discover us and successfully fetch /api/sessions etc. - desc.lastHeartbeat = Date.now(); - writeDaemonDescriptor(desc); - const descriptorHeartbeat = setInterval(() => { - desc.lastHeartbeat = Date.now(); - try { writeDaemonDescriptor(desc); } catch { /* best effort */ } - }, 30_000); - // Don't keep the event loop alive on this interval alone. - if (typeof descriptorHeartbeat.unref === 'function') descriptorHeartbeat.unref(); - // Reap a dead prior daemon's detached classifier before any prepared // proposal can be recovered. Per-proposal generation locks still serialize // live rolling-restart overlap; this sweep handles only dead-owner markers. @@ -16681,6 +18066,11 @@ export async function startDaemon(botIndex?: number): Promise { ); } + // Do not start any Lark dispatcher until durable sessions have been restored + // into the routing registry. Otherwise an inbound same-anchor turn can create + // a second owner while the old unsettled row exists only on disk. + const startEventDispatchers: Array<() => void> = []; + // Per-bot initialization for (const bot of getAllBots()) { const cfg = bot.config; @@ -16764,34 +18154,44 @@ export async function startDaemon(botIndex?: number): Promise { ); } - // Start event dispatcher for this bot + // Build the dispatcher now for authorization replay, but start it only + // after restore has published every durable route owner. const botEventHandlers: EventHandlers = { - handleCardAction: (data, appId) => handleCardAction(data, cardDeps, appId), + handleCardAction: (data, appId) => withBotTurnAdmission( + appId, + () => handleCardAction(data, cardDeps, appId), + ), handleNewTopic: (data, ctx) => handleNewTopic(data, ctx), handleThreadReply: (data, ctx) => handleThreadReply(data, ctx), - handleBotAdded: (chatId, operatorOpenId, appId) => handleBotAdded(chatId, operatorOpenId, appId), + handleBotAdded: (chatId, operatorOpenId, appId) => withBotTurnAdmission( + appId, + () => handleBotAdded(chatId, operatorOpenId, appId), + ), handleDocComment: (ctx) => handleDocComment(ctx), - handleVcMeetingPush: (ctx) => handleVcMeetingPush(ctx), + handleVcMeetingPush: (ctx) => withBotTurnAdmission( + ctx.larkAppId, + () => handleVcMeetingPush(ctx), + ), beforeSessionTurn: (data, ctx) => maybeCatchUpVcMeetingConsumerBeforeTurn(data, ctx), isSessionOwner: (anchor, appId) => activeSessions.has(sessionKey(anchor, appId)), resolveReplyThreadAlias: (rootId, chatId, appId) => findChatReplyAlias(rootId, chatId, appId), // Chat was converted 普通群 → 话题群 while we held a chat-scope session. - // Evict it from the routing map so subsequent inbound messages can land - // on a fresh thread-scope session (dispatcher already rerouted this turn - // to handleNewTopic). The worker is left running on purpose: the user may - // still have its web terminal open, and `/close` is the canonical cleanup - // path. Scheduler tasks tied to this session keep their `scope='chat'` - // semantics — that's an edge case worth following up on, not blocking - // the main fix. + // Idle legacy owners are evicted so subsequent inbound messages land on + // fresh thread-scope sessions. Owners with accepted/pending work remain + // addressable; the dispatcher already reroutes converted turns by their + // new thread root, so preserving that old chat key cannot steal them. onChatModeConverted: (chatId, appId) => { - const key = sessionKey(chatId, appId); - const evicted = activeSessions.delete(key); - logger.info(`[chat-mode-converted] ${chatId.substring(0, 12)} evicted=${evicted}; worker (if any) keeps running until /close`); + handleChatModeConverted(chatId, appId); }, }; // 存起来供授权成功后重放消息用（replayGrantedMessage → replayMessageEvent）。 botHandlers.set(cfg.larkAppId, botEventHandlers); - startLarkEventDispatcher(cfg.larkAppId, cfg.larkAppSecret, botEventHandlers, normalizeBrand(cfg.brand)); + startEventDispatchers.push(() => startLarkEventDispatcher( + cfg.larkAppId, + cfg.larkAppSecret, + botEventHandlers, + normalizeBrand(cfg.brand), + )); // A distillation command is durably prepared before its model run/card // delivery. Resume active prepared/proposed allocations after a daemon @@ -16824,6 +18224,12 @@ export async function startDaemon(botIndex?: number): Promise { for (const bot of getAllBots()) { markForwardFollowupsSessionsReady(bot.config.larkAppId); } + // The descriptor was intentionally published before restore so offline CLI + // mutations delegate to this daemon. Release those queued IPC calls only + // after every durable owner is visible in the canonical registry. + markIpcReady(); + + for (const startDispatcher of startEventDispatchers) startDispatcher(); try { await reconcileVcMeetingManagedActionsOnBoot(cfg.larkAppId); @@ -16864,7 +18270,11 @@ export async function startDaemon(botIndex?: number): Promise { || receiver.memberEpoch !== ref.memberEpoch) { // Corrupt identity must not restart an unrelated session or silently // un-gate delivery. Keep the daemon fail-closed for operator repair. - addVcMeetingReceiverRecoveryPending(recoveryKey, vcMeetingDeliveryScopeFromRef(ref)); + addVcMeetingReceiverRecoveryPending(recoveryKey, { + ...vcMeetingDeliveryScopeFromRef(ref), + turnId: ref.deliveryKey, + dispatchAttempt: ref.dispatchAttempt, + }); logger.error( `[vc-delivery] boot receiver identity conflict; delivery remains gated ` + `session=${ref.receiverSessionId}`, @@ -16877,7 +18287,11 @@ export async function startDaemon(botIndex?: number): Promise { vcMeetingRuntimeLeaseRecovery.arm(ref, cfg.larkAppId); continue; } - addVcMeetingReceiverRecoveryPending(recoveryKey, vcMeetingDeliveryScopeFromRef(ref)); + addVcMeetingReceiverRecoveryPending(recoveryKey, { + ...vcMeetingDeliveryScopeFromRef(ref), + turnId: ref.deliveryKey, + dispatchAttempt: ref.dispatchAttempt, + }); armVcMeetingReceiverRecoveryTimeout(recoveryKey, ref.receiverSessionId); try { ds.worker.send({ @@ -16965,7 +18379,10 @@ export async function startDaemon(botIndex?: number): Promise { // each filters to only execute tasks whose `larkAppId` matches its bot // (unmatched tasks are handled by the owning bot's daemon instead; a // missing larkAppId falls through to bot-0 as a legacy fallback). - scheduler.setExecuteCallback((task) => executeScheduledTask(task, activeSessions, refreshCliVersion)); + scheduler.setExecuteCallback((task) => withBotTurnAdmission( + task.larkAppId ?? cfg.larkAppId, + () => executeScheduledTask(task, activeSessions, refreshCliVersion), + )); scheduler.setOwnerFilter(cfg.larkAppId, idx === 0); scheduler.startScheduler(); @@ -16976,7 +18393,7 @@ export async function startDaemon(botIndex?: number): Promise { try { let busy = 0; for (const [, ds] of activeSessions) { - if (ds.worker && !ds.worker.killed && ds.lastScreenStatus === 'working') busy++; + if (ds.worker && !ds.worker.killed && isMaintenanceBusyStatus(ds.lastScreenStatus)) busy++; } writeHeartbeat(cfg.larkAppId, busy); } catch { /* best-effort */ } @@ -17013,21 +18430,161 @@ export async function startDaemon(botIndex?: number): Promise { }, 5_000).unref?.(); } - // Graceful shutdown. Sends SIGTERM (or `{type:'close'}` IPC via killWorker) - // to every worker, then waits up to SHUTDOWN_GRACE_MS for them to exit + // Graceful shutdown. Riff owners first run a three-phase non-cancelling drain + // that durably ACKs every exact final task lineage as one fleet transaction. + // No worker exits until ALL participants are prepared and fresh-verified. + // Only after every Riff + // owner is safe do ordinary workers receive SIGTERM / close IPC. Then wait + // up to DAEMON_WORKER_EXIT_GRACE_MS for process exit // before sending SIGKILL to stragglers. Without the wait, daemon - // `process.exit(0)` races worker signal delivery — and any worker whose + // `process.exit(DAEMON_GRACEFUL_EXIT_CODE)` races worker signal delivery — and any worker whose // main thread is in a sync code path (e.g. the bridge fingerprint scan // bug fixed in v2.9.2) loses the signal and survives as a ppid=1 orphan // forever (we'd accumulated 841 such orphans across daemon restarts, // consuming ~65 GB of RAM until manually SIGKILL'd). - const SHUTDOWN_GRACE_MS = 3000; let shuttingDown = false; const shutdown = async () => { if (shuttingDown) return; shuttingDown = true; + const shutdownDeadlineMs = Date.now() + DAEMON_SHUTDOWN_MAX_MS; + const mutationResult = await tryWithBotTurnMutation( + cfg.larkAppId, + BOT_TURN_MUTATION_SHUTDOWN_ACQUIRE_TIMEOUT_MS, + async () => { setSessionLifecycleShutdown(true); logger.info(`Daemon shutting down... (active: ${getActiveCount()})`); + + const initialShutdownFleet = collectUniqueDaemonShutdownSessions(activeSessions.values()); + if (!initialShutdownFleet.ok) { + setSessionLifecycleShutdown(false); + shuttingDown = false; + logger.error(`Daemon remains online: ${initialShutdownFleet.error}`); + return; + } + + // Preflight Riff BEFORE stopping services/removing descriptors. A failed + // drain or persistence write aborts every successfully prepared peer, so a + // refused shutdown returns to a fully live fleet rather than leaving early + // sessions workerless. Workerless active Riff rows participate too: their + // runtime lineage may be newer than disk after an earlier save failure. + const riffCandidates = initialShutdownFleet.sessions + .filter(ds => shutdownBackendDisposition(ds) === 'riff-drain-detach'); + const riffPrepareResults = await prepareRiffFleetForShutdown(riffCandidates, { + deadlineMs: shutdownDeadlineMs, + }); + const riffPrepared = riffPrepareResults.filter( + (entry): entry is { ds: DaemonSession; result: PreparedRiffShutdown } => entry.result.ok, + ); + const riffFenced = riffPrepareResults.filter( + (entry): entry is { ds: DaemonSession; result: FencedRiffShutdownParticipant } => + entry.result.fence !== 'none', + ); + + const abortRiffFleet = async ( + reason: string, + retainFenced: ReadonlySet = new Set(), + ): Promise => { + for (const ds of retainFenced) { + logger.error( + `[${ds.session.sessionId.slice(0, 8)}] Riff shutdown participant remains fenced: ` + + 'durable/runtime ownership could not be reconciled safely', + ); + } + const aborts = await abortRiffShutdownFleet(riffFenced + .filter(({ ds }) => !retainFenced.has(ds)) + .map(({ ds, result }) => ({ ds, result })), { + deadlineMs: shutdownDeadlineMs, + }); + for (const { ds, result } of aborts) { + if (result.ok) continue; + logger.error( + `[${ds.session.sessionId.slice(0, 8)}] Riff shutdown rollback was not ACKed: ` + + `${result.error ?? 'unknown'}; admission remains fail-closed`, + ); + } + setSessionLifecycleShutdown(false); + shuttingDown = false; + logger.error(`Daemon remains online: ${reason}`); + }; + + const riffPrepareFailures = riffPrepareResults.filter(entry => !entry.result.ok); + if (riffPrepareFailures.length > 0) { + for (const { ds, result } of riffPrepareFailures) { + if (result.ok) continue; + logger.error( + `[${ds.session.sessionId.slice(0, 8)}] Daemon shutdown prepare refused: ` + + `${result.error}${result.taskId ? ` (task ${result.taskId})` : ''}`, + ); + } + await abortRiffFleet( + `${riffPrepareFailures.length} Riff owner(s) could not be safely drained`, + ); + return; + } + + // Phase 2 is one all-or-none lock/CAS/rename/readback transaction. A + // pre-write ownership conflict retains only affected fences; ordinary + // pre-write I/O restores all peers; post-rename ambiguity retains all. + const riffPersistence = persistPreparedRiffShutdownFleet(riffPrepared, { + deadlineMs: shutdownDeadlineMs, + }); + if (!riffPersistence.ok) { + const retainIds = new Set(riffPersistence.retainFencedSessionIds); + const retainFenced = new Set(riffPrepared + .filter(({ ds }) => retainIds.has(ds.session.sessionId)) + .map(({ ds }) => ds)); + logger.error(`Daemon shutdown lineage verification refused: ${riffPersistence.error}`); + await abortRiffFleet( + `${riffPersistence.sessionIds.length} Riff lineage row(s) could not be fresh-verified`, + retainFenced, + ); + return; + } + + // Second generation/snapshot check while every participant is still fenced + // and every daemon service remains live. This catches a future ungated fork, + // route replacement, or disappeared generation before any peer is committed. + const currentShutdownFleet = collectUniqueDaemonShutdownSessions(activeSessions.values()); + if (!currentShutdownFleet.ok) { + const ambiguousSessionId = currentShutdownFleet.sessionId; + await abortRiffFleet( + `Riff ownership became ambiguous after shutdown preflight: ${currentShutdownFleet.error}`, + new Set(riffPrepared + .filter(({ ds }) => ds.session.sessionId === ambiguousSessionId) + .map(({ ds }) => ds)), + ); + return; + } + const currentRiffOwners = currentShutdownFleet.sessions + .filter(ds => shutdownBackendDisposition(ds) === 'riff-drain-detach'); + const riffPreparedOwners = new Set(riffPrepared.map(entry => entry.ds)); + const riffGenerationMismatch = currentRiffOwners.some(ds => !riffPreparedOwners.has(ds)) + || riffPrepared.some(({ ds, result }) => + !currentRiffOwners.includes(ds) || !isPreparedRiffSessionCurrent(ds, result)); + if (riffGenerationMismatch) { + const retainFenced = new Set(riffPrepared + .filter(({ ds, result }) => + (!currentRiffOwners.includes(ds) || !isPreparedRiffSessionCurrent(ds, result)) + && !canAbortVerifiedExitedRiffPreparation(ds, result)) + .map(({ ds }) => ds)); + await abortRiffFleet( + 'Riff ownership changed after shutdown preflight', + retainFenced, + ); + return; + } + + // Validate-all above, then commit-all synchronously with no await or callback + // boundary between peers. commit cannot fail after that validation: IPC send + // failure falls back to a safe direct signal because lineage is already exact. + const riffRetiredWorkers: ChildProcess[] = []; + for (const { ds, result } of riffPrepared) { + if (!commitPreparedRiffShutdown(ds, result)) { + throw new Error(`Riff shutdown commit invariant lost for ${ds.session.sessionId}`); + } + if (result.worker) riffRetiredWorkers.push(result.worker); + } + scheduler.stopScheduler(); stopMaintenance(); vcMeetingTerminalReconciler?.stop(); @@ -17061,23 +18618,38 @@ export async function startDaemon(botIndex?: number): Promise { const pendingExits: Array> = []; const survivors: ChildProcess[] = []; - for (const [, ds] of activeSessions) { + const trackWorkerExit = (w: ChildProcess): void => { + if (w.exitCode !== null || w.signalCode !== null) return; + pendingExits.push(new Promise(resolve => { + const onExit = () => resolve(); + w.once('exit', onExit); + // Close the tiny check/listener race: if exit landed between the first + // status read and once(), resolve immediately (the later once callback + // is harmless because Promise resolution is idempotent). + if (w.exitCode !== null || w.signalCode !== null) resolve(); + })); + survivors.push(w); + }; + for (const worker of riffRetiredWorkers) trackWorkerExit(worker); + for (const ds of currentShutdownFleet.sessions) { if (ds.worker && !ds.worker.killed) { logger.info(`Shutting down worker for session ${ds.session.sessionId}`); const w = ds.worker; - // Capture the exit promise BEFORE killWorker nulls ds.worker. - if (w.exitCode === null && w.signalCode === null) { - pendingExits.push(new Promise(resolve => { - w.once('exit', () => resolve()); - })); - survivors.push(w); + const disposition = shutdownBackendDisposition(ds); + if (disposition === 'riff-drain-detach') { + // No await occurs between the generation check, commit-all, service + // teardown, and this loop. Reaching this branch is an internal invariant + // violation, never a recoverable half-torn-down refusal. + throw new Error(`undrained Riff generation after atomic commit: ${ds.session.sessionId}`); } + // Capture the exit promise BEFORE killWorker nulls ds.worker. + trackWorkerExit(w); // Branch by the session's FROZEN backend (stamped on Session.backendType // at spawn), NOT the bot's live config — a dashboard backendType edit must // not change how a running session is torn down, or we'd e.g. try to // detach-preserve a "herdr" session whose real pane is tmux (freeze-once). // undefined (frozen pty, or unresolvable legacy) → non-persistent → killWorker. - if (shutdownBackendDisposition(ds) === 'detach') { + if (disposition === 'detach') { // Persistent backends (tmux / herdr / zellij): just kill the worker process — // the multiplexer session survives for re-attach. The worker's SIGTERM // handler calls backend.kill(), which only DETACHES. Going through @@ -17097,7 +18669,11 @@ export async function startDaemon(botIndex?: number): Promise { } if (pendingExits.length > 0) { - const timeout = new Promise(resolve => setTimeout(resolve, SHUTDOWN_GRACE_MS)); + const exitGraceMs = Math.min( + DAEMON_WORKER_EXIT_GRACE_MS, + Math.max(0, shutdownDeadlineMs - Date.now()), + ); + const timeout = new Promise(resolve => setTimeout(resolve, exitGraceMs)); await Promise.race([Promise.all(pendingExits), timeout]); let stragglers = 0; for (const w of survivors) { @@ -17107,7 +18683,7 @@ export async function startDaemon(botIndex?: number): Promise { } } if (stragglers > 0) { - logger.warn(`${stragglers}/${survivors.length} worker(s) didn't exit within ${SHUTDOWN_GRACE_MS}ms — SIGKILL'd to prevent ppid=1 orphans.`); + logger.warn(`${stragglers}/${survivors.length} worker(s) didn't exit within ${exitGraceMs}ms — SIGKILL'd to prevent ppid=1 orphans.`); } } @@ -17117,15 +18693,44 @@ export async function startDaemon(botIndex?: number): Promise { flushIdentityCacheSync(); removePidFile(); - process.exit(0); + // This reserved non-zero code is the only PM2 stop_exit_codes proof that + // the authenticated prepare/persist/commit shutdown reached its end. + // PM2 maps abrupt signal death to 0, which must remain autorestartable. + process.exit(DAEMON_GRACEFUL_EXIT_CODE); + }, + ); + if (!mutationResult.acquired) { + shuttingDown = false; + logger.error( + `Daemon remains online: could not acquire exclusive shutdown mutation lease ` + + `(${mutationResult.reason})`, + ); + } }; process.on('SIGTERM', () => { shutdown().catch(err => { logger.error(`shutdown failed: ${err?.message ?? err}`); process.exit(1); }); }); process.on('SIGINT', () => { shutdown().catch(err => { logger.error(`shutdown failed: ${err?.message ?? err}`); process.exit(1); }); }); + // Capability publication is the final startup commit for supervisor-driven + // shutdown. The early descriptor intentionally lacks it: a new CLI that + // observes this daemon before both handlers/state closures exist must refuse + // to signal. Atomic rewrite makes the capability visible only afterward. + if (readSupervisorProcessStartIdentity(process.pid) !== desc.processStartIdentity) { + throw new Error('daemon process-start identity changed before shutdown capability commit'); + } + setSupervisorShutdownHandler({ + larkAppId: cfg.larkAppId, + bootInstanceId: desc.bootInstanceId, + processStartIdentity: desc.processStartIdentity, + shutdown, + }); + desc.supervisorShutdownProtocol = SUPERVISOR_SHUTDOWN_PROTOCOL; + desc.lastHeartbeat = Date.now(); + writeDaemonDescriptor(desc); // Best-effort cleanup on plain `exit` (e.g. uncaught fatal). No worker // shutdown here since the process is already on its way out — just remove // the descriptor so the dashboard doesn't see a phantom daemon. process.on('exit', () => { + setSupervisorShutdownHandler(null); clearInterval(descriptorHeartbeat); clearInterval(idleWorkerSweepTimer); clearInterval(docCommentPollTimer); diff --git a/src/dashboard/session-card-model.ts b/src/dashboard/session-card-model.ts index 54115593d..9530baa2e 100644 --- a/src/dashboard/session-card-model.ts +++ b/src/dashboard/session-card-model.ts @@ -21,6 +21,7 @@ export type SessionStatus = | 'working' | 'idle' | 'analyzing' + | 'stalled' | 'limited' | 'starting' | 'dormant' @@ -83,6 +84,8 @@ export function statusToDot(status: string): StatusDot { return { tone: 'success', pulse: true, label: 'sessions.status.working' }; case 'analyzing': return { tone: 'info', pulse: true, label: 'sessions.status.analyzing' }; + case 'stalled': + return { tone: 'danger', pulse: false, label: 'sessions.status.stalled' }; case 'starting': return { tone: 'info', pulse: true, label: 'sessions.status.starting' }; case 'idle': @@ -183,11 +186,12 @@ export function filterByCli(entries: ReadonlyArray, cliId: string const STATUS_ORDER: Record = { working: 0, analyzing: 1, - starting: 2, - idle: 3, - dormant: 4, - limited: 5, - closed: 6, + stalled: 2, + starting: 3, + idle: 4, + dormant: 5, + limited: 6, + closed: 7, }; function statusRank(status: string): number { diff --git a/src/dashboard/web/i18n.ts b/src/dashboard/web/i18n.ts index 1fb47de78..56290549c 100644 --- a/src/dashboard/web/i18n.ts +++ b/src/dashboard/web/i18n.ts @@ -289,6 +289,7 @@ const zh: DashboardMessages = { 'sessions.status.idle': '空闲', 'sessions.status.dormant': '休眠', 'sessions.status.analyzing': '分析中', + 'sessions.status.stalled': '长时间无进展', 'sessions.status.active': '活动中', 'sessions.status.limited': '限额已达', 'sessions.status.closed': '已关闭', @@ -962,7 +963,7 @@ const zh: DashboardMessages = { 'sessions.topic.empty': '暂无匹配的话题会话', 'sessions.selectSession': '选择会话', 'sessions.board.needsYou': '需要你', - 'sessions.board.needsYouHint': '等待仓库、TUI 选择或额度处理', + 'sessions.board.needsYouHint': '等待仓库、TUI 选择、额度处理或无进展检查', 'sessions.board.starting': '启动中', 'sessions.board.startingHint': 'CLI 正在创建或恢复', 'sessions.board.working': '干活中', @@ -973,6 +974,7 @@ const zh: DashboardMessages = { 'sessions.board.signalRepo': '待选仓库', 'sessions.board.signalPrompt': '等待 TUI 选择', 'sessions.board.signalLimited': '额度受限', + 'sessions.board.signalStalled': '长时间无进展', 'sessions.board.signalAgent': '需要人工介入', 'sessions.board.waiting': '等待', 'sessions.board.dragHint': '拖动列头调整列顺序', @@ -2198,6 +2200,7 @@ const en: DashboardMessages = { 'sessions.status.idle': 'Idle', 'sessions.status.dormant': 'Dormant', 'sessions.status.analyzing': 'Analyzing', + 'sessions.status.stalled': 'No recent progress', 'sessions.status.active': 'Active', 'sessions.status.limited': 'Limit reached', 'sessions.status.closed': 'Closed', @@ -2871,7 +2874,7 @@ const en: DashboardMessages = { 'sessions.topic.empty': 'No matching topic sessions', 'sessions.selectSession': 'Select session', 'sessions.board.needsYou': 'Needs You', - 'sessions.board.needsYouHint': 'Repo, TUI, or usage limit waiting', + 'sessions.board.needsYouHint': 'Repo, TUI, usage limit, or no-progress attention', 'sessions.board.starting': 'Starting', 'sessions.board.startingHint': 'CLI is spawning or resuming', 'sessions.board.working': 'Working', @@ -2882,6 +2885,7 @@ const en: DashboardMessages = { 'sessions.board.signalRepo': 'Repo needed', 'sessions.board.signalPrompt': 'TUI choice needed', 'sessions.board.signalLimited': 'Usage limited', + 'sessions.board.signalStalled': 'No recent progress', 'sessions.board.signalAgent': 'Needs human input', 'sessions.board.waiting': 'Waiting', 'sessions.board.dragHint': 'Drag header to reorder columns', diff --git a/src/dashboard/web/kanban-model.ts b/src/dashboard/web/kanban-model.ts index 3cf616f10..846f94366 100644 --- a/src/dashboard/web/kanban-model.ts +++ b/src/dashboard/web/kanban-model.ts @@ -25,7 +25,7 @@ export function deriveKanbanColumn(s: KanbanRowLike): SessionKanbanColumn { if (s.status === 'closed') return 'done'; const manual = normalizeKanbanColumn(s.kanbanColumn); if (manual) return manual; - if (s.pendingRepo || s.tuiPromptActive || s.agentAttention || s.status === 'limited') return 'in_review'; + if (s.pendingRepo || s.tuiPromptActive || s.agentAttention || s.status === 'limited' || s.status === 'stalled') return 'in_review'; if (s.status === 'starting' || s.status === 'working' || s.status === 'analyzing' || s.status === 'active') { return 'in_progress'; } diff --git a/src/dashboard/web/sessions-kanban.tsx b/src/dashboard/web/sessions-kanban.tsx index 10b3c6b41..65e2d63bf 100644 --- a/src/dashboard/web/sessions-kanban.tsx +++ b/src/dashboard/web/sessions-kanban.tsx @@ -143,6 +143,7 @@ function boardSignalLabel(s: any): string { if (s.pendingRepo) return t('sessions.board.signalRepo'); if (s.tuiPromptActive) return t('sessions.board.signalPrompt'); if (s.status === 'limited') return t('sessions.board.signalLimited'); + if (s.status === 'stalled') return t('sessions.board.signalStalled'); return ''; } diff --git a/src/dashboard/web/sessions-page.tsx b/src/dashboard/web/sessions-page.tsx index 2fc43331e..a3b05f5af 100644 --- a/src/dashboard/web/sessions-page.tsx +++ b/src/dashboard/web/sessions-page.tsx @@ -334,6 +334,7 @@ function boardSignalLabel(s: any): string { if (s.pendingRepo) return t('sessions.board.signalRepo'); if (s.tuiPromptActive) return t('sessions.board.signalPrompt'); if (s.status === 'limited') return t('sessions.board.signalLimited'); + if (s.status === 'stalled') return t('sessions.board.signalStalled'); return ''; } diff --git a/src/dashboard/web/sessions.ts b/src/dashboard/web/sessions.ts index 80a5f81b7..27a33ff2b 100644 --- a/src/dashboard/web/sessions.ts +++ b/src/dashboard/web/sessions.ts @@ -40,6 +40,7 @@ export const SESSION_STATUS_OPTIONS = [ 'idle', 'dormant', 'analyzing', + 'stalled', 'active', 'limited', 'closed', @@ -279,7 +280,7 @@ export function historySenderKey(message: any): string { export function deriveSessionBoardColumn(s: any): BoardColumnId | null { if (s.status === 'closed') return null; - if (s.pendingRepo || s.tuiPromptActive || s.agentAttention || s.status === 'limited') return 'needs-you'; + if (s.pendingRepo || s.tuiPromptActive || s.agentAttention || s.status === 'limited' || s.status === 'stalled') return 'needs-you'; if (s.status === 'starting') return 'starting'; if (s.status === 'working' || s.status === 'analyzing' || s.status === 'active') return 'working'; if (s.status === 'dormant') return 'idle'; diff --git a/src/dashboard/web/style.css b/src/dashboard/web/style.css index 5de190089..827b7301f 100644 --- a/src/dashboard/web/style.css +++ b/src/dashboard/web/style.css @@ -2159,6 +2159,7 @@ button.primary { .status-analyzing { background: var(--warning-soft); color: var(--warning); } .status-starting { background: var(--success-soft); color: var(--success); } .status-limited { background: var(--danger-soft); color: var(--danger); } +.status-stalled { background: var(--danger-soft); color: var(--danger); } /* ── Sessions board — "control-room signal wall" ───────────────────────────── Each column is a signal channel: a LED dot + uppercase channel label in the @@ -8239,6 +8240,7 @@ button.contrast:hover { background: var(--danger-soft); border-color: var(--dang color: var(--warning); } .status-limited { background: var(--danger-soft); color: var(--danger); } +.status-stalled { background: var(--danger-soft); color: var(--danger); } /* ── 玻璃面板（dark 下生效；light 保持实底）─────────────────────────────── */ :root[data-theme="dark"] .panel, diff --git a/src/dashboard/web/ui.ts b/src/dashboard/web/ui.ts index 4d2bc0e2f..3a15ad5e3 100644 --- a/src/dashboard/web/ui.ts +++ b/src/dashboard/web/ui.ts @@ -368,6 +368,7 @@ export function attentionReason(s: Record): string | null { if (s.pendingRepo) return t('sessions.board.signalRepo'); if (s.tuiPromptActive) return t('sessions.board.signalPrompt'); if (s.status === 'limited') return t('sessions.board.signalLimited'); + if (s.status === 'stalled') return t('sessions.board.signalStalled'); return null; } diff --git a/src/i18n/en.ts b/src/i18n/en.ts index 0cea646b2..ba7898d0d 100644 --- a/src/i18n/en.ts +++ b/src/i18n/en.ts @@ -33,9 +33,13 @@ export const messages: Record = { 'card.status.idle': 'Awaiting input', 'card.status.dormant': 'Dormant', 'card.status.analyzing': 'Analyzing…', + 'card.status.stalled': 'No recent progress', 'card.status.limited': 'Limit reached', 'card.status.retry_ready': 'Ready to retry', 'card.status.executing': 'Executing…', + 'worker.codex_app.no_progress': '⚠️ This Codex App turn has shown no observable progress for {seconds} seconds. It may still be running; check the web terminal and, if needed, restart only this session. Botmux will not restart or replay the task automatically.', + 'worker.riff_close_in_progress': '⚠️ This Riff session is currently closing, so the new message was not submitted. Retry after the close finishes or reports a retryable failure.', + 'worker.riff_restart_unsupported': '⚠️ In-place restart is not supported for a live Riff task. The session and task were left unchanged; use /close to terminate it.', 'card.status.session_closed': '🛑 Session Closed', 'card.status.relay_frozen': '🔄 Relayed away', 'card.status.selected': 'Selected', @@ -187,6 +191,7 @@ export const messages: Record = { 'card.relay.field_location': 'Where', 'card.relay.field_time': 'Active', 'card.relay.toast_adopt_not_relayable': 'That session was attached via /adopt to an external tmux — it cannot be relayed.', + 'card.relay.toast_riff_not_relayable': 'A Riff sandbox is bound to its original reply route and cannot be relayed yet.', 'card.relay.toast_not_found': 'Session not found or already closed.', 'card.relay.toast_not_owner': 'Only the session owner can relay it.', 'card.relay.toast_not_invoker': 'This menu was summoned by someone else. Run /relay yourself to get your own.', @@ -247,7 +252,9 @@ export const messages: Record = { 'cmd.substitute.usage': 'Usage: @me /substitute status | on | off', 'cmd.restart.in_progress': '🔄 Restarting {cliName}…', 'cmd.restart.terminated': '{cliName} has been terminated; it will auto-resume on your next message.', + 'cmd.restart.riff_unsupported': '⚠️ In-place restart is not supported for a live Riff task. The session and task were left unchanged; use /close to terminate it.', 'cmd.cd.usage': 'Usage: /cd \nExample: /cd ~/projects/my-app', + 'cmd.cd.riff_unsupported': '⚠️ A Riff session’s repository and directory are pinned in its remote task, so /cd is not supported. The session and task were left unchanged; /close it before starting in a new directory.', 'cmd.cd.switched': 'Working directory switched to {path}. It will resume there on your next message.', 'cmd.cd.created_switched': '📁 Directory did not exist — created it and switched to {path}. It will resume there on your next message.', 'cmd.cd.dir_not_exist': 'Directory does not exist: {path}', @@ -458,6 +465,7 @@ export const messages: Record = { 'cmd.relay.no_mentions': '⚠️ /relay --create requires at least one @-mentioned bot as a collaborator in the new group.', 'cmd.relay.not_owner': '⚠️ Only the original session owner can /relay --create this session to a new group.', 'cmd.relay.adopt_not_relayable': '⚠️ This session was attached via /adopt to an external tmux. The CLI runs on your machine and botmux does not control its tmux lifecycle, so it cannot be relayed.', + 'cmd.relay.riff_not_relayable': '⚠️ A Riff sandbox is bound to its original reply route and cannot be relayed yet. The session and remote task were left in place.', 'cmd.relay.not_started_yet': '⚠️ The session has not picked a repo and the CLI has never started — nothing to relay yet. Pick a repo in this session first.', 'cmd.relay.worker_busy': '⚠️ The session is mid-turn (working). Wait for the current turn to finish (idle), then try /relay again.', 'cmd.relay.resolve_failed': '⚠️ Could not resolve the @-mentioned bots; /relay cancelled. Please retry shortly.', @@ -698,6 +706,7 @@ export const messages: Record = { 'card.voice.toast_session_gone': '⚠️ Session is offline; cannot generate a voice summary.', 'card.voice.toast_need_auth': '🔒 You are not authorized to use this bot, so you cannot generate a voice summary. Ask an admin for access.', 'card.voice.toast_worker_busy': '⚠️ The session is still running. Wait for it to go idle, then generate the voice summary.', + 'card.repo.toast_stale_picker': '⚠️ This repo picker is no longer current. Use the latest card.', 'card.voice.user_message': 'Generate a voice summary', 'card.action.takeover_retired': '⚠️ The old "Take Over" button is retired. In bridge mode, botmux bridges the original CLI so replies still come back to Lark — no takeover needed. Full takeover (`/adopt --takeover`) is on the roadmap.', 'card.action.terminal_not_ready': '⚠️ Terminal is not ready yet, please try again shortly.', @@ -790,6 +799,7 @@ export const messages: Record = { 'daemon.download_failed_need_login': '⚠️ Some images/files failed to download (missing User Token). Send `/login` in this topic to authorize, then resend.', 'daemon.foreign_bot_mention_prefix': '[@mention from {botName}]', 'daemon.cmd_needs_active_cli': '{cmd} needs an active CLI process; no running session in this topic.', + 'daemon.cmd_activation_pending': '{cmd} cannot be sent yet because the previous turn is still being submitted. Retry shortly.', 'daemon.enriched_mentions_label': '@mentions in this message:', 'daemon.choose_repo_first': 'Pick a repo from the card above first — your message is queued and will be sent once a repo is chosen.', 'daemon.worktree_building_wait': 'Creating a worktree (includes a git fetch, may take a few seconds) — your message is queued and will be sent together once it is ready.', diff --git a/src/i18n/zh.ts b/src/i18n/zh.ts index 74c555566..340277d63 100644 --- a/src/i18n/zh.ts +++ b/src/i18n/zh.ts @@ -36,9 +36,13 @@ export const messages: Record = { 'card.status.idle': '等待输入', 'card.status.dormant': '休眠', 'card.status.analyzing': '正在分析…', + 'card.status.stalled': '长时间无进展', 'card.status.limited': '限额已达', 'card.status.retry_ready': '可重试', 'card.status.executing': '正在执行…', + 'worker.codex_app.no_progress': '⚠️ Codex App 此轮已连续 {seconds} 秒没有可观察进展。任务可能仍在运行；请检查 Web 终端，必要时仅重启当前会话。Botmux 不会自动重启或重放任务。', + 'worker.riff_close_in_progress': '⚠️ Riff 会话正在关闭，本条消息未提交。请等待关闭完成，或在关闭返回“可重试”失败后重发。', + 'worker.riff_restart_unsupported': '⚠️ Riff 远端任务暂不支持原地重启。会话和任务均保持不变；如需终止，请使用 /close。', 'card.status.session_closed': '🛑 会话已关闭', 'card.status.relay_frozen': '🔄 会话已搬迁', 'card.status.selected': '已选择', @@ -190,6 +194,7 @@ export const messages: Record = { 'card.relay.field_location': '位置', 'card.relay.field_time': '活跃', 'card.relay.toast_adopt_not_relayable': '该会话是 /adopt 接入的外部 tmux，无法接力。', + 'card.relay.toast_riff_not_relayable': 'Riff 远端沙箱绑定了原会话的回传地址，暂不支持接力。', 'card.relay.toast_not_found': '会话已不存在或已关闭。', 'card.relay.toast_not_owner': '只有会话发起人能接力它。', 'card.relay.toast_not_invoker': '这张菜单是别人召唤的，请自己发 /relay 召唤一张。', @@ -250,7 +255,9 @@ export const messages: Record = { 'cmd.substitute.usage': '用法：@我 /substitute status｜on｜off', 'cmd.restart.in_progress': '🔄 正在重启 {cliName}...', 'cmd.restart.terminated': '{cliName} 进程已终止，下次发消息时将自动恢复。', + 'cmd.restart.riff_unsupported': '⚠️ Riff 远端任务暂不支持原地重启。会话和任务均保持不变；如需终止，请使用 /close。', 'cmd.cd.usage': '用法：/cd \n例如：/cd ~/projects/my-app', + 'cmd.cd.riff_unsupported': '⚠️ Riff 会话的仓库与目录已固定在远端任务中，暂不支持 /cd。会话和任务均保持不变；如需换目录，请先 /close 后新建会话。', 'cmd.cd.switched': '工作目录已切换到 {path}，下次发消息时将在新目录下恢复。', 'cmd.cd.created_switched': '📁 目录不存在，已自动创建并切换到 {path}，下次发消息时将在新目录下恢复。', 'cmd.cd.dir_not_exist': '目录不存在：{path}', @@ -461,6 +468,7 @@ export const messages: Record = { 'cmd.relay.no_mentions': '⚠️ /relay --create 必须 @ 至少一个机器人作为新群的协作伙伴。', 'cmd.relay.not_owner': '⚠️ 只有会话发起人能用 /relay --create 把会话搬到新群。', 'cmd.relay.adopt_not_relayable': '⚠️ 该会话是 /adopt 接入的外部 tmux 进程，CLI 在你电脑里跑，botmux 控不了 tmux 生命周期，无法接力。', + 'cmd.relay.riff_not_relayable': '⚠️ Riff 远端沙箱绑定了原会话的回传地址，暂不支持接力。会话和任务均保持在原位置。', 'cmd.relay.not_started_yet': '⚠️ 会话还没选仓库、CLI 没起来，无法接力。请先在本会话里选仓库把 CLI 起起来。', 'cmd.relay.worker_busy': '⚠️ 会话正在处理中（mid-turn），无法接力。请等当前回合结束（idle）后再发 /relay。', 'cmd.relay.resolve_failed': '⚠️ 无法解析被 @ 的机器人，/relay 取消。请稍后重试。', @@ -701,6 +709,7 @@ export const messages: Record = { 'card.voice.toast_session_gone': '⚠️ 会话已不在线，无法生成语音总结', 'card.voice.toast_need_auth': '🔒 你没有该机器人的使用权限，无法生成语音总结，请联系管理员授权', 'card.voice.toast_worker_busy': '⚠️ 会话当前还在执行中，请等它空闲后再生成语音总结。', + 'card.repo.toast_stale_picker': '⚠️ 此仓库选择卡已失效，请使用最新卡片。', 'card.voice.user_message': '生成语音总结', 'card.action.takeover_retired': '⚠️ 旧版"接管"按钮已停用。bridge 模式下原 CLI 由 botmux 桥接，无需接管即可在飞书中收到回答。如需 /resume 完整接管能力，请等待 /adopt --takeover 命令上线。', 'card.action.terminal_not_ready': '⚠️ 终端尚未就绪，请稍后再试。', @@ -793,6 +802,7 @@ export const messages: Record = { 'daemon.download_failed_need_login': '⚠️ 部分图片/文件下载失败（缺少 User Token）。请在话题中发送 /login 授权后重新发送。', 'daemon.foreign_bot_mention_prefix': '[来自 {botName} 的 @mention]', 'daemon.cmd_needs_active_cli': '{cmd} 需要活跃的 CLI 进程，当前话题无运行中的会话。', + 'daemon.cmd_activation_pending': '{cmd} 暂不能发送：上一条消息仍在提交中，请稍后重试。', 'daemon.enriched_mentions_label': '消息中的 @mention：', 'daemon.choose_repo_first': '请先在上方卡片中选择仓库，您的消息已暂存，选择后会自动发送。', 'daemon.worktree_building_wait': '正在创建 worktree（含 git fetch，可能需要几秒），您的消息已暂存，创建完成后会自动一并发送。', diff --git a/src/im/lark/card-builder.ts b/src/im/lark/card-builder.ts index 5c9483b8a..4d51b5957 100644 --- a/src/im/lark/card-builder.ts +++ b/src/im/lark/card-builder.ts @@ -637,7 +637,7 @@ export function truncateContent(content: string, locale?: Locale, maxBytes: numb const PRIVATE_SNAPSHOT_TEXT_MAX = 50_000; const STREAM_TEMPLATE_MAP = { - starting: 'yellow', working: 'blue', idle: 'green', analyzing: 'purple', limited: 'red', retry_ready: 'green', + starting: 'yellow', working: 'blue', idle: 'green', analyzing: 'purple', stalled: 'red', limited: 'red', retry_ready: 'green', } as const; /** Header status label for a streaming/snapshot card. Shared by the live card @@ -648,6 +648,7 @@ function streamStatusLabel(status: StreamStatus, usageLimit: CliUsageLimitState case 'working': return t('card.status.working', undefined, locale); case 'idle': return t('card.status.idle', undefined, locale); case 'analyzing': return t('card.status.analyzing', undefined, locale); + case 'stalled': return t('card.status.stalled', undefined, locale); case 'limited': return usageLimit?.retryReady ? t('card.status.retry_ready', undefined, locale) : t('card.status.limited', undefined, locale); diff --git a/src/im/lark/card-handler.ts b/src/im/lark/card-handler.ts index f93d1398c..b65818454 100644 --- a/src/im/lark/card-handler.ts +++ b/src/im/lark/card-handler.ts @@ -59,13 +59,13 @@ import { ttadkConfigModelChoices } from '../../setup/cli-selection.js'; import { logger } from '../../utils/logger.js'; import * as sessionStore from '../../services/session-store.js'; import { loadFrozenCards, saveFrozenCards } from '../../services/frozen-card-store.js'; -import { forkWorker, sendWorkerInput, killWorker, scheduleCardPatch, parkStreamCard, clearUsageLimitState, cardUsageLimit, writableTerminalLinkFor, resolvePrivateCardAudience, deliverWriteLinkCard, deliverEphemeralOrReply, CARD_POSTING_SENTINEL } from '../../core/worker-pool.js'; +import { forkWorker, sendWorkerInput, killWorker, scheduleCardPatch, parkStreamCard, clearUsageLimitState, cardUsageLimit, writableTerminalLinkFor, resolvePrivateCardAudience, deliverWriteLinkCard, deliverEphemeralOrReply, CARD_POSTING_SENTINEL, closeSession as closeWorkerPoolSession, withActiveSessionKeyLock } from '../../core/worker-pool.js'; import { getSessionWorkingDir, buildNewTopicCliInput, getAvailableBots, persistStreamCardState, resumeSession, rememberLastCliInput, ensureSessionWhiteboard } from '../../core/session-manager.js'; import { publishAttentionPatch, announcePendingRepoSession } from '../../core/session-activity.js'; import { fallbackTurnId } from '../../core/reply-target.js'; import { validateWorkingDir } from '../../core/working-dir.js'; import type { DaemonToWorker, DisplayMode, TermActionKey } from '../../types.js'; -import { sessionKey, sessionAnchorId, frozenDisplayMode, markRepoCardConsumed, isRepoCardConsumed, isActiveRepoCard, claimCurrentRepoCard } from '../../core/types.js'; +import { activeSessionKey, sessionKey, sessionAnchorId, frozenDisplayMode, markRepoCardConsumed, isActiveRepoCard } from '../../core/types.js'; import type { DaemonSession } from '../../core/types.js'; import { buildTerminalUrl } from '../../core/terminal-url.js'; import type { ProjectInfo } from '../../services/project-scanner.js'; @@ -82,6 +82,9 @@ import { openLocalCliInIterm, preflightLocalCliOpen, } from '../../services/local-cli-opener.js'; +import { hasProtectedSessionMutationOwnership } from '../../core/session-mutation-guard.js'; +import { persistPendingRepoCardMessageId } from '../../core/pending-repo-journal.js'; +import { runDetachedBotTurnAdmission, withBotTurnAdmission, withBotTurnMutation } from '../../core/bot-turn-mutation-gate.js'; // ─── Types ──────────────────────────────────────────────────────────────── @@ -362,199 +365,183 @@ export async function commitRepoSelection( // The worktree flow already posted a precise "worktree 已创建：path 分支 …" // line before funnelling in here — suppress the redundant "已选择/已切换" // confirmation so the user sees a single message, not two. - // pinWorkingDir=false: "直接开始"/skip keeps the default cwd for this launch - // without persisting it (esp. $HOME) onto session.workingDir — sibling bots - // must still get their own repo card instead of inheriting HOME. - // confirmReplyText: optional confirmation to send under the same claim - // (used by skip_repo so the post-fork reply window cannot race a second click). opts?: { suppressConfirmReply?: boolean; confirmReplyText?: string; pinWorkingDir?: boolean; riffRepoDirs?: string[]; }, -): Promise { +): Promise { const { ds, rootId, cardMessageId, larkAppId, operatorOpenId, activeSessions, sessionReply } = ctx; const locTarget = localeForBot(ds.larkAppId); // `/close` deletes the active-map entry without touching sessionId or // pendingRepo — identity against the map is the only tell that the session // this flow captured is gone. Checked alongside the generation snapshots. - const repoSessionKey = larkAppId ? sessionKey(rootId, larkAppId) : rootId; + const repoSessionKey = activeSessionKey(ds); const sessionStillActive = () => activeSessions.get(repoSessionKey) === ds; const commitGenSessionId = ds.session.sessionId; const pinWorkingDir = opts?.pinWorkingDir !== false; - if (ds.pendingRepo) { - // Card select/skip and auto-worktree converge here; the text /repo path - // mirrors this transaction through the same in-memory claim. Only one may - // own the pending -> worker transition, including prompt preparation. - if (ds.pendingRepoCommitInFlight) { - logger.info(`[${tag(ds)}] Pending repo commit already in flight — ignoring duplicate selection`); - return; - } - ds.pendingRepoCommitInFlight = true; - let committed = false; - try { - // Claim before mutating the selected directory so two different card - // choices cannot overwrite each other while one prompt is being built. - // Skip/bare-start may intentionally leave workingDir unpinned so the - // launch uses the bot default without persisting $HOME for siblings. - if (pinWorkingDir) { - ds.workingDir = dirPath; - ds.session.workingDir = dirPath; - } - ds.session.riffRepoDirs = opts?.riffRepoDirs; - sessionStore.updateSession(ds.session); - const selfBot = getBot(ds.larkAppId); - const botCfg = selfBot.config; - const effectiveCliId = sessionCliId(ds); - - // Keep pendingRepo=true across this await. New topic messages therefore - // remain buffered on this same session instead of racing a worker-null - // safety-net fork. Snapshot every pending field only after it settles. - const needsPromptContext = !ds.pendingRawInput || - (ds.pendingPrompt?.trim().length ?? 0) > 0 || - (ds.pendingAttachments?.length ?? 0) > 0 || - (ds.pendingFollowUps?.length ?? 0) > 0; - const availableBots = needsPromptContext - ? await getAvailableBots(ds.larkAppId, ds.chatId) - : []; - if (!sessionStillActive() || ds.session.sessionId !== commitGenSessionId || - !ds.pendingRepo || (ds.worker && !ds.worker.killed)) { - logger.warn(`[${tag(ds)}] Session changed while preparing the pending-CLI prompt (${commitGenSessionId} → ${ds.session.sessionId}, active=${sessionStillActive()}, pending=${!!ds.pendingRepo}, worker=${!!ds.worker}) — aborting this fork`); - return; - } + // Card callbacks are replayable. Only the currently published picker may + // mutate this session; a withdrawn/replaced card must stay stale even after + // the activation FIFO eventually settles. Internal auto-worktree commits do + // not originate from a card and deliberately omit cardMessageId. + if (cardMessageId !== undefined + && (!ds.repoCardMessageId || cardMessageId !== ds.repoCardMessageId)) { + logger.warn( + `[${tag(ds)}] Ignoring stale repo-card callback ${cardMessageId} ` + + `(current=${ds.repoCardMessageId ?? 'none'})`, + ); + return false; + } - const pendingPrompt = ds.pendingPrompt ?? ''; - const pendingRawInput = ds.pendingRawInput; - const hasBufferedInput = - pendingPrompt.trim().length > 0 || - (ds.pendingAttachments?.length ?? 0) > 0 || - (ds.pendingFollowUps?.length ?? 0) > 0; - if (!pendingRawInput || hasBufferedInput) ensureSessionWhiteboard(ds); - const wrappedInput = (!pendingRawInput || hasBufferedInput) - ? buildNewTopicCliInput( - pendingPrompt, - ds.session.sessionId, - effectiveCliId, - botCfg.cliPathOverride, - ds.pendingAttachments, - ds.pendingMentions, - availableBots, - ds.pendingFollowUps, - { name: selfBot.botName, openId: selfBot.botOpenId }, - locTarget, - ds.pendingSender, - { - larkAppId: ds.larkAppId, - chatId: ds.chatId, - whiteboardId: ds.session.whiteboardId, - substituteTrigger: ds.pendingSubstituteTrigger, - codexAppText: ds.pendingCodexAppText, - codexAppApplicationContext: ds.pendingCodexAppApplicationContext, - codexAppMessageContext: ds.pendingCodexAppMessageContext, - codexAppFollowUps: ds.pendingCodexAppFollowUps, - codexAppFollowUpContexts: ds.pendingCodexAppFollowUpContexts, - }, - ) - : { content: '' }; - const prompt = pendingRawInput ? '' : wrappedInput; - const pendingTurnId = ds.pendingTurnId; - const previousPendingFollowUpInput = ds.pendingFollowUpInput; - if (pendingRawInput && hasBufferedInput) { - ds.pendingFollowUpInput = { - userPrompt: ds.pendingCodexAppText !== undefined || ds.pendingCodexAppFollowUps - ? [ds.pendingCodexAppText ?? '', ...(ds.pendingCodexAppFollowUps ?? [])].filter(Boolean).join('\n\n') - : pendingPrompt || ds.pendingFollowUps?.join('\n\n') || '', - cliInput: wrappedInput.content, - ...(ds.pendingFollowUpTurnId ? { turnId: ds.pendingFollowUpTurnId } : {}), - ...(effectiveCliId === 'codex-app' && botCfg.codexAppCleanInput === true && wrappedInput.codexAppInput - ? { codexAppInput: wrappedInput.codexAppInput } - : {}), - codexAppInputGateFrozen: true, - }; - } + if (!ds.pendingRepo + && hasProtectedSessionMutationOwnership(ds)) { + await sessionReply( + rootId, + '当前会话仍有待提交消息，暂不能切换仓库；请等待提交完成或关闭会话。', + ); + return false; + } - // From here through forkWorker there is no await: publish the state - // transition atomically from the router's point of view. - ds.pendingRepo = false; - publishAttentionPatch(ds); - try { - if (pendingRawInput) { - // pendingRawTurnId is applied at the literal PTY write boundary. - forkWorker(ds, '', false); - } else if (pendingTurnId && hasBufferedInput) { - forkWorker(ds, prompt, { turnId: pendingTurnId }); - } else { - forkWorker(ds, prompt); - } - } catch (e) { - ds.pendingRepo = true; - ds.pendingFollowUpInput = previousPendingFollowUpInput; - publishAttentionPatch(ds); - throw e; - } - rememberLastCliInput(ds, pendingRawInput ?? pendingPrompt, pendingRawInput ?? wrappedInput); - ds.pendingPrompt = undefined; - ds.pendingCodexAppText = undefined; - ds.pendingCodexAppApplicationContext = undefined; - ds.pendingCodexAppMessageContext = undefined; - ds.pendingAttachments = undefined; - ds.pendingMentions = undefined; - ds.pendingSubstituteTrigger = undefined; - ds.pendingSender = undefined; - ds.pendingFollowUps = undefined; - ds.pendingCodexAppFollowUps = undefined; - ds.pendingCodexAppFollowUpContexts = undefined; - ds.pendingFollowUpTurnId = undefined; - ds.pendingTurnId = undefined; - committed = true; - - // Local one-shot consume BEFORE any network await. Feishu withdraw is - // best-effort and may lag/fail; correctness for stale card clicks depends - // on this mark so a second callback cannot mid-session-switch after claim - // release (and so a sessionReply throw that jumps to finally still leaves - // the card locally invalid). - const cardToWithdraw = cardMessageId ?? ds.repoCardMessageId; - markRepoCardConsumed(ds, cardToWithdraw); - ds.repoCardMessageId = undefined; - - // Hold the claim through confirmation + best-effort card withdraw so a - // concurrent in-flight select still sees pendingRepoCommitInFlight. - try { - if (!opts?.suppressConfirmReply) { - // A card click has no turn of its own — anchor the confirmation to the - // session's current reply-target turn so a shared fold-back topic keeps - // it in-thread (same leak as the /repo command path). - await sessionReply(rootId, t('cmd.repo.selected_in_pending', { name: dirLabel }, locTarget), undefined, fallbackTurnId(ds, undefined)); - } else if (opts.confirmReplyText) { - await sessionReply(rootId, opts.confirmReplyText, undefined, fallbackTurnId(ds, undefined)); - } - } catch (e) { - logger.warn(`[${tag(ds)}] Confirm reply after pending repo commit failed: ${e instanceof Error ? e.message : e}`); - } - if (cardToWithdraw && larkAppId) { - try { await deleteMessage(larkAppId, cardToWithdraw); } catch { /* card may already be gone */ } - } - } finally { - ds.pendingRepoCommitInFlight = false; + if (ds.pendingRepo) { + const targetSessionId = ds.session.sessionId; + const started = await withBotTurnMutation(ds.larkAppId, async () => { + const current = [...activeSessions.values()].find( + candidate => candidate.session.sessionId === targetSessionId + && candidate.session.status === 'active', + ); + if (!current || current !== ds || !current.pendingRepo) return false; + // "Start directly" launches in the resolved default cwd without pinning + // HOME onto the session for sibling-bot inheritance. + if (pinWorkingDir) { + ds.workingDir = dirPath; + ds.session.workingDir = dirPath; + } + // riff 多仓 stamp：只有多仓 worktree 流显式传入（保留用户选择顺序，首仓=primary）； + // 其它选仓路径一律清除旧 stamp——workingDir 变了，旧的多仓组合不再成立。 + ds.session.riffRepoDirs = opts?.riffRepoDirs; + sessionStore.updateSession(ds.session); + const selfBot = getBot(ds.larkAppId); + const botCfg = selfBot.config; + const effectiveCliId = sessionCliId(ds); + // First-time repo selection — now spawn CLI with the original prompt + const pendingPrompt = ds.pendingPrompt ?? ''; + const pendingRawInput = ds.pendingRawInput; + // Raw-input cold start still wraps any input buffered while the repo card + // was pending — see the skip_repo branch for the rationale. + const hasBufferedInput = + pendingPrompt.trim().length > 0 || + ds.pendingCodexAppText !== undefined || + (ds.pendingAttachments?.length ?? 0) > 0 || + (ds.pendingFollowUps?.length ?? 0) > 0; + if (hasBufferedInput) ensureSessionWhiteboard(ds); + const wrappedInput = hasBufferedInput + ? buildNewTopicCliInput( + pendingPrompt, + ds.session.sessionId, + effectiveCliId, + botCfg.cliPathOverride, + ds.pendingAttachments, + ds.pendingMentions, + await getAvailableBots(ds.larkAppId, ds.chatId), + ds.pendingFollowUps, + { name: selfBot.botName, openId: selfBot.botOpenId }, + locTarget, + ds.pendingSender, + { + larkAppId: ds.larkAppId, + chatId: ds.chatId, + whiteboardId: ds.session.whiteboardId, + substituteTrigger: ds.pendingSubstituteTrigger, + codexAppText: ds.pendingCodexAppText, + codexAppApplicationContext: ds.pendingCodexAppApplicationContext, + codexAppMessageContext: ds.pendingCodexAppMessageContext, + codexAppFollowUps: ds.pendingCodexAppFollowUps, + codexAppFollowUpContexts: ds.pendingCodexAppFollowUpContexts, + }, + ) + : undefined; + const prompt = pendingRawInput ? '' : (wrappedInput ?? ''); + // Last-line defence: prompt prep awaited above — if anything replaced + // OR closed the session in that window, forking now would clobber it + // (or resurrect a /close'd session). + if (!sessionStillActive() || ds.session.sessionId !== commitGenSessionId) { + logger.warn(`[${tag(ds)}] Session replaced or closed while preparing the pending-CLI prompt (${commitGenSessionId} → ${ds.session.sessionId}, active=${sessionStillActive()}) — aborting this fork`); + return false; + } + if (pendingRawInput && hasBufferedInput && wrappedInput) { + ds.pendingFollowUpInput = { + userPrompt: ds.pendingCodexAppText !== undefined || ds.pendingCodexAppFollowUps + ? [ds.pendingCodexAppText ?? '', ...(ds.pendingCodexAppFollowUps ?? [])].filter(Boolean).join('\n\n') + : pendingPrompt || ds.pendingFollowUps?.join('\n\n') || '', + cliInput: wrappedInput.content, + ...((ds.pendingFollowUpTurnIds?.at(-1) ?? ds.pendingFollowUpTurnId) + ? { turnId: ds.pendingFollowUpTurnIds?.at(-1) ?? ds.pendingFollowUpTurnId } + : {}), + ...(effectiveCliId === 'codex-app' && botCfg.codexAppCleanInput === true && wrappedInput.codexAppInput + ? { codexAppInput: wrappedInput.codexAppInput } + : {}), + codexAppInputGateFrozen: true, + }; + } + if (pendingRawInput) rememberLastCliInput(ds, pendingRawInput, pendingRawInput); + else if (hasBufferedInput && wrappedInput) rememberLastCliInput(ds, pendingPrompt, wrappedInput); + // Keep the reservation and every buffered opening field intact through + // forkWorker's synchronous pre-accept/write-ahead phase. If it throws, + // the user can retry this exact selection without losing the first turn. + const pendingTurnId = ds.pendingTurnId ?? ds.session.pendingRepoSetup?.turnId; + forkWorker( + ds, + prompt, + !pendingRawInput && pendingTurnId ? { turnId: pendingTurnId } : false, + ); + ds.pendingRepo = false; + // A queued activation owns the route through its adapter-level ACK. Every + // buffer below was synchronously folded into prompt N and is safe to clear; + // later inbounds observe this gate and enter the separate exact staged FIFO. + ds.initialStartPending = ds.session.queuedActivationPending === true; + publishAttentionPatch(ds); + ds.pendingPrompt = undefined; + ds.pendingCodexAppText = undefined; + ds.pendingCodexAppApplicationContext = undefined; + ds.pendingCodexAppMessageContext = undefined; + ds.pendingAttachments = undefined; + ds.pendingMentions = undefined; + ds.pendingSubstituteTrigger = undefined; + ds.pendingSender = undefined; + ds.pendingFollowUps = undefined; + ds.pendingFollowUpTurnId = undefined; + ds.pendingFollowUpTurnIds = undefined; + ds.pendingCodexAppFollowUps = undefined; + ds.pendingCodexAppFollowUpContexts = undefined; + ds.pendingCodexAppFollowUpGateAccepted = undefined; + ds.pendingTurnId = undefined; + return true; + }); + if (!started) return false; + // Invalidate synchronously at the successful commit boundary. Card delete + // and confirmation delivery are best effort and may await/fail; neither is + // allowed to leave a replayable mutation capability behind. + const cardToWithdraw = cardMessageId ?? ds.repoCardMessageId; + markRepoCardConsumed(ds, cardToWithdraw); + ds.repoCardMessageId = undefined; + // A card click has no turn of its own — anchor the confirmation to the + // session's current reply-target turn so a shared fold-back topic keeps + // it in-thread (same leak as the /repo command path). + if (!opts?.suppressConfirmReply) { + await sessionReply(rootId, t('cmd.repo.selected_in_pending', { name: dirLabel }, locTarget), undefined, fallbackTurnId(ds, undefined)); + } else if (opts.confirmReplyText) { + await sessionReply(rootId, opts.confirmReplyText, undefined, fallbackTurnId(ds, undefined)); + } + if (cardToWithdraw && larkAppId) { + try { await deleteMessage(larkAppId, cardToWithdraw); } + catch { /* best-effort */ } } - if (!committed) return; logger.info(`[${tag(ds)}] Repo selected: ${dirPath}, spawning CLI`); - return; + return true; } else { // Mid-session repo switch — close old session, start fresh. - // Claim the current card BEFORE killWorker / any await. A concurrent click - // on the same Feishu card must not pass a second kill+fork while the first - // switch is still awaiting confirm replies. Correctness does not depend on - // deleteMessage succeeding. - const claimedCard = claimCurrentRepoCard(ds, cardMessageId); - if (cardMessageId && !claimedCard) { - // Stale / already-claimed card (entry check races can still reach here). - logger.info(`[${tag(ds)}] Ignoring stale mid-session repo card ${cardMessageId}`); - return; - } - // Safety net (mirrors the `/repo` text-command path): build the same // "session closed" card `/close` emits BEFORE displacing the old session // (it reads the live session's identity off `ds`). The new session reuses @@ -566,73 +553,98 @@ export async function commitRepoSelection( // the displaced session's stored workingDir (and the closed card), so // `claude --resume` later would reopen the old context in the new repo's // cwd. The new repo is pinned onto the fresh session below instead. - const closedCard = buildClosedSessionCard(ds, locTarget); - - killWorker(ds); - // Park the current card in `frozenCards` so the next POST under the new - // session sweeps it via recall. closeSession() wipes the on-disk - // frozen-cards file under the OLD sessionId, but the in-memory Map - // travels with `ds` into the new session and still carries the - // old messageId for deletion. If fork or POST fails, the parked card - // stays in the thread instead of vanishing prematurely. - parkStreamCard(ds); - sessionStore.closeSession(ds.session.sessionId); - + const targetSessionId = ds.session.sessionId; + const switched = await withBotTurnMutation(ds.larkAppId, async () => { + const candidate = [...activeSessions.values()].find( + candidate => candidate.session.sessionId === targetSessionId, + ); + if (!candidate || candidate !== ds || candidate.session.status !== 'active') { + return { ok: false as const, error: 'session_replaced' as const }; + } + const key = activeSessionKey(candidate); + return withActiveSessionKeyLock(activeSessions, key, async () => { + const current = [...activeSessions.values()].find( + owner => owner.session.sessionId === targetSessionId, + ); + if (!current || current !== candidate + || activeSessions.get(key) !== current + || current.session.status !== 'active') { + return { ok: false as const, error: 'session_replaced' as const }; + } + if (hasProtectedSessionMutationOwnership(current)) { + return { ok: false as const, error: 'dispatch_pending' as const }; + } + const closedCard = buildClosedSessionCard(current, locTarget); + // Preserve the old card in memory before durable close clears its old + // frozen-card file; it is re-keyed under the replacement session below. + parkStreamCard(current); + const oldSession = current.session; + const closeResult = await closeWorkerPoolSession(targetSessionId); + if (!closeResult.ok) { + return { ok: false as const, error: 'close_failed' as const, closeResult }; + } + if (activeSessions.get(key) === current) activeSessions.delete(key); + if (activeSessions.has(key)) { + return { ok: false as const, error: 'session_replaced' as const }; + } + const cardToWithdraw = cardMessageId ?? current.repoCardMessageId; + markRepoCardConsumed(current, cardToWithdraw); + current.repoCardMessageId = undefined; + + const session = sessionStore.createSession( + current.chatId, + current.scope === 'chat' ? oldSession.rootMessageId : rootId, + dirLabel, + current.chatType, + current.scope, + ); + current.session = session; + current.lastUserPrompt = undefined; + current.lastCliInput = undefined; + current.workingDir = dirPath; + session.workingDir = dirPath; + session.larkAppId = current.larkAppId; + session.chatDisplayName = oldSession.chatDisplayName; + session.ownerOpenId = oldSession.ownerOpenId; + session.creatorOpenId = oldSession.creatorOpenId; + session.lastCallerOpenId = oldSession.lastCallerOpenId; + session.riffRepoDirs = opts?.riffRepoDirs; + sessionStore.updateSession(session); + current.hasHistory = false; + if (current.frozenCards && current.frozenCards.size > 0) { + saveFrozenCards(session.sessionId, current.frozenCards); + } + current.streamCardId = undefined; + current.streamCardNonce = undefined; + current.streamCardPending = undefined; + current.lastScreenContent = undefined; + current.lastScreenStatus = undefined; + activeSessions.set(key, current); + forkWorker(current, '', false); + return { ok: true as const, current, closedCard, cardToWithdraw }; + }); + }); + if (!switched.ok) { + if (switched.error === 'dispatch_pending') { + await sessionReply( + rootId, + '当前 Codex App 仍有未结算消息，暂不能切换仓库；请等待本轮完成或关闭会话。', + ); + } else if (switched.error === 'close_failed') { + await sessionReply( + rootId, + 'Riff 远端任务取消失败，原会话仍保持活跃；未切换仓库，请重试。', + ); + } + return false; + } await deliverEphemeralOrReply( - ds, + switched.current, operatorOpenId, - closedCard, + switched.closedCard, 'interactive', - () => sessionReply(rootId, closedCard, 'interactive'), + () => sessionReply(rootId, switched.closedCard, 'interactive'), ); - - const oldSession = ds.session; - // `rootId` is the routing anchor. For chat-scope sessions it is the - // `oc_...` chat id, not the traceable `om_...` message root stored on - // Session. Preserve the old identity and explicitly persist scope so card - // switches cannot recreate the session as a legacy scope-less thread. - const session = sessionStore.createSession( - ds.chatId, - ds.scope === 'chat' ? oldSession.rootMessageId : rootId, - dirLabel, - ds.chatType, - ds.scope, - ); - ds.session = session; - ds.lastUserPrompt = undefined; - ds.lastCliInput = undefined; - // Pin workingDir + larkAppId onto the new session before forkWorker. - // Without this, a daemon restart restores the session with an empty - // workingDir and the worker spawns in the bot's default cwd, so - // `claude --resume` looks in the wrong .claude/projects// dir and - // exits code 0 immediately, crash-looping until the rate-limiter trips. - ds.workingDir = dirPath; - ds.session.workingDir = dirPath; - ds.session.larkAppId = ds.larkAppId; - ds.session.chatDisplayName = oldSession.chatDisplayName; - ds.session.ownerOpenId = oldSession.ownerOpenId; - ds.session.creatorOpenId = oldSession.creatorOpenId; - ds.session.lastCallerOpenId = oldSession.lastCallerOpenId; - // Stamp the newly-created session, not the displaced session that was just - // closed. Plain/single-repo switches pass undefined and clear stale state. - ds.session.riffRepoDirs = opts?.riffRepoDirs; - sessionStore.updateSession(ds.session); - ds.hasHistory = false; - // Re-persist the parked card under the NEW sessionId so a daemon crash - // before the next POST doesn't strand it. closeSession() above wiped - // the on-disk file under the OLD sessionId; without this re-save, the - // in-memory Map only survives in process memory. - if (ds.frozenCards && ds.frozenCards.size > 0) { - saveFrozenCards(ds.session.sessionId, ds.frozenCards); - } - // Drop the old turn's streaming-card reference so worker_ready POSTs a - // fresh card for the new session instead of PATCHing the previous one. - ds.streamCardId = undefined; - ds.streamCardNonce = undefined; - ds.streamCardPending = undefined; - ds.lastScreenContent = undefined; - ds.lastScreenStatus = undefined; - forkWorker(ds, '', false); if (!opts?.suppressConfirmReply) { try { await sessionReply(rootId, t('cmd.repo.switched_to', { name: dirLabel }, locTarget)); @@ -640,12 +652,14 @@ export async function commitRepoSelection( logger.warn(`[${tag(ds)}] Confirm reply after mid-session repo switch failed: ${e instanceof Error ? e.message : e}`); } } - logger.info(`[${tag(ds)}] Repo switched to ${dirPath}, new session created`); - // Best-effort withdraw — card already claimed above. - if (claimedCard && larkAppId) { - try { await deleteMessage(larkAppId, claimedCard); } catch { /* best-effort */ } + if (switched.cardToWithdraw && larkAppId) { + try { await deleteMessage(larkAppId, switched.cardToWithdraw); } + catch { /* best-effort */ } } + logger.info(`[${tag(switched.current)}] Repo switched to ${dirPath}, new session created`); } + + return true; } /** @@ -690,7 +704,11 @@ export async function runAutoWorktreeCommit(deps: { // pendingRepo session, folds any messages buffered during creation (pendingPrompt // + pendingFollowUps) into the first turn. suppressConfirmReply: the worktree // helper already posted the '已创建/回退' line, so skip the '已选择' confirmation. - await commitRepoSelection( + // The worktree build is intentionally detached from its caller's inbound + // admission. Re-enter with a fresh lease at the delayed commit/fork edge; + // the outer lease may have ended minutes ago and must not authorize this + // descendant across a bot-wide config mutation. + await runDetachedBotTurnAdmission(larkAppId, () => commitRepoSelection( { ds, rootId: anchor, larkAppId, operatorOpenId, activeSessions, // Never reached under suppressConfirmReply for a pendingRepo session. @@ -699,7 +717,7 @@ export async function runAutoWorktreeCommit(deps: { wt.dir, pathBasename(wt.dir), { suppressConfirmReply: true }, - ); + )); } catch (e) { // No recovery fork here: forking with an empty prompt would DROP the buffered // first turn (pendingPrompt lives only in-memory, not the message queue). Leave @@ -1229,6 +1247,10 @@ export async function handleCardAction(data: CardActionData, deps: CardHandlerDe if (sourceDs.session.ownerOpenId && sourceDs.session.ownerOpenId !== operatorOpenId) { return { toast: { type: 'error', content: t('card.relay.toast_not_owner', undefined, loc) } }; } + if ((sourceDs.initConfig?.backendType ?? sourceDs.session.backendType) === 'riff' + || sourceDs.session.cliId === 'riff') { + return { toast: { type: 'error', content: t('card.relay.toast_riff_not_relayable', undefined, loc) } }; + } // Anchor-based self-relay guard: a thread-scope source in the SAME chat // (different 话题) is a legitimate cross-topic move, so refuse only when the // source and target anchors are identical. @@ -1315,6 +1337,9 @@ export async function handleCardAction(data: CardActionData, deps: CardHandlerDe if (r.error === 'adopt_not_relayable') { return { toast: { type: 'error', content: t('card.relay.toast_adopt_not_relayable', undefined, loc) } }; } + if (r.error === 'riff_not_relayable') { + return { toast: { type: 'error', content: t('card.relay.toast_riff_not_relayable', undefined, loc) } }; + } if (r.error === 'worker_busy') { return { toast: { type: 'error', content: t('card.relay.toast_worker_busy', undefined, loc) } }; } @@ -1586,8 +1611,10 @@ export async function handleCardAction(data: CardActionData, deps: CardHandlerDe logger.info(`[${tag(ds)}] voice_summary blocked because worker is busy: ${ds.lastScreenStatus ?? 'unknown'}`); return { toast: { type: 'warning', content: t('card.voice.toast_worker_busy', undefined, locDs) } }; } - voicedCardIds.add(dedupeKey); - if (voicedCardIds.size > 5000) { voicedCardIds.clear(); voicedCardIds.add(dedupeKey); } + if ((!ds.worker || ds.worker.killed) && hasProtectedSessionMutationOwnership(ds)) { + logger.info(`[${tag(ds)}] voice_summary deferred behind durable opening ownership`); + return { toast: { type: 'warning', content: t('card.voice.toast_worker_busy', undefined, locDs) } }; + } const instruction = voiceSummaryInstruction(locDs); const voiceInput = { content: instruction, @@ -1598,8 +1625,26 @@ export async function handleCardAction(data: CardActionData, deps: CardHandlerDe 'application', ), }; - if (ds.worker && !ds.worker.killed) sendWorkerInput(ds, voiceInput); - else forkWorker(ds, voiceInput, ds.hasHistory); + let accepted = false; + try { + if (ds.worker && !ds.worker.killed) accepted = sendWorkerInput(ds, voiceInput); + else { + forkWorker(ds, voiceInput, ds.hasHistory); + accepted = true; + } + } catch (err) { + logger.warn( + `[${tag(ds)}] voice_summary failed before acceptance: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + } + if (!accepted) { + return { toast: { type: 'warning', content: t('card.voice.toast_worker_busy', undefined, locDs) } }; + } + // Burn the replay key only after live IPC or the durable activation tail + // accepted the exact instruction. + voicedCardIds.add(dedupeKey); + if (voicedCardIds.size > 5000) { voicedCardIds.clear(); voicedCardIds.add(dedupeKey); } logger.info(`[${tag(ds)}] voice_summary triggered by ${operatorOpenId ?? '?'}`); return { toast: { type: 'success', content: t('card.voice.toast_wait', undefined, locDs) } }; } @@ -1610,26 +1655,63 @@ export async function handleCardAction(data: CardActionData, deps: CardHandlerDe // which violates the bridge invariant. Defense in depth — buildSessionCard // already omits the restart button when adoptMode=true, but a stale // pre-fix card or a malformed action payload could still arrive. - const locDs = localeForBot(ds.larkAppId); - if (ds.adoptedFrom) { - logger.warn(`[${tag(ds)}] Rejected restart on adopt session — would kill user's pane`); + const targetSessionId = ds.session.sessionId; + const targetAppId = ds.larkAppId; + const locDs = localeForBot(targetAppId); + const restart = await withBotTurnMutation(targetAppId, async () => { + const current = [...activeSessions.values()].find( + candidate => candidate.session.sessionId === targetSessionId + && candidate.session.status === 'active', + ); + if (!current) return { status: 'gone' as const }; + if (current.adoptedFrom || current.initConfig?.adoptMode) { + logger.warn(`[${tag(current)}] Rejected restart on adopt session — would kill user's pane`); + return { status: 'adopted' as const }; + } + if ((current.initConfig?.backendType ?? current.session.backendType) === 'riff' + || sessionCliId(current) === 'riff') { + logger.warn(`[${tag(current)}] Rejected restart on Riff session — lineage owner retained`); + return { status: 'riff' as const }; + } + if (hasProtectedSessionMutationOwnership(current)) { + return { status: 'pending' as const }; + } + const effectiveCliId = sessionCliId(current); + if (current.worker && !current.worker.killed) { + logger.info(`[${tag(current)}] Restart via card button`); + current.worker.send({ type: 'restart', reason: 'operator' } as DaemonToWorker); + return { status: 'restarted' as const, current, effectiveCliId }; + } + logger.info(`[${tag(current)}] Re-forking worker via card button`); + forkWorker(current, '', current.hasHistory); + return { status: 'reforked' as const, current, effectiveCliId }; + }); + if (restart.status === 'gone') { + return { toast: { type: 'warning', content: t('card.action.session_gone', undefined, locDs) } }; + } + if (restart.status === 'adopted') { await sessionReply(rootId, t('card.action.adopt_no_restart', undefined, locDs)); return; } - const botCfg = getBot(ds.larkAppId).config; - const effectiveCliId = sessionCliId(ds); - if (ds.worker) { - logger.info(`[${tag(ds)}] Restart via card button`); - ds.worker.send({ type: 'restart' } as DaemonToWorker); - const cliName = getCliDisplayName(effectiveCliId); + if (restart.status === 'riff') { + await sessionReply(rootId, t('cmd.restart.riff_unsupported', undefined, locDs)); + return; + } + if (restart.status === 'pending') { + await sessionReply( + rootId, + '当前 Codex App 仍有未结算消息，暂不能重启；请等待本轮完成或关闭会话。', + ); + return; + } + if (restart.status === 'restarted') { + const cliName = getCliDisplayName(restart.effectiveCliId); const restartedMsg = t('card.action.restarted', { cliName }, locDs); - await deliverEphemeralOrReply(ds, operatorOpenId, restartedMsg, 'text', () => sessionReply(rootId, restartedMsg)); + await deliverEphemeralOrReply(restart.current, operatorOpenId, restartedMsg, 'text', () => sessionReply(rootId, restartedMsg)); } else { - logger.info(`[${tag(ds)}] Re-forking worker via card button`); - forkWorker(ds, '', ds.hasHistory); - const cliName = getCliDisplayName(effectiveCliId); + const cliName = getCliDisplayName(restart.effectiveCliId); const restartedFreshMsg = t('card.action.restarted_fresh', { cliName }, locDs); - await deliverEphemeralOrReply(ds, operatorOpenId, restartedFreshMsg, 'text', () => sessionReply(rootId, restartedFreshMsg)); + await deliverEphemeralOrReply(restart.current, operatorOpenId, restartedFreshMsg, 'text', () => sessionReply(rootId, restartedFreshMsg)); // DM card will be sent by the ready handler when worker starts } } @@ -1640,13 +1722,38 @@ export async function handleCardAction(data: CardActionData, deps: CardHandlerDe // 会话」却静默无反应会让人以为按钮坏了，给一条失败 toast（成功路径不弹，已关卡即反馈）。 return { toast: { type: 'warning', content: t('card.action.session_gone', undefined, localeForBot(larkAppId)) } }; } - const botCfg = getBot(ds.larkAppId).config; - // Build the closed card BEFORE killWorker/closeSession — it reads the - // live session's identity off `ds`. - const card = buildClosedSessionCard(ds, localeForBot(ds.larkAppId)); - killWorker(ds); - sessionStore.closeSession(ds.session.sessionId); - activeSessions.delete(sKey); + const targetSessionId = ds.session.sessionId; + const closed = await withBotTurnMutation(ds.larkAppId, async () => { + // Card payload roots survive transfers. Re-resolve by immutable session + // id after draining admissions, then let the pool remove the target's + // CURRENT activeSessionKey. Never delete the stale payload root, which + // may now belong to an unrelated pending FIFO owner. + const current = [...activeSessions.values()].find( + candidate => candidate.session.sessionId === targetSessionId, + ); + if (!current) return undefined; + const botCfg = getBot(current.larkAppId).config; + const card = buildClosedSessionCard(current, localeForBot(current.larkAppId)); + const result = await closeWorkerPoolSession(targetSessionId); + if (!result.ok) return { status: 'failed' as const, current, failure: result }; + return { status: 'closed' as const, current, botCfg, card }; + }); + if (!closed) { + return { toast: { type: 'warning', content: t('card.action.session_gone', undefined, localeForBot(larkAppId)) } }; + } + if (closed.status === 'failed') { + logger.warn( + `[${tag(closed.current)}] close card refused: ${closed.failure.error} ` + + `task=${closed.failure.taskId}`, + ); + return { + toast: { + type: 'error', + content: 'Riff 远端任务取消失败；会话仍保持活跃，请重试关闭。', + }, + }; + } + const { current, botCfg, card } = closed; // The closed card carries session title / CLI name / workingDir / resume // command. In private-card mode those must not leak to the group — send the // closed card ephemeral to the same owner audience instead. No group @@ -1655,15 +1762,15 @@ export async function handleCardAction(data: CardActionData, deps: CardHandlerDe // clicked, so a card built in private mode stays ephemeral even if the // bot's `privateCard` config was turned off in the meantime. if (value?.visibility === 'private' || botCfg.privateCard) { - const audience = resolvePrivateCardAudience(ds); + const audience = resolvePrivateCardAudience(current); for (const openId of audience) { - await sendEphemeralCard(ds.larkAppId, ds.chatId, openId, card).catch(err => - logger.warn(`[${tag(ds)}] private close card ephemeral send to ${openId.substring(0, 8)}… failed: ${err}`)); + await sendEphemeralCard(current.larkAppId, current.chatId, openId, card).catch(err => + logger.warn(`[${tag(current)}] private close card ephemeral send to ${openId.substring(0, 8)}… failed: ${err}`)); } - logger.info(`[${tag(ds)}] Closed via card button (private close card → ${audience.length} owner(s))`); + logger.info(`[${tag(current)}] Closed via card button (private close card → ${audience.length} owner(s))`); } else { - await deliverEphemeralOrReply(ds, operatorOpenId, card, 'interactive', () => sessionReply(rootId, card, 'interactive')); - logger.info(`[${tag(ds)}] Closed via card button`); + await deliverEphemeralOrReply(current, operatorOpenId, card, 'interactive', () => sessionReply(rootId, card, 'interactive')); + logger.info(`[${tag(current)}] Closed via card button`); } } @@ -1697,9 +1804,18 @@ export async function handleCardAction(data: CardActionData, deps: CardHandlerDe } if (actionType === 'disconnect' && ds) { - killWorker(ds); - sessionStore.closeSession(ds.session.sessionId); - activeSessions.delete(sKey); + const targetSessionId = ds.session.sessionId; + const disconnected = await withBotTurnMutation(ds.larkAppId, async () => { + const current = [...activeSessions.values()].find( + candidate => candidate.session.sessionId === targetSessionId, + ); + if (!current) return undefined; + await closeWorkerPoolSession(targetSessionId); + return current; + }); + if (!disconnected) { + return { toast: { type: 'warning', content: t('card.action.session_gone', undefined, localeForBot(larkAppId)) } }; + } await sessionReply(rootId, t('card.action.disconnected', undefined, localeForBot(ds.larkAppId))); logger.info(`[${tag(ds)}] Disconnected (adopt) via card button`); } @@ -1725,6 +1841,36 @@ export async function handleCardAction(data: CardActionData, deps: CardHandlerDe return; } + const retryCodexAppInput = ds.lastCodexAppInput + ? (({ clientUserMessageId: _priorMessageId, ...input }) => input)(ds.lastCodexAppInput) + : undefined; + const retryInput = { + content: cliInput, + ...(retryCodexAppInput ? { codexAppInput: retryCodexAppInput } : {}), + }; + if ((!ds.worker || ds.worker.killed) && hasProtectedSessionMutationOwnership(ds)) { + await sessionReply(rootId, t('card.action.retry_last_task_unavailable', undefined, locDs)); + return; + } + let accepted = false; + try { + if (ds.worker && !ds.worker.killed) accepted = sendWorkerInput(ds, retryInput); + else { + forkWorker(ds, retryInput, ds.hasHistory); + accepted = true; + } + } catch (err) { + logger.warn( + `[${tag(ds)}] retry_last_task failed before acceptance: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + } + if (!accepted) { + await sessionReply(rootId, t('card.action.retry_last_task_unavailable', undefined, locDs)); + return; + } + // Only now consume the one-shot retry state and advertise working: live + // IPC or the durable activation tail already owns the exact retry input. clearUsageLimitState(ds); ds.lastScreenStatus = 'working'; ds.streamCardPending = true; @@ -1755,16 +1901,6 @@ export async function handleCardAction(data: CardActionData, deps: CardHandlerDe ); scheduleCardPatch(ds, cardJson); } - - const retryCodexAppInput = ds.lastCodexAppInput - ? (({ clientUserMessageId: _priorMessageId, ...input }) => input)(ds.lastCodexAppInput) - : undefined; - const retryInput = { - content: cliInput, - ...(retryCodexAppInput ? { codexAppInput: retryCodexAppInput } : {}), - }; - if (ds.worker && !ds.worker.killed) sendWorkerInput(ds, retryInput); - else forkWorker(ds, retryInput, ds.hasHistory); logger.info(`[${tag(ds)}] Retrying last task after usage limit`); if (cardJson) { try { return JSON.parse(cardJson); } catch { /* fall through */ } @@ -2181,31 +2317,27 @@ export async function handleCardAction(data: CardActionData, deps: CardHandlerDe return { toast: { type: 'info', content: t('cmd.repo.card_already_consumed', undefined, locDs) } }; } if (ds.pendingRepo) { - if (ds.worktreeCreating || ds.pendingRepoCommitInFlight) { - return { toast: { type: 'info', content: t('cmd.repo.worktree_in_progress', undefined, locDs) } }; - } const cwd = getSessionWorkingDir(ds); - // Reuse the same claimed pending->worker transition as a normal repo - // selection. This keeps buffering active across prompt preparation and - // makes skip/select races single-winner. Do NOT pin the resolved default - // cwd (often $HOME) onto session.workingDir — that would let sibling - // bots inherit HOME instead of getting their own repo card. Confirmation - // + card withdraw run under the claim inside commitRepoSelection. - await commitRepoSelection( - { ds, rootId, cardMessageId, larkAppId, operatorOpenId, activeSessions, sessionReply }, + // "Start directly" is the same pending-start commit as choosing a + // directory, just pinned to the current/default cwd. Reusing the shared + // helper gives this card path the bot mutation, exact-owner recheck, + // buffered-sidecar handling, and fork-before-release ordering. + const started = await commitRepoSelection( + { ds, rootId, cardMessageId, larkAppId: larkAppId ?? ds.larkAppId, operatorOpenId, activeSessions, sessionReply }, + cwd, cwd, - pathBasename(cwd) || cwd, { suppressConfirmReply: true, confirmReplyText: t('cmd.skip.opened', { cwd }, locDs), pinWorkingDir: false, }, ); - if (!ds.pendingRepo) { + if (started) { logger.info(`[${tag(ds)}] Skip repo, spawning CLI in ${cwd}`); } } else { await sessionReply(rootId, t('card.action.continue_using_current_repo', { cwd: getSessionWorkingDir(ds) }, locDs)); + markRepoCardConsumed(ds, cardMessageId); if (cardMessageId && larkAppId) deleteMessage(larkAppId, cardMessageId); ds.repoCardMessageId = undefined; } @@ -2248,10 +2380,12 @@ export async function handleCardAction(data: CardActionData, deps: CardHandlerDe // re-send a fresh repo card in the new mode — a form can't ride an // in-place patch, so the old card is withdrawn and a new one posted. const locDs = localeForBot(ds.larkAppId); - // Same active-card gate as skip/manual/worktree: a stale card must not - // flip bot config or replace the live repo card after claim/restart. - if (cardMessageId && !isActiveRepoCard(ds, cardMessageId)) { - return { toast: { type: 'info', content: t('cmd.repo.card_already_consumed', undefined, locDs) } }; + if (!cardMessageId || !ds.repoCardMessageId || cardMessageId !== ds.repoCardMessageId) { + logger.warn( + `[${tag(ds)}] Ignoring stale worktree-toggle picker ${cardMessageId ?? 'none'} ` + + `(current=${ds.repoCardMessageId ?? 'none'})`, + ); + return { toast: { type: 'warning', content: t('card.repo.toast_stale_picker', undefined, locDs) } }; } const spec = findConfigField('worktreeMultiPicker'); if (!spec) return; @@ -2259,18 +2393,47 @@ export async function handleCardAction(data: CardActionData, deps: CardHandlerDe const r = await applyConfigField(ds.larkAppId, spec, next); if (!r.ok) return { toast: { type: 'error', content: t('cmd.config.write_failed', { reason: r.reason }, locDs) } }; const projects = lastRepoScan.get(ds.chatId) ?? []; - // await so a rejected delete is caught here (not an unhandled rejection); - // a missing/already-gone card is fine — we post the fresh one regardless. - if (ds.repoCardMessageId && ds.larkAppId) { try { await deleteMessage(ds.larkAppId, ds.repoCardMessageId); } catch { /* card already gone */ } } const newCard = buildRepoSelectCard(projects, getSessionWorkingDir(ds), rootId, locDs, next); - ds.repoCardMessageId = await sessionReply(rootId, newCard, 'interactive'); + const oldCardMessageId = ds.repoCardMessageId; + let newCardMessageId: string; + try { + newCardMessageId = await sessionReply(rootId, newCard, 'interactive'); + } catch (err) { + logger.warn( + `[${tag(ds)}] Failed to publish replacement repo picker; old card remains authoritative: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + return { toast: { type: 'error', content: t('cmd.config.write_failed', { reason: err instanceof Error ? err.message : String(err) }, locDs) } }; + } + try { + // Switch durable authority before changing runtime identity or + // withdrawing the old card. A failed write leaves the old picker fully + // usable and makes the newly-published card stale by exact-ID guard. + persistPendingRepoCardMessageId(ds, newCardMessageId); + } catch (err) { + logger.error( + `[${tag(ds)}] Failed to persist replacement repo picker ${newCardMessageId}; ` + + `old picker ${oldCardMessageId} retained: ` + + `${err instanceof Error ? err.message : String(err)}`, + ); + return { toast: { type: 'error', content: t('cmd.config.write_failed', { reason: err instanceof Error ? err.message : String(err) }, locDs) } }; + } + ds.repoCardMessageId = newCardMessageId; + // The fresh ID is now durable. Withdrawal is best-effort and cannot + // invalidate the new authority if Lark reports the old card missing. + try { await deleteMessage(ds.larkAppId, oldCardMessageId); } + catch { /* card already gone */ } return { toast: { type: 'info', content: t(next ? 'card.repo.toast_worktree_mode_switched' : 'card.repo.toast_worktree_mode_switched_back', undefined, locDs) } }; } if (actionType === 'repo_worktree_submit' && ds) { const locDs = localeForBot(ds.larkAppId); - if (cardMessageId && !isActiveRepoCard(ds, cardMessageId)) { - return { toast: { type: 'info', content: t('cmd.repo.card_already_consumed', undefined, locDs) } }; + if (!cardMessageId || !ds.repoCardMessageId || cardMessageId !== ds.repoCardMessageId) { + logger.warn( + `[${tag(ds)}] Ignoring stale worktree-submit picker ${cardMessageId ?? 'none'} ` + + `(current=${ds.repoCardMessageId ?? 'none'})`, + ); + return { toast: { type: 'warning', content: t('card.repo.toast_stale_picker', undefined, locDs) } }; } const selectedPaths = stringListFromLarkMultiSelect(action?.form_value?.repo_worktree_paths); if (selectedPaths.length === 0) { @@ -2486,11 +2649,22 @@ export async function handleCardAction(data: CardActionData, deps: CardHandlerDe return; } - // Only the live posted card may drive selection. Covers: already-consumed, - // wrong/old card, and post-restart (repoCardMessageId is in-memory). Must - // reject before mid-session switch can killWorker. - if (cardMessageId && !isActiveRepoCard(targetDs, cardMessageId)) { - return { toast: { type: 'info', content: t('cmd.repo.card_already_consumed', undefined, localeForBot(targetDs.larkAppId)) } }; + // The picker message id is the capability for every repo dropdown. Check it + // before slug generation, worktreeCreating, git creation/push, or commit. + // In particular the direct single-select `repo_worktree` callback does not + // pass through the form-submit branch above, so relying on the later commit + // check would allow a stale card to create an orphan worktree first. + if (!cardMessageId || !targetDs.repoCardMessageId || cardMessageId !== targetDs.repoCardMessageId) { + logger.warn( + `[${tag(targetDs)}] Ignoring stale ${repoKey} picker ${cardMessageId ?? 'none'} ` + + `(current=${targetDs.repoCardMessageId ?? 'none'})`, + ); + return { + toast: { + type: 'warning', + content: t('card.repo.toast_stale_picker', undefined, localeForBot(targetDs.larkAppId)), + }, + }; } // 权限边界：pendingRepo（首次选 repo 才能 spawn）放行「会话发起人 或 canOperate」， diff --git a/src/im/lark/client.ts b/src/im/lark/client.ts index 1a063e8ce..e32d3b1f6 100644 --- a/src/im/lark/client.ts +++ b/src/im/lark/client.ts @@ -2,10 +2,16 @@ import { readFileSync, writeFileSync, createWriteStream, mkdirSync, existsSync } import { dirname, extname, basename, join } from 'node:path'; import { pipeline } from 'node:stream/promises'; import { Client } from '@larksuiteoapi/node-sdk'; -import { getBotClient, getAllBots, getBot, formatLarkError } from '../../bot-registry.js'; +import { + configureLarkClientHttpTimeout, + getBotClient, + getAllBots, + getBot, + formatLarkError, +} from '../../bot-registry.js'; import { loadBotConfigs } from '../../bot-registry.js'; import { config } from '../../config.js'; -import { emitHookEvent } from '../../services/hook-runner.js'; +import { emitHookEvent, type ManagedHookOrigin } from '../../services/hook-runner.js'; import { logger } from '../../utils/logger.js'; import { BoundedMap } from '../../utils/bounded-map.js'; import { resolveUserToken } from '../../utils/user-token.js'; @@ -67,11 +73,16 @@ function getAllBotClients() { // 降级用注册表里的配置，`botmux bots list` 等只读探测照常可用。 cfgs = getAllBots().map((b) => b.config); } - allBotClients = cfgs.map((cfg) => ({ - appId: cfg.larkAppId, - cliId: cfg.cliId, - client: new Client({ appId: cfg.larkAppId, appSecret: cfg.larkAppSecret, domain: sdkDomain(normalizeBrand(cfg.brand as any)), logger: probeLarkLogger }), - })); + allBotClients = cfgs.map((cfg) => { + const client = new Client({ + appId: cfg.larkAppId, + appSecret: cfg.larkAppSecret, + domain: sdkDomain(normalizeBrand(cfg.brand as any)), + logger: probeLarkLogger, + }); + configureLarkClientHttpTimeout(client); + return { appId: cfg.larkAppId, cliId: cfg.cliId, client }; + }); } return allBotClients; } @@ -128,6 +139,31 @@ export interface OutboundMessageOptions { * Lark deduplicates the message, but the local outbound hook is a separate * side effect and must not be fired twice. */ suppressHook?: boolean; + /** Fence the distinct post-provider hook effect. A failure drops only the + * hook because the Lark message has already been accepted and must not be + * reported as failed (which would invite a duplicate retry). */ + beforeHook?: () => void | Promise; + /** Frozen protected origin used by read-isolated hook forwarding. */ + hookOrigin?: ManagedHookOrigin; +} + +async function emitOutboundHookIfAllowed( + options: OutboundMessageOptions | undefined, + event: 'outbound.send' | 'outbound.reply', + payload: Record, +): Promise { + if (options?.suppressHook) return; + try { + await options?.beforeHook?.(); + } catch (err) { + logger.warn(`Dropped ${event} hook after authority changed: ${err instanceof Error ? err.message : String(err)}`); + return; + } + if (options?.hookOrigin) { + emitHookEvent(event, payload, { managedOrigin: options.hookOrigin }); + } else { + emitHookEvent(event, payload); + } } export async function sendMessage( @@ -168,8 +204,7 @@ export async function sendMessage( const messageId = res.data?.message_id; if (!messageId) throw new Error('No message_id in response'); logger.info(`Sent message ${messageId} to chat ${chatId}`); - if (!options?.suppressHook) { - emitHookEvent('outbound.send', { + await emitOutboundHookIfAllowed(options, 'outbound.send', { ...hookContext, larkAppId, chatId, @@ -178,7 +213,6 @@ export async function sendMessage( uuid, content, }); - } return messageId; } @@ -228,8 +262,7 @@ export async function replyMessage( const replyId = res.data?.message_id; if (!replyId) throw new Error('No message_id in reply response'); logger.info(`Replied ${replyId} to message ${messageId} [msgType=${msgType}, replyInThread=${replyInThread}]`); - if (!options?.suppressHook) { - emitHookEvent('outbound.reply', { + await emitOutboundHookIfAllowed(options, 'outbound.reply', { ...hookContext, larkAppId, messageId, @@ -239,7 +272,6 @@ export async function replyMessage( uuid, content, }); - } return replyId; } diff --git a/src/im/lark/doc-comment.ts b/src/im/lark/doc-comment.ts index 7264ff64b..98f639710 100644 --- a/src/im/lark/doc-comment.ts +++ b/src/im/lark/doc-comment.ts @@ -181,6 +181,14 @@ interface DriveCallOpts { preferTenant?: boolean; /** 订阅 API 专用：把 1069603 连同实际失败身份归一化为结构化异常。 */ classifySubscriptionPermission?: boolean; + /** Managed-origin fence invoked immediately before every actual tenant/user + * provider request. It deliberately sits outside fallback catch blocks so a + * revoked origin aborts instead of being mistaken for an identity failure. */ + beforeProviderEffect?: () => void | Promise; +} + +export interface DocProviderEffectOptions { + beforeProviderEffect?: () => void | Promise; } const DOC_SUBSCRIPTION_PERMISSION_CODE = 1069603; @@ -297,8 +305,13 @@ async function driveApiCall(larkAppId: string, opts: DriveCallOpts): Promise { + // Token resolution may refresh an expired user token over the network. + // Fence both that refresh and the actual Drive request: revocation in + // either interval must stop before the next provider-visible effect. + await opts.beforeProviderEffect?.(); const userToken = await resolveUserToken(bot.config.larkAppId, bot.config.larkAppSecret, brand); if (!userToken) throw new UserTokenMissingError('该操作需要 User Token（请在话题中 /login 授权）。'); + await opts.beforeProviderEffect?.(); return fetchWithUserToken(brand, userToken, opts); }; @@ -306,6 +319,7 @@ async function driveApiCall(larkAppId: string, opts: DriveCallOpts): Promise { const elements = buildCommentElements(text, mentionOpenId); let res: any; @@ -571,13 +589,14 @@ export async function replyToDocComment( params: { file_type: file.fileType, user_id_type: 'open_id' }, data: { content: { elements } }, preferTenant: true, // 回复显示为 bot 本身（应用身份）；bot 无访问权时回退 user + beforeProviderEffect: options.beforeProviderEffect, }); } catch (err) { // 有的评论不允许被回复（飞书 1069302：全文评论 / 已解决 / 文档评论设置受限）。 // 退回新建一条全文评论，保证 bot 的答复总能落到文档（不嵌套但仍在评论区）。 if (isReplyNotAllowed(err)) { logger.warn(`[doc-comment] comment=${commentId.slice(0, 12)} 不允许回复，退回新建全文评论`); - const c = await createDocComment(larkAppId, file, text, mentionOpenId); + const c = await createDocComment(larkAppId, file, text, mentionOpenId, options); return { replyId: c.replyId, commentId: c.commentId }; } throw err; @@ -586,7 +605,7 @@ export async function replyToDocComment( if (res?.code !== 0) { if (isReplyNotAllowed(res)) { logger.warn(`[doc-comment] comment=${commentId.slice(0, 12)} 不允许回复(code=${res?.code})，退回新建全文评论`); - const c = await createDocComment(larkAppId, file, text, mentionOpenId); + const c = await createDocComment(larkAppId, file, text, mentionOpenId, options); return { replyId: c.replyId, commentId: c.commentId }; } throw new Error(`回复评论 失败: ${res?.msg ?? 'unknown'} (code: ${res?.code})`); @@ -624,6 +643,7 @@ export async function createDocComment( file: ResolvedDocFile, text: string, mentionOpenId?: string, + options: DocProviderEffectOptions = {}, ): Promise<{ commentId: string; replyId?: string }> { const elements = buildCommentElements(text, mentionOpenId); const res = await driveApiCall(larkAppId, { @@ -632,6 +652,7 @@ export async function createDocComment( params: { file_type: file.fileType, user_id_type: 'open_id' }, data: { reply_list: { replies: [{ content: { elements } }] } }, preferTenant: true, // 评论显示为 bot 本身（应用身份）；bot 无访问权时回退 user + beforeProviderEffect: options.beforeProviderEffect, }); const data = ensureOk(res, '发表评论'); const commentId: string = data?.comment_id ?? ''; @@ -678,6 +699,7 @@ export async function addCommentReaction( commentId: string, replyId: string, reactionType: string, + options: DocProviderEffectOptions = {}, ): Promise { try { const res = await driveApiCall(larkAppId, { @@ -686,6 +708,7 @@ export async function addCommentReaction( params: { file_type: file.fileType }, data: { action: 'add', comment_id: commentId, reply_id: replyId, reaction_type: reactionType }, preferTenant: true, + beforeProviderEffect: options.beforeProviderEffect, }); const reactionId: string | undefined = res?.data?.reaction_id; if (reactionId) { @@ -710,6 +733,7 @@ export async function removeCommentReaction( commentId: string, replyId: string, reactionId: string, + options: DocProviderEffectOptions = {}, ): Promise { if (!reactionId) return; try { @@ -723,6 +747,7 @@ export async function removeCommentReaction( reaction_id: reactionId, }, preferTenant: true, + beforeProviderEffect: options.beforeProviderEffect, }); logger.info(`[doc-comment] removed reaction=${reactionId.slice(0, 12)} on reply=${replyId.slice(0, 12)}`); } catch (err) { diff --git a/src/im/lark/event-dispatcher.ts b/src/im/lark/event-dispatcher.ts index bdfc51fbc..e1aa08bdb 100644 --- a/src/im/lark/event-dispatcher.ts +++ b/src/im/lark/event-dispatcher.ts @@ -600,6 +600,28 @@ function eventIdForKey(data: any): string | undefined { return data?.event_id ?? data?.uuid ?? data?.header?.event_id ?? data?.event?.event_id; } +/** + * Synchronous first-stage routing lane for inbound Lark messages. + * + * Routing can await bot identity, chat mode, oncall binding, summary expansion, + * and VC catch-up before it discovers the canonical daemon anchor. Reserving + * only at that later anchor lets a faster N+1 reach the session before N. + * + * This barrier must be chat-wide, not thread-wide: a topic seed has no thread_id + * and initially looks chat-shaped, while its first reply carries thread_id; both + * later canonicalize to the seed message_id. Shared-topic aliases likewise fold + * a thread-shaped event back into a chat owner. Per-thread raw lanes therefore + * cannot prove arrival order. The barrier is released as soon as canonical work + * is synchronously enqueued below, so handlers for distinct canonical topics can + * still run concurrently. + */ +export function rawMessageIngressAnchor(larkAppId: string, message: any): string { + const chatId = typeof message?.chat_id === 'string' && message.chat_id.trim() + ? message.chat_id.trim() + : '__chatless__'; + return `lark-message-routing:${larkAppId}:${chatId}`; +} + // card.action.trigger is a synchronous callback with a 3s deadline and NO // re-push (unlike events). If a handler (e.g. restart, which spawns a worker) // might exceed the budget, we ACK before 3s with a toast and patch the card @@ -2055,14 +2077,13 @@ export function startLarkEventDispatcher(larkAppId: string, larkAppSecret: strin const sessionsReady = new Promise(resolve => { resolveSessionsReady = resolve; }); if (sessionsReadyApps.has(larkAppId)) resolveSessionsReady(); sessionsReadyBarriers.set(larkAppId, { promise: sessionsReady, resolve: resolveSessionsReady }); - const dispatchHumanMessage = async (payload: PendingForwardTopicPayload): Promise => { - await serializeByAnchor(payload.ctx.anchor, () => { + const dispatchHumanMessage = (payload: PendingForwardTopicPayload): Promise => + serializeByAnchor(payload.ctx.anchor, () => { const ownsSession = handlers.isSessionOwner?.(payload.ctx.anchor, larkAppId) ?? payload.ownsSession; return ownsSession ? handlers.handleThreadReply(payload.data, payload.ctx) : handlers.handleNewTopic(payload.data, payload.ctx); - }); - }; + }, 0); const dispatchPersistedForwardFollowup = async ( seedMessageId: string, payload: PendingForwardTopicPayload, @@ -2102,6 +2123,26 @@ export function startLarkEventDispatcher(larkAppId: string, larkAppSecret: strin } } const seedRoutingGates = new Map; complete: () => void }>(); + const waitForSeedRoutingGate = async (messageId: string): Promise => { + const gate = seedRoutingGates.get(messageId); + if (!gate) return; + const timeoutMs = Math.max(1, config.daemon.forwardFollowupWaitMs); + let timer: NodeJS.Timeout | undefined; + const outcome = await Promise.race([ + gate.ready.then(() => 'ready' as const), + new Promise<'timeout'>(resolve => { + timer = setTimeout(() => resolve('timeout'), timeoutMs); + timer.unref?.(); + }), + ]); + if (timer) clearTimeout(timer); + if (outcome === 'timeout') { + logger.warn( + `[forward-followup] seed routing gate timed out after ${timeoutMs}ms ` + + `for root=${messageId.substring(0, 12)}; continuing without pairing`, + ); + } + }; const registerSeedRoutingGate = (messageId: string) => { let resolveReady!: () => void; const ready = new Promise(resolve => { resolveReady = resolve; }); @@ -2188,8 +2229,8 @@ export function startLarkEventDispatcher(larkAppId: string, larkAppSecret: strin // Serialize per anchor so back-to-back messages to the same thread // (e.g. dispatch's /repo prime + brief kickoff) don't interleave with // the first's async session-spawn. See anchor-serializer.ts. - await serializeByAnchor(ctx.anchor, () => - handlers.handleThreadReply(data, { ...ctx, chatId, messageId, chatType, larkAppId })) + void serializeByAnchor(ctx.anchor, () => + handlers.handleThreadReply(data, { ...ctx, chatId, messageId, chatType, larkAppId }), 0) .catch(err => logger.error(`Error handling message event: ${err}`)); return; } @@ -2319,8 +2360,8 @@ export function startLarkEventDispatcher(larkAppId: string, larkAppSecret: strin logger.info(`Bot-to-bot @mention detected (scope=${ctx.scope}): routing to handleThreadReply`); // Serialize per anchor — a sub-bot dispatched a /repo prime + kickoff // back-to-back into this thread must be handled in order, not raced. - await serializeByAnchor(ctx.anchor, () => - handlers.handleThreadReply(data, { ...ctx, chatId, messageId, chatType, larkAppId, replyRootId })) + void serializeByAnchor(ctx.anchor, () => + handlers.handleThreadReply(data, { ...ctx, chatId, messageId, chatType, larkAppId, replyRootId }), 0) .catch(err => logger.error(`Error handling bot @mention: ${err}`)); return; } @@ -2592,6 +2633,7 @@ export function startLarkEventDispatcher(larkAppId: string, larkAppSecret: strin : ''; const isControlCommand = strippedRoutingText.startsWith('/'); let pairedForwardSeed; + let stalePendingSeed; // Require isAllowed before pairing: a root-linked clarification from a // sender who was /revoked within the grace window must not consume the // seed from the buffer or overwrite the durable paired record. The seed @@ -2599,7 +2641,7 @@ export function startLarkEventDispatcher(larkAppId: string, larkAppSecret: strin // only arise under never/ambient modes, where a legitimate merge always // requires isAllowed anyway. if (senderOpenId && isAllowed && message.root_id && !isControlCommand) { - await seedRoutingGates.get(message.root_id)?.ready; + await waitForSeedRoutingGate(message.root_id); const pairingInput = { larkAppId, chatId, @@ -2614,17 +2656,7 @@ export function startLarkEventDispatcher(larkAppId: string, larkAppSecret: strin if (pairingDelayEnabled && !ambientRedirect) { pairedForwardSeed = forwardFollowups.take(pairingInput); } else if (!pairingDelayEnabled) { - const stalePendingSeed = forwardFollowups.take(pairingInput); - if (stalePendingSeed) { - try { - await dispatchPersistedForwardFollowup(stalePendingSeed.messageId, stalePendingSeed.payload); - } catch (err) { - logger.warn( - `[forward-followup] failed to flush stale seed=${stalePendingSeed.messageId.substring(0, 12)}; ` + - `continuing current msg=${messageId.substring(0, 12)}: ${err}`, - ); - } - } + stalePendingSeed = forwardFollowups.take(pairingInput); } } if (pairedForwardSeed) { @@ -2828,16 +2860,40 @@ export function startLarkEventDispatcher(larkAppId: string, larkAppSecret: strin } catch (err) { logger.warn(`[forward-followup] failed to persist paired payload before dispatch: ${err}`); } - await dispatchPersistedForwardFollowup(pairedForwardSeed.messageId, payload) + void dispatchPersistedForwardFollowup(pairedForwardSeed.messageId, payload) .catch(err => logger.error(`Error handling paired message event: ${err}`)); return; } + if (stalePendingSeed) { + // The mention mode changed while a seed was held. Release the raw chat + // ingress lane immediately, but enqueue the stale seed before the + // current reply once session restoration is complete. Both calls append + // synchronously to the same canonical FIFO; their handler completion is + // deliberately not awaited by the raw lane. + void sessionsReady.then(() => { + void dispatchHumanMessage(stalePendingSeed.payload) + .then(() => removeForwardFollowup(larkAppId, stalePendingSeed.messageId)) + .catch(err => logger.warn( + `[forward-followup] failed to flush stale seed=${stalePendingSeed.messageId.substring(0, 12)}; ` + + `continuing current msg=${messageId.substring(0, 12)}: ${err}`, + )); + void dispatchHumanMessage(payload) + .catch(err => logger.error(`Error handling message event: ${err}`)); + }).catch(err => logger.error(`Error awaiting restored sessions for stale seed: ${err}`)); + return; + } + // Serialize per anchor so two messages to the same thread/chat are // processed in arrival order — never concurrently. Without this a fast // second message interleaves with the first's async session-spawn and is // dropped (worker-not-ready → re-fork branch). See anchor-serializer.ts. - await dispatchHumanMessage(payload) + // The chat-wide ingress lane protects only asynchronous routing. Once + // canonical work has been synchronously appended to its own anchor FIFO, + // release the raw lane so independent topics in the same chat can run + // concurrently. Same-anchor handlers remain strictly serialized by + // dispatchHumanMessage's canonical queue. + void dispatchHumanMessage(payload) .catch(err => logger.error(`Error handling message event: ${err}`)); } catch (err) { logger.error(`Error handling message event: ${err}`); @@ -2905,9 +2961,21 @@ export function startLarkEventDispatcher(larkAppId: string, larkAppSecret: strin const claim = messageIdForKey ? () => claimMessageOnce(larkAppId, messageIdForKey) : () => claimEventOnce(eventKey); - let seedRoutingGate: { ready: Promise; complete: () => void } | undefined; - const scheduled = scheduleAckSafeEvent(eventKey, () => processMessageEvent(data, seedRoutingGate), 'message event', claim); const rawMessage = data?.message; + const ingressAnchor = rawMessageIngressAnchor(larkAppId, rawMessage); + let seedRoutingGate: { ready: Promise; complete: () => void } | undefined; + // Reserve the app/chat ingress lane before any routing await. Canonical + // per-anchor serialization inside processMessageEvent remains the final + // execution fence after aliases and chat mode are resolved. + const scheduled = scheduleAckSafeEvent( + eventKey, + () => serializeByAnchor( + ingressAnchor, + () => processMessageEvent(data, seedRoutingGate), + ), + 'message event', + claim, + ); const rawSenderType = data?.sender?.sender_type; if ( scheduled diff --git a/src/im/lark/overview-card.ts b/src/im/lark/overview-card.ts index bc9c8d5cf..22c6f3132 100644 --- a/src/im/lark/overview-card.ts +++ b/src/im/lark/overview-card.ts @@ -31,10 +31,11 @@ export const OVERVIEW_ACTION_GOTO_SCHEDULES = 'dash_overview_goto_schedules' as export const OVERVIEW_ACTION_GOTO_SETTINGS = 'dash_overview_goto_settings' as const; export const OVERVIEW_ACTION_GOTO_GROUPS = 'dash_overview_goto_groups' as const; -/** Status set treated as "active" (working / analyzing / starting / limited). */ +/** Status set treated as active rather than falsely counted as idle. */ const ACTIVE_STATUSES: ReadonlySet = new Set([ 'working', 'analyzing', + 'stalled', 'starting', 'limited', ]); diff --git a/src/services/adopt-input-sequence.ts b/src/services/adopt-input-sequence.ts new file mode 100644 index 000000000..f11106d04 --- /dev/null +++ b/src/services/adopt-input-sequence.ts @@ -0,0 +1,87 @@ +import { AsyncSerialQueue } from '../utils/async-serial-queue.js'; + +export interface AdoptQueuedWriteSequence { + queue: AsyncSerialQueue; + /** Rechecked at the actual dequeue/execution point, not at IPC receipt. */ + isCurrent: () => boolean; + write: () => Promise; + /** Requeue the exact untouched input when its captured backend generation + * became stale before the task ever started writing. */ + onStale: () => void | Promise; +} + +export type AdoptQueuedWriteResult = + | { status: 'ran'; value: T } + | { status: 'stale-before-write' }; + +/** Fence a process-lifetime serial queue task to the backend generation that + * accepted it. AsyncSerialQueue intentionally survives CLI restarts; without + * this dequeue-time check an old task can mark with backend=null or write into + * a replacement CLI before its prompt is ready. */ +export function runAdoptQueuedWriteSequence( + input: AdoptQueuedWriteSequence, +): Promise> { + return input.queue.run(async () => { + if (!input.isCurrent()) { + await input.onStale(); + return { status: 'stale-before-write' }; + } + return { status: 'ran', value: await input.write() }; + }); +} + +export interface AdoptRawInputSequence { + queue: AsyncSerialQueue; + /** Resolves true only after the literal command's submit key has landed. */ + writeRawInput: () => Promise; + /** Resolves only after the bundled follow-up's complete adapter lifecycle. */ + writeFollowUp?: () => Promise; + /** Same generation/backend fence used by ordinary adopt writes. */ + isCurrent?: () => boolean; + onStaleBeforeWrite?: () => void | Promise; + onStaleBeforeFollowUp?: () => void | Promise; +} + +/** + * Keep an adopt slash command and its bundled follow-up in one serial-queue + * transaction. The follow-up callback must cover the complete adapter write, + * history verification and lifecycle settlement; awaiting it here prevents a + * later process-message listener from sharing or overwriting the TUI composer. + */ +export function runAdoptRawInputSequence(input: AdoptRawInputSequence): Promise { + return input.queue.run(async () => { + if (input.isCurrent && !input.isCurrent()) { + await input.onStaleBeforeWrite?.(); + return; + } + const rawInputSent = await input.writeRawInput(); + if (!rawInputSent) return; + if (input.writeFollowUp) { + if (input.isCurrent && !input.isCurrent()) { + await input.onStaleBeforeFollowUp?.(); + return; + } + await input.writeFollowUp(); + } + }); +} + +export interface AdoptSessionRenameSequence { + queue: AsyncSerialQueue; + /** Rechecked after every earlier adopt composer write has settled. */ + isPromptReady: () => boolean; + /** Resolves only after the rename command's literal text and Enter land. */ + writeRename: () => Promise; +} + +/** Serialize native rename with every adopt composer write. A readiness check + * made before waiting on the queue can become stale when an earlier adopt + * message starts first, so recheck inside the queue and let the worker retry + * the rename at the next genuine prompt instead of steering it into a turn. */ +export function runAdoptSessionRenameSequence(input: AdoptSessionRenameSequence): Promise { + return input.queue.run(async () => { + if (!input.isPromptReady()) return false; + await input.writeRename(); + return true; + }); +} diff --git a/src/services/codex-bridge-queue.ts b/src/services/codex-bridge-queue.ts index 63c57a171..7e66edbde 100644 --- a/src/services/codex-bridge-queue.ts +++ b/src/services/codex-bridge-queue.ts @@ -33,16 +33,32 @@ * , or replayed history) is IGNORED and does NOT * drop the collecting turn — keying HOL-drop off the turn-start * decision reuses its tooOld/fingerprint freshness as one invariant. - * * 'assistant_final' event → the currently-collecting turn closes - * with finalText set; eligible for emit on the next drain. - * - drainEmittable() — pop FIFO any leading turn that is started AND - * has finalText. + * * terminal event → the currently-collecting turn closes with + * finalText set; an abort uses empty text plus an ambiguous terminal + * outcome so durable delivery settles without inventing a reply. + * - drainEmittable() — pop FIFO any leading turn that is started AND has + * reached either terminal edge. */ import { makeFingerprint, normaliseForFingerprint } from './bridge-turn-queue.js'; import type { CodexBridgeEvent } from './codex-transcript.js'; const UNMATCHED_REPLAY_WINDOW_MS = 5_000; const MAX_BUFFERED_UNMATCHED_EVENTS = 20; +/** A verified submit may be parked in a type-ahead queue before its transcript + * user event is written. Keep that hand-off busy for a bounded interval: long + * enough for the active turn to finish and dequeue it, but never forever if + * the CLI accepted the keypress without producing a structured event. */ +export const STRUCTURED_SUBMIT_START_GRACE_MS = 20_000; +/** Maximum time an unconfirmed worker mark may remain at the attribution + * head after the adapter write/verification path stops. This lease never + * contributes to lifecycle busy: it exists only so a late transcript user + * event can still claim the mark without allowing a silent write to wedge + * every later turn forever. */ +export const STRUCTURED_UNCONFIRMED_ATTRIBUTION_GRACE_MS = 20_000; +/** Maximum time a worker may wait for an adapter/history verification call. + * This covers Codex/CoCo's in-band polling plus the 20s deferred recheck, + * while remaining bounded if an adapter promise or recheck is stranded. */ +export const STRUCTURED_SUBMIT_VERIFICATION_GRACE_MS = 30_000; export interface CodexPendingTurn { turnId: string; @@ -53,6 +69,17 @@ export interface CodexPendingTurn { * the lower bound of the "did `botmux send` happen for this turn?" * window. Optional only for legacy / test-injected turns. */ markTimeMs?: number; + /** Wall-clock millis when an authoritative adapter/history check confirmed + * the submit. Unverified writes deliberately leave this unset. */ + submitConfirmedAtMs?: number; + /** Wall-clock millis anchoring the bounded attribution-only lease for an + * unconfirmed mark. Unlike verification/confirmation leases, this never + * gates screen-ready or reports the CLI busy. */ + unconfirmedAttributionStartedAtMs?: number; + /** Wall-clock millis when worker-side authoritative submit verification + * began. This closes the race where screen-ready arrives while writeInput + * is still polling history, before it can return `submitted: true`. */ + submitVerificationStartedAtMs?: number; /** Set once an assistant_final event closes this turn. */ finalText?: string; /** Explicit transcript terminal semantics. Undefined keeps the historical @@ -68,6 +95,14 @@ export interface CodexPendingTurn { * assistant reply so the Lark thread sees both sides of the exchange. */ userText?: string; sourceSessionId?: string; + /** True when the turn was delivered via Codex RPC (turn/start) and the + * app-server has acknowledged it. RPC turns have no local transcript to + * ingest, so they can never reach the started state; this flag keeps the + * lifecycle gate asserted for the full server-side execution instead of + * letting the bounded 20s confirmation lease expire mid-turn (which would + * falsely release idle and prune a still-running turn). Cleared by the + * terminal edge or an explicit stop. */ + rpcActive?: boolean; } export class CodexBridgeQueue { @@ -82,6 +117,8 @@ export class CodexBridgeQueue { * "live" local input. Typically set to the moment adopt was wired up. */ private localLowerBoundMs = 0; + constructor(private readonly now: () => number = Date.now) {} + /** Register events as historical without producing pending-turn side * effects. Used at attach time when resume mode wants to swallow prior * conversation as already-processed. */ @@ -92,7 +129,7 @@ export class CodexBridgeQueue { /** Toggle adopt-mode local-turn synthesis. `lowerBoundMs` (typically * Date.now() at adopt-time) protects against a fresh-empty attach * feeding historical user_messages back as "live" local turns. */ - setLocalTurns(enabled: boolean, lowerBoundMs: number = Date.now()): void { + setLocalTurns(enabled: boolean, lowerBoundMs: number = this.now()): void { this.localTurnsEnabled = enabled; this.localLowerBoundMs = lowerBoundMs; if (enabled) this.bufferedUnmatched = []; @@ -103,13 +140,14 @@ export class CodexBridgeQueue { * to start this turn. Pre-path-known marking is allowed: the worker can * call this before late-attach has located the rollout file, and the * ingest call after attach will still match correctly. */ - mark(turnId: string, message: string, markTimeMs: number = Date.now(), dispatchAttempt?: number): void { + mark(turnId: string, message: string, markTimeMs: number = this.now(), dispatchAttempt?: number): void { this.queue.push({ turnId, dispatchAttempt, started: false, contentFingerprint: makeFingerprint(message), markTimeMs, + unconfirmedAttributionStartedAtMs: markTimeMs, }); this.replayBufferedUnmatched(markTimeMs); } @@ -124,20 +162,217 @@ export class CodexBridgeQueue { return dropped; } - /** Remove one exact worker delivery attempt after a failed/ambiguous - * terminal. A replay may reuse turnId with a higher dispatchAttempt; if - * the retired mark stayed queued, its fingerprint would claim the replay's - * transcript events and permanently head-of-line block the live attempt. */ - dropPendingTurn(turnId: string, dispatchAttempt?: number): CodexPendingTurn | null { - const idx = this.queue.findIndex(turn => - turn.turnId === turnId && turn.dispatchAttempt === dispatchAttempt, + /** Remove one exact worker delivery attempt. Submit-confirmation cleanup is + * pre-start-only by default: once the transcript has started a turn, only a + * structured terminal may retire it. An authoritative failed/ambiguous + * terminal may opt into removing a started attempt via `allowStarted`. + * Matching dispatchAttempt keeps a replay of the same turnId isolated from + * the retired delivery attempt. */ + dropPendingTurn( + turnId: string, + dispatchAttempt?: number, + allowStarted = false, + ): CodexPendingTurn | null { + const index = this.queue.findIndex(turn => + turn.turnId === turnId + && turn.dispatchAttempt === dispatchAttempt + && (allowStarted || !turn.started), ); - if (idx < 0) return null; - const [dropped] = this.queue.splice(idx, 1); + if (index < 0) return null; + const [dropped] = this.queue.splice(index, 1); if (this.collecting === dropped) this.collecting = null; + if (allowStarted) this.refreshNextPreStartLease(); + // A later turn's user event can already be buffered behind this failed + // head mark: ingestOne only matches the first unstarted fingerprint. Once + // the failed head is gone, replay recent events against the new head or it + // can remain unstarted forever and wedge fallback delivery. + const next = this.queue.find(turn => !turn.started); + if (next?.markTimeMs !== undefined) this.replayBufferedUnmatched(next.markTimeMs); return dropped ?? null; } + /** Remove expired pre-start queue heads that never reached transcript + * start, whether positively confirmed or only retained for attribution. + * Only the first unresolved fingerprint(s) are eligible, and never + * while an earlier started turn is still running: that predecessor's final + * is the dequeue boundary that refreshes the next legitimate type-ahead + * lease. Dropping a stale head replays buffered events immediately, so a + * later real turn can become started instead of remaining hidden behind a + * dead fingerprint. */ + pruneExpiredPreStartHeads(nowMs: number = this.now()): CodexPendingTurn[] { + const dropped: CodexPendingTurn[] = []; + for (;;) { + // A long-running predecessor legitimately keeps later confirmed input in + // the CLI's type-ahead queue. Its assistant_final refreshes the next head + // from local observation time, so expiring anything before that boundary + // would drop valid queued work. + if (this.queue.some(turn => turn.started && turn.finalText === undefined)) break; + + const head = this.queue.find(turn => !turn.started && turn.finalText === undefined); + if (!head) break; + + // An RPC turn is running server-side with no local transcript to ingest. + // It can never reach started, so the bounded lease must NOT be used to + // prune it — that would falsely release idle while the app-server is still + // executing. Keep it until the terminal edge or explicit stop. + if (head.rpcActive) break; + + // An authoritative adapter/history check can legitimately outlive the + // shorter attribution lease. Do not prune under that in-flight await; + // once it finishes, finishSubmitVerification refreshes attribution from + // local observation time, and once this bounded verification lease + // expires the old mark becomes eligible again. + const verificationActive = head.submitVerificationStartedAtMs !== undefined + && nowMs - head.submitVerificationStartedAtMs <= STRUCTURED_SUBMIT_VERIFICATION_GRACE_MS; + if (verificationActive) break; + + const leaseStartedAtMs = head.submitConfirmedAtMs + ?? head.unconfirmedAttributionStartedAtMs; + if (leaseStartedAtMs === undefined) break; + const leaseGraceMs = head.submitConfirmedAtMs !== undefined + ? STRUCTURED_SUBMIT_START_GRACE_MS + : STRUCTURED_UNCONFIRMED_ATTRIBUTION_GRACE_MS; + if (nowMs - leaseStartedAtMs <= leaseGraceMs) break; + + const removed = this.dropPendingTurn(head.turnId, head.dispatchAttempt); + if (!removed) break; + dropped.push(removed); + // dropPendingTurn replays buffered user/final events. If that starts the + // next real turn, the started-turn guard above stops the loop. + } + return dropped; + } + + /** Record positive submit evidence from an adapter/history check. The turn + * can still be waiting in the CLI's type-ahead queue, so this starts a + * bounded hand-off lease until its transcript user event appears. */ + confirmPendingTurn( + turnId: string, + confirmedAtMs: number = this.now(), + dispatchAttempt?: number, + ): boolean { + const turn = this.queue.find(candidate => candidate.turnId === turnId + && candidate.dispatchAttempt === dispatchAttempt + && candidate.finalText === undefined); + if (!turn) return false; + turn.submitVerificationStartedAtMs = undefined; + turn.unconfirmedAttributionStartedAtMs = undefined; + turn.submitConfirmedAtMs = confirmedAtMs; + return true; + } + + /** Mark a turn as actively running server-side via Codex RPC. The app-server + * ack for turn/start is authoritative confirmation that execution has begun, + * but no local transcript event will follow to flip started. This flag keeps + * the lifecycle gate asserted and protects the turn from lease expiry pruning + * until the terminal edge (or an explicit stop) clears it. */ + markRpcActive(turnId: string, dispatchAttempt?: number): boolean { + const turn = this.queue.find(candidate => candidate.turnId === turnId + && candidate.dispatchAttempt === dispatchAttempt + && candidate.finalText === undefined); + if (!turn) return false; + turn.rpcActive = true; + turn.submitVerificationStartedAtMs = undefined; + turn.unconfirmedAttributionStartedAtMs = undefined; + turn.submitConfirmedAtMs = undefined; + return true; + } + + /** Clear the server-side active flag when an RPC turn reaches a terminal edge + * or is otherwise retired. Without this, a completed RPC turn would keep the + * lifecycle gate asserted forever (permanent false-busy). */ + stopRpcActive(turnId: string, dispatchAttempt?: number): boolean { + const turn = this.queue.find(candidate => candidate.turnId === turnId + && candidate.dispatchAttempt === dispatchAttempt); + if (!turn || !turn.rpcActive) return false; + turn.rpcActive = undefined; + return true; + } + + /** Start bounded adapter/history verification before awaiting writeInput. */ + beginSubmitVerification( + turnId: string, + startedAtMs: number = this.now(), + dispatchAttempt?: number, + ): boolean { + const turn = this.queue.find(candidate => candidate.turnId === turnId + && candidate.dispatchAttempt === dispatchAttempt + && candidate.finalText === undefined); + if (!turn) return false; + turn.submitVerificationStartedAtMs = startedAtMs; + return true; + } + + /** Finish verification without positive submit evidence. A bare mark remains + * available for transcript attribution but no longer gates screen-ready. */ + finishSubmitVerification( + turnId: string, + finishedAtMs: number = this.now(), + dispatchAttempt?: number, + ): boolean { + const turn = this.queue.find(candidate => candidate.turnId === turnId + && candidate.dispatchAttempt === dispatchAttempt); + if (!turn || turn.submitVerificationStartedAtMs === undefined) return false; + turn.submitVerificationStartedAtMs = undefined; + if (!turn.started && turn.submitConfirmedAtMs === undefined) { + turn.unconfirmedAttributionStartedAtMs = finishedAtMs; + } + return true; + } + + /** Exact-attempt existence check for deferred callbacks. A replay reuses + * turnId with a higher dispatchAttempt, so an old timer must treat that as + * a missing target rather than mutating the new delivery generation. */ + hasPendingTurn(turnId: string, dispatchAttempt?: number): boolean { + return this.queue.some(candidate => candidate.turnId === turnId + && candidate.dispatchAttempt === dispatchAttempt + && candidate.finalText === undefined); + } + + /** True when buffered transcript replay has already closed this exact turn + * before its RPC turn/start continuation installs rpcActive. */ + hasTerminalTurn(turnId: string, dispatchAttempt?: number): boolean { + return this.queue.some(candidate => candidate.turnId === turnId + && candidate.dispatchAttempt === dispatchAttempt + && candidate.finalText !== undefined); + } + + /** True while the transcript proves a turn is running, or while a verified + * submit is in the bounded pre-start hand-off window. A bare worker mark is + * never authoritative, preventing a dropped Enter from causing permanent + * false-busy. */ + hasBlockingTurn(nowMs: number = this.now()): boolean { + return this.queue.some(turn => { + if (turn.finalText !== undefined) return false; + if (turn.started) return true; + if (turn.rpcActive) return true; + const confirmed = turn.submitConfirmedAtMs !== undefined + && nowMs - turn.submitConfirmedAtMs <= STRUCTURED_SUBMIT_START_GRACE_MS; + const verifying = turn.submitVerificationStartedAtMs !== undefined + && nowMs - turn.submitVerificationStartedAtMs <= STRUCTURED_SUBMIT_VERIFICATION_GRACE_MS; + return confirmed || verifying; + }); + } + + /** Remaining bounded pre-start verification/confirmation lease. The worker + * uses this to re-drive a previously rejected ready signal once every active + * lease expires. Started turns return undefined because their eventual + * assistant_final is the authoritative re-drive. */ + preStartLeaseRemainingMs(nowMs: number = this.now()): number | undefined { + if (this.queue.some(turn => (turn.started || turn.rpcActive) && turn.finalText === undefined)) return undefined; + const activeRemaining = this.queue.flatMap(candidate => { + if (candidate.started || candidate.finalText !== undefined) return []; + const leases: number[] = []; + if (candidate.submitConfirmedAtMs !== undefined) { + leases.push(STRUCTURED_SUBMIT_START_GRACE_MS - (nowMs - candidate.submitConfirmedAtMs)); + } + if (candidate.submitVerificationStartedAtMs !== undefined) { + leases.push(STRUCTURED_SUBMIT_VERIFICATION_GRACE_MS - (nowMs - candidate.submitVerificationStartedAtMs)); + } + return leases.filter(remaining => remaining >= 0); + }); + return activeRemaining.length > 0 ? Math.max(...activeRemaining) : undefined; + } /** Process newly-appended events. Idempotent on uuid: events with seen * uuids are skipped, so callers can replay safely. */ ingest(events: CodexBridgeEvent[]): void { @@ -152,7 +387,10 @@ export class CodexBridgeQueue { if (this.bufferedUnmatched.length === 0) return; const replay = this.bufferedUnmatched.filter(ev => ev.timestampMs >= markTimeMs - UNMATCHED_REPLAY_WINDOW_MS); this.bufferedUnmatched = []; - for (const ev of replay) this.ingestOne(ev, false); + // Keep still-unmatched events buffered: more than one failed head can sit + // ahead of the successful turn. Each drop gets another chance to replay + // the same bounded event set against the new head. + for (const ev of replay) this.ingestOne(ev, true); } private rememberUnmatched(ev: CodexBridgeEvent): void { @@ -162,6 +400,24 @@ export class CodexBridgeQueue { } } + /** Refresh the next queued submit from the locally-observed terminal edge. + * External transcript clocks may be skewed, so lease boundedness must use + * this process's clock for both successful and aborted predecessors. */ + private refreshNextPreStartLease(): void { + const nextPending = this.queue.find(turn => !turn.started + && turn.finalText === undefined); + if (!nextPending) return; + // RPC turns keep their own rpcActive flag for lifecycle gating; do not + // overwrite it with a bounded confirmation/attribution lease. + if (nextPending.rpcActive) return; + const observedAtMs = this.now(); + if (nextPending.submitConfirmedAtMs !== undefined) { + nextPending.submitConfirmedAtMs = observedAtMs; + } else { + nextPending.unconfirmedAttributionStartedAtMs = observedAtMs; + } + } + private ingestOne(ev: CodexBridgeEvent, bufferUnmatched: boolean): void { if (ev.kind === 'user') { // First decide whether this user event is a REAL turn-start: either it @@ -199,6 +455,8 @@ export class CodexBridgeQueue { if (willStartNext) { next!.started = true; + next!.submitVerificationStartedAtMs = undefined; + next!.unconfirmedAttributionStartedAtMs = undefined; next!.sourceSessionId = ev.sourceSessionId; // Anchor the bridge-fallback suppression window to when the turn // ACTUALLY started processing (the transcript user event's @@ -266,14 +524,37 @@ export class CodexBridgeQueue { this.collecting.terminalErrorCode = ev.terminalErrorCode; this.lastClosedAssistantFinalTimeMs = ev.timestampMs; this.collecting = null; + // CoCo-style type-ahead writes the next user event only after this + // final dequeues it. Refresh the next turn's pre-start lease at that + // hand-off boundary instead of letting either its confirmed lease or + // attribution-only lease expire while the predecessor was legitimately + // running. + this.refreshNextPreStartLease(); } else if (bufferUnmatched && !this.localTurnsEnabled) { this.rememberUnmatched(ev); } + } else if (ev.kind === 'turn_aborted') { + if (!this.collecting) { + if (bufferUnmatched && !this.localTurnsEnabled) this.rememberUnmatched(ev); + return; + } + if (this.collecting.sourceSessionId && ev.sourceSessionId && this.collecting.sourceSessionId !== ev.sourceSessionId) return; + // Interrupted Codex turns have no assistant_final. Close with empty text + // so the worker skips final_output but still publishes the authoritative + // exact-attempt turn_terminal required by reliable durable delivery. + // Side effects may already have happened, so mirror TRAE-X and classify + // an otherwise-untyped abort as ambiguous rather than completed/failed. + this.collecting.finalText = ''; + this.collecting.terminalStatus = ev.terminalStatus ?? 'ambiguous'; + this.collecting.terminalErrorCode = ev.terminalErrorCode ?? 'structured_turn_aborted'; + this.collecting = null; + this.lastClosedAssistantFinalTimeMs = ev.timestampMs; + this.refreshNextPreStartLease(); } } - /** Pop FIFO any leading turn that is started AND observed assistant_final. - * Empty final text still closes a durable turn. */ + /** Pop FIFO any leading turn that is started AND observed a terminal edge. + * Empty final text closes a durable turn without producing final_output. */ drainEmittable(): CodexPendingTurn[] { const out: CodexPendingTurn[] = []; while (this.queue.length > 0) { @@ -295,3 +576,24 @@ export class CodexBridgeQueue { return this.queue; } } + +/** Explicit mutation boundary for lease expiry. Pruning can replay a buffered + * successor user+final pair, so callers must drain/emit in the same call + * stack; keeping that invariant here prevents a status/query path from + * silently creating an unconsumed completion. */ +export function pruneExpiredPreStartHeadsAndEmit( + queue: CodexBridgeQueue, + emitReady: () => void, + nowMs?: number, + /** Settle exact durable attempts before a replayed successor is emitted. + * The queue removal can expose buffered successor user/final events, so + * running this after emitReady would publish N+1 ahead of N's terminal. */ + onDropped?: (dropped: readonly CodexPendingTurn[]) => void, +): CodexPendingTurn[] { + const dropped = queue.pruneExpiredPreStartHeads(nowMs); + if (dropped.length > 0) { + onDropped?.(dropped); + emitReady(); + } + return dropped; +} diff --git a/src/services/codex-transcript.ts b/src/services/codex-transcript.ts index 8547740f4..8f525fa92 100644 --- a/src/services/codex-transcript.ts +++ b/src/services/codex-transcript.ts @@ -4,19 +4,24 @@ * Codex stores each session's full transcript at * ~/.codex/sessions///
/rollout--.jsonl * and creates the file lazily on the first user submit. Inside, the bridge - * fallback only cares about two `response_item.payload.type === 'message'` - * shapes: + * fallback consumes two `response_item.payload.type === 'message'` shapes and + * one terminal UI-event shape: * * - role=user → the user's prompt text (input_text content) * - role=assistant + * phase=final_answer → the model's final reply (output_text content) + * - event_msg.turn_aborted → the active turn ended without a final reply * - * Why these and not `event_msg`: + * Why final text still comes from `response_item`, while abort comes from + * `event_msg`: * - `response_item` is the canonical transcript record; `event_msg` is a * UI-event stream that can carry the same final text via two channels * (`agent_message phase=final_answer` AND `task_complete.last_agent_message`). - * Picking `response_item` keeps the reader to a single source of truth - * and avoids any chance of double-emit if both paths are present. + * Picking `response_item` keeps final text to a single source of truth and + * avoids any chance of double-emit if both paths are present. + * - An interrupted turn has no assistant response_item at all. Codex records + * only `event_msg.payload.type === 'turn_aborted'`; without that terminal + * edge a transcript-started lifecycle would remain busy forever. * - Skipping role=developer (system instructions), phase=commentary * (mid-turn status), reasoning, and function_call* keeps the bridge * focused on what the user actually said and what the model finally @@ -107,8 +112,9 @@ export interface CodexBridgeEvent { timestampMs: number; /** Discriminator for the queue layer: * - 'user' starts a pending Lark turn (fingerprint-matched) - * - 'assistant_final' closes the currently-collecting turn */ - kind: 'user' | 'assistant_final'; + * - 'assistant_final' closes the currently-collecting turn with output + * - 'turn_aborted' closes it without producing fallback output */ + kind: 'user' | 'assistant_final' | 'turn_aborted'; /** Concatenated text from the message's content blocks (input_text for * user, output_text for assistant). */ text: string; @@ -125,6 +131,11 @@ export interface CodexBridgeEvent { preserveMarkTimeMs?: boolean; } +/** Terminal lifecycle edges understood by the shared structured bridge. */ +export function isStructuredTerminalEvent(event: Pick): boolean { + return event.kind === 'assistant_final' || event.kind === 'turn_aborted'; +} + /** Extract the last completed user/assistant turn from a Codex / CoCo bridge * event sequence. Used by /adopt to surface the previous turn as a * preamble card in the Lark thread — gives the user context to continue @@ -136,7 +147,7 @@ export interface CodexBridgeEvent { * undefined when either side is missing — typically a fresh session whose * user typed something but the model hasn't replied yet. */ export function extractLastCodexTurn( - events: readonly { kind: 'user' | 'assistant_final'; text: string }[], + events: readonly { kind: 'user' | 'assistant_final' | 'turn_aborted'; text: string }[], ): { userText: string; assistantText: string } | undefined { let assistantIdx = -1; for (let i = events.length - 1; i >= 0; i--) { @@ -324,18 +335,26 @@ export function drainCodexRollout(path: string, fromOffset: number): CodexDrainR cursor += lineByteLen; let obj: any; try { obj = JSON.parse(line); } catch { continue; } - if (obj?.type !== 'response_item') continue; const p = obj.payload; - if (!p || typeof p !== 'object' || p.type !== 'message') continue; const ts = typeof obj.timestamp === 'string' ? Date.parse(obj.timestamp) : NaN; const timestampMs = Number.isFinite(ts) ? ts : Date.now(); + if (obj?.type === 'event_msg' && p?.type === 'turn_aborted') { + const reason = typeof p.reason === 'string' ? p.reason : 'aborted'; + events.push({ uuid: `${path}:${lineStart}`, timestampMs, kind: 'turn_aborted', text: reason }); + continue; + } + if (obj?.type !== 'response_item') continue; + if (!p || typeof p !== 'object' || p.type !== 'message') continue; if (p.role === 'user') { const text = joinTextBlocks(p.content, 'input_text'); if (!text) continue; events.push({ uuid: `${path}:${lineStart}`, timestampMs, kind: 'user', text }); } else if (p.role === 'assistant' && p.phase === 'final_answer') { const text = joinTextBlocks(p.content, 'output_text'); - if (!text) continue; + // final_answer is the authoritative normal terminal boundary even when + // Codex records an empty output_text. The worker deliberately suppresses + // an empty final_output, but reliable delivery still needs a completed + // exact-attempt turn_terminal and the lifecycle gate must be released. events.push({ uuid: `${path}:${lineStart}`, timestampMs, kind: 'assistant_final', text }); } // Skip role=developer (instructions), phase=commentary (mid-turn diff --git a/src/services/hook-runner.ts b/src/services/hook-runner.ts index 79ff5aeea..99bc6a074 100644 --- a/src/services/hook-runner.ts +++ b/src/services/hook-runner.ts @@ -45,6 +45,21 @@ export type HookPayload = Record & { sender_open_id?: string; }; +/** Frozen authority for a read-isolated post-provider hook. Every value comes + * from the original protected claim/attestation; forwarding must not rediscover + * a daemon or reread a rotating capability after the fence. */ +export interface ManagedHookOrigin { + ipcPort: number; + sessionId: string; + capability: string; + turnId: string; + dispatchAttempt?: number; +} + +export interface EmitHookEventOptions { + managedOrigin?: ManagedHookOrigin; +} + export type ParsedHookCommand = { file: string; args: string[]; @@ -361,7 +376,11 @@ async function runHookCommand( }); } -export function emitHookEvent(event: HookEvent, body: Record = {}): void { +export function emitHookEvent( + event: HookEvent, + body: Record = {}, + options: EmitHookEventOptions = {}, +): void { try { const payload: HookPayload = { ...body, @@ -378,8 +397,14 @@ export function emitHookEvent(event: HookEvent, body: Record = // session-scoped env scrubbed (index-daemon.ts) and its /api/hooks/emit // handler calls emitHookEventLocal, so the gate can't self-forward even // if leaked env survives somewhere. - if (process.env.BOTMUX_SESSION_ID && process.env.BOTMUX_LARK_APP_ID) { - void forwardEmitToDaemon(event, payload, process.env.BOTMUX_LARK_APP_ID); + if (options.managedOrigin + || (process.env.BOTMUX_SESSION_ID && process.env.BOTMUX_LARK_APP_ID)) { + void forwardEmitToDaemon( + event, + payload, + process.env.BOTMUX_LARK_APP_ID ?? '', + options.managedOrigin, + ); return; } @@ -442,10 +467,16 @@ const HOOK_FORWARD_FETCH_TIMEOUT_MS = 2_000; * process-group cleanup work. Best-effort — daemon unreachable / 4xx / 5xx * just log and drop, hooks are best-effort by contract. */ -async function forwardEmitToDaemon(event: HookEvent, payload: HookPayload, larkAppId: string): Promise { +export async function forwardEmitToDaemon( + event: HookEvent, + payload: HookPayload, + larkAppId: string, + managedOrigin?: ManagedHookOrigin, +): Promise { try { - const daemon = findOnlineDaemon(larkAppId); - if (!daemon) { + const daemon = managedOrigin ? undefined : findOnlineDaemon(larkAppId); + const ipcPort = managedOrigin?.ipcPort ?? daemon?.ipcPort; + if (!ipcPort) { logger.debug(`[hooks] CLI forward: no daemon for ${larkAppId}, dropping ${event}`); return; } @@ -453,13 +484,16 @@ async function forwardEmitToDaemon(event: HookEvent, payload: HookPayload, larkA const timer = setTimeout(() => ctrl.abort(), HOOK_FORWARD_FETCH_TIMEOUT_MS); timer.unref(); try { - const sessionId = process.env.BOTMUX_SESSION_ID; - const origin = resolveSessionContext(config.session.dataDir, sessionId); - const originCapability = readManagedOriginCapability( - config.session.dataDir, - sessionId, - process.env.BOTMUX_SEND_RELAY, - )?.capability; + const sessionId = managedOrigin?.sessionId ?? process.env.BOTMUX_SESSION_ID; + const origin = managedOrigin + ? undefined + : resolveSessionContext(config.session.dataDir, sessionId); + const originCapability = managedOrigin?.capability ?? readManagedOriginCapability( + config.session.dataDir, + sessionId, + process.env.BOTMUX_SEND_RELAY, + process.env.BOTMUX_ORIGIN_CHANNEL_ID, + )?.capability; const envAttempt = Number(process.env.BOTMUX_DISPATCH_ATTEMPT); const request = { method: 'POST', @@ -469,8 +503,8 @@ async function forwardEmitToDaemon(event: HookEvent, payload: HookPayload, larkA payload, sessionId, originCapability, - originTurnId: origin?.turnId ?? process.env.BOTMUX_TURN_ID, - originDispatchAttempt: origin?.dispatchAttempt + originTurnId: managedOrigin?.turnId ?? origin?.turnId ?? process.env.BOTMUX_TURN_ID, + originDispatchAttempt: managedOrigin?.dispatchAttempt ?? origin?.dispatchAttempt ?? (Number.isSafeInteger(envAttempt) && envAttempt > 0 ? envAttempt : undefined), }), signal: ctrl.signal, @@ -478,8 +512,8 @@ async function forwardEmitToDaemon(event: HookEvent, payload: HookPayload, larkA let secret: string | undefined; try { secret = loadDaemonIpcSecret(); } catch { /* Seatbelt/read-isolated CLI */ } const res = secret - ? await fetchDaemonIpc(daemon.ipcPort, '/api/hooks/emit', request, secret) - : await fetch(`http://127.0.0.1:${daemon.ipcPort}/api/hooks/emit`, request); + ? await fetchDaemonIpc(ipcPort, '/api/hooks/emit', request, secret) + : await fetch(`http://127.0.0.1:${ipcPort}/api/hooks/emit`, request); if (!res.ok) { logger.warn(`[hooks] CLI forward ${event} → daemon: HTTP ${res.status}`); } diff --git a/src/services/relay-picker.ts b/src/services/relay-picker.ts index d415354c4..bfb0aac91 100644 --- a/src/services/relay-picker.ts +++ b/src/services/relay-picker.ts @@ -38,6 +38,11 @@ export async function collectRelayPickerEntries( if (sessionAnchorId(c) === currentAnchor) continue; if (c.session.ownerOpenId !== operatorOpenId) continue; if (c.session.adoptedFrom) continue; + // A Riff sandbox keeps the BOTMUX reply route injected at task creation; + // follow-ups cannot retarget it. Do not offer an action transferSession + // must reject. + if ((c.initConfig?.backendType ?? c.session.backendType) === 'riff' + || c.session.cliId === 'riff') continue; // Daemon-command scratches (worker:null + no persisted CLI markers) // are placeholder records for /help / unfinished /relay etc. — they // have no real conversation to bring along. Don't surface them in diff --git a/src/services/restart-intent-store.ts b/src/services/restart-intent-store.ts index d453cf632..81764150c 100644 --- a/src/services/restart-intent-store.ts +++ b/src/services/restart-intent-store.ts @@ -14,10 +14,11 @@ import { randomBytes } from 'node:crypto'; import { join } from 'node:path'; import { config } from '../config.js'; import { readProcessStartIdentity } from '../core/session-marker.js'; +import { withFileLockSync } from '../utils/file-lock.js'; export type RestartKind = 'manual' | 'update' | 'rollback'; -export interface RestartIntent { +export interface RestartIntentPayload { kind: RestartKind; /** Present for an update or rollback: the version delta to report. */ oldVersion?: string; @@ -26,6 +27,22 @@ export interface RestartIntent { at: string; } +export interface RestartIntent extends RestartIntentPayload { + /** Unique CLI start attempt that may remove only its own failed breadcrumb. */ + attemptId?: string; + /** A prepared attempt is deliberately invisible to daemons until the CLI + * verifies the complete configured fleet and atomically commits it. */ + attemptState?: 'prepared' | 'committed' | 'aborted'; + /** A newer maintenance writer is held behind the active prepared attempt so + * a partially-started primary daemon cannot consume it either. */ + deferredIntent?: RestartIntentPayload; +} + +export type RestartIntentReportClaim = + | { state: 'claimed'; intent: RestartIntent } + | { state: 'prepared' } + | { state: 'absent' }; + const FILE = 'restart-intent.json'; const LEASE_FILE = 'restart-lease.json'; @@ -39,7 +56,7 @@ export function restartIntentPathIn(dir: string): string { return join(dir, FILE); } -export function writeRestartIntentTo(dir: string, intent: RestartIntent): void { +function writeRestartIntentUnlocked(dir: string, intent: RestartIntent): void { if (!existsSync(dir)) mkdirSync(dir, { recursive: true }); const path = restartIntentPathIn(dir); const tmp = `${path}.${process.pid}.tmp`; @@ -48,7 +65,37 @@ export function writeRestartIntentTo(dir: string, intent: RestartIntent): void { } export function clearRestartIntentTo(dir: string): void { - try { rmSync(restartIntentPathIn(dir)); } catch { /* absent / best-effort */ } + if (!existsSync(dir)) return; + withFileLockSync(restartIntentPathIn(dir), () => { + try { rmSync(restartIntentPathIn(dir)); } catch { /* absent / best-effort */ } + }); +} + +function payloadOf(intent: RestartIntent): RestartIntentPayload { + return { + kind: intent.kind, + at: intent.at, + ...(intent.oldVersion !== undefined ? { oldVersion: intent.oldVersion } : {}), + ...(intent.newVersion !== undefined ? { newVersion: intent.newVersion } : {}), + }; +} + +export function writeRestartIntentTo(dir: string, intent: RestartIntent): void { + if (!existsSync(dir)) mkdirSync(dir, { recursive: true }); + withFileLockSync(restartIntentPathIn(dir), () => { + const current = readRaw(dir); + const intentAt = Date.parse(intent.at); + const writerNow = Number.isFinite(intentAt) ? intentAt : Date.now(); + if ((current?.attemptState === 'prepared' || current?.attemptState === 'aborted') + && isFresh(current, writerNow)) { + writeRestartIntentUnlocked(dir, { + ...current, + deferredIntent: payloadOf(intent), + }); + return; + } + writeRestartIntentUnlocked(dir, payloadOf(intent)); + }); } function readRaw(dir: string): RestartIntent | null { @@ -152,22 +199,134 @@ export function clearRestartLeaseTo(dir: string, id: string): void { * it fires at most once and never lingers into a later restart. Returns the * intent only when it is fresh. */ export function consumeRestartIntentTo(dir: string, nowMs: number): RestartIntent | null { - const intent = readRaw(dir); - const path = restartIntentPathIn(dir); - if (existsSync(path)) { - try { rmSync(path); } catch { /* best-effort */ } - } - if (!intent) return null; - return isFresh(intent, nowMs) ? intent : null; + if (!existsSync(dir)) mkdirSync(dir, { recursive: true }); + return withFileLockSync(restartIntentPathIn(dir), () => { + const intent = readRaw(dir); + const path = restartIntentPathIn(dir); + if ((intent?.attemptState === 'prepared' || intent?.attemptState === 'aborted') + && isFresh(intent, nowMs)) { + return null; + } + if (existsSync(path)) { + try { rmSync(path); } catch { /* best-effort */ } + } + if (!intent) return null; + return isFresh(intent, nowMs) ? intent : null; + }); +} + +/** Atomically observe the restart-attempt state and, only for a visible fresh + * intent, claim it by deleting the breadcrumb in the same lock acquisition. + * Report polling must use this single state transition instead of composing + * `consume()` and a later read-only prepared check: prepared can commit between + * those two reads. */ +export function claimRestartIntentForReportTo( + dir: string, + nowMs: number, +): RestartIntentReportClaim { + if (!existsSync(dir)) mkdirSync(dir, { recursive: true }); + return withFileLockSync(restartIntentPathIn(dir), () => { + const intent = readRaw(dir); + const path = restartIntentPathIn(dir); + if (!intent || !isFresh(intent, nowMs)) { + if (existsSync(path)) { + try { rmSync(path); } catch { /* best-effort stale/corrupt cleanup */ } + } + return { state: 'absent' }; + } + if (intent.attemptState === 'prepared') return { state: 'prepared' }; + if (intent.attemptState === 'aborted') return { state: 'absent' }; + if (existsSync(path)) { + try { rmSync(path); } + catch { return { state: 'absent' }; } + } + return { state: 'claimed', intent }; + }); +} + +/** Read-only admission used by the primary daemon to distinguish a prepared + * restart (wait for commit/abort) from a crash with no breadcrumb. */ +export function hasPreparedRestartIntentTo(dir: string, nowMs: number): boolean { + if (!existsSync(dir)) return false; + return withFileLockSync(restartIntentPathIn(dir), () => { + const intent = readRaw(dir); + if (intent?.attemptState !== 'prepared') return false; + if (isFresh(intent, nowMs)) return true; + try { rmSync(restartIntentPathIn(dir)); } catch { /* best-effort stale cleanup */ } + return false; + }); } /** Write a `manual` breadcrumb only when no *fresh* breadcrumb already exists — * so a maintenance-written `update` breadcrumb is not clobbered * by the `botmux restart` it spawns. */ export function writeManualIntentIfAbsentTo(dir: string, nowMs: number, atIso: string): void { - const existing = readRaw(dir); - if (existing && isFresh(existing, nowMs)) return; - writeRestartIntentTo(dir, { kind: 'manual', at: atIso }); + if (!existsSync(dir)) mkdirSync(dir, { recursive: true }); + withFileLockSync(restartIntentPathIn(dir), () => { + const existing = readRaw(dir); + if (existing && isFresh(existing, nowMs)) return; + writeRestartIntentUnlocked(dir, { kind: 'manual', at: atIso }); + }); +} + +/** Atomically preserve any newer/richer fresh intent while tagging the exact + * CLI start attempt that owns rollback responsibility. */ +export function writeRestartAttemptIntentTo( + dir: string, + preferred: RestartIntent, + nowMs: number, + attemptId: string, +): RestartIntent { + if (!existsSync(dir)) mkdirSync(dir, { recursive: true }); + return withFileLockSync(restartIntentPathIn(dir), () => { + const existing = readRaw(dir); + const selected = existing && isFresh(existing, nowMs) + ? ((existing.attemptState === 'prepared' || existing.attemptState === 'aborted') + && existing.deferredIntent + ? existing.deferredIntent + : payloadOf(existing)) + : payloadOf(preferred); + const written: RestartIntent = { ...selected, attemptId, attemptState: 'prepared' }; + writeRestartIntentUnlocked(dir, written); + return written; + }); +} + +/** Make exactly one successfully verified start attempt visible to the primary + * daemon. A newer ordinary writer is folded into the same commit, but can + * never bypass the prepared fence while the fleet is partial. */ +export function commitRestartIntentAttemptTo(dir: string, attemptId: string): boolean { + if (!existsSync(dir)) return false; + return withFileLockSync(restartIntentPathIn(dir), () => { + const current = readRaw(dir); + if (current?.attemptId !== attemptId || current.attemptState !== 'prepared') return false; + const selected = current.deferredIntent ?? payloadOf(current); + writeRestartIntentUnlocked(dir, { + ...selected, + attemptId, + attemptState: 'committed', + }); + return true; + }); +} + +/** Abort only the breadcrumb still owned by this failed start attempt. The + * restart-intent lock prevents a read/unlink race with a newer update writer. + * The result remains an aborted/non-consumable fence: a stranded partial + * primary must stay silent and even a later maintenance writer is deferred, + * while a later verified restart can adopt the newest payload. */ +export function removeRestartIntentAttemptTo(dir: string, attemptId: string): boolean { + if (!existsSync(dir)) return false; + return withFileLockSync(restartIntentPathIn(dir), () => { + const current = readRaw(dir); + if (current?.attemptId !== attemptId) return false; + writeRestartIntentUnlocked(dir, { + ...(current.deferredIntent ?? payloadOf(current)), + attemptId: `aborted:${attemptId}`, + attemptState: 'aborted', + }); + return true; + }); } // ---- default-dir wrappers (production wiring) ---- @@ -196,6 +355,16 @@ export function clearRestartLease(id: string): void { clearRestartLeaseTo(config.session.dataDir, id); } +export function claimRestartIntentForReport( + nowMs: number = Date.now(), +): RestartIntentReportClaim { + return claimRestartIntentForReportTo(config.session.dataDir, nowMs); +} + +export function hasPreparedRestartIntent(nowMs: number = Date.now()): boolean { + return hasPreparedRestartIntentTo(config.session.dataDir, nowMs); +} + export function writeManualIntentIfAbsent(nowMs: number = Date.now()): void { writeManualIntentIfAbsentTo(config.session.dataDir, nowMs, new Date(nowMs).toISOString()); } diff --git a/src/services/sandbox-store.ts b/src/services/sandbox-store.ts index bdaf28a07..beb994285 100644 --- a/src/services/sandbox-store.ts +++ b/src/services/sandbox-store.ts @@ -37,24 +37,46 @@ export function getBotReadIsolation(larkAppId: string): boolean { try { return getBot(larkAppId).config.readIsolation === true; } catch { return false; } } -/** Per-bot read-isolation toggle (macOS Seatbelt read-deny). Same persistence - * contract as {@link updateBotSandbox}: atomic bots.json write + in-memory sync, - * so the next session spawn reads `botCfg.readIsolation` without a daemon restart. */ -export async function updateBotReadIsolation( +/** Restore only the daemon's live spawn view before an async persistence + * rollback. This closes the otherwise observable window where a cold refork + * could consume a transient read-isolation value while bots.json is being + * rolled back after a Codex App ownership conflict. */ +export function setBotReadIsolationRuntime(larkAppId: string, enabled: boolean): void { + try { + const cfg = getBot(larkAppId).config; + if (enabled) cfg.readIsolation = true; + else delete cfg.readIsolation; + } catch { /* the preceding successful update already proves this in production */ } +} + +/** Persist the desired flag without exposing it to worker admission yet. + * Dashboard cold-restart mutations use this staged form so the daemon can + * atomically fence all old generations before publishing the new spawn view. */ +export async function persistBotReadIsolation( larkAppId: string, enabled: boolean, ): Promise<{ ok: true; readIsolation: boolean } | { ok: false; reason: string }> { - let bot; - try { bot = getBot(larkAppId); } catch { return { ok: false, reason: 'bot_not_registered' }; } + try { getBot(larkAppId); } catch { return { ok: false, reason: 'bot_not_registered' }; } const r = await rmwBotEntry(larkAppId, (entry) => { if (enabled) entry.readIsolation = true; - else delete entry.readIsolation; // omit key when off → preserves "absent = off" + else delete entry.readIsolation; return { write: true, result: enabled }; }); if (!r.ok) return { ok: false, reason: r.reason }; + return { ok: true, readIsolation: enabled }; +} - bot.config.readIsolation = enabled; +/** Per-bot read-isolation toggle (macOS Seatbelt read-deny). Same persistence + * contract as {@link updateBotSandbox}: atomic bots.json write + in-memory sync, + * so the next session spawn reads `botCfg.readIsolation` without a daemon restart. */ +export async function updateBotReadIsolation( + larkAppId: string, + enabled: boolean, +): Promise<{ ok: true; readIsolation: boolean } | { ok: false; reason: string }> { + const r = await persistBotReadIsolation(larkAppId, enabled); + if (!r.ok) return { ok: false, reason: r.reason }; + setBotReadIsolationRuntime(larkAppId, enabled); logger.info(`[read-isolation:${larkAppId}] readIsolation → ${enabled}`); return { ok: true, readIsolation: enabled }; } diff --git a/src/services/session-store.ts b/src/services/session-store.ts index e70c649b8..472f1da15 100644 --- a/src/services/session-store.ts +++ b/src/services/session-store.ts @@ -1,4 +1,4 @@ -import { readFileSync, writeFileSync, mkdirSync, existsSync, renameSync, readdirSync } from 'node:fs'; +import { readFileSync, writeFileSync, mkdirSync, existsSync, renameSync, readdirSync, unlinkSync } from 'node:fs'; import { join, dirname } from 'node:path'; import { randomUUID } from 'node:crypto'; import { config } from '../config.js'; @@ -6,6 +6,7 @@ import { logger } from '../utils/logger.js'; import { cleanupMaterializedDashboardImages } from '../core/dashboard-images.js'; import { deleteFrozenCards } from './frozen-card-store.js'; import type { Session } from '../types.js'; +import { withFileLockSync } from '../utils/file-lock.js'; let sessions: Map = new Map(); let loaded = false; @@ -20,6 +21,65 @@ export function stripLegacyPendingCardFields(session: Record): for (const f of LEGACY_PENDING_CARD_FIELDS) delete session[f]; } +/** The exact active row no longer has the lineage/ownership sampled by the + * caller. Unlike an I/O failure, retrying against the caller's live backend + * could overwrite or resume a different durable owner. */ +export class RiffLineageOwnershipError extends Error { + override readonly name = 'RiffLineageOwnershipError'; +} + +export type RiffDurableOwner = { + pid: number | null; + larkAppId: string | null; + backendType: string | null; +}; + +export type ActiveRiffShutdownSnapshot = { + sessionId: string; + taskId: string | null; + owner: RiffDurableOwner; +}; + +export type ActiveRiffLineageBatchUpdate = ActiveRiffShutdownSnapshot & { + targetTaskId: string | null; + expectedCurrentTaskIds: readonly (string | null)[]; +}; + +export type RiffLineageBatchFailureStage = + | 'prewrite_ownership' + | 'prewrite_io' + | 'postrename_ambiguity'; + +export class RiffLineageBatchError extends Error { + override readonly name = 'RiffLineageBatchError'; + constructor( + readonly stage: RiffLineageBatchFailureStage, + readonly sessionIds: readonly string[], + message: string, + ) { + super(message); + } +} + +function riffDurableOwner(session: Session): RiffDurableOwner { + return { + pid: session.pid ?? null, + larkAppId: session.larkAppId ?? null, + backendType: session.backendType ?? null, + }; +} + +function riffOwnersEqual(left: RiffDurableOwner, right: RiffDurableOwner): boolean { + return left.pid === right.pid + && left.larkAppId === right.larkAppId + && left.backendType === right.backendType; +} + +let testOnlyAfterRiffBatchRename: (() => void) | undefined; +export function __testOnly_setAfterRiffBatchRename(hook: (() => void) | undefined): void { + testOnlyAfterRiffBatchRename = hook; +} + /** * Initialise session store for a specific bot (multi-daemon mode). * When appId is set, sessions are stored in `sessions-{appId}.json`. @@ -78,54 +138,58 @@ function load(): void { if (loaded) return; ensureDir(); const fp = getFilePath(); - if (existsSync(fp)) { - try { - const data = JSON.parse(readFileSync(fp, 'utf-8')); - sessions = new Map(Object.entries(data)); - } catch (err) { - logger.error(`Failed to load sessions: ${err}`); - sessions = new Map(); - loaded = true; - return; - } - const repaired = repairMissingChatScopes(); - if (repaired > 0) { + withFileLockSync(fp, () => { + if (existsSync(fp)) { try { - save(); - logger.info(`Repaired ${repaired} scope-less chat session(s) in ${fp}`); + const data = JSON.parse(readFileSync(fp, 'utf-8')); + sessions = new Map(Object.entries(data)); + const repaired = repairMissingChatScopes(); + if (repaired > 0) { + try { + const tmpFp = `${fp}.${process.pid}.${randomUUID()}.tmp`; + writeFileSync(tmpFp, JSON.stringify(Object.fromEntries(sessions), null, 2), 'utf-8'); + renameSync(tmpFp, fp); + logger.info(`Repaired ${repaired} scope-less chat session(s) in ${fp}`); + } catch (err) { + // Loading succeeded, so keep the in-memory sessions available even + // if the best-effort repair cannot be persisted yet. + logger.error(`Failed to persist repaired chat session scopes: ${err}`); + } + } + logger.info(`Loaded ${sessions.size} sessions from ${fp}`); } catch (err) { - // Loading succeeded, so keep the in-memory sessions available even if - // this best-effort migration cannot be persisted yet (ENOSPC/EACCES). - logger.error(`Failed to persist repaired chat session scopes: ${err}`); - } - } - logger.info(`Loaded ${sessions.size} sessions from ${fp}`); - } else if (currentAppId) { - // Per-bot file doesn't exist — migrate matching sessions from legacy sessions.json - const legacyFp = join(config.session.dataDir, 'sessions.json'); - if (existsSync(legacyFp)) { - try { - const data: Record = JSON.parse(readFileSync(legacyFp, 'utf-8')); + logger.error(`Failed to load sessions: ${err}`); sessions = new Map(); - for (const [k, v] of Object.entries(data)) { - if (v.larkAppId === currentAppId) { - sessions.set(k, v); + } + } else if (currentAppId) { + // Per-bot file doesn't exist — migrate matching legacy rows while still + // holding the same lock used by daemon saves and offline CLI mutations. + const legacyFp = join(config.session.dataDir, 'sessions.json'); + if (existsSync(legacyFp)) { + try { + const data: Record = JSON.parse(readFileSync(legacyFp, 'utf-8')); + sessions = new Map(); + for (const [k, v] of Object.entries(data)) { + if (v.larkAppId === currentAppId) sessions.set(k, v); } - } - if (sessions.size > 0) { - const repaired = repairMissingChatScopes(); - save(); - logger.info(`Migrated ${sessions.size} sessions from sessions.json to ${fp}`); - if (repaired > 0) { - logger.info(`Repaired ${repaired} scope-less chat session(s) during migration`); + if (sessions.size > 0) { + const repaired = repairMissingChatScopes(); + const obj = Object.fromEntries(sessions); + const tmpFp = `${fp}.${process.pid}.${randomUUID()}.tmp`; + writeFileSync(tmpFp, JSON.stringify(obj, null, 2), 'utf-8'); + renameSync(tmpFp, fp); + logger.info(`Migrated ${sessions.size} sessions from sessions.json to ${fp}`); + if (repaired > 0) { + logger.info(`Repaired ${repaired} scope-less chat session(s) during migration`); + } } + } catch (err) { + logger.error(`Failed to migrate sessions from legacy file: ${err}`); + sessions = new Map(); } - } catch (err) { - logger.error(`Failed to migrate sessions from legacy file: ${err}`); - sessions = new Map(); } } - } + }); loaded = true; } @@ -139,26 +203,224 @@ function readExistingSessionsFromDisk(fp: string): { raw: string; parsed: Record } } -function save(): void { +function readSessionsProjectionStrict(fp: string): { raw: string; parsed: Record } { + if (!existsSync(fp)) return { raw: '', parsed: {} }; + const raw = readFileSync(fp, 'utf-8'); + const value = JSON.parse(raw) as unknown; + if (!value || typeof value !== 'object' || Array.isArray(value)) { + throw new Error(`invalid sessions projection at ${fp}`); + } + return { raw, parsed: value as Record }; +} + +function duplicateIds(ids: readonly string[]): string[] { + const seen = new Set(); + const duplicates = new Set(); + for (const id of ids) { + if (seen.has(id)) duplicates.add(id); + else seen.add(id); + } + return [...duplicates]; +} + +/** + * Sample every active Riff participant from one fresh sessions projection. + * Graceful fleet shutdown must take this snapshot before it fences any worker: + * sampling rows one-at-a-time can otherwise mix owners from different durable + * generations and leave an already-fenced peer without a trustworthy abort + * handle. + */ +export function getActiveRiffShutdownSnapshotsBatch( + sessionIds: readonly string[], + options: { maxWaitMs?: number } = {}, +): ActiveRiffShutdownSnapshot[] { + if (sessionIds.length === 0) return []; + const duplicates = duplicateIds(sessionIds); + if (duplicates.length > 0) { + throw new RiffLineageBatchError( + 'prewrite_ownership', + duplicates, + `duplicate Riff shutdown session ids: ${duplicates.join(', ')}`, + ); + } + ensureDir(); const fp = getFilePath(); - const { raw: existingRaw } = readExistingSessionsFromDisk(fp); - const obj: Record = {}; - for (const [k, v] of sessions) { - stripLegacyPendingCardFields(v as unknown as Record); - obj[k] = v; + try { + return withFileLockSync(fp, () => { + const { parsed } = readSessionsProjectionStrict(fp); + const invalid = sessionIds.filter((sessionId) => { + const session = parsed[sessionId]; + return !session || session.status !== 'active'; + }); + if (invalid.length > 0) { + throw new RiffLineageBatchError( + 'prewrite_ownership', + invalid, + `cannot snapshot non-active Riff sessions: ${invalid.join(', ')}`, + ); + } + return sessionIds.map((sessionId) => { + const session = parsed[sessionId]!; + return { + sessionId, + taskId: session.riffParentTaskId ?? null, + owner: riffDurableOwner(session), + }; + }); + }, { maxWaitMs: options.maxWaitMs }); + } catch (error) { + if (error instanceof RiffLineageBatchError) throw error; + throw new RiffLineageBatchError( + 'prewrite_io', + [...sessionIds], + `failed to snapshot active Riff sessions: ${String(error)}`, + ); + } +} + +/** + * Commit every prepared Riff lineage as one compare-and-set transaction. + * Every row is checked before the single rename and every published row is + * read back under the same lock before any worker may be ACKed to exit. + */ +export function persistActiveRiffLineagesExactBatch( + updates: readonly ActiveRiffLineageBatchUpdate[], + options: { maxWaitMs?: number } = {}, +): ActiveRiffShutdownSnapshot[] { + if (updates.length === 0) return []; + const sessionIds = updates.map(update => update.sessionId); + const duplicates = duplicateIds(sessionIds); + if (duplicates.length > 0) { + throw new RiffLineageBatchError( + 'prewrite_ownership', + duplicates, + `duplicate Riff lineage batch session ids: ${duplicates.join(', ')}`, + ); + } + + ensureDir(); + const fp = getFilePath(); + let published = false; + let tmpFp: string | undefined; + try { + return withFileLockSync(fp, () => { + const { raw, parsed } = readSessionsProjectionStrict(fp); + const conflicts: string[] = []; + for (const update of updates) { + const durable = parsed[update.sessionId]; + const durableTaskId = durable?.riffParentTaskId ?? null; + if (!durable + || durable.status !== 'active' + || !update.expectedCurrentTaskIds.some(candidate => candidate === durableTaskId) + || !riffOwnersEqual(riffDurableOwner(durable), update.owner)) { + conflicts.push(update.sessionId); + } + } + if (conflicts.length > 0) { + throw new RiffLineageBatchError( + 'prewrite_ownership', + conflicts, + `Riff lineage batch compare-and-set failed for: ${conflicts.join(', ')}`, + ); + } + + for (const update of updates) { + const durable = parsed[update.sessionId]!; + const next: Session = { + ...durable, + riffParentTaskId: update.targetTaskId ?? undefined, + }; + stripLegacyPendingCardFields(next as unknown as Record); + parsed[update.sessionId] = next; + } + + const json = JSON.stringify(parsed, null, 2); + if (json !== raw) { + tmpFp = `${fp}.${process.pid}.${randomUUID()}.tmp`; + writeFileSync(tmpFp, json, 'utf-8'); + renameSync(tmpFp, fp); + tmpFp = undefined; + published = true; + testOnlyAfterRiffBatchRename?.(); + } + + let verifiedProjection: Record; + try { + verifiedProjection = readSessionsProjectionStrict(fp).parsed; + } catch (error) { + throw new RiffLineageBatchError( + published ? 'postrename_ambiguity' : 'prewrite_io', + [...sessionIds], + `failed to read back Riff lineage batch: ${String(error)}`, + ); + } + + const ambiguous = updates.filter((update) => { + const durable = verifiedProjection[update.sessionId]; + return !durable + || durable.status !== 'active' + || (durable.riffParentTaskId ?? null) !== update.targetTaskId + || !riffOwnersEqual(riffDurableOwner(durable), update.owner); + }).map(update => update.sessionId); + if (ambiguous.length > 0) { + throw new RiffLineageBatchError( + published ? 'postrename_ambiguity' : 'prewrite_ownership', + ambiguous, + `Riff lineage batch readback mismatch for: ${ambiguous.join(', ')}`, + ); + } + + const verified = updates.map((update) => ({ + sessionId: update.sessionId, + taskId: update.targetTaskId, + owner: riffDurableOwner(verifiedProjection[update.sessionId]!), + })); + + // The durable projection remains authoritative. Only mirror the exact + // verified rows into an already-loaded process cache after readback. + if (loaded) { + for (const update of updates) { + const cached = sessions.get(update.sessionId); + if (cached) cached.riffParentTaskId = update.targetTaskId ?? undefined; + } + } + return verified; + }, { maxWaitMs: options.maxWaitMs }); + } catch (error) { + if (error instanceof RiffLineageBatchError) throw error; + throw new RiffLineageBatchError( + published ? 'postrename_ambiguity' : 'prewrite_io', + [...sessionIds], + `failed to persist Riff lineage batch: ${String(error)}`, + ); + } finally { + if (tmpFp) { + try { unlinkSync(tmpFp); } catch { /* best-effort orphan cleanup */ } + } } - const json = JSON.stringify(obj, null, 2); - // The daemon fires several updateSession()/save() calls per inbound message - // (activity bump, pid, stream-card state, …) and many leave the serialized - // file byte-identical. Skipping the temp-file write + rename in that case - // elides the bulk of the redundant disk I/O — and writing identical bytes is - // a guaranteed no-op, so this can't drop state or race a concurrent writer - // (we compare against what's actually on disk right now). - if (json === existingRaw) return; - const tmpFp = `${fp}.${process.pid}.${randomUUID()}.tmp`; - writeFileSync(tmpFp, json, 'utf-8'); - renameSync(tmpFp, fp); +} + +function save(): void { + ensureDir(); + const fp = getFilePath(); + withFileLockSync(fp, () => { + const { raw: existingRaw } = readExistingSessionsFromDisk(fp); + const obj: Record = {}; + for (const [k, v] of sessions) { + stripLegacyPendingCardFields(v as unknown as Record); + obj[k] = v; + } + const json = JSON.stringify(obj, null, 2); + // The daemon fires several updateSession()/save() calls per inbound message + // (activity bump, pid, stream-card state, …) and many leave the serialized + // file byte-identical. Skipping the temp-file write + rename in that case + // elides the bulk of the redundant disk I/O. + if (json === existingRaw) return; + const tmpFp = `${fp}.${process.pid}.${randomUUID()}.tmp`; + writeFileSync(tmpFp, json, 'utf-8'); + renameSync(tmpFp, fp); + }); } export function createSession( @@ -190,6 +452,24 @@ export function getSession(sessionId: string): Session | undefined { return sessions.get(sessionId) ?? findInOtherFiles(sessionId); } +/** Cross-process fresh read for worker-side authorization. Workers initialise + * this store once, but the daemon rotates Codex dispatch ledgers in a different + * process; the ordinary in-memory cache is therefore not an authority for a + * later turn. The shared file lock orders this read after daemon/CLI writes. */ +export function getSessionFresh(sessionId: string): Session | undefined { + ensureDir(); + const fp = getFilePath(); + return withFileLockSync(fp, () => { + if (!existsSync(fp)) return undefined; + try { + const data = JSON.parse(readFileSync(fp, 'utf-8')) as Record; + return data[sessionId]; + } catch { + return undefined; + } + }); +} + /** * Search all session files for a session not found in the current file. * @@ -215,27 +495,160 @@ function findInOtherFiles(sessionId: string): Session | undefined { return undefined; } -export function closeSession(sessionId: string): void { +export function closeSession( + sessionId: string, + opts: { clearRiffParentTaskId?: boolean } = {}, +): void { load(); const session = sessions.get(sessionId); if (session) { - if (session.larkAppId && session.dashboardAttachments?.length) { + const priorStatus = session.status; + const priorClosedAt = session.closedAt; + const priorPid = session.pid; + const priorLedger = session.codexAppDispatchLedger; + const priorCommits = session.codexAppGenerationCommits; + const priorRiffParentTaskId = session.riffParentTaskId; + const priorDashboardAttachments = session.dashboardAttachments; + const priorQueuedAttachments = session.queuedAttachments; + const priorQueuedOwnership = { + queued: session.queued, + queuedPrompt: session.queuedPrompt, + queuedCodexAppText: session.queuedCodexAppText, + queuedCodexAppMessageContext: session.queuedCodexAppMessageContext, + queuedActivationPending: session.queuedActivationPending, + queuedActivationToken: session.queuedActivationToken, + queuedActivationInput: session.queuedActivationInput, + queuedActivationTurnId: session.queuedActivationTurnId, + queuedActivationDispatchAttempt: session.queuedActivationDispatchAttempt, + queuedActivationResume: session.queuedActivationResume, + queuedActivationTail: session.queuedActivationTail, + queuedActivationTailNextOrder: session.queuedActivationTailNextOrder, + pendingRepoSetup: session.pendingRepoSetup, + }; + session.status = 'closed'; + session.closedAt = new Date().toISOString(); + // A persisted worker pid is not a teardown identity: after close it can be + // reused by an unrelated process and make later policy checks report a + // permanent false-live row. Runtime worker/pane teardown remains tracked by + // the active registry and stamped persistent backend. + session.pid = undefined; + // Closing is an explicit abandon boundary, unlike suspend/daemon crash. + // Never let a later generic resume replay prepared Codex App input the + // user intentionally discarded, and retire generation ACK history with + // the abandoned FIFO. + session.codexAppDispatchLedger = undefined; + session.codexAppGenerationCommits = undefined; + session.queued = undefined; + session.queuedPrompt = undefined; + session.queuedCodexAppText = undefined; + session.queuedCodexAppMessageContext = undefined; + session.queuedActivationPending = undefined; + session.queuedActivationToken = undefined; + session.queuedActivationInput = undefined; + session.queuedActivationTurnId = undefined; + session.queuedActivationDispatchAttempt = undefined; + session.queuedActivationResume = undefined; + session.queuedActivationTail = undefined; + session.queuedActivationTailNextOrder = undefined; + session.pendingRepoSetup = undefined; + session.dashboardAttachments = undefined; + session.queuedAttachments = undefined; + // A successful explicit Riff close has already confirmed remote + // cancellation. Clear its retry handle in the SAME atomic file replace as + // the closed status; clearing it in a prior update loses lineage when this + // save fails between prepare and commit. + if (opts.clearRiffParentTaskId) session.riffParentTaskId = undefined; + try { + save(); + } catch (err) { + session.status = priorStatus; + session.closedAt = priorClosedAt; + session.pid = priorPid; + session.codexAppDispatchLedger = priorLedger; + session.codexAppGenerationCommits = priorCommits; + session.riffParentTaskId = priorRiffParentTaskId; + session.dashboardAttachments = priorDashboardAttachments; + session.queuedAttachments = priorQueuedAttachments; + Object.assign(session, priorQueuedOwnership); + throw err; + } + if (session.larkAppId && priorDashboardAttachments?.length) { try { - cleanupMaterializedDashboardImages(session.larkAppId, session.dashboardAttachments); - session.dashboardAttachments = undefined; - session.queuedAttachments = undefined; + cleanupMaterializedDashboardImages(session.larkAppId, priorDashboardAttachments); } catch (error: any) { logger.warn(`Failed to clean Dashboard images for session ${sessionId}: ${error?.message ?? error}`); } } - session.status = 'closed'; - session.closedAt = new Date().toISOString(); - save(); deleteFrozenCards(sessionId); logger.info(`Closed session ${sessionId}`); } } +/** + * Reactivate one explicitly closed row and discard every queued/setup owner in + * the same durable file replacement. The close path has cleared these fields + * since 2026-07, but older closed rows can still contain prepared input. A + * generic resume is an explicit new lifecycle and must never revive that + * abandoned FIFO. + */ +export function reactivateClosedSession( + sessionId: string, +): { ok: true; session: Session } +| { ok: false; error: 'not_found' | 'not_closed' } { + load(); + const session = sessions.get(sessionId); + if (!session) return { ok: false, error: 'not_found' }; + if (session.status !== 'closed') return { ok: false, error: 'not_closed' }; + + const prior = { + status: session.status, + closedAt: session.closedAt, + lastMessageAt: session.lastMessageAt, + codexAppDispatchLedger: session.codexAppDispatchLedger, + codexAppGenerationCommits: session.codexAppGenerationCommits, + queued: session.queued, + queuedPrompt: session.queuedPrompt, + queuedCodexAppText: session.queuedCodexAppText, + queuedCodexAppMessageContext: session.queuedCodexAppMessageContext, + queuedActivationPending: session.queuedActivationPending, + queuedActivationToken: session.queuedActivationToken, + queuedActivationInput: session.queuedActivationInput, + queuedActivationTurnId: session.queuedActivationTurnId, + queuedActivationDispatchAttempt: session.queuedActivationDispatchAttempt, + queuedActivationResume: session.queuedActivationResume, + queuedActivationTail: session.queuedActivationTail, + queuedActivationTailNextOrder: session.queuedActivationTailNextOrder, + pendingRepoSetup: session.pendingRepoSetup, + }; + + session.status = 'active'; + session.closedAt = undefined; + session.lastMessageAt = new Date().toISOString(); + session.codexAppDispatchLedger = undefined; + session.codexAppGenerationCommits = undefined; + session.queued = undefined; + session.queuedPrompt = undefined; + session.queuedCodexAppText = undefined; + session.queuedCodexAppMessageContext = undefined; + session.queuedActivationPending = undefined; + session.queuedActivationToken = undefined; + session.queuedActivationInput = undefined; + session.queuedActivationTurnId = undefined; + session.queuedActivationDispatchAttempt = undefined; + session.queuedActivationResume = undefined; + session.queuedActivationTail = undefined; + session.queuedActivationTailNextOrder = undefined; + session.pendingRepoSetup = undefined; + + try { + save(); + } catch (err) { + Object.assign(session, prior); + throw err; + } + return { ok: true, session }; +} + export function updateSessionPid(sessionId: string, pid: number | null): void { load(); const session = sessions.get(sessionId); @@ -251,6 +664,89 @@ export function updateSession(session: Session): void { save(); } +/** Persist the exact Riff follow-up lineage for an ACTIVE row, transactionally + * with respect to this process's in-memory store. Graceful shutdown uses this + * before ACKing a worker detach: `updateSession()` cannot provide that ACK + * because it installs the caller's object in the Map before save(), leaving a + * false in-memory success when the atomic file replace throws. + * + * `null` is authoritative and clears a stale parent. The caller should follow + * with getSessionFresh() when it needs proof that the on-disk row contains the + * exact value before allowing a remote owner to exit. */ +export function persistActiveRiffLineageExact( + sessionId: string, + taskId: string | null, + options: { + expectedCurrentTaskIds?: readonly (string | null)[]; + expectedOwner?: { + pid: number | null; + larkAppId: string | null; + backendType: string | null; + }; + } = {}, +): Session { + load(); + ensureDir(); + const fp = getFilePath(); + return withFileLockSync(fp, () => { + const { raw, parsed } = readExistingSessionsFromDisk(fp); + const durable = parsed[sessionId]; + if (!durable || durable.status !== 'active') { + throw new RiffLineageOwnershipError( + `cannot persist Riff lineage for non-active session ${sessionId}`, + ); + } + const durableTaskId = durable.riffParentTaskId ?? null; + const expected = options.expectedCurrentTaskIds; + if (expected && !expected.some(candidate => candidate === durableTaskId)) { + throw new RiffLineageOwnershipError( + `Riff lineage compare-and-set failed for ${sessionId} ` + + `(current=${durableTaskId ?? 'none'}, expected=${expected.map(id => id ?? 'none').join('|')})`, + ); + } + const expectedOwner = options.expectedOwner; + const durableOwner = { + pid: durable.pid ?? null, + larkAppId: durable.larkAppId ?? null, + backendType: durable.backendType ?? null, + }; + if (expectedOwner + && (durableOwner.pid !== expectedOwner.pid + || durableOwner.larkAppId !== expectedOwner.larkAppId + || durableOwner.backendType !== expectedOwner.backendType)) { + throw new RiffLineageOwnershipError( + `Riff owner compare-and-set failed for ${sessionId} ` + + `(current=${JSON.stringify(durableOwner)}, expected=${JSON.stringify(expectedOwner)})`, + ); + } + + // Merge only this exact fresh row. Serializing the process-wide `sessions` + // cache here can leak unrelated runtime-only mutations from another + // prepared fleet participant into the durable file before its own commit. + const next: Session = { + ...durable, + riffParentTaskId: taskId ?? undefined, + }; + stripLegacyPendingCardFields(next as unknown as Record); + parsed[sessionId] = next; + const json = JSON.stringify(parsed, null, 2); + if (json !== raw) { + const tmpFp = `${fp}.${process.pid}.${randomUUID()}.tmp`; + writeFileSync(tmpFp, json, 'utf-8'); + renameSync(tmpFp, fp); + } + + // Publish into the local cache only after the atomic replace succeeded. + const cached = sessions.get(sessionId); + if (cached) { + cached.riffParentTaskId = taskId ?? undefined; + return cached; + } + sessions.set(sessionId, next); + return next; + }); +} + export function listSessions(): Session[] { load(); return [...sessions.values()]; diff --git a/src/services/structured-bridge-clis.ts b/src/services/structured-bridge-clis.ts index a0b378955..d4dbd3701 100644 --- a/src/services/structured-bridge-clis.ts +++ b/src/services/structured-bridge-clis.ts @@ -49,6 +49,21 @@ export const STRUCTURED_BRIDGE_ADOPT_CLI_IDS = [ const ALWAYS_SET: ReadonlySet = new Set(STRUCTURED_BRIDGE_ALWAYS_CLI_IDS); const ADOPT_SET: ReadonlySet = new Set(STRUCTURED_BRIDGE_ADOPT_CLI_IDS); +/** Drivers whose transcript contract exposes every terminal edge needed for a + * strong started-turn status gate. Codex has final_answer plus explicit + * turn_aborted parsing. Other structured drivers still + * use the queue for attribution, but their interrupted/error shapes are not + * yet complete enough to let a started turn suppress screen-ready forever. */ +export const STRUCTURED_BRIDGE_LIFECYCLE_BLOCKING_CLI_IDS = [ + 'codex', +] as const satisfies readonly CliId[]; + +const LIFECYCLE_BLOCKING_SET: ReadonlySet = new Set(STRUCTURED_BRIDGE_LIFECYCLE_BLOCKING_CLI_IDS); + +export function isStructuredBridgeLifecycleBlockingCli(cliId: string | undefined): boolean { + return !!cliId && LIFECYCLE_BLOCKING_SET.has(cliId); +} + /** Worker `codexBridgeFallbackActive` — cursor only when adoptMode. */ export function isStructuredBridgeFallbackActive( cliId: string | undefined, @@ -90,6 +105,7 @@ export function isStructuredBridgeAdoptIdleCli(cliId: string | undefined): boole export const STRUCTURED_BRIDGE_ADOPT_INPUT_CLI_IDS = [ 'codex', 'traex', + 'coco', 'pi', 'grok', 'mtr', diff --git a/src/services/submit-confirmation.ts b/src/services/submit-confirmation.ts index 1aaf1aa80..2fe6080ee 100644 --- a/src/services/submit-confirmation.ts +++ b/src/services/submit-confirmation.ts @@ -1,3 +1,5 @@ +import type { SubmitRecheckResult } from '../adapters/cli/types.js'; + export type SubmitActivityEvidence = 'pty-output' | 'structured-transcript' | 'botmux-send'; export type SubmitConfirmationAction = @@ -21,3 +23,136 @@ export function decideSubmitConfirmationAction(input: SubmitConfirmationDecision if (input.activityEvidence) return { kind: 'suppress-active', evidence: input.activityEvidence }; return { kind: 'notify-stuck' }; } + +export interface StructuredSubmitLifecycleController { + hasPendingTurn(turnId: string, dispatchAttempt?: number): boolean; + confirmPendingTurn(turnId: string, confirmedAtMs?: number, dispatchAttempt?: number): boolean; + finishSubmitVerification(turnId: string, finishedAtMs?: number, dispatchAttempt?: number): boolean; +} + +export interface DeferredSubmitConfirmationInput { + turnId?: string; + dispatchAttempt?: number; + /** True only when turnId names a CodexBridgeQueue mark. Claude fallback + * turns use BridgeTurnQueue IDs, so treating every bridge ID as a + * structured target would incorrectly discard every Claude recheck. */ + structuredTarget?: boolean; + recheck?: () => SubmitRecheckResult | Promise; + usageLimitDetected: () => boolean; + activityEvidence: () => SubmitActivityEvidence | undefined; + /** Generation/backend fence owned by the worker. It is checked both before + * invoking the old recheck closure and after its await resolves. */ + isCurrent?: () => boolean; +} + +export interface CurrentDeferredSubmitConfirmationSettlement { + stale: false; + action: SubmitConfirmationAction; + cliSessionId?: string; + recheckError?: unknown; + lifecycle: 'confirmed' | 'verification-finished' | 'unchanged'; +} + +export interface StaleDeferredSubmitConfirmationSettlement { + stale: true; + staleReason: 'generation' | 'attempt'; + lifecycle: 'unchanged'; +} + +export type DeferredSubmitConfirmationSettlement = + | CurrentDeferredSubmitConfirmationSettlement + | StaleDeferredSubmitConfirmationSettlement; + +export type StaleWriteContinuationDisposition = 'ambiguous-terminal' | 'ordinary-carryover'; + +/** Settle an await continuation whose CLI generation is no longer current. + * Ordinary inputs are already owned by InflightInputTracker's crash carryover: + * touching the process-global bridge queues or warning the user here could + * delete the replacement generation's same-ID mark or invite a duplicate + * manual retry. Durable inputs are not auto-replayed and therefore need their + * exact attempt terminal reconciled by the receiver. */ +export function settleStaleWriteContinuation( + identity: { turnId?: string; dispatchAttempt?: number }, + errorCode: string, + emitAmbiguous: (turnId: string, errorCode: string, dispatchAttempt: number) => void, +): StaleWriteContinuationDisposition { + if (identity.turnId && identity.dispatchAttempt !== undefined) { + emitAmbiguous(identity.turnId, errorCode, identity.dispatchAttempt); + return 'ambiguous-terminal'; + } + return 'ordinary-carryover'; +} + +/** Execute the deferred submit-recheck callback and settle the structured + * pre-start lifecycle atomically with its decision. Both an actual history + * recheck hit and weaker "CLI is active" evidence become a bounded confirmed + * lease. The latter must not merely clear verification: that would leave a + * bare, unstarted fingerprint which no expiry pruning can ever remove. */ +export async function settleDeferredSubmitConfirmation( + controller: StructuredSubmitLifecycleController, + input: DeferredSubmitConfirmationInput, +): Promise { + const targetStillCurrent = (): StaleDeferredSubmitConfirmationSettlement | undefined => { + if (input.isCurrent && !input.isCurrent()) { + return { stale: true, staleReason: 'generation', lifecycle: 'unchanged' }; + } + if (input.structuredTarget + && input.turnId + && !controller.hasPendingTurn(input.turnId, input.dispatchAttempt)) { + return { stale: true, staleReason: 'attempt', lifecycle: 'unchanged' }; + } + return undefined; + }; + + // A durable attempt can expire/restart during the 20s delay before this + // function is called. Never run its old adapter closure against a new CLI. + const staleBeforeRecheck = targetStillCurrent(); + if (staleBeforeRecheck) return staleBeforeRecheck; + + let recheckSubmitted = false; + let cliSessionId: string | undefined; + let recheckError: unknown; + if (input.recheck) { + try { + const result = await input.recheck(); + recheckSubmitted = typeof result === 'boolean' ? result : result.submitted === true; + cliSessionId = typeof result === 'object' && result && typeof result.cliSessionId === 'string' + ? result.cliSessionId + : undefined; + } catch (err) { + recheckError = err; + } + } + + // The old closure can itself await filesystem/CLI state. Re-check both the + // CLI generation and exact attempt before any lifecycle or caller effect. + const staleAfterRecheck = targetStillCurrent(); + if (staleAfterRecheck) return staleAfterRecheck; + + const action = decideSubmitConfirmationAction({ + recheckSubmitted, + usageLimitDetected: input.usageLimitDetected(), + activityEvidence: input.activityEvidence(), + }); + + let lifecycle: DeferredSubmitConfirmationSettlement['lifecycle'] = 'unchanged'; + if (input.structuredTarget && input.turnId) { + if (action.kind === 'suppress-confirmed' || action.kind === 'suppress-active') { + if (controller.confirmPendingTurn(input.turnId, undefined, input.dispatchAttempt)) { + lifecycle = 'confirmed'; + } else if (controller.finishSubmitVerification(input.turnId, undefined, input.dispatchAttempt)) { + lifecycle = 'verification-finished'; + } + } else if (controller.finishSubmitVerification(input.turnId, undefined, input.dispatchAttempt)) { + lifecycle = 'verification-finished'; + } + } + + return { + stale: false, + action, + ...(cliSessionId ? { cliSessionId } : {}), + ...(recheckError !== undefined ? { recheckError } : {}), + lifecycle, + }; +} diff --git a/src/services/voice/index.ts b/src/services/voice/index.ts index 39a32473b..a140acd24 100644 --- a/src/services/voice/index.ts +++ b/src/services/voice/index.ts @@ -9,7 +9,7 @@ * default). Neither pulls a heavyweight dependency into the package. */ import { encodePcmToOpus, toSpoken, type OpusResult, type Pcm } from './audio.js'; -import { samiSynthesizePcm, type SamiCreds } from './sami.js'; +import { samiSynthesizePcm, type SamiCreds, type VoiceProviderEffectOptions } from './sami.js'; import { openaiSynthesizePcm, type OpenAITtsConfig } from './openai.js'; import { readGlobalConfig } from '../../global-config.js'; import { loadBotConfigs } from '../../bot-registry.js'; @@ -84,7 +84,11 @@ function effectiveSpeaker(v: VoiceConfig): string { * Synthesize `text` into raw signed-16 PCM. Realtime VC voice uses this * intermediate directly; IM voice bubbles encode it to ogg/opus below. */ -export async function synthesizeVoicePcmForMessage(larkAppId: string | undefined, text: string): Promise { +export async function synthesizeVoicePcmForMessage( + larkAppId: string | undefined, + text: string, + effects: VoiceProviderEffectOptions = {}, +): Promise { const cfg = resolveVoiceConfig(larkAppId); if (!cfg) throw new Error('未配置语音引擎：在 ~/.botmux/config.json 的 voice 块或 bots.json 里配置 SAMI / OpenAI 兼容引擎。'); const spoken = toSpoken(text); @@ -92,8 +96,8 @@ export async function synthesizeVoicePcmForMessage(larkAppId: string | undefined const speaker = effectiveSpeaker(cfg); return cfg.engine === 'openai' - ? await openaiSynthesizePcm(cfg.openai as OpenAITtsConfig, spoken, { speaker, rate: cfg.rate }) - : await samiSynthesizePcm(cfg.sami as SamiCreds, spoken, { speaker, rate: cfg.rate }); + ? await openaiSynthesizePcm(cfg.openai as OpenAITtsConfig, spoken, { speaker, rate: cfg.rate }, effects) + : await samiSynthesizePcm(cfg.sami as SamiCreds, spoken, { speaker, rate: cfg.rate }, effects); } /** @@ -101,7 +105,11 @@ export async function synthesizeVoicePcmForMessage(larkAppId: string | undefined * Caller owns the returned temp dir and must rm -rf it after sending. * Throws when voice isn't configured for this bot or synthesis fails. */ -export async function synthesizeVoiceOpus(larkAppId: string | undefined, text: string): Promise { - const pcm = await synthesizeVoicePcmForMessage(larkAppId, text); +export async function synthesizeVoiceOpus( + larkAppId: string | undefined, + text: string, + effects: VoiceProviderEffectOptions = {}, +): Promise { + const pcm = await synthesizeVoicePcmForMessage(larkAppId, text, effects); return encodePcmToOpus(pcm); } diff --git a/src/services/voice/openai.ts b/src/services/voice/openai.ts index a17ad101e..468705476 100644 --- a/src/services/voice/openai.ts +++ b/src/services/voice/openai.ts @@ -10,6 +10,7 @@ * that don't implement pcm should be configured against one that does. */ import type { Pcm } from './audio.js'; +import type { VoiceProviderEffectOptions } from './sami.js'; export interface OpenAITtsConfig { baseUrl: string; // e.g. https://api.openai.com/v1 or http://127.0.0.1:8880/v1 @@ -25,7 +26,12 @@ export interface OpenAISynthOpts { timeoutMs?: number; } -export async function openaiSynthesizePcm(cfg: OpenAITtsConfig, text: string, opts: OpenAISynthOpts): Promise { +export async function openaiSynthesizePcm( + cfg: OpenAITtsConfig, + text: string, + opts: OpenAISynthOpts, + effects: VoiceProviderEffectOptions = {}, +): Promise { const clean = text.trim(); if (!clean) throw new Error('没有要合成的文字'); if (!cfg.baseUrl || !cfg.model) throw new Error('OpenAI 兼容引擎配置不完整（需要 baseUrl / model）。'); @@ -41,6 +47,7 @@ export async function openaiSynthesizePcm(cfg: OpenAITtsConfig, text: string, op const controller = new AbortController(); const timer = setTimeout(() => controller.abort(), opts.timeoutMs ?? 60000); try { + await effects.beforeProviderEffect?.(); const res = await fetch(url, { method: 'POST', headers: { diff --git a/src/services/voice/sami.ts b/src/services/voice/sami.ts index 5d9589c66..08b4d2a0d 100644 --- a/src/services/voice/sami.ts +++ b/src/services/voice/sami.ts @@ -29,6 +29,10 @@ export interface SamiCreds { wsUrl?: string; } +export interface VoiceProviderEffectOptions { + beforeProviderEffect?: () => void | Promise; +} + const AUTH_VERSION = 'auth-v1'; const OK = 20000000; // SAMI success status_code const SAMI_SR = 24000; @@ -42,7 +46,11 @@ function samiEndpoint(fromConfig: string | undefined, envName: string, fallback: /** Mint a short-lived SAMI token (default 24h). Two-step HMAC: sign the * canonical string with the SecretKey, then HMAC an empty body with that. */ -export async function mintSamiToken(creds: SamiCreds, expiration = 86400): Promise { +export async function mintSamiToken( + creds: SamiCreds, + expiration = 86400, + effects: VoiceProviderEffectOptions = {}, +): Promise { if (!creds.accessKey || !creds.secretKey || !creds.appkey) { throw new Error('SAMI 凭证不完整（需要 accessKey / secretKey / appkey）。'); } @@ -60,6 +68,7 @@ export async function mintSamiToken(creds: SamiCreds, expiration = 86400): Promi signature, }); const tokenUrl = samiEndpoint(creds.tokenUrl, 'SAMI_TOKEN_URL', DEFAULT_TOKEN_URL); + await effects.beforeProviderEffect?.(); const res = await fetch(`${tokenUrl}?${qs.toString()}`, { cache: 'no-store' }); const data = (await res.json()) as { token?: string; status_text?: string }; if (!data.token) throw new Error(`SAMI token 签发失败：${data.status_text ?? JSON.stringify(data).slice(0, 160)}`); @@ -74,10 +83,15 @@ export interface SamiSynthOpts { } /** Synthesize text → raw PCM (s16le mono, 24kHz) via the tts.sync WebSocket. */ -export async function samiSynthesizePcm(creds: SamiCreds, text: string, opts: SamiSynthOpts): Promise { +export async function samiSynthesizePcm( + creds: SamiCreds, + text: string, + opts: SamiSynthOpts, + effects: VoiceProviderEffectOptions = {}, +): Promise { const clean = text.trim(); if (!clean) throw new Error('没有要合成的文字'); - const token = await mintSamiToken(creds); + const token = await mintSamiToken(creds, 86400, effects); const audio_config: Record = { format: 'pcm', sample_rate: SAMI_SR }; const config: Record = { text: clean, speaker: opts.speaker, audio_config }; @@ -102,6 +116,7 @@ export async function samiSynthesizePcm(creds: SamiCreds, text: string, opts: Sa }; const wsUrl = samiEndpoint(creds.wsUrl, 'SAMI_WS_URL', DEFAULT_WS_URL); + await effects.beforeProviderEffect?.(); return await new Promise((resolve, reject) => { const ws = new WebSocket(wsUrl); const chunks: Buffer[] = []; @@ -117,7 +132,18 @@ export async function samiSynthesizePcm(creds: SamiCreds, text: string, opts: Sa }; const timer = setTimeout(() => finish(new Error('SAMI 合成超时')), opts.timeoutMs ?? 60000); - ws.on('open', () => ws.send(JSON.stringify(req))); + ws.on('open', async () => { + try { + // The socket may spend an arbitrary amount of time connecting after + // the pre-connect fence above. Revalidate at the last async boundary + // before the provider-visible request leaves this process. + await effects.beforeProviderEffect?.(); + if (done) return; + ws.send(JSON.stringify(req)); + } catch (e) { + finish(e instanceof Error ? e : new Error(String(e))); + } + }); ws.on('message', (data: Buffer, isBinary: boolean) => { if (isBinary) { chunks.push(data); return; } let m: any; diff --git a/src/setup/bots-store.ts b/src/setup/bots-store.ts index 12835d3a1..33746b052 100644 --- a/src/setup/bots-store.ts +++ b/src/setup/bots-store.ts @@ -5,12 +5,18 @@ * - 文件权限 0o600 (只有用户自己能读), secret 不外泄给同机器人其它用户 */ import { writeFileSync, renameSync, existsSync, readFileSync } from 'node:fs'; +import { withFileLockSync } from '../utils/file-lock.js'; export function writeBotsJsonAtomic(botsJsonPath: string, bots: any[]): void { - // 注意: tmp 必须在同一目录下 (同 fs), 否则 rename 可能跨文件系统失败. - const tmp = botsJsonPath + '.tmp'; - writeFileSync(tmp, JSON.stringify(bots, null, 2) + '\n', { mode: 0o600 }); - renameSync(tmp, botsJsonPath); + // PM2 start surfaces hold this same generation lock from snapshot through + // post-start verification/rollback, so the ecosystem and its expected names + // can never be built from different bots.json generations. + withFileLockSync(botsJsonPath, () => { + // 注意: tmp 必须在同一目录下 (同 fs), 否则 rename 可能跨文件系统失败. + const tmp = botsJsonPath + '.tmp'; + writeFileSync(tmp, JSON.stringify(bots, null, 2) + '\n', { mode: 0o600 }); + renameSync(tmp, botsJsonPath); + }); } export function readBotsJsonOrEmpty(botsJsonPath: string): any[] { diff --git a/src/types.ts b/src/types.ts index 85f27bb04..a2cfdbd7d 100644 --- a/src/types.ts +++ b/src/types.ts @@ -98,7 +98,7 @@ export interface VcMeetingConsumerConfig { } /** Runtime status the worker derives from screen content. */ -export type ScreenStatus = 'working' | 'idle' | 'analyzing' | 'limited'; +export type ScreenStatus = 'working' | 'idle' | 'analyzing' | 'limited' | 'stalled'; /** Status shown on a streaming card — adds the pre-spawn 'starting' phase. */ export type StreamStatus = ScreenStatus | 'starting'; @@ -170,6 +170,28 @@ export interface Session { /** Dashboard-created image files retained while the session is active so * the CLI can read them, then removed by session-store on close. */ dashboardAttachments?: LarkAttachment[]; + /** Durable journal for a queued activation until the worker confirms that + * the exact initial input crossed its adapter submission boundary. */ + queuedActivationPending?: boolean; + /** Generation-scoped identity for the exact queued activation journal. */ + queuedActivationToken?: string; + /** Exact final worker input retained for retry. This may include a triggering + * group reply in addition to the original dashboard backlog prompt. */ + queuedActivationInput?: CliTurnPayload; + queuedActivationTurnId?: string; + queuedActivationDispatchAttempt?: number; + /** Frozen resume mode for the exact journal head. Backlog activations are + * fresh (`false`); post-history promoted successors resume (`true`). */ + queuedActivationResume?: boolean; + /** Durable FIFO of exact turns accepted while an activation/setup gate owns + * the route. Entries are removed only when promoted into the tokened journal. */ + queuedActivationTail?: QueuedActivationTailEntry[]; + /** Monotonic per-session reservation cursor. Reserved synchronously before + * async prompt materialization so completion order cannot reorder arrivals. */ + queuedActivationTailNextOrder?: number; + /** Crash-safe pending-repo owner. Presence means picker/worktree setup must + * be restored instead of classifying the active worker:null row as scratch. */ + pendingRepoSetup?: PendingRepoSetup; createdAt: string; /** Last user/bot/scheduler input that was routed into this session. */ lastMessageAt?: string; @@ -247,6 +269,9 @@ export interface Session { replyTargets?: Record; /** True once a substitute-mode control card has been DM'd to the owner(s). Persisted to avoid re-sends on worker restart or daemon recovery. */ substituteControlCardSent?: boolean; + /** Bounded exact destination captured at inbound turn start. Codex App copies + * this into its durable dispatch ledger after any intervening awaits. */ + turnReplyContexts?: Record; /** * 文档评论入口（/watch-comment / /subscribe-lark-doc）：当本会话「当前这一轮」由飞书文档评论 * 触发时，`botmux send` 的用户可见回复要回到该文档评论（而非飞书）。因 botmux @@ -278,6 +303,10 @@ export interface Session { /** Structured companion for lastCliInput so retry_last_task can preserve a * clean Codex App turn. The legacy string remains authoritative fallback. */ lastCodexAppInput?: CodexAppTurnInput; + /** Crash-safe Codex App accepted/prepared FIFO; daemon is the sole writer. */ + codexAppDispatchLedger?: CodexAppDispatchLedgerEntry[]; + /** Cumulative ACK boundary retained until a fresh runner retires the generation. */ + codexAppGenerationCommits?: CodexAppGenerationCommit[]; /** Default local project whiteboard bound to this session when the optional whiteboard feature is enabled. */ whiteboardId?: string; /** CLI-native resume id when it differs from botmux's sessionId (for example Codex thread id). */ @@ -534,30 +563,138 @@ export interface CodexAppTurnInput { clientUserMessageId?: string; } +/** Daemon-frozen Lark destination for one accepted turn. This is separate + * from `currentReplyTarget`, which is mutable session UI state. */ +export type FrozenSessionReplyTarget = + | { mode: 'plain'; chatId: string } + | { mode: 'thread'; rootMessageId: string } + | { mode: 'quote'; rootMessageId: string }; + +export interface FrozenSessionReplyContext { + target: FrozenSessionReplyTarget; + quoteTargetId?: string; + replyTargetSenderOpenId?: string; + replyTargetSenderIsBot?: boolean; +} + +/** Host-side destination frozen when a Codex App turn crosses daemon + * acceptance. Transient HTTP/silent sinks deliberately carry only their kind: + * after daemon restart their in-memory consumer no longer exists, so recovery + * must fail closed instead of leaking the result into ordinary Lark IM. */ +export type CodexAppDeliverySink = + | 'lark' + | 'doc_comment' + | 'http_wait' + | 'http_async' + | 'suppressed'; + +/** + * Daemon-owned durable attribution for one Codex App runner submission. + * + * `accepted` means the daemon accepted the IM turn but the worker has not yet + * crossed the runner write boundary. `prepared` is published by the worker + * immediately before that write. It is restored into a replacement FIFO, but + * remains uncertain until a signed final arrives. A replacement runner's idle + * marker cannot prove the prior generation never buffered the frame, so warm + * prepared recovery fails closed and requires exact generation fencing. + */ +export interface CodexAppDispatchLedgerEntry { + dispatchId: string; + turnId: string; + /** Links a queued activation journal to the exact accepted FIFO item. */ + queuedActivationToken?: string; + /** Frozen chat/thread routing identity. May differ from daemon-minted + * turnId for scheduler/card inputs that arrived without a native message id. */ + replyTurnId?: string; + /** Exact daemon-owned destination captured when the inbound turn began. */ + replyTarget?: FrozenSessionReplyTarget; + quoteTargetId?: string; + replyTargetSenderOpenId?: string; + replyTargetSenderIsBot?: boolean; + /** Exact output channel selected at acceptance. Undefined is legacy Lark. */ + deliverySink?: CodexAppDeliverySink; + dispatchAttempt?: number; + state: 'accepted' | 'prepared'; + content: string; + codexAppInput?: CodexAppTurnInput; + vcMeetingImTurnOrigin?: VcMeetingImTurnOrigin; +} + +/** Monotonic cumulative ACK boundary for one authenticated runner generation. */ +export interface CodexAppGenerationCommit { + generation: string; + committedThrough: number; +} + /** A legacy CLI prompt plus an optional backend-specific structured sidecar. */ export interface CliTurnPayload { content: string; codexAppInput?: CodexAppTurnInput; } +/** Exact ordered successor retained behind an adapter-ACKed activation. */ +export interface QueuedActivationTailEntry { + id: string; + order: number; + userPrompt: string; + cliInput: CliTurnPayload; + turnId: string; + dispatchAttempt?: number; +} + +/** Durable opening/setup state while a repository picker or detached default + * worktree owns the first turn. Runtime DaemonSession buffers are rebuilt from + * this record after restart; the record is cleared only with the activation + * journal's adapter-level ACK. */ +export interface PendingRepoSetup { + mode: 'picker' | 'auto_worktree'; + prompt: string; + rawInput?: string; + turnId?: string; + baseDir?: string; + repoCardMessageId?: string; + codexAppText?: string; + codexAppApplicationContext?: string; + codexAppMessageContext?: string; + attachments?: LarkAttachment[]; + mentions?: LarkMention[]; + substituteTrigger?: SubstituteTrigger; + sender?: { openId: string; type: 'user' | 'bot'; name?: string }; +} + /** Messages sent from Daemon to Worker */ export type DaemonToWorker = - | { type: 'init'; sessionId: string; chatId: string; chatType?: 'group' | 'p2p'; rootMessageId: string; workingDir: string; cliId: string; cliPathOverride?: string; wrapperCli?: string; launchShell?: string; model?: string; disableCliBypass?: boolean; codexRpcInput?: boolean; startupCommands?: string[]; env?: Record; sandbox?: boolean; sandboxHidePaths?: string[]; sandboxReadonlyPaths?: string[]; sandboxNetwork?: boolean; readIsolation?: boolean; readDenyExtraPaths?: string[]; daemonBootId?: string; backendType: BackendType; backendConfig?: RiffBackendConfig; riffParentTaskId?: string; riffRepoDirs?: string[]; deferredScheduleRun?: Session['deferredScheduleRun']; prompt: string; promptCodexAppInput?: CodexAppTurnInput; resume?: boolean; cliSessionId?: string; originalSessionId?: string; ownerOpenId?: string; webPort?: number; larkAppId: string; larkAppSecret: string; brand?: 'feishu' | 'lark'; botName?: string; botOpenId?: string; locale?: 'zh' | 'en'; turnId?: string; dispatchAttempt?: number; vcMeetingImTurnOrigin?: VcMeetingImTurnOrigin; pluginBindings?: string[]; skillPolicy?: BotSkillPolicy; skillPluginDir?: string; skillReadonlyRoots?: string[]; adoptMode?: boolean; adoptSource?: 'tmux' | 'herdr' | 'zellij'; adoptTmuxTarget?: string; adoptZellijSession?: string; adoptZellijPaneId?: string; adoptHerdrSessionName?: string; adoptHerdrTarget?: string; adoptHerdrPaneId?: string; adoptPaneCols?: number; adoptPaneRows?: number; bridgeJsonlPath?: string; adoptCliPid?: number; adoptCwd?: string; adoptRestoredFromMetadata?: boolean } - | { type: 'message'; content: string; codexAppInput?: CodexAppTurnInput; turnId?: string; dispatchAttempt?: number; vcMeetingImTurnOrigin?: VcMeetingImTurnOrigin } + | { type: 'init'; sessionId: string; chatId: string; chatType?: 'group' | 'p2p'; rootMessageId: string; workingDir: string; cliId: string; cliPathOverride?: string; wrapperCli?: string; launchShell?: string; model?: string; disableCliBypass?: boolean; codexRpcInput?: boolean; startupCommands?: string[]; env?: Record; sandbox?: boolean; sandboxHidePaths?: string[]; sandboxReadonlyPaths?: string[]; sandboxNetwork?: boolean; readIsolation?: boolean; readDenyExtraPaths?: string[]; daemonBootId?: string; backendType: BackendType; backendConfig?: RiffBackendConfig; riffParentTaskId?: string; riffRepoDirs?: string[]; deferredScheduleRun?: Session['deferredScheduleRun']; prompt: string; promptCodexAppInput?: CodexAppTurnInput; queuedActivationToken?: string; resume?: boolean; cliSessionId?: string; originalSessionId?: string; ownerOpenId?: string; webPort?: number; larkAppId: string; larkAppSecret: string; brand?: 'feishu' | 'lark'; botName?: string; botOpenId?: string; locale?: 'zh' | 'en'; turnId?: string; replyTurnId?: string; dispatchAttempt?: number; codexAppDispatchId?: string; codexAppRecoveredDispatches?: CodexAppDispatchLedgerEntry[]; codexAppGenerationCommits?: CodexAppGenerationCommit[]; vcMeetingImTurnOrigin?: VcMeetingImTurnOrigin; pluginBindings?: string[]; skillPolicy?: BotSkillPolicy; skillPluginDir?: string; skillReadonlyRoots?: string[]; adoptMode?: boolean; adoptSource?: 'tmux' | 'herdr' | 'zellij'; adoptTmuxTarget?: string; adoptZellijSession?: string; adoptZellijPaneId?: string; adoptHerdrSessionName?: string; adoptHerdrTarget?: string; adoptHerdrPaneId?: string; adoptPaneCols?: number; adoptPaneRows?: number; bridgeJsonlPath?: string; adoptCliPid?: number; adoptCwd?: string; adoptRestoredFromMetadata?: boolean } + | { type: 'message'; content: string; codexAppInput?: CodexAppTurnInput; turnId?: string; replyTurnId?: string; dispatchAttempt?: number; codexAppDispatchId?: string; queuedActivationToken?: string; vcMeetingImTurnOrigin?: VcMeetingImTurnOrigin } + | { type: 'codex_app_dispatch_persisted'; requestId: string; ok: boolean; error?: string } /** Literal slash-command passthrough. `followUpContent` rides along so the * worker enqueues it strictly AFTER the slash command's Enter — two separate * IPCs would race: process.on('message') handlers don't serialize, and the * raw_input branch awaits 200ms between sendText and Enter, a window where * a separate `message` IPC could write into the PTY first. */ - | { type: 'raw_input'; content: string; turnId?: string; followUpContent?: string; followUpTurnId?: string; followUpCodexAppInput?: CodexAppTurnInput } + | { type: 'raw_input'; content: string; turnId?: string; followUpContent?: string; followUpTurnId?: string; followUpCodexAppInput?: CodexAppTurnInput; queuedActivationToken?: string } /** Rename the current CLI-native interactive session. The worker queues this * administrative slash command until the TUI is idle and does not treat it * as a model turn. Only adapters declaring buildSessionRenameCommand handle * it; all other CLIs ignore it. */ | { type: 'rename_session'; title: string } - | { type: 'close' } + | { type: 'close'; requestId?: string } + /** Commit a successful Riff prepare-close handshake after the daemon has + * durably closed the row. The worker must not exit between prepare ACK and + * this commit or its exit handler could race a replacement generation. */ + | { type: 'close_commit'; requestId: string } + /** Abort a Riff prepare after durable close/lineage persistence failed. */ + | { type: 'close_abort'; requestId: string } + /** Graceful daemon shutdown: fence new Riff writes and drain all writes + * accepted before this prepare without cancelling the remote task. */ + | { type: 'riff_shutdown_prepare'; requestId: string } + /** The daemon durably persisted the exact drained lineage; the worker may + * now detach and exit. */ + | { type: 'riff_shutdown_commit'; requestId: string } + /** Persistence/coordination failed; restore the prepared Riff owner. */ + | { type: 'riff_shutdown_abort'; requestId: string } | { type: 'suspend' } - | { type: 'restart' } + | { type: 'restart'; reason?: 'operator' | 'cli_crash' } /** Lease watchdog fencing: only the exact still-running durable attempt may * tear down/restart the CLI. A late command after terminal/current-turn * advance is ignored worker-side. */ @@ -604,6 +741,15 @@ export type WorkerToDaemon = cliPid?: number; cliProcStart?: string; } + | { + type: 'queued_activation_submitted'; + sessionId: string; + activationToken: string; + /** Riff accepts asynchronously. When present, daemon persistence must + * commit this exact new lineage atomically with clearing the activation + * journal; generic adapter write-return is not acceptance. */ + riffTaskId?: string; + } | { type: 'cli_session_id'; cliSessionId: string; turnId?: string; dispatchAttempt?: number } | { type: 'claude_exit'; code: number | null; signal: string | null; logTail?: string; canParkDiagnostic?: boolean; turnId?: string; dispatchAttempt?: number } | { type: 'prompt_ready' } @@ -627,12 +773,20 @@ export type WorkerToDaemon = dispatchAttempt: number; disposition: 'queued_removed' | 'cli_fenced'; } - | { type: 'managed_turn_origin'; sessionId: string; capability: string; turnId?: string; dispatchAttempt?: number } + | { type: 'managed_turn_origin'; sessionId: string; capability: string; originChannelId?: string; turnId?: string; dispatchAttempt?: number } /** An in-worker CLI restart rotates the managed-send authority without * replacing the Node worker. Carry the old token so the daemon can revoke * exactly that generation and ignore a delayed revoke after the next turn * has already published a fresh token. */ - | { type: 'managed_turn_origin_revoked'; sessionId: string; capability?: string; turnId?: string; dispatchAttempt?: number } + | { type: 'managed_turn_origin_revoked'; sessionId: string; capability?: string; originChannelId?: string; turnId?: string; dispatchAttempt?: number } + | { + type: 'codex_app_dispatch_transition'; + sessionId: string; + requestId: string; + operation: 'submit' | 'cancel' | 'retry'; + entries: Array>; + } + | { type: 'codex_app_generation_active'; sessionId: string; generation: string; fresh: boolean } | { type: 'final_output'; /** Worker-side botmux session identity. Daemon validates this before @@ -647,6 +801,7 @@ export type WorkerToDaemon = content: string; lastUuid: string; turnId: string; + replyTurnId?: string; /** Durable receiver attempt attribution. Final output suppression is * attempt-scoped so a late attempt-N event cannot affect attempt N+1. */ dispatchAttempt?: number; @@ -658,6 +813,15 @@ export type WorkerToDaemon = // which mixes presentation with payload). kind?: 'bridge' | 'local-turn' | 'local-turn-headless'; userText?: string; + /** Two-phase Codex App final settlement; daemon persists before ACKing worker. */ + codexAppSettlement?: { + requestId: string; + generation: string; + seq: number; + dispatchId: string; + }; + /** The model already delivered through botmux send; settle without fallback output. */ + suppressDelivery?: boolean; } | { type: 'turn_terminal'; @@ -680,4 +844,27 @@ export type WorkerToDaemon = | { type: 'adopt_preamble'; userText: string; assistantText: string; turnId?: string } | { type: 'deferred_topic_materialized'; sessionId: string; turnId: string; rootMessageId: string } | { type: 'riff_access_url'; accessUrl: string; directAccessUrl?: string; turnId?: string; dispatchAttempt?: number } - | { type: 'riff_task_id'; taskId: string | null }; + | { type: 'riff_task_id'; taskId: string | null } + | { + type: 'riff_shutdown_result'; + requestId: string; + phase: 'prepare' | 'abort'; + ok: boolean; + /** Required exact lineage. `null` authoritatively clears stale state. */ + taskId: string | null; + error?: string; + } + | { + type: 'close_abort_result'; + requestId: string; + ok: boolean; + error?: string; + } + | { + type: 'close_result'; + requestId: string; + ok: boolean; + /** Exact Riff task that remains live when cancellation failed. */ + taskId?: string; + error?: string; + }; diff --git a/src/utils/async-serial-queue.ts b/src/utils/async-serial-queue.ts new file mode 100644 index 000000000..1adac53ae --- /dev/null +++ b/src/utils/async-serial-queue.ts @@ -0,0 +1,13 @@ +/** Minimal per-worker promise-chain mutex. Each task begins only after the + * previous task settles; rejection never poisons the tail. Used for adopt + * writes where concurrent paste -> delay -> Enter/history verification would + * otherwise interleave in the same external CLI composer. */ +export class AsyncSerialQueue { + private tail: Promise = Promise.resolve(); + + run(task: () => Promise): Promise { + const result = this.tail.then(task, task); + this.tail = result.then(() => undefined, () => undefined); + return result; + } +} diff --git a/src/utils/bot-routing.ts b/src/utils/bot-routing.ts index 587e8ec4e..02dfe48fb 100644 --- a/src/utils/bot-routing.ts +++ b/src/utils/bot-routing.ts @@ -139,7 +139,7 @@ export function stripCodeSpans(text: string): string { * default addressing (owner / oncall last-caller) is unchanged. */ export function buildFooterAddressing( - s: { ownerOpenId?: string; lastCallerOpenId?: string }, + s: { ownerOpenId?: string; lastCallerOpenId?: string; lastCallerIsBot?: boolean }, opts: { isOncall: boolean; isSubstitute?: boolean; @@ -159,7 +159,7 @@ export function buildFooterAddressing( if (!opts.isOncall && !opts.isSubstitute) return { sendTo: ownerHuman, cc: [] }; const caller = s.lastCallerOpenId ?? owner; - const callerIsBot = !!caller && botIds.has(caller); + const callerIsBot = !!caller && (s.lastCallerIsBot === true || botIds.has(caller)); // Oncall + last caller is a bot (but this reply itself has no explicit bot // target) → fall back to the human owner rather than pinging the bot caller. diff --git a/src/utils/child-env.ts b/src/utils/child-env.ts index 9c8ad7c6c..727b0df61 100644 --- a/src/utils/child-env.ts +++ b/src/utils/child-env.ts @@ -114,10 +114,19 @@ export const BOTMUX_INJECTED_ENV_KEYS = [ // session-scoped, capability-gated routes (v3 workflow relay, vc-agent). // A port marker, not a credential — every route authenticates independently. 'BOTMUX_DAEMON_IPC_PORT', + // Fail-closed classification hint for macOS read isolation. This is never + // authority; cmdSend also checks the host-owned marker + live challenge. + 'BOTMUX_READ_ISOLATED', + // Unguessable pane/profile authority channel. It selects an exact read carve + // but is not sufficient without the daemon-rotated capability inside it. + 'BOTMUX_ORIGIN_CHANNEL_ID', // Keep `botmux bots list` and ready-gated CLIs aligned with daemon config. 'BOTMUX_LARK_LIST_BOTS_API_ENABLED', 'BOTMUX_LARK_LIST_BOTS_API_TIMEOUT_MS', 'BOTMUX_READY_COMMAND', + // Path to a one-shot 0600 Codex App control bootstrap. Only the path reaches + // the pane; the runner consumes+unlinks the file before app-server starts. + 'BOTMUX_CODEX_APP_CONTROL_BOOTSTRAP', // Hermes profile roots must match the worker-side transcript reader. 'HERMES_HOME', 'HERMES_BOTMUX_SOURCE_HOME', diff --git a/src/utils/codex-app-control.ts b/src/utils/codex-app-control.ts new file mode 100644 index 000000000..a2a2a3262 --- /dev/null +++ b/src/utils/codex-app-control.ts @@ -0,0 +1,3100 @@ +import { + createHash, + createPrivateKey, + createPublicKey, + generateKeyPairSync, + randomBytes, + sign, + verify, + type KeyObject, +} from 'node:crypto'; +import { spawnSync, type SpawnSyncReturns } from 'node:child_process'; +import { + chmodSync, + closeSync, + constants as fsConstants, + fchmodSync, + fstatSync, + fsyncSync, + existsSync, + lstatSync, + mkdirSync, + openSync, + readFileSync, + readdirSync, + renameSync, + rmdirSync, + unlinkSync, + writeSync, + type BigIntStats, + type Stats, +} from 'node:fs'; +import { homedir } from 'node:os'; +import { basename, dirname, isAbsolute, join, win32 } from 'node:path'; +import type { BackendType } from '../adapters/backend/types.js'; +import type { ScreenStatus } from '../types.js'; + +/** + * The launch boundary carries only a path to an owner-only, read-once file. + * That file contains a fresh Ed25519 private key. The runner consumes and + * unlinks it before app-server (and therefore model tools) can start. Only the + * public key is persisted by the worker. + */ +export const CODEX_APP_CONTROL_BOOTSTRAP_ENV = 'BOTMUX_CODEX_APP_CONTROL_BOOTSTRAP'; + +export const CODEX_APP_CONTROL_LINE_MAX_BYTES = 4_096; +export const CODEX_APP_CONTROL_FINAL_MAX_BYTES = 1_048_576; +export const CODEX_APP_CONTROL_FINAL_CHUNK_BYTES = 1_536; +/** Keep control bootstrap/proof alive for the worker's existing cold-start cap. */ +export const CODEX_APP_CONTROL_STARTUP_TIMEOUT_MS = 90_000; +/** Absolute per-connection challenge/accept deadline; transport activity does not extend it. */ +export const CODEX_APP_CONTROL_HANDSHAKE_TIMEOUT_MS = 5_000; + +const CONTROL_STATE_VERSION = 3 as const; +const CONTROL_WIRE_VERSION = 2 as const; +const CONTROL_BOOTSTRAP_VERSION = 3 as const; +const CONTROL_LOCATOR_VERSION = 1 as const; +const PRIVATE_DIRECTORY_MODE = 0o700; +const PRIVATE_FILE_MODE = 0o600; +const BOOTSTRAP_MAX_BYTES = 8_192; +const LOCATOR_MAX_BYTES = 4_096; +const WINDOWS_SYSTEM_SID = 'S-1-5-18'; +const WINDOWS_PIPE_PREFIX = '\\\\?\\pipe\\botmux-codex-app-'; +const WINDOWS_OWNER_PIPE_PREFIX = '\\\\?\\pipe\\botmux-codex-app-owner-'; +const POSIX_SOCKET_LEAF_RE = /^endpoint-[a-f0-9]{32}\.sock$/; +const POSIX_OWNER_RECORD_RE = /^owner-[a-f0-9]{64}\.json$/; +const POSIX_OWNER_PENDING_RE = /^owner-[a-f0-9]{64}\.pending$/; +const POSIX_REAPER_RECORD_RE = /^reap-[a-f0-9]{64}\.json$/; +const POSIX_REAPER_PENDING_RE = /^reap-[a-f0-9]{64}\.pending$/; +const POSIX_DIRECTORY_GENERATION_RE = /^generation-([a-f0-9]{64})\.json$/; +const POSIX_LEASE_RECORD_MAX_BYTES = 4_096; +const windowsHardenedDirectories = new Set(); + +/** + * Arm the shared Codex App startup deadline. Keeping the scheduling in one + * helper prevents the runner auth gate, bootstrap cleanup, worker proof gate, + * and first-prompt hard cap from silently drifting to different lifetimes. + */ +export function armCodexAppControlStartupTimeout( + onTimeout: () => void, + timeoutMs = CODEX_APP_CONTROL_STARTUP_TIMEOUT_MS, +): ReturnType { + return setTimeout(onTimeout, timeoutMs); +} + +export function armCodexAppControlHandshakeTimeout( + onTimeout: () => void, + timeoutMs = CODEX_APP_CONTROL_HANDSHAKE_TIMEOUT_MS, +): ReturnType { + return setTimeout(onTimeout, timeoutMs); +} + +export type CodexAppSignedStateReadiness = 'invalid' | 'waiting' | 'ready'; + +/** Authentication and a signed busy/idle bit are not sufficient readiness. + * The runner must explicitly assert that its initialized stdin/app-server path + * accepts input. Missing/non-boolean values are protocol violations; `false` + * is a valid not-ready state that leaves the absolute proof deadline armed. */ +export function codexAppSignedStateReadiness(payload: unknown): CodexAppSignedStateReadiness { + if (!payload || typeof payload !== 'object') return 'invalid'; + const acceptingInput = (payload as Record).acceptingInput; + if (typeof acceptingInput !== 'boolean') return 'invalid'; + return acceptingInput ? 'ready' : 'waiting'; +} + +/** Re-armable proof deadline shared by startup and authenticated disconnects. */ +export class CodexAppControlProofDeadline { + private timer: ReturnType | undefined; + + arm(onTimeout: () => void, timeoutMs = CODEX_APP_CONTROL_STARTUP_TIMEOUT_MS): void { + this.clear(); + this.timer = armCodexAppControlStartupTimeout(() => { + this.timer = undefined; + onTimeout(); + }, timeoutMs); + this.timer.unref?.(); + } + + clear(): void { + if (this.timer) clearTimeout(this.timer); + this.timer = undefined; + } + + get armed(): boolean { + return this.timer !== undefined; + } +} + +/** + * Coalesce one signed control record while its worker-side effects are still + * awaiting a durable daemon ACK. Endpoint replacement can replay the same + * generation/sequence before the replay window is committed; every replay + * must observe the first application result instead of running queue or + * liveness effects again. + * + * The caller releases the entry only after committing the replay window (or + * after a rejected application). Keeping release outside the promise closes + * the resolution -> replay-commit gap. + */ +export class CodexAppControlRecordApplicationGate { + private readonly inFlight = new Map>(); + + run( + generation: string, + seq: number, + apply: () => boolean | Promise, + ): Promise { + const key = this.key(generation, seq); + const existing = this.inFlight.get(key); + if (existing) return existing; + const application = Promise.resolve().then(apply); + this.inFlight.set(key, application); + return application; + } + + release(generation: string, seq: number, application: Promise): void { + const key = this.key(generation, seq); + if (this.inFlight.get(key) === application) this.inFlight.delete(key); + } + + private key(generation: string, seq: number): string { + return `${generation}\0${seq}`; + } +} + +export type CodexAppRuntimeScreenStatus = Exclude; + +/** Authentication alone is not readiness. Until the active generation has + * published a signed accepting-input state, an idle screen is fail-closed to + * working so reconnect replay cannot expose a false-idle interval. */ +export function projectCodexAppControlReadinessStatus( + base: CodexAppRuntimeScreenStatus, + readiness: { + controlProven: boolean; + signedStateObserved: boolean; + inputReady: boolean; + }, +): CodexAppRuntimeScreenStatus { + if (base !== 'idle') return base; + return readiness.controlProven + && readiness.signedStateObserved + && readiness.inputReady + ? base + : 'working'; +} +const STATE_MAX_BYTES = 16_384; +const GENERATION_RE = /^[A-Za-z0-9_-]{43}$/; +const CHALLENGE_RE = /^[A-Za-z0-9_-]{43}$/; +const KEY_RE = /^[A-Za-z0-9_-]{32,512}$/; +const SIGNATURE_RE = /^[A-Za-z0-9_-]{86}$/; +const MAX_STATE_IDENTITIES = 4; + +export interface CodexAppControlPathOptions { + platform?: NodeJS.Platform; + localAppData?: string; + homeDirectory?: string; +} + +export interface CodexAppControlFilesystemPolicy { + useNoFollow: boolean; + verifyUid: boolean; + verifyExactMode: boolean; + chmodAfterCreate: boolean; + verifyPostUnlinkLinkCount: boolean; + fsyncDirectory: boolean; +} + +export function codexAppControlFilesystemPolicy( + platform: NodeJS.Platform = process.platform, +): CodexAppControlFilesystemPolicy { + if (platform === 'win32') { + return { + useNoFollow: false, + verifyUid: false, + verifyExactMode: false, + chmodAfterCreate: false, + verifyPostUnlinkLinkCount: false, + fsyncDirectory: false, + }; + } + return { + useNoFollow: true, + verifyUid: true, + verifyExactMode: true, + chmodAfterCreate: true, + verifyPostUnlinkLinkCount: true, + fsyncDirectory: true, + }; +} + +function ownerUid(): number | undefined { + return process.geteuid?.() ?? process.getuid?.(); +} + +function noFollowFlag(platform: NodeJS.Platform): number { + if (!codexAppControlFilesystemPolicy(platform).useNoFollow) return 0; + const flag = (fsConstants as typeof fsConstants & { O_NOFOLLOW?: number }).O_NOFOLLOW; + if (typeof flag !== 'number') throw new Error('O_NOFOLLOW is required for Codex App control files'); + return flag; +} + +function exactMode(mode: number): number { + return mode & 0o777; +} + +function assertOwned(stat: { uid: number }, label: string, platform: NodeJS.Platform): void { + if (!codexAppControlFilesystemPolicy(platform).verifyUid) return; + const uid = ownerUid(); + if (uid !== undefined && stat.uid !== uid) throw new Error(`${label} must be owned by the current uid`); +} + +function isAbsoluteForPlatform(path: string, platform: NodeJS.Platform): boolean { + return platform === 'win32' ? win32.isAbsolute(path) : isAbsolute(path); +} + +function joinForPlatform(platform: NodeJS.Platform, ...parts: string[]): string { + return platform === 'win32' ? win32.join(...parts) : join(...parts); +} + +function dirnameForPlatform(platform: NodeJS.Platform, path: string): string { + return platform === 'win32' ? win32.dirname(path) : dirname(path); +} + +/** + * Windows never places control material under SESSION_DATA_DIR: that setting + * is bot/user supplied and may point at a shared tree. The fixed root is + * instead anchored in the current Windows profile and hardened before any + * secret or locator is created beneath it. + */ +export function codexAppWindowsControlRoot( + options: CodexAppControlPathOptions = {}, +): string { + const base = options.localAppData?.trim() + || (options.platform === 'win32' || options.platform === undefined + ? process.env.LOCALAPPDATA?.trim() + : undefined) + || options.homeDirectory?.trim() + || homedir(); + // The locator protocol relies on local NTFS-style ACL and atomic-replace + // semantics. A UNC/redirected profile can have server-defined ACL and + // rename behavior, so do not silently place capabilities on it. The drive + // root itself remains configurable through LOCALAPPDATA/home for normal + // per-user profiles. + if (!base || !/^[A-Za-z]:[\\/]/.test(base) || !win32.isAbsolute(base)) { + throw new Error( + 'Windows Codex App control root requires a local drive-qualified LOCALAPPDATA or home directory', + ); + } + return win32.join(base, 'Botmux', 'codex-app-control'); +} + +function parseStrictCsvRow(row: string): string[] | undefined { + const fields: string[] = []; + let cursor = 0; + while (cursor < row.length) { + if (row[cursor] !== '"') return undefined; + cursor++; + let field = ''; + let closed = false; + while (cursor < row.length) { + const ch = row[cursor++]!; + if (ch !== '"') { + field += ch; + continue; + } + if (row[cursor] === '"') { + field += '"'; + cursor++; + continue; + } + closed = true; + break; + } + if (!closed) return undefined; + fields.push(field); + if (cursor === row.length) break; + if (row[cursor] !== ',') return undefined; + cursor++; + if (cursor === row.length) return undefined; + } + return fields; +} + +export function parseWindowsCurrentSid(output: string): string | undefined { + const rows = output.replace(/^\uFEFF/, '').split(/\r?\n/).filter(row => row.length > 0); + if (rows.length !== 1) return undefined; + const fields = parseStrictCsvRow(rows[0]!); + if (fields?.length !== 2) return undefined; + const sid = fields[1]!; + return /^S-\d-(?:\d+-)+\d+$/i.test(sid) ? sid.toUpperCase() : undefined; +} + +interface WindowsAclAce { + type: string; + flags: string; + rights: string; + objectGuid: string; + inheritObjectGuid: string; + trustee: string; +} + +export function decodeWindowsAclSnapshot(raw: Buffer): string { + if (raw.length >= 2 && raw[0] === 0xff && raw[1] === 0xfe) { + return raw.subarray(2).toString('utf16le'); + } + if (raw.length >= 4 && raw[1] === 0 && raw[3] === 0) return raw.toString('utf16le'); + return raw.toString('utf8'); +} + +function parseWindowsDaclSnapshot(snapshot: string): { flags: string; aces: WindowsAclAce[] } | undefined { + // `/save` layout is not a documented API. Locate the SDDL token by grammar, + // whether it follows the path on the same or next line. `D:\path` cannot + // match because a DACL has only uppercase control flags before its first ACE. + const daclMatch = snapshot.match(/D:[A-Z]*\([^\r\n]*/); + if (!daclMatch) return undefined; + let dacl = daclMatch[0].slice(2).trim(); + const saclStart = dacl.indexOf('S:'); + if (saclStart >= 0) dacl = dacl.slice(0, saclStart); + const firstAce = dacl.indexOf('('); + if (firstAce < 0) return undefined; + const flags = dacl.slice(0, firstAce); + const aceText = dacl.slice(firstAce); + const aces: WindowsAclAce[] = []; + let cursor = 0; + while (cursor < aceText.length) { + if (aceText[cursor] !== '(') return undefined; + const end = aceText.indexOf(')', cursor + 1); + if (end < 0) return undefined; + const fields = aceText.slice(cursor + 1, end).split(';'); + if (fields.length !== 6) return undefined; + aces.push({ + type: fields[0]!, + flags: fields[1]!, + rights: fields[2]!, + objectGuid: fields[3]!, + inheritObjectGuid: fields[4]!, + trustee: fields[5]!, + }); + cursor = end + 1; + } + return { flags, aces }; +} + +/** Locale-independent verification of the SDDL emitted by `icacls /save`. */ +export function verifyWindowsCodexAppControlDacl( + snapshot: string, + currentSid: string, + kind: 'directory' | 'file' = 'directory', +): boolean { + const parsed = parseWindowsDaclSnapshot(snapshot); + if (!parsed || !parsed.flags.includes('P')) return false; + // P = protected DACL. AI/AR are canonical auto-inheritance metadata that + // icacls may retain even after inherited ACEs are removed; reject anything + // else and separately reject every inherited ACE below. + if (parsed.flags.replace(/AI|AR|P/g, '')) return false; + const expected = new Set( + currentSid.toUpperCase() === WINDOWS_SYSTEM_SID + ? [WINDOWS_SYSTEM_SID] + : [currentSid.toUpperCase(), WINDOWS_SYSTEM_SID], + ); + if (parsed.aces.length !== expected.size) return false; + for (const ace of parsed.aces) { + const trustee = ace.trustee.toUpperCase() === 'SY' + ? WINDOWS_SYSTEM_SID + : ace.trustee.toUpperCase(); + if (ace.type !== 'A' || ace.rights !== 'FA' || ace.objectGuid || ace.inheritObjectGuid + || ace.flags.includes('ID') || !expected.delete(trustee)) return false; + if (kind === 'directory') { + if (!ace.flags.includes('OI') || !ace.flags.includes('CI') + || ace.flags.replace(/OI|CI/g, '')) return false; + } else if (ace.flags) return false; + } + return expected.size === 0; +} + +export type WindowsControlCommandRunner = ( + command: string, + args: string[], +) => Pick, 'status' | 'stdout' | 'stderr' | 'error'>; + +function defaultWindowsControlCommandRunner( + command: string, + args: string[], +): Pick, 'status' | 'stdout' | 'stderr' | 'error'> { + return spawnSync(command, args, { + encoding: 'utf8', + windowsHide: true, + shell: false, + timeout: 10_000, + maxBuffer: 1_048_576, + }); +} + +function runWindowsControlCommand( + runner: WindowsControlCommandRunner, + command: string, + args: string[], + label: string, +): string { + const result = runner(command, args); + if (result.error || result.status !== 0) { + throw new Error(`${label} failed closed (status=${String(result.status)})`); + } + return result.stdout ?? ''; +} + +/** + * Remove inherited ACLs, grant only the current SID and SYSTEM full control, + * then validate the resulting protected DACL via locale-independent SDDL. + * Unknown explicit ACEs are never silently retained: verification fails. + */ +function hardenWindowsCodexAppControlAcl( + path: string, + kind: 'directory' | 'file', + runner: WindowsControlCommandRunner = defaultWindowsControlCommandRunner, +): void { + if (!path || !win32.isAbsolute(path)) { + throw new Error(`Windows Codex App control ${kind} must be absolute`); + } + if (kind === 'directory') mkdirSync(path, { recursive: true }); + const stat = lstatSync(path); + if ((kind === 'directory' ? !stat.isDirectory() : !stat.isFile()) + || stat.isSymbolicLink() || (kind === 'file' && stat.nlink !== 1)) { + throw new Error(`Windows Codex App control ${kind} must be a real single-link ${kind}`); + } + const systemRoot = process.env.SystemRoot?.trim() || process.env.WINDIR?.trim(); + if (!systemRoot || !win32.isAbsolute(systemRoot)) { + throw new Error('Windows SystemRoot is required for trusted ACL tools'); + } + const whoamiCommand = win32.join(systemRoot, 'System32', 'whoami.exe'); + const icaclsCommand = win32.join(systemRoot, 'System32', 'icacls.exe'); + const whoami = runWindowsControlCommand( + runner, + whoamiCommand, + ['/user', '/fo', 'csv', '/nh'], + 'Windows current SID lookup', + ); + const sid = parseWindowsCurrentSid(whoami); + if (!sid) throw new Error('Windows current SID lookup returned no SID'); + let lastVerificationError: Error | undefined; + // Several workers can harden the shared per-user root concurrently. The + // transaction is idempotent; bounded full retries avoid snapshotting another + // worker's inheritance-removal→grant gap without ever accepting a loose ACL. + for (let attempt = 0; attempt < 3; attempt++) { + const snapshotPath = win32.join( + kind === 'directory' ? path : win32.dirname(path), + `.botmux-acl-${process.pid}-${randomBytes(16).toString('hex')}.txt`, + ); + try { + runWindowsControlCommand( + runner, + icaclsCommand, + [path, '/setowner', `*${sid}`, '/q'], + 'Windows control owner assignment', + ); + runWindowsControlCommand( + runner, + icaclsCommand, + [path, '/inheritance:r', '/q'], + 'Windows control ACL inheritance removal', + ); + runWindowsControlCommand( + runner, + icaclsCommand, + [path, '/grant:r', `*${sid}:${kind === 'directory' ? '(OI)(CI)F' : 'F'}`, + ...(sid === WINDOWS_SYSTEM_SID + ? [] + : [`*${WINDOWS_SYSTEM_SID}:${kind === 'directory' ? '(OI)(CI)F' : 'F'}`]), '/q'], + 'Windows control ACL grant', + ); + runWindowsControlCommand( + runner, + icaclsCommand, + [path, '/verify', '/q'], + 'Windows control ACL canonical verification', + ); + runWindowsControlCommand( + runner, + icaclsCommand, + [path, '/save', snapshotPath, '/q'], + 'Windows control ACL snapshot', + ); + const snapshot = decodeWindowsAclSnapshot(readFileSync(snapshotPath)); + if (verifyWindowsCodexAppControlDacl(snapshot, sid, kind)) return; + lastVerificationError = new Error( + `Windows Codex App control ${kind} ACL is not exactly current SID + SYSTEM`, + ); + } catch (err) { + lastVerificationError = err instanceof Error ? err : new Error(String(err)); + } finally { + try { unlinkSync(snapshotPath); } catch { /* no snapshot on command failure */ } + } + } + throw lastVerificationError ?? new Error(`Windows Codex App control ${kind} ACL verification failed`); +} + +/** + * Remove inherited ACLs, set the current SID as owner, grant only that SID and + * SYSTEM full control, and verify the protected DACL. Unknown explicit ACEs + * are retained by icacls but cause exact verification to fail closed. + */ +export function hardenWindowsCodexAppControlDirectory( + directory: string, + runner: WindowsControlCommandRunner = defaultWindowsControlCommandRunner, +): void { + hardenWindowsCodexAppControlAcl(directory, 'directory', runner); +} + +/** Validate/harden a pre-existing fixed-name state file before trusting it. */ +export function hardenWindowsCodexAppControlFile( + path: string, + runner: WindowsControlCommandRunner = defaultWindowsControlCommandRunner, +): void { + hardenWindowsCodexAppControlAcl(path, 'file', runner); +} + +/** Create/validate a non-symlink owner-only directory. */ +export function ensureCodexAppControlDirectory( + directory: string, + platform: NodeJS.Platform = process.platform, +): void { + if (!directory || !isAbsoluteForPlatform(directory, platform)) { + throw new Error('Codex App control directory must be absolute'); + } + if (platform === 'win32') { + const cacheKey = win32.normalize(directory).toLowerCase(); + if (windowsHardenedDirectories.has(cacheKey)) { + const stat = lstatSync(directory); + if (!stat.isDirectory() || stat.isSymbolicLink()) { + windowsHardenedDirectories.delete(cacheKey); + throw new Error('Windows Codex App control directory changed after ACL hardening'); + } + return; + } + hardenWindowsCodexAppControlDirectory(directory); + // Re-running whoami/icacls for every atomic locator replacement would + // block the worker event loop and amplify the named-pipe DoS boundary. + // Once exact ACL verification succeeds, cross-user mutation is excluded; + // same-SID compromise is outside this control plane's trust boundary. + windowsHardenedDirectories.add(cacheKey); + return; + } + mkdirSync(directory, { recursive: true, mode: PRIVATE_DIRECTORY_MODE }); + const stat = lstatSync(directory); + if (!stat.isDirectory() || stat.isSymbolicLink()) { + throw new Error('Codex App control directory must be a real directory'); + } + assertOwned(stat, 'Codex App control directory', platform); + if (exactMode(stat.mode) !== PRIVATE_DIRECTORY_MODE) { + chmodSync(directory, PRIVATE_DIRECTORY_MODE); + const tightened = lstatSync(directory); + assertOwned(tightened, 'Codex App control directory', platform); + if (!tightened.isDirectory() || tightened.isSymbolicLink() + || exactMode(tightened.mode) !== PRIVATE_DIRECTORY_MODE) { + throw new Error('Codex App control directory could not be secured to 0700'); + } + } +} + +function sessionKey(sessionId: string): string { + return createHash('sha256').update(sessionId, 'utf8').digest('hex'); +} + +export function codexAppControlStatePath(sessionDataDir: string, sessionId: string): string { + if (process.platform === 'win32') { + return win32.join(codexAppWindowsControlRoot(), 'state', `${sessionKey(sessionId)}.json`); + } + return join(sessionDataDir, 'codex-app-control', `${sessionKey(sessionId)}.json`); +} + +export function codexAppControlStatePathForPlatform( + sessionDataDir: string, + sessionId: string, + options: CodexAppControlPathOptions = {}, +): string { + const platform = options.platform ?? process.platform; + if (platform === 'win32') { + return win32.join(codexAppWindowsControlRoot(options), 'state', `${sessionKey(sessionId)}.json`); + } + return join(sessionDataDir, 'codex-app-control', `${sessionKey(sessionId)}.json`); +} + +export function codexAppControlSocketPath(directory: string, sessionId: string): string { + // Unix-domain paths are short on macOS. A fixed 32-hex leaf keeps the path + // comfortably below the platform limit for normal BOT_HOME/tmp roots. + return join(directory, `${sessionKey(sessionId).slice(0, 32)}.sock`); +} + +export function codexAppPosixControlRoot(uid = process.getuid?.()): string { + return join('/tmp', `botmux-codex-app-${uid ?? 'unknown'}`); +} + +export function codexAppPosixOwnerLeaseDirectory(controlRoot: string, sessionId: string): string { + return join(controlRoot, 'leases', `${sessionKey(sessionId).slice(0, 32)}.lease`); +} + +export function generateCodexAppPosixSocketEndpoint(socketDirectory: string): string { + return join(socketDirectory, `endpoint-${randomBytes(16).toString('hex')}.sock`); +} + +export function codexAppControlLocatorPath( + controlRoot: string, + sessionId: string, + platform: NodeJS.Platform = process.platform, +): string { + return joinForPlatform(platform, controlRoot, 'locators', `${sessionKey(sessionId)}.json`); +} + +export function generateCodexAppWindowsPipeEndpoint(): string { + return `${WINDOWS_PIPE_PREFIX}${randomBytes(32).toString('hex')}`; +} + +/** + * A fixed coordination pipe is held for the lifetime of one Windows worker + * process. It is not a control endpoint and is never put in a runner locator. + * Its only purpose is to serialize locator publishers across the daemon's + * kill-then-fork overlap window. The OS releases it if the old worker is + * terminated, avoiding a crash-stale filesystem lock. + */ +export function codexAppWindowsOwnerPipeEndpoint(sessionId: string): string { + return `${WINDOWS_OWNER_PIPE_PREFIX}${sessionKey(sessionId)}`; +} + +export interface CodexAppControlOwnerLeaseOptions { + bind: () => Promise; + timeoutMs?: number; + retryDelayMs?: number; + now?: () => number; + wait?: (delayMs: number) => Promise; +} + +/** Retry only a competing bind; every other owner-lease error fails closed. */ +export async function acquireCodexAppControlOwnerLease( + options: CodexAppControlOwnerLeaseOptions, +): Promise { + const timeoutMs = options.timeoutMs ?? 10_000; + const retryDelayMs = options.retryDelayMs ?? 50; + const now = options.now ?? Date.now; + const wait = options.wait ?? (delayMs => new Promise(resolve => setTimeout(resolve, delayMs))); + const deadline = now() + timeoutMs; + let lastBusyError: unknown; + while (true) { + try { + return await options.bind(); + } catch (err) { + if ((err as NodeJS.ErrnoException)?.code !== 'EADDRINUSE') throw err; + lastBusyError = err; + } + if (now() >= deadline) { + throw lastBusyError instanceof Error + ? lastBusyError + : new Error('Windows Codex App control owner lease timed out'); + } + await wait(Math.min(retryDelayMs, Math.max(0, deadline - now()))); + } +} + +interface CodexAppPosixFileIdentity { + dev: string; + ino: string; +} + +interface CodexAppPosixDirectoryIdentity extends CodexAppPosixFileIdentity { + generation?: string; +} + +interface CodexAppPosixOwnerTarget { + nonce: string; + recordIdentity: CodexAppPosixFileIdentity; + pid: number | null; + processStartToken: string | null; +} + +interface CodexAppPosixOwnerRecord { + version: 2 | 3; + role: 'owner' | 'reaper'; + sessionId: string; + nonce: string; + pid: number; + processStartToken: string; + createdAtMs: number; + directoryIdentity: CodexAppPosixDirectoryIdentity; + targetOwner?: CodexAppPosixOwnerTarget; +} + +export type CodexAppOwnerProcessStatus = 'alive' | 'dead' | 'unknown'; + +export interface CodexAppPosixOwnerLease { + directory: string; + ownerRecordPath: string; + isOwned(): boolean; + release(): void; +} + +export interface CodexAppPosixOwnerLeaseOptions { + controlRoot: string; + sessionId: string; + platform?: NodeJS.Platform; + timeoutMs?: number; + retryDelayMs?: number; + initializationGraceMs?: number; + pid?: number; + processStartToken?: string; + now?: () => number; + wait?: (delayMs: number) => Promise; + inspectOwner?: (pid: number, processStartToken: string) => CodexAppOwnerProcessStatus; + /** Deterministic handoff-race seam: invoked after mkdir(EEXIST), before observation. */ + onContended?: () => void; + /** Deterministic crash-resume seam: invoked after this creator pins the new directory inode. */ + onOwnerDirectoryCreated?: (directory: string) => void | Promise; + /** Deterministic crash-window seam: pauses after owner publication, before the directory CAS. */ + onOwnerRecordPublished?: (directory: string, ownerRecordPath: string) => void | Promise; + /** Deterministic replacement seam: pauses after a stale owner is pinned, before reaper publication. */ + onBeforeReaperRecordPublished?: (directory: string) => void | Promise; + /** Deterministic crash-window seam: pauses after reaper publication, before the directory CAS. */ + onReaperRecordPublished?: (directory: string, reaperRecordPath: string) => void | Promise; +} + +function linuxProcessStartToken(raw: string): string | undefined { + const closeParen = raw.lastIndexOf(')'); + if (closeParen < 0) return undefined; + const fields = raw.slice(closeParen + 2).trim().split(/\s+/); + return fields[19] || undefined; +} + +export function codexAppPosixProcessProbeEnv( + base: NodeJS.ProcessEnv = process.env, +): NodeJS.ProcessEnv { + // `ps -o lstart` is formatted through locale and timezone. Pin both so the + // same live process cannot acquire a different token after the worker's + // ambient LC_*/TZ changes. Preserve PATH and the remaining launch context. + return { ...base, LC_ALL: 'C', LANG: 'C', TZ: 'UTC' }; +} + +export function readCodexAppProcessStartToken( + pid: number, + platform: NodeJS.Platform = process.platform, +): string | undefined { + if (!Number.isSafeInteger(pid) || pid <= 0 || platform === 'win32') return undefined; + if (platform === 'linux') { + try { return linuxProcessStartToken(readFileSync(`/proc/${pid}/stat`, 'utf8')); } + catch { return undefined; } + } + const result = spawnSync('/bin/ps', ['-p', String(pid), '-o', 'lstart='], { + encoding: 'utf8', + env: codexAppPosixProcessProbeEnv(), + shell: false, + timeout: 5_000, + maxBuffer: 64 * 1024, + }); + if (result.error || result.status !== 0) return undefined; + const token = result.stdout.trim(); + return token || undefined; +} + +function inspectCodexAppOwnerProcess( + pid: number, + expectedStartToken: string, + platform: NodeJS.Platform, +): CodexAppOwnerProcessStatus { + try { + process.kill(pid, 0); + } catch (err) { + const code = (err as NodeJS.ErrnoException).code; + if (code === 'ESRCH') return 'dead'; + // EPERM proves that a process occupies the pid. Any other probe failure is + // also fail-closed: it must not authorize stale-owner deletion. + if (code === 'EPERM') return 'alive'; + return 'unknown'; + } + const actualStartToken = readCodexAppProcessStartToken(pid, platform); + if (!actualStartToken) return 'unknown'; + // lstart has one-second resolution on non-Linux POSIX. A same-second PID + // reuse can therefore look alive and block stale recovery, but cannot grant + // ownership or reap a live owner: the ambiguity remains fail-closed. + return actualStartToken === expectedStartToken ? 'alive' : 'dead'; +} + +function validPosixFileIdentity(value: unknown): value is CodexAppPosixFileIdentity { + if (!value || typeof value !== 'object') return false; + const identity = value as Record; + return typeof identity.dev === 'string' && /^(?:0|[1-9]\d*)$/.test(identity.dev) + && typeof identity.ino === 'string' && /^(?:0|[1-9]\d*)$/.test(identity.ino); +} + +function validPosixDirectoryIdentity( + value: unknown, + requireGeneration: boolean, +): value is CodexAppPosixDirectoryIdentity { + if (!validPosixFileIdentity(value)) return false; + const generation = (value as unknown as Record).generation; + return requireGeneration + ? typeof generation === 'string' && /^[a-f0-9]{64}$/.test(generation) + : generation === undefined; +} + +function validPosixOwnerTarget(value: unknown): value is CodexAppPosixOwnerTarget { + if (!value || typeof value !== 'object') return false; + const target = value as Record; + const processIdentityValid = (target.pid === null && target.processStartToken === null) + || (Number.isSafeInteger(target.pid) && (target.pid as number) > 0 + && typeof target.processStartToken === 'string' + && target.processStartToken.length > 0 && target.processStartToken.length <= 256); + return typeof target.nonce === 'string' && /^[a-f0-9]{64}$/.test(target.nonce) + && validPosixFileIdentity(target.recordIdentity) + && processIdentityValid; +} + +function validPosixOwnerRecord( + value: unknown, + sessionId: string, + expectedRole?: 'owner' | 'reaper', +): value is CodexAppPosixOwnerRecord { + if (!value || typeof value !== 'object') return false; + const record = value as Record; + const roleValid = record.role === 'owner' || record.role === 'reaper'; + const targetValid = record.role === 'owner' + ? record.targetOwner === undefined + : validPosixOwnerTarget(record.targetOwner); + const versionValid = record.version === 2 + ? validPosixDirectoryIdentity(record.directoryIdentity, false) + : record.version === 3 && validPosixDirectoryIdentity(record.directoryIdentity, true); + return versionValid + && roleValid + && (expectedRole === undefined || record.role === expectedRole) + && record.sessionId === sessionId + && typeof record.nonce === 'string' && /^[a-f0-9]{64}$/.test(record.nonce) + && Number.isSafeInteger(record.pid) && (record.pid as number) > 0 + && typeof record.processStartToken === 'string' + && record.processStartToken.length > 0 && record.processStartToken.length <= 256 + && typeof record.createdAtMs === 'number' && Number.isFinite(record.createdAtMs) + && targetValid; +} + +function readPosixOwnerRecord( + path: string, + sessionId: string, + platform: NodeJS.Platform, + expectedRole: 'owner' | 'reaper' = 'owner', +): CodexAppPosixOwnerRecord | undefined { + try { + const parsed: unknown = JSON.parse(readOwnedRegularFile( + path, + 4_096, + 'Codex App POSIX owner record', + platform, + )); + return validPosixOwnerRecord(parsed, sessionId, expectedRole) ? parsed : undefined; + } catch { + return undefined; + } +} + +interface ObservedPosixLeaseRecord { + path: string; + name: string; + nonce: string; + role: 'owner' | 'reaper'; + stat: BigIntStats; + ageMs: number; + record?: CodexAppPosixOwnerRecord; +} + +interface ObservedPosixLeaseDirectory { + stat: BigIntStats; + identity: CodexAppPosixDirectoryIdentity; + names: string[]; + generationName?: string; + generationStat?: BigIntStats; + generationNames?: string[]; + generationStats?: Map; +} + +function errnoCode(err: unknown): string | undefined { + return (err as NodeJS.ErrnoException)?.code; +} + +function posixLeaseRaceError(message: string): NodeJS.ErrnoException { + return Object.assign(new Error(message), { code: 'EAGAIN' }); +} + +function posixFileIdentity(stat: BigIntStats): CodexAppPosixFileIdentity { + return { dev: stat.dev.toString(10), ino: stat.ino.toString(10) }; +} + +function samePosixFileIdentity( + left: CodexAppPosixFileIdentity, + right: CodexAppPosixFileIdentity, +): boolean { + return left.dev === right.dev && left.ino === right.ino; +} + +function samePosixStatIdentity(before: BigIntStats, after: BigIntStats): boolean { + return before.dev === after.dev && before.ino === after.ino; +} + +function exactBigIntMode(mode: bigint): number { + return Number(mode & 0o777n); +} + +function assertPosixOwned( + stat: { uid: bigint }, + label: string, + platform: NodeJS.Platform, +): void { + if (!codexAppControlFilesystemPolicy(platform).verifyUid) return; + const uid = ownerUid(); + if (uid !== undefined && stat.uid !== BigInt(uid)) { + throw new Error(`${label} must be owned by the current uid`); + } +} + +function assertPrivatePosixLeaseRecord( + stat: BigIntStats, + label: string, + platform: NodeJS.Platform, +): void { + if (!stat.isFile() || stat.isSymbolicLink() || stat.nlink !== 1n + || exactBigIntMode(stat.mode) !== PRIVATE_FILE_MODE) { + throw new Error(`${label} must be a single-link regular 0600 file`); + } + assertPosixOwned(stat, label, platform); +} + +function posixLeaseRecordNonce(name: string): string | undefined { + const match = name.match(/^(?:owner|reap)-([a-f0-9]{64})\.(?:json|pending)$/); + return match?.[1]; +} + +function posixLeaseRecordRole(name: string): 'owner' | 'reaper' | undefined { + if (POSIX_OWNER_RECORD_RE.test(name) || POSIX_OWNER_PENDING_RE.test(name)) return 'owner'; + if (POSIX_REAPER_RECORD_RE.test(name) || POSIX_REAPER_PENDING_RE.test(name)) return 'reaper'; + return undefined; +} + +/** + * Observe one lease actor record without treating a crash-partial payload as + * insecure metadata. Wrong type/owner/mode/link count fails hard; a secure + * empty/truncated/oversized file is returned as a grace-reclaimable record. + */ +function observePosixLeaseRecord( + path: string, + name: string, + sessionId: string, + platform: NodeJS.Platform, + nowMs: number, + label: string, +): ObservedPosixLeaseRecord | undefined { + let before: BigIntStats; + try { + before = lstatSync(path, { bigint: true }); + } catch (err) { + if (errnoCode(err) === 'ENOENT') return undefined; + throw err; + } + assertPrivatePosixLeaseRecord(before, label, platform); + const nonce = posixLeaseRecordNonce(name); + if (!nonce) throw new Error(`${label} has an invalid filename`); + const role = posixLeaseRecordRole(name); + if (!role) throw new Error(`${label} has an invalid role`); + + let record: CodexAppPosixOwnerRecord | undefined; + if (before.size > 0n && before.size <= BigInt(POSIX_LEASE_RECORD_MAX_BYTES)) { + try { + const parsed: unknown = JSON.parse(readOwnedRegularFile( + path, + POSIX_LEASE_RECORD_MAX_BYTES, + label, + platform, + )); + if (validPosixOwnerRecord(parsed, sessionId, role) && parsed.nonce === nonce) { + record = parsed; + } + } catch (err) { + // JSON syntax failures are crash-partial data. A path race is handled by + // the post-read identity check below and never reclassified as stale. + if (errnoCode(err) === 'ENOENT') return undefined; + } + } + + let after: BigIntStats; + try { + after = lstatSync(path, { bigint: true }); + } catch (err) { + if (errnoCode(err) === 'ENOENT') return undefined; + throw err; + } + assertPrivatePosixLeaseRecord(after, label, platform); + if (!samePosixStatIdentity(before, after)) return undefined; + return { + path, + name, + nonce, + role, + stat: after, + ageMs: Math.max(0, nowMs - Number(after.mtimeMs)), + ...(record ? { record } : {}), + }; +} + +function observePosixLeaseDirectory( + directory: string, + platform: NodeJS.Platform, +): ObservedPosixLeaseDirectory | undefined { + let before: BigIntStats; + try { + before = lstatSync(directory, { bigint: true }); + } catch (err) { + if (errnoCode(err) === 'ENOENT') return undefined; + throw err; + } + assertPosixLeaseDirectoryStat(before, platform); + let names: string[]; + try { + names = readdirSync(directory).sort(); + } catch (err) { + if (errnoCode(err) === 'ENOENT') return undefined; + throw err; + } + let after: BigIntStats; + try { + after = lstatSync(directory, { bigint: true }); + } catch (err) { + if (errnoCode(err) === 'ENOENT') return undefined; + throw err; + } + if (!samePosixStatIdentity(before, after)) return undefined; + assertPosixLeaseDirectoryStat(after, platform); + const generationNames = names.filter(name => POSIX_DIRECTORY_GENERATION_RE.test(name)); + const generationStats = new Map(); + for (const candidateName of generationNames) { + const candidateGeneration = candidateName.match(POSIX_DIRECTORY_GENERATION_RE)?.[1]; + const candidatePath = join(directory, candidateName); + try { + const candidateStat = lstatSync(candidatePath, { bigint: true }); + assertPrivatePosixLeaseRecord( + candidateStat, + 'Codex App POSIX directory generation', + platform, + ); + const persistedGeneration = readOwnedRegularFile( + candidatePath, + 128, + 'Codex App POSIX directory generation', + platform, + ); + const verifiedStat = lstatSync(candidatePath, { bigint: true }); + assertPrivatePosixLeaseRecord( + verifiedStat, + 'Codex App POSIX directory generation', + platform, + ); + if (!candidateGeneration || persistedGeneration !== candidateGeneration + || !samePosixStatIdentity(candidateStat, verifiedStat)) { + throw new Error('Codex App POSIX directory generation verification failed'); + } + generationStats.set(candidateName, verifiedStat); + } catch (err) { + if (errnoCode(err) === 'ENOENT') return undefined; + throw err; + } + } + if (generationNames.length > 1) { + return { + stat: after, + identity: posixFileIdentity(after), + names, + generationNames, + generationStats, + }; + } + const generationName = generationNames[0]; + if (!generationName) return { stat: after, identity: posixFileIdentity(after), names }; + const generation = generationName.match(POSIX_DIRECTORY_GENERATION_RE)?.[1]; + return { + stat: after, + identity: { ...posixFileIdentity(after), generation }, + names, + generationName, + generationStat: generationStats.get(generationName), + }; +} + +function createStagedPosixLeaseRecord(input: { + directory: string; + finalPath: string; + pendingPath: string; + record: CodexAppPosixOwnerRecord; + sessionId: string; + label: string; + platform: NodeJS.Platform; +}): void { + try { + createExclusiveFile( + input.pendingPath, + JSON.stringify(input.record), + `${input.label} pending file`, + input.platform, + ); + // A complete synced inode becomes authoritative in one namespace step. + // SIGKILL during the write can therefore leave only a `.pending` file. + renameSync(input.pendingPath, input.finalPath); + fsyncDirectory(input.directory, input.platform); + const persisted = readPosixOwnerRecord( + input.finalPath, + input.sessionId, + input.platform, + input.record.role, + ); + if (!persisted || persisted.nonce !== input.record.nonce) { + throw new Error(`${input.label} verification failed`); + } + } finally { + try { unlinkSync(input.pendingPath); } catch { /* renamed or crash-partial cleanup */ } + } +} + +/** + * Rename the exact observed random-name inode out of the lease directory. + * Random actor filenames make this a compare-and-swap boundary: a concurrent + * cleaner can only receive ENOENT and can never unlink a successor's record. + */ +function retireObservedPosixLeaseRecord( + observed: ObservedPosixLeaseRecord, + leasesRoot: string, + sessionId: string, + platform: NodeJS.Platform, +): boolean { + let current: BigIntStats; + try { + current = lstatSync(observed.path, { bigint: true }); + } catch (err) { + if (errnoCode(err) === 'ENOENT') return false; + throw err; + } + assertPrivatePosixLeaseRecord(current, 'Codex App POSIX lease record', platform); + if (!samePosixStatIdentity(observed.stat, current)) return false; + const retired = join( + leasesRoot, + `.retired-${sessionKey(sessionId).slice(0, 16)}-${observed.nonce}-${randomBytes(16).toString('hex')}`, + ); + try { + renameSync(observed.path, retired); + } catch (err) { + if (errnoCode(err) === 'ENOENT') return false; + throw err; + } + try { + const moved = lstatSync(retired, { bigint: true }); + assertPrivatePosixLeaseRecord(moved, 'Retired Codex App POSIX lease record', platform); + if (!samePosixStatIdentity(observed.stat, moved)) { + throw new Error('Codex App POSIX lease record changed during retirement'); + } + } finally { + try { unlinkSync(retired); } catch { /* crash residue is outside the authority directory */ } + } + return true; +} + +/** Remove the exact generation marker only while it is the directory's sole + * remaining entry. The marker gives each mkdir generation an unforgeable + * identity even when the filesystem immediately reuses the same dev+ino. */ +function retirePosixDirectoryGeneration( + directory: string, + observed: ObservedPosixLeaseDirectory, + leasesRoot: string, + sessionId: string, + platform: NodeJS.Platform, +): boolean { + if (!observed.generationName || !observed.generationStat) return false; + const current = observePosixLeaseDirectory(directory, platform); + if (!current || current.names.length !== 1 + || current.generationName !== observed.generationName + || current.identity.generation !== observed.identity.generation + || !samePosixFileIdentity(current.identity, observed.identity) + || !current.generationStat + || !samePosixStatIdentity(current.generationStat, observed.generationStat)) return false; + const markerPath = join(directory, observed.generationName); + const retired = join( + leasesRoot, + `.retired-generation-${sessionKey(sessionId).slice(0, 16)}-${observed.identity.generation}`, + ); + try { + renameSync(markerPath, retired); + } catch (err) { + if (errnoCode(err) === 'ENOENT') return false; + throw err; + } + try { + const moved = lstatSync(retired, { bigint: true }); + assertPrivatePosixLeaseRecord(moved, 'Retired Codex App POSIX directory generation', platform); + if (!samePosixStatIdentity(observed.generationStat, moved)) { + throw new Error('Codex App POSIX directory generation changed during retirement'); + } + } finally { + try { unlinkSync(retired); } catch { /* crash residue is outside authority */ } + } + try { rmdirSync(directory); } catch (err) { + if (errnoCode(err) !== 'ENOENT' && errnoCode(err) !== 'ENOTEMPTY') throw err; + } + return true; +} + +/** Recover only the crash residue in which the directory contains two or more + * individually valid generation markers and no actor can still be live. Each + * exact marker inode is moved out before unlink, so a concurrent replacement + * turns into a retry instead of deleting successor authority. */ +function retireAmbiguousPosixDirectoryGenerations( + directory: string, + observed: ObservedPosixLeaseDirectory, + leasesRoot: string, + sessionId: string, + platform: NodeJS.Platform, +): boolean { + const observedNames = observed.generationNames; + const observedStats = observed.generationStats; + if (!observedNames || observedNames.length < 2 || !observedStats) return false; + const current = observePosixLeaseDirectory(directory, platform); + if (!current + || !samePosixFileIdentity(current.identity, observed.identity) + || current.names.length !== observedNames.length + || current.generationNames?.length !== observedNames.length + || current.generationNames.some((name, index) => name !== observedNames[index])) return false; + for (const name of observedNames) { + const expectedStat = observedStats.get(name); + const currentStat = current.generationStats?.get(name); + if (!expectedStat || !currentStat || !samePosixStatIdentity(expectedStat, currentStat)) { + return false; + } + } + for (const name of observedNames) { + const expectedStat = observedStats.get(name)!; + const markerPath = join(directory, name); + const retired = join( + leasesRoot, + `.retired-generation-${sessionKey(sessionId).slice(0, 16)}-${randomBytes(16).toString('hex')}`, + ); + try { + renameSync(markerPath, retired); + } catch (err) { + if (errnoCode(err) === 'ENOENT') return false; + throw err; + } + try { + const moved = lstatSync(retired, { bigint: true }); + assertPrivatePosixLeaseRecord(moved, 'Retired Codex App POSIX directory generation', platform); + if (!samePosixStatIdentity(expectedStat, moved)) { + throw new Error('Codex App POSIX directory generation changed during retirement'); + } + } finally { + try { unlinkSync(retired); } catch { /* residue is outside authority */ } + } + } + try { rmdirSync(directory); } catch (err) { + if (errnoCode(err) !== 'ENOENT' && errnoCode(err) !== 'ENOTEMPTY') throw err; + } + return true; +} + +function assertPosixLeaseDirectoryStat(stat: BigIntStats, platform: NodeJS.Platform): void { + if (!stat.isDirectory() || stat.isSymbolicLink() || exactBigIntMode(stat.mode) !== PRIVATE_DIRECTORY_MODE) { + throw new Error('Codex App POSIX owner lease must be a real 0700 directory'); + } + assertPosixOwned(stat, 'Codex App POSIX owner lease', platform); +} + +function posixLeaseRecordMatchesDirectory( + observed: ObservedPosixLeaseRecord, + directory: ObservedPosixLeaseDirectory | CodexAppPosixDirectoryIdentity, +): boolean { + if (!observed.record) return false; + const identity = 'identity' in directory ? directory.identity : directory; + if (!samePosixFileIdentity(observed.record.directoryIdentity, identity)) return false; + return observed.record.version === 2 + ? identity.generation === undefined + : observed.record.directoryIdentity.generation === identity.generation; +} + +function posixOwnerTarget(observed: ObservedPosixLeaseRecord): CodexAppPosixOwnerTarget { + return { + nonce: observed.nonce, + recordIdentity: posixFileIdentity(observed.stat), + pid: observed.record?.pid ?? null, + processStartToken: observed.record?.processStartToken ?? null, + }; +} + +function posixReaperTargetsOwner( + reaper: ObservedPosixLeaseRecord, + owner: ObservedPosixLeaseRecord, +): boolean { + const target = reaper.record?.targetOwner; + if (!target || target.nonce !== owner.nonce + || !samePosixFileIdentity(target.recordIdentity, posixFileIdentity(owner.stat))) return false; + if (!owner.record) return target.pid === null && target.processStartToken === null; + return target.pid === owner.record.pid + && target.processStartToken === owner.record.processStartToken; +} + +/** + * Process-lifetime publisher lease for POSIX. + * + * Both owners and stale cleaners publish complete, random-name actor records. + * An actor record is written to `.pending`, synced, and atomically renamed to + * `.json`, so SIGKILL cannot create a partial authoritative record. Existing + * partial records remain recoverable after a grace interval. Every complete + * actor is bound to the exact directory dev+ino it intended to mutate; reapers + * additionally bind the exact owner-record inode/process tuple. A cleaner moves + * the exact observed random-name inode out of the authority directory before + * deleting it; that rename is the CAS boundary which prevents a delayed actor + * from poisoning or deleting a successor's record after path replacement. + */ +export async function acquireCodexAppPosixOwnerLease( + options: CodexAppPosixOwnerLeaseOptions, +): Promise { + const platform = options.platform ?? process.platform; + if (platform === 'win32') throw new Error('POSIX owner lease is unavailable on Windows'); + const now = options.now ?? Date.now; + const wait = options.wait ?? (delayMs => new Promise(resolve => setTimeout(resolve, delayMs))); + const timeoutMs = options.timeoutMs ?? 10_000; + const retryDelayMs = options.retryDelayMs ?? 50; + const initializationGraceMs = options.initializationGraceMs ?? 1_000; + const pid = options.pid ?? process.pid; + const processStartToken = options.processStartToken + ?? readCodexAppProcessStartToken(pid, platform); + if (!processStartToken) { + throw new Error('Cannot verify current process start token for Codex App POSIX owner lease'); + } + const inspectOwner = options.inspectOwner + ?? ((ownerPid: number, token: string) => inspectCodexAppOwnerProcess(ownerPid, token, platform)); + ensureCodexAppControlDirectory(options.controlRoot, platform); + const leasesRoot = join(options.controlRoot, 'leases'); + ensureCodexAppControlDirectory(leasesRoot, platform); + const directory = codexAppPosixOwnerLeaseDirectory(options.controlRoot, options.sessionId); + const deadline = now() + timeoutMs; + + const retry = async (): Promise => { + if (now() >= deadline) throw new Error('Codex App POSIX owner lease timed out'); + await wait(Math.min(retryDelayMs, Math.max(0, deadline - now()))); + }; + + acquireLoop: while (true) { + const nonce = randomBytes(32).toString('hex'); + const directoryGeneration = randomBytes(32).toString('hex'); + const generationName = `generation-${directoryGeneration}.json`; + const generationPath = join(directory, generationName); + const ownerRecordPath = join(directory, `owner-${nonce}.json`); + const ownerPendingPath = join(directory, `owner-${nonce}.pending`); + let createdDirectory = false; + try { + mkdirSync(directory, { mode: PRIVATE_DIRECTORY_MODE }); + createdDirectory = true; + const ownedDirectoryStat = lstatSync(directory, { bigint: true }); + assertPosixLeaseDirectoryStat(ownedDirectoryStat, platform); + createExclusiveFile( + generationPath, + directoryGeneration, + 'Codex App POSIX directory generation', + platform, + ); + fsyncDirectory(directory, platform); + const ownedDirectoryIdentity: CodexAppPosixDirectoryIdentity = { + ...posixFileIdentity(ownedDirectoryStat), + generation: directoryGeneration, + }; + if (options.onOwnerDirectoryCreated) { + await options.onOwnerDirectoryCreated(directory); + } + const record: CodexAppPosixOwnerRecord = { + version: 3, + role: 'owner', + sessionId: options.sessionId, + nonce, + pid, + processStartToken, + createdAtMs: now(), + directoryIdentity: ownedDirectoryIdentity, + }; + try { + createStagedPosixLeaseRecord({ + directory, + finalPath: ownerRecordPath, + pendingPath: ownerPendingPath, + record, + sessionId: options.sessionId, + label: 'Codex App POSIX owner record', + platform, + }); + const persisted = readPosixOwnerRecord(ownerRecordPath, options.sessionId, platform, 'owner'); + if (!persisted || persisted.nonce !== nonce) { + throw new Error('Codex App POSIX owner record verification failed'); + } + if (options.onOwnerRecordPublished) { + await options.onOwnerRecordPublished(directory, ownerRecordPath); + } + const installed = observePosixLeaseDirectory(directory, platform); + if (!installed || !samePosixStatIdentity(installed.stat, ownedDirectoryStat) + || installed.identity.generation !== directoryGeneration + || installed.names.length !== 2 + || !installed.names.includes(generationName) + || !installed.names.includes(basename(ownerRecordPath))) { + throw posixLeaseRaceError( + 'Codex App POSIX owner directory changed during record publication', + ); + } + } catch (err) { + try { unlinkSync(ownerPendingPath); } catch { /* not installed */ } + try { unlinkSync(ownerRecordPath); } catch { /* not installed */ } + try { unlinkSync(generationPath); } catch { /* replacement directory owns a different generation */ } + try { rmdirSync(directory); } catch { /* competing reaper or residue */ } + throw err; + } + let released = false; + const isOwned = (): boolean => { + if (released) return false; + try { + const currentDirectory = observePosixLeaseDirectory(directory, platform); + if (!currentDirectory + || !samePosixFileIdentity(currentDirectory.identity, ownedDirectoryIdentity)) return false; + const currentOwner = observePosixLeaseRecord( + ownerRecordPath, + basename(ownerRecordPath), + options.sessionId, + platform, + now(), + 'Codex App POSIX owner record', + ); + if (!currentOwner || currentOwner.record?.nonce !== nonce + || !posixLeaseRecordMatchesDirectory(currentOwner, ownedDirectoryIdentity)) return false; + for (const name of currentDirectory.names.filter(candidate => ( + POSIX_REAPER_RECORD_RE.test(candidate) || POSIX_REAPER_PENDING_RE.test(candidate) + ))) { + const reaper = observePosixLeaseRecord( + join(directory, name), + name, + options.sessionId, + platform, + now(), + 'Codex App POSIX reaper record', + ); + if (!reaper) return false; + // A delayed cleaner may publish into a replacement directory. Only + // a complete actor bound to this inode and this exact owner tuple + // can suspend authority. Partial/foreign actors never authorize a + // takeover and are reclaimed by the acquisition loop. + if (posixLeaseRecordMatchesDirectory(reaper, currentDirectory) + && posixReaperTargetsOwner(reaper, currentOwner)) return false; + } + return true; + } catch { + return false; + } + }; + return { + directory, + ownerRecordPath, + isOwned, + release: () => { + if (released) return; + released = true; + const persisted = readPosixOwnerRecord(ownerRecordPath, options.sessionId, platform, 'owner'); + if (persisted?.nonce === nonce + && samePosixFileIdentity(persisted.directoryIdentity, ownedDirectoryIdentity)) { + try { unlinkSync(ownerRecordPath); } catch { /* already reaped */ } + } + try { unlinkSync(ownerPendingPath); } catch { /* never published or already retired */ } + try { + const currentDirectory = observePosixLeaseDirectory(directory, platform); + if (currentDirectory + && currentDirectory.identity.generation === directoryGeneration + && currentDirectory.names.length === 1 + && currentDirectory.names[0] === generationName) { + retirePosixDirectoryGeneration( + directory, + currentDirectory, + leasesRoot, + options.sessionId, + platform, + ); + } + } catch { /* another contender owns cleanup */ } + }, + }; + } catch (err) { + if (createdDirectory && (errnoCode(err) === 'EAGAIN' || errnoCode(err) === 'ENOENT')) { + await retry(); + continue; + } + if (createdDirectory || errnoCode(err) !== 'EEXIST') throw err; + } + + options.onContended?.(); + const snapshot = observePosixLeaseDirectory(directory, platform); + if (!snapshot) { + await retry(); + continue; + } + + const knownNames = snapshot.names.filter(name => ( + POSIX_OWNER_RECORD_RE.test(name) + || POSIX_OWNER_PENDING_RE.test(name) + || POSIX_REAPER_RECORD_RE.test(name) + || POSIX_REAPER_PENDING_RE.test(name) + || POSIX_DIRECTORY_GENERATION_RE.test(name) + )); + if (knownNames.length !== snapshot.names.length) { + throw new Error('Codex App POSIX owner lease contains an unknown record'); + } + + if ((snapshot.generationNames?.length ?? 0) > 1) { + let retiredActor = false; + for (const name of snapshot.names.filter(candidate => ( + POSIX_OWNER_RECORD_RE.test(candidate) + || POSIX_OWNER_PENDING_RE.test(candidate) + || POSIX_REAPER_RECORD_RE.test(candidate) + || POSIX_REAPER_PENDING_RE.test(candidate) + ))) { + const observed = observePosixLeaseRecord( + join(directory, name), + name, + options.sessionId, + platform, + now(), + 'Codex App POSIX ambiguous-generation actor', + ); + if (!observed) { + await retry(); + continue acquireLoop; + } + if (observed.record) { + const status = inspectOwner( + observed.record.pid, + observed.record.processStartToken, + ); + if (status !== 'dead') { + await retry(); + continue acquireLoop; + } + } else if (observed.ageMs < initializationGraceMs) { + await retry(); + continue acquireLoop; + } + if (!retireObservedPosixLeaseRecord( + observed, + leasesRoot, + options.sessionId, + platform, + )) { + await retry(); + continue acquireLoop; + } + retiredActor = true; + } + if (retiredActor) continue; + const ageMs = Math.max(0, now() - Number(snapshot.stat.mtimeMs)); + if (ageMs < initializationGraceMs + || !retireAmbiguousPosixDirectoryGenerations( + directory, + snapshot, + leasesRoot, + options.sessionId, + platform, + )) { + await retry(); + } + continue; + } + + // A live/unknown cleaner is authority and blocks fail-closed. A dead or + // crash-partial cleaner has a unique filename, so retiring that exact inode + // cannot clobber a successor cleaner. + const reaperNames = snapshot.names.filter(name => ( + POSIX_REAPER_RECORD_RE.test(name) || POSIX_REAPER_PENDING_RE.test(name) + )); + if (reaperNames.length > 0) { + let retiredAny = false; + for (const name of reaperNames) { + const observed = observePosixLeaseRecord( + join(directory, name), + name, + options.sessionId, + platform, + now(), + 'Codex App POSIX reaper record', + ); + if (!observed) { + await retry(); + continue acquireLoop; + } + if (observed.record && !posixLeaseRecordMatchesDirectory(observed, snapshot)) { + // The actor was published by a delayed cleaner that pinned an older + // directory inode. It has no authority in this replacement directory, + // regardless of whether that process is still alive. + } else if (observed.record) { + const status = inspectOwner( + observed.record.pid, + observed.record.processStartToken, + ); + if (status !== 'dead') { + await retry(); + continue acquireLoop; + } + } else if (observed.ageMs < initializationGraceMs) { + await retry(); + continue acquireLoop; + } + if (!retireObservedPosixLeaseRecord( + observed, + leasesRoot, + options.sessionId, + platform, + )) { + await retry(); + continue acquireLoop; + } + retiredAny = true; + } + if (retiredAny) continue; + } + + const ownerNames = snapshot.names.filter(name => ( + POSIX_OWNER_RECORD_RE.test(name) || POSIX_OWNER_PENDING_RE.test(name) + )); + const observedOwners: ObservedPosixLeaseRecord[] = []; + let retiredForeignOwner = false; + for (const name of ownerNames) { + const observed = observePosixLeaseRecord( + join(directory, name), + name, + options.sessionId, + platform, + now(), + 'Codex App POSIX owner record', + ); + if (!observed) { + await retry(); + continue acquireLoop; + } + if (observed.record && !posixLeaseRecordMatchesDirectory(observed, snapshot)) { + // This can only be a creator that pinned an older directory and resumed + // after the fixed path was replaced. Its process liveness is irrelevant: + // the record never held authority over this inode. + if (!retireObservedPosixLeaseRecord( + observed, + leasesRoot, + options.sessionId, + platform, + )) { + await retry(); + continue acquireLoop; + } + retiredForeignOwner = true; + continue; + } + observedOwners.push(observed); + } + if (retiredForeignOwner) continue; + if (observedOwners.length > 1) { + let retiredNonAuthoritativeOwner = false; + for (const observed of observedOwners) { + const reclaimable = observed.record + ? inspectOwner(observed.record.pid, observed.record.processStartToken) === 'dead' + : observed.ageMs >= initializationGraceMs; + if (!reclaimable) continue; + // A creator can be paused after mkdir while another contender replaces + // the empty inode, then resume and publish into that replacement before + // noticing another owner. Exact actor-file CAS makes dead/partial losing + // candidates independently reclaimable without touching a live/unknown + // same-directory candidate. + if (!retireObservedPosixLeaseRecord( + observed, + leasesRoot, + options.sessionId, + platform, + )) { + await retry(); + continue acquireLoop; + } + retiredNonAuthoritativeOwner = true; + } + if (retiredNonAuthoritativeOwner) continue; + // Multiple live/unknown candidates are ambiguous. None can be newly + // granted by this contender; wait for a creator to finish its directory + // CAS or for liveness to become provable. + await retry(); + continue; + } + + if (observedOwners.length === 0) { + const ageMs = Math.max(0, now() - Number(snapshot.stat.mtimeMs)); + if (ageMs < initializationGraceMs) { + await retry(); + continue; + } + if (snapshot.generationName) { + if (!retirePosixDirectoryGeneration( + directory, + snapshot, + leasesRoot, + options.sessionId, + platform, + )) await retry(); + } else { + try { + rmdirSync(directory); + } catch (err) { + if (errnoCode(err) !== 'ENOENT' && errnoCode(err) !== 'ENOTEMPTY') throw err; + await retry(); + } + } + continue; + } + + const observedOwner = observedOwners[0]!; + if (observedOwner.record) { + const status = inspectOwner( + observedOwner.record.pid, + observedOwner.record.processStartToken, + ); + if (status !== 'dead') { + await retry(); + continue; + } + } else if (observedOwner.ageMs < initializationGraceMs) { + await retry(); + continue; + } + + const reaperNonce = randomBytes(32).toString('hex'); + const reaperPath = join(directory, `reap-${reaperNonce}.json`); + const reaperPendingPath = join(directory, `reap-${reaperNonce}.pending`); + const reaperRecord: CodexAppPosixOwnerRecord = { + version: snapshot.identity.generation ? 3 : 2, + role: 'reaper', + sessionId: options.sessionId, + nonce: reaperNonce, + pid, + processStartToken, + createdAtMs: now(), + directoryIdentity: snapshot.identity, + targetOwner: posixOwnerTarget(observedOwner), + }; + try { + if (options.onBeforeReaperRecordPublished) { + await options.onBeforeReaperRecordPublished(directory); + } + createStagedPosixLeaseRecord({ + directory, + finalPath: reaperPath, + pendingPath: reaperPendingPath, + record: reaperRecord, + sessionId: options.sessionId, + label: 'Codex App POSIX reaper record', + platform, + }); + if (options.onReaperRecordPublished) { + await options.onReaperRecordPublished(directory, reaperPath); + } + const reaperDirectory = observePosixLeaseDirectory(directory, platform); + if (!reaperDirectory || !samePosixStatIdentity(reaperDirectory.stat, snapshot.stat)) { + throw posixLeaseRaceError( + 'Codex App POSIX owner directory changed during reaper election', + ); + } + } catch (err) { + try { unlinkSync(reaperPendingPath); } catch { /* never installed */ } + try { unlinkSync(reaperPath); } catch { /* never installed */ } + if (errnoCode(err) === 'EEXIST' || errnoCode(err) === 'ENOENT' + || errnoCode(err) === 'EAGAIN') { + await retry(); + continue; + } + throw err; + } + + try { + const electedSnapshot = observePosixLeaseDirectory(directory, platform); + if (!electedSnapshot) continue; + const electedReapers: ObservedPosixLeaseRecord[] = []; + let electionMustRetry = false; + for (const name of electedSnapshot.names.filter(candidate => ( + POSIX_REAPER_RECORD_RE.test(candidate) || POSIX_REAPER_PENDING_RE.test(candidate) + ))) { + const observed = observePosixLeaseRecord( + join(directory, name), + name, + options.sessionId, + platform, + now(), + 'Codex App POSIX reaper record', + ); + if (!observed || !observed.record) { + electionMustRetry = true; + break; + } + if (!posixLeaseRecordMatchesDirectory(observed, electedSnapshot) + || !posixReaperTargetsOwner(observed, observedOwner)) { + electionMustRetry = true; + break; + } + if (observed.nonce !== reaperNonce) { + const status = inspectOwner( + observed.record.pid, + observed.record.processStartToken, + ); + if (status !== 'alive') { + // Unknown actors block fail-closed; dead actors are reclaimed by + // the outer observation loop before a new election. + electionMustRetry = true; + break; + } + } + electedReapers.push(observed); + } + if (electionMustRetry) continue; + electedReapers.sort((a, b) => ( + (a.record!.createdAtMs - b.record!.createdAtMs) || a.nonce.localeCompare(b.nonce) + )); + // Simultaneous contenders may both have published before observing each + // other. The oldest deterministic actor proceeds; all others withdraw. + if (electedReapers[0]?.nonce !== reaperNonce) continue; + + const currentOwner = observePosixLeaseRecord( + observedOwner.path, + observedOwner.name, + options.sessionId, + platform, + now(), + 'Codex App POSIX owner record', + ); + if (!currentOwner || !samePosixStatIdentity(currentOwner.stat, observedOwner.stat) + || (currentOwner.record && !posixLeaseRecordMatchesDirectory(currentOwner, electedSnapshot)) + || !posixReaperTargetsOwner( + electedReapers.find(candidate => candidate.nonce === reaperNonce)!, + currentOwner, + )) continue; + if (currentOwner.record) { + if (inspectOwner( + currentOwner.record.pid, + currentOwner.record.processStartToken, + ) !== 'dead') continue; + } else if (currentOwner.ageMs < initializationGraceMs) { + continue; + } + if (!retireObservedPosixLeaseRecord( + currentOwner, + leasesRoot, + options.sessionId, + platform, + )) { + continue; + } + } finally { + try { unlinkSync(reaperPendingPath); } catch { /* renamed or already retired */ } + try { unlinkSync(reaperPath); } catch { /* another cleaner retired it */ } + } + const emptied = observePosixLeaseDirectory(directory, platform); + if (emptied?.generationName && emptied.names.length === 1) { + retirePosixDirectoryGeneration( + directory, + emptied, + leasesRoot, + options.sessionId, + platform, + ); + } else { + try { rmdirSync(directory); } catch (err) { + if (errnoCode(err) !== 'ENOENT' && errnoCode(err) !== 'ENOTEMPTY') throw err; + } + } + } +} + +export function generateCodexAppControlEpoch(): string { + return randomBytes(32).toString('base64url'); +} + +export interface CodexAppControlLocator { + version: typeof CONTROL_LOCATOR_VERSION; + sessionId: string; + epoch: string; + endpoint: string; +} + +export function isValidCodexAppWindowsPipeEndpoint(endpoint: unknown): endpoint is string { + return typeof endpoint === 'string' + && /^\\\\\?\\pipe\\botmux-codex-app-[a-f0-9]{64}$/.test(endpoint); +} + +function isValidCodexAppPosixSocketEndpoint( + endpoint: unknown, + locatorPath: string | undefined, + expectedControlRoot: string | undefined, + expectedSessionId: string, + platform: NodeJS.Platform, +): endpoint is string { + if (typeof endpoint !== 'string' || !locatorPath || !expectedControlRoot + || !isAbsolute(endpoint) || !isAbsolute(locatorPath) || !isAbsolute(expectedControlRoot)) { + return false; + } + // Do not infer trust from an attacker-selected locator path. The caller must + // supply the already trusted control root and both paths must be the exact + // canonical children for this session. + if (locatorPath !== codexAppControlLocatorPath( + expectedControlRoot, + expectedSessionId, + platform, + )) return false; + const endpointDirectory = join(expectedControlRoot, 'sockets'); + const leaf = basename(endpoint); + return POSIX_SOCKET_LEAF_RE.test(leaf) + && endpoint === join(endpointDirectory, leaf) + // Darwin's sockaddr_un.sun_path is 104 bytes including the terminator. + && Buffer.byteLength(endpoint, 'utf8') < 104; +} + +export interface CodexAppControlLocatorValidationOptions { + platform?: NodeJS.Platform; + locatorPath?: string; + /** Trusted, worker-owned root; never derive this authority from locatorPath. */ + expectedControlRoot?: string; +} + +export function validateCodexAppControlLocator( + value: unknown, + expectedSessionId: string, + options: CodexAppControlLocatorValidationOptions = {}, +): CodexAppControlLocator | undefined { + if (!value || typeof value !== 'object') return undefined; + const record = value as Record; + const platform = options.platform ?? process.platform; + const endpointValid = platform === 'win32' + ? isValidCodexAppWindowsPipeEndpoint(record.endpoint) + : isValidCodexAppPosixSocketEndpoint( + record.endpoint, + options.locatorPath, + options.expectedControlRoot, + expectedSessionId, + platform, + ); + if (record.version !== CONTROL_LOCATOR_VERSION + || record.sessionId !== expectedSessionId + || !isValidGeneration(record.epoch) + || !endpointValid) return undefined; + return record as unknown as CodexAppControlLocator; +} + +export interface CodexAppControlIdentity { + generation: string; + publicKey: string; + createdAtMs: number; +} + +/** + * pending may contain an old live identity plus a fresh spawn candidate. The + * first identity that answers this worker's fresh socket challenge is atomically + * collapsed to the sole active identity. No private material is persisted. + */ +export interface CodexAppControlState { + version: typeof CONTROL_STATE_VERSION; + status: 'pending' | 'active'; + identities: CodexAppControlIdentity[]; + updatedAtMs: number; + activatedAtMs?: number; +} + +function isValidGeneration(value: unknown): value is string { + return typeof value === 'string' && GENERATION_RE.test(value); +} + +function isValidChallenge(value: unknown): value is string { + return typeof value === 'string' && CHALLENGE_RE.test(value); +} + +function importPublicKey(encoded: string): KeyObject { + return createPublicKey({ key: Buffer.from(encoded, 'base64url'), format: 'der', type: 'spki' }); +} + +function importPrivateKey(encoded: string): KeyObject { + return createPrivateKey({ key: Buffer.from(encoded, 'base64url'), format: 'der', type: 'pkcs8' }); +} + +function validIdentity(value: unknown): value is CodexAppControlIdentity { + if (!value || typeof value !== 'object') return false; + const record = value as Record; + if (!isValidGeneration(record.generation) + || typeof record.publicKey !== 'string' || !KEY_RE.test(record.publicKey) + || typeof record.createdAtMs !== 'number' || !Number.isFinite(record.createdAtMs)) return false; + try { + return importPublicKey(record.publicKey).asymmetricKeyType === 'ed25519'; + } catch { + return false; + } +} + +function validState(value: unknown): value is CodexAppControlState { + if (!value || typeof value !== 'object') return false; + const record = value as Record; + if (record.version !== CONTROL_STATE_VERSION + || (record.status !== 'pending' && record.status !== 'active') + || !Array.isArray(record.identities) + || record.identities.length < 1 + || record.identities.length > MAX_STATE_IDENTITIES + || !record.identities.every(validIdentity) + || new Set(record.identities.map(identity => identity.generation)).size !== record.identities.length + || typeof record.updatedAtMs !== 'number' || !Number.isFinite(record.updatedAtMs) + || (record.status === 'active' && record.identities.length !== 1) + || (record.activatedAtMs !== undefined + && (typeof record.activatedAtMs !== 'number' || !Number.isFinite(record.activatedAtMs)))) return false; + return true; +} + +function assertRegularSingleLinkWithinLimit( + stat: Stats, + maxBytes: number, + label: string, + platform: NodeJS.Platform, +): void { + const policy = codexAppControlFilesystemPolicy(platform); + if (!stat.isFile() || stat.isSymbolicLink() || stat.nlink !== 1 + || (policy.verifyExactMode && exactMode(stat.mode) !== PRIVATE_FILE_MODE) + || stat.size <= 0 || stat.size > maxBytes) { + const mode = policy.verifyExactMode ? ' 0600' : ''; + throw new Error(`${label} must be a single-link regular${mode} file within the size limit`); + } + assertOwned(stat, label, platform); +} + +function sameFileIdentity( + before: Stats, + after: Stats, +): boolean { + return Number.isFinite(before.dev) && Number.isFinite(before.ino) + && before.dev === after.dev && before.ino === after.ino; +} + +function readOwnedRegularFile( + path: string, + maxBytes: number, + label: string, + platform: NodeJS.Platform = process.platform, +): string { + const before = lstatSync(path); + assertRegularSingleLinkWithinLimit(before, maxBytes, label, platform); + const fd = openSync(path, fsConstants.O_RDONLY | noFollowFlag(platform)); + try { + const stat = fstatSync(fd); + assertRegularSingleLinkWithinLimit(stat, maxBytes, label, platform); + if (!sameFileIdentity(before, stat)) throw new Error(`${label} changed between lstat and open`); + return readFileSync(fd, 'utf8'); + } finally { + closeSync(fd); + } +} + +export function readCodexAppControlState( + path: string, + platform: NodeJS.Platform = process.platform, +): CodexAppControlState | undefined { + try { + const parsed: unknown = JSON.parse(readOwnedRegularFile( + path, + STATE_MAX_BYTES, + 'Codex App control state', + platform, + )); + return validState(parsed) ? parsed : undefined; + } catch { + return undefined; + } +} + +function createExclusiveFile( + path: string, + contents: string, + label: string, + platform: NodeJS.Platform = process.platform, +): void { + const policy = codexAppControlFilesystemPolicy(platform); + const fd = openSync( + path, + fsConstants.O_WRONLY | fsConstants.O_CREAT | fsConstants.O_EXCL | noFollowFlag(platform), + PRIVATE_FILE_MODE, + ); + try { + if (policy.chmodAfterCreate) fchmodSync(fd, PRIVATE_FILE_MODE); + const stat = fstatSync(fd); + if (!stat.isFile() || stat.nlink !== 1 + || (policy.verifyExactMode && exactMode(stat.mode) !== PRIVATE_FILE_MODE)) { + throw new Error(`${label} was not created as a single-link regular private file`); + } + assertOwned(stat, label, platform); + const data = Buffer.from(contents, 'utf8'); + let offset = 0; + while (offset < data.length) offset += writeSync(fd, data, offset, data.length - offset); + fsyncSync(fd); + } finally { + closeSync(fd); + } +} + +function fsyncDirectory(directory: string, platform: NodeJS.Platform): void { + if (!codexAppControlFilesystemPolicy(platform).fsyncDirectory) return; + const fd = openSync(directory, fsConstants.O_RDONLY | noFollowFlag(platform)); + try { + const stat = fstatSync(fd); + if (!stat.isDirectory()) throw new Error('Codex App control state parent must be a directory'); + assertOwned(stat, 'Codex App control state parent', platform); + fsyncSync(fd); + } finally { + closeSync(fd); + } +} + +/** Symlink-safe atomic state replacement inside a private directory. */ +export function writeCodexAppControlState(path: string, state: CodexAppControlState): void { + writeCodexAppControlStateForPlatform(path, state, process.platform); +} + +export function writeCodexAppControlStateForPlatform( + path: string, + state: CodexAppControlState, + platform: NodeJS.Platform, +): void { + if (!validState(state)) throw new Error('Codex App control state is invalid'); + const directory = dirnameForPlatform(platform, path); + ensureCodexAppControlDirectory(directory, platform); + const tmp = joinForPlatform( + platform, + directory, + `.${sessionKey(path)}.${process.pid}.${randomBytes(8).toString('hex')}.tmp`, + ); + try { + createExclusiveFile(tmp, JSON.stringify(state), 'Codex App control state temp file', platform); + renameSync(tmp, path); + // The file itself was synced before rename; sync the parent as well so a + // proved generation does not silently roll back after a host crash. + fsyncDirectory(directory, platform); + const persisted = readCodexAppControlState(path, platform); + if (!persisted || persisted.status !== state.status + || persisted.identities.map(identity => identity.generation).join(',') + !== state.identities.map(identity => identity.generation).join(',')) { + throw new Error('Codex App control state verification failed after rename'); + } + } finally { + try { unlinkSync(tmp); } catch { /* renamed or never created */ } + } +} + +export function readCodexAppControlLocator( + path: string, + expectedSessionId: string, + platform: NodeJS.Platform = process.platform, + expectedControlRoot: string | undefined = platform === 'win32' + ? undefined + : codexAppPosixControlRoot(), +): CodexAppControlLocator | undefined { + try { + const value: unknown = JSON.parse(readOwnedRegularFile( + path, + LOCATOR_MAX_BYTES, + 'Codex App control locator', + platform, + )); + return validateCodexAppControlLocator(value, expectedSessionId, { + platform, + locatorPath: path, + expectedControlRoot, + }); + } catch { + return undefined; + } +} + +export function writeCodexAppControlLocator( + path: string, + locator: CodexAppControlLocator, + platform: NodeJS.Platform = process.platform, + expectedControlRoot: string | undefined = platform === 'win32' + ? undefined + : codexAppPosixControlRoot(), +): void { + if (!validateCodexAppControlLocator(locator, locator.sessionId, { + platform, + locatorPath: path, + expectedControlRoot, + })) { + throw new Error('Codex App control locator is invalid'); + } + const directory = dirnameForPlatform(platform, path); + ensureCodexAppControlDirectory(directory, platform); + const tmp = joinForPlatform( + platform, + directory, + `.${sessionKey(path)}.${process.pid}.${randomBytes(8).toString('hex')}.tmp`, + ); + try { + createExclusiveFile(tmp, JSON.stringify(locator), 'Codex App control locator temp file', platform); + renameSync(tmp, path); + fsyncDirectory(directory, platform); + const persisted = readCodexAppControlLocator( + path, + locator.sessionId, + platform, + expectedControlRoot, + ); + if (!persisted || persisted.epoch !== locator.epoch || persisted.endpoint !== locator.endpoint) { + throw new Error('Codex App control locator verification failed after rename'); + } + } finally { + try { unlinkSync(tmp); } catch { /* renamed or never created */ } + } +} + +/** + * The ordering contract is security-sensitive on every platform: the random + * pipe/socket must already be bound before its locator becomes visible. + */ +export async function bindThenPublishCodexAppControlLocator(input: { + sessionId: string; + epoch: string; + endpoint: string; + listen: (endpoint: string) => Promise; + publish: (locator: CodexAppControlLocator) => void; + platform?: NodeJS.Platform; + locatorPath?: string; + expectedControlRoot?: string; + isCurrent?: () => boolean; + retire?: () => void; +}): Promise { + const locator = validateCodexAppControlLocator({ + version: CONTROL_LOCATOR_VERSION, + sessionId: input.sessionId, + epoch: input.epoch, + endpoint: input.endpoint, + }, input.sessionId, { + platform: input.platform, + locatorPath: input.locatorPath, + expectedControlRoot: input.expectedControlRoot, + }); + if (!locator) throw new Error('Codex App control endpoint publication metadata is invalid'); + await input.listen(locator.endpoint); + if (input.isCurrent && !input.isCurrent()) { + input.retire?.(); + return undefined; + } + try { + input.publish(locator); + } catch (err) { + input.retire?.(); + throw err; + } + return locator; +} + +export function shouldFailCodexAppControlChannel(input: { + channelId: number; + currentChannelId: number; + stopping: boolean; +}): boolean { + return !input.stopping && input.channelId === input.currentChannelId; +} + +/** + * Runner-side endpoint policy for locator rotations. A never-accepted + * locator may be retried (the independently random, protected epoch is still + * required for acceptance) until the shared startup deadline. Once accepted, + * the pipe name is permanently burned and only a newly published endpoint can + * be used. + */ +export class CodexAppControlEndpointTracker { + private readonly attemptsByEndpoint = new Map(); + private readonly acceptedEndpoints = new Set(); + + take(locator: CodexAppControlLocator): string | undefined { + if (this.acceptedEndpoints.has(locator.endpoint)) return undefined; + const attempts = this.attemptsByEndpoint.get(locator.endpoint) ?? 0; + this.attemptsByEndpoint.set(locator.endpoint, attempts + 1); + return locator.endpoint; + } + + noteAccepted(endpoint: string): void { + if (!this.attemptsByEndpoint.has(endpoint)) { + throw new Error('Cannot accept an unattempted Codex App control endpoint'); + } + this.acceptedEndpoints.add(endpoint); + } + + wasAttempted(endpoint: string): boolean { + return this.attemptsByEndpoint.has(endpoint); + } + + attemptCount(endpoint: string): number { + return this.attemptsByEndpoint.get(endpoint) ?? 0; + } + + wasAccepted(endpoint: string): boolean { + return this.acceptedEndpoints.has(endpoint); + } +} + +/** Read and select one locator endpoint; missing/corrupt files are a poll miss. */ +export function takeCodexAppControlLocatorEndpoint(input: { + locatorPath: string; + sessionId: string; + tracker: CodexAppControlEndpointTracker; + platform?: NodeJS.Platform; + expectedControlRoot?: string; +}): { endpoint: string; epoch: string } | undefined { + const platform = input.platform ?? process.platform; + const locator = readCodexAppControlLocator( + input.locatorPath, + input.sessionId, + platform, + input.expectedControlRoot ?? (platform === 'win32' ? undefined : codexAppPosixControlRoot()), + ); + if (!locator) return undefined; + const endpoint = input.tracker.take(locator); + return endpoint ? { endpoint, epoch: locator.epoch } : undefined; +} + +const PERSISTENT_BACKEND_TYPES: ReadonlySet = new Set(['tmux', 'herdr', 'zellij']); + +/** Missing/corrupt/legacy generations cannot prove a warm reattach. */ +export function shouldColdStartCodexAppReattach(input: { + cliId?: string; + backendType: BackendType; + isReattach: boolean; + persistedState?: CodexAppControlState; +}): boolean { + return input.cliId === 'codex-app' + && input.isReattach + && PERSISTENT_BACKEND_TYPES.has(input.backendType) + && !input.persistedState; +} + +export interface CodexAppControlBootstrap { + path: string; + identity: CodexAppControlIdentity; +} + +export interface ConsumedCodexAppControlBootstrap { + generation: string; + privateKey: KeyObject; + socketPath?: string; + locatorPath?: string; +} + +export type CodexAppControlBootstrapTarget = + | { kind: 'endpoint'; socketPath: string } + | { kind: 'locator'; locatorPath: string }; + +/** Create a fresh asymmetric candidate and one O_EXCL private bootstrap. */ +export function createCodexAppControlBootstrap( + directory: string, + sessionId: string, + target: string | CodexAppControlBootstrapTarget = codexAppControlSocketPath(directory, sessionId), + platform: NodeJS.Platform = process.platform, +): CodexAppControlBootstrap { + ensureCodexAppControlDirectory(directory, platform); + const resolvedTarget: CodexAppControlBootstrapTarget = typeof target === 'string' + ? { kind: 'endpoint', socketPath: target } + : target; + if (resolvedTarget.kind === 'endpoint' + && !isAbsoluteForPlatform(resolvedTarget.socketPath, platform)) { + throw new Error('Codex App control socket path must be absolute'); + } + if (resolvedTarget.kind === 'locator' + && !isAbsoluteForPlatform(resolvedTarget.locatorPath, platform)) { + throw new Error('Codex App control locator path must be absolute'); + } + const generation = randomBytes(32).toString('base64url'); + const pair = generateKeyPairSync('ed25519'); + const publicKey = pair.publicKey.export({ format: 'der', type: 'spki' }).toString('base64url'); + const privateKey = pair.privateKey.export({ format: 'der', type: 'pkcs8' }).toString('base64url'); + const identity: CodexAppControlIdentity = { generation, publicKey, createdAtMs: Date.now() }; + const key = sessionKey(sessionId).slice(0, 16); + const path = joinForPlatform(platform, directory, `${key}.${randomBytes(16).toString('hex')}.bootstrap`); + createExclusiveFile( + path, + JSON.stringify({ + version: CONTROL_BOOTSTRAP_VERSION, + sessionId, + generation, + privateKey, + ...(resolvedTarget.kind === 'endpoint' + ? { socketPath: resolvedTarget.socketPath } + : { locatorPath: resolvedTarget.locatorPath }), + }), + 'Codex App control bootstrap', + platform, + ); + return { path, identity }; +} + +/** Remove crash-orphaned one-shot files for exactly one session generation. */ +export function cleanupStaleCodexAppControlBootstraps( + directory: string, + sessionId: string, + platform: NodeJS.Platform = process.platform, +): void { + ensureCodexAppControlDirectory(directory, platform); + const key = sessionKey(sessionId).slice(0, 16); + const pattern = new RegExp(`^${key}\\.[a-f0-9]{32}\\.bootstrap$`); + for (const name of readdirSync(directory)) { + if (!pattern.test(name)) continue; + try { unlinkSync(joinForPlatform(platform, directory, name)); } catch { /* raced with runner consume */ } + } +} + +/** + * One-shot consume on one O_NOFOLLOW fd. The file is unlinked before its bytes + * are read and the post-unlink link count must be zero. The private key is + * imported here so callers do not retain or forward its encoded form. + */ +export function consumeCodexAppControlBootstrap( + path: string, + expectedSessionId?: string, + platform: NodeJS.Platform = process.platform, +): ConsumedCodexAppControlBootstrap { + if (!path || !isAbsoluteForPlatform(path, platform)) { + throw new Error('Codex App control bootstrap path must be absolute'); + } + let fd: number | undefined; + try { + const policy = codexAppControlFilesystemPolicy(platform); + const beforePath = lstatSync(path); + assertRegularSingleLinkWithinLimit( + beforePath, + BOOTSTRAP_MAX_BYTES, + 'Codex App control bootstrap', + platform, + ); + fd = openSync(path, fsConstants.O_RDONLY | noFollowFlag(platform)); + const before = fstatSync(fd); + assertRegularSingleLinkWithinLimit( + before, + BOOTSTRAP_MAX_BYTES, + 'Codex App control bootstrap', + platform, + ); + if (!sameFileIdentity(beforePath, before)) { + throw new Error('Codex App control bootstrap changed between lstat and open'); + } + unlinkSync(path); + const after = fstatSync(fd); + if (policy.verifyPostUnlinkLinkCount && after.nlink !== 0) { + throw new Error('Codex App control bootstrap was not consumed by unlink'); + } + const decoded = JSON.parse(readFileSync(fd, 'utf8')) as Record; + const hasSocketPath = typeof decoded.socketPath === 'string' + && isAbsoluteForPlatform(decoded.socketPath, platform); + const hasLocatorPath = typeof decoded.locatorPath === 'string' + && isAbsoluteForPlatform(decoded.locatorPath, platform); + const posixLocatorPathValid = platform === 'win32' || !hasLocatorPath + || decoded.locatorPath === codexAppControlLocatorPath( + codexAppPosixControlRoot(), + typeof decoded.sessionId === 'string' ? decoded.sessionId : '', + platform, + ); + if (decoded.version !== CONTROL_BOOTSTRAP_VERSION + || typeof decoded.sessionId !== 'string' + || (expectedSessionId !== undefined && decoded.sessionId !== expectedSessionId) + || !isValidGeneration(decoded.generation) + || typeof decoded.privateKey !== 'string' || !KEY_RE.test(decoded.privateKey) + || hasSocketPath === hasLocatorPath + // Windows always follows its protected locator. Current POSIX workers + // also emit locators; a legacy direct socket bootstrap remains accepted + // because the read-once bootstrap itself is the launch capability. Any + // POSIX locator is nevertheless pinned to the fixed per-UID root here. + || (platform === 'win32' && !hasLocatorPath) + || !posixLocatorPathValid) { + throw new Error('Codex App control bootstrap is invalid'); + } + const privateKey = importPrivateKey(decoded.privateKey); + if (privateKey.asymmetricKeyType !== 'ed25519') throw new Error('Codex App control private key is invalid'); + return { + generation: decoded.generation, + privateKey, + ...(hasSocketPath ? { socketPath: decoded.socketPath as string } : {}), + ...(hasLocatorPath ? { locatorPath: decoded.locatorPath as string } : {}), + }; + } catch (err) { + try { unlinkSync(path); } catch { /* remove a rejected bootstrap/symlink */ } + throw err; + } finally { + if (fd !== undefined) closeSync(fd); + } +} + +export function mergeCodexAppControlCandidate( + existing: CodexAppControlState | undefined, + candidate: CodexAppControlIdentity, + nowMs = Date.now(), +): CodexAppControlState { + if (!validIdentity(candidate)) { + throw new Error('Codex App control candidate is invalid'); + } + const identities = [candidate, ...(existing?.identities ?? [])] + .filter((identity, index, all) => all.findIndex(item => item.generation === identity.generation) === index) + .slice(0, MAX_STATE_IDENTITIES); + return { + version: CONTROL_STATE_VERSION, + status: 'pending', + identities, + updatedAtMs: nowMs, + }; +} + +export function activateCodexAppControlIdentity( + state: CodexAppControlState, + generation: string, + nowMs = Date.now(), +): CodexAppControlState { + const identity = state.identities.find(candidate => candidate.generation === generation); + if (!identity) throw new Error('Codex App control generation is not a persisted candidate'); + return { + version: CONTROL_STATE_VERSION, + status: 'active', + identities: [identity], + updatedAtMs: nowMs, + activatedAtMs: nowMs, + }; +} + +export function generateCodexAppControlChallenge(): string { + return randomBytes(32).toString('base64url'); +} + +function canonicalJson(value: unknown): string { + if (value === null || typeof value === 'string' || typeof value === 'boolean') return JSON.stringify(value); + if (typeof value === 'number') { + if (!Number.isFinite(value)) throw new Error('non-finite control payload number'); + return JSON.stringify(value); + } + if (Array.isArray(value)) return `[${value.map(canonicalJson).join(',')}]`; + if (value && typeof value === 'object') { + const record = value as Record; + return `{${Object.keys(record).sort().map(key => ( + `${JSON.stringify(key)}:${canonicalJson(record[key])}` + )).join(',')}}`; + } + throw new Error('unsupported control payload value'); +} + +function signingBytes(domain: 'auth' | 'marker', value: Record): Buffer { + return Buffer.from(`botmux/codex-app/control/${domain}/v2\0${canonicalJson(value)}`, 'utf8'); +} + +function signature(privateKey: KeyObject, domain: 'auth' | 'marker', value: Record): string { + if (privateKey.type !== 'private' || privateKey.asymmetricKeyType !== 'ed25519') { + throw new Error('Codex App control signer must be an Ed25519 private key'); + } + return sign(null, signingBytes(domain, value), privateKey).toString('base64url'); +} + +function signatureValid( + publicKey: string, + signatureValue: string, + domain: 'auth' | 'marker', + value: Record, +): boolean { + if (!KEY_RE.test(publicKey) || !SIGNATURE_RE.test(signatureValue)) return false; + try { + return verify( + null, + signingBytes(domain, value), + importPublicKey(publicKey), + Buffer.from(signatureValue, 'base64url'), + ); + } catch { + return false; + } +} + +export interface CodexAppControlChallenge { + version: typeof CONTROL_WIRE_VERSION; + type: 'challenge'; + sessionId: string; + challenge: string; +} + +export interface CodexAppControlAuth { + version: typeof CONTROL_WIRE_VERSION; + type: 'auth'; + sessionId: string; + generation: string; + challenge: string; + signature: string; +} + +export interface CodexAppControlAccepted { + version: typeof CONTROL_WIRE_VERSION; + type: 'accepted'; + sessionId: string; + generation: string; + challenge: string; + endpointEpoch?: string; +} + +export interface CodexAppControlAck { + version: typeof CONTROL_WIRE_VERSION; + type: 'ack'; + sessionId: string; + generation: string; + challenge: string; + seq: number; +} + +export interface CodexAppSignedControlMarker { + version: typeof CONTROL_WIRE_VERSION; + type: 'marker'; + sessionId: string; + generation: string; + challenge: string; + seq: number; + kind: string; + payload: Record; + signature: string; +} + +export type CodexAppControlWireRecord = + | CodexAppControlChallenge + | CodexAppControlAuth + | CodexAppControlAccepted + | CodexAppControlAck + | CodexAppSignedControlMarker; + +export function encodeCodexAppControlChallenge(sessionId: string, challenge: string): string { + if (!sessionId || !isValidChallenge(challenge)) throw new Error('Codex App control challenge is invalid'); + return JSON.stringify({ version: CONTROL_WIRE_VERSION, type: 'challenge', sessionId, challenge }); +} + +export function encodeCodexAppControlAccepted( + sessionId: string, + generation: string, + challenge: string, + endpointEpoch?: string, +): string { + if (!sessionId || !isValidGeneration(generation) || !isValidChallenge(challenge)) { + throw new Error('Codex App control acceptance metadata is invalid'); + } + if (endpointEpoch !== undefined && !isValidGeneration(endpointEpoch)) { + throw new Error('Codex App control endpoint epoch is invalid'); + } + return JSON.stringify({ + version: CONTROL_WIRE_VERSION, + type: 'accepted', + sessionId, + generation, + challenge, + ...(endpointEpoch ? { endpointEpoch } : {}), + }); +} + +export function encodeCodexAppControlAck( + sessionId: string, + generation: string, + challenge: string, + seq: number, +): string { + if (!sessionId || !isValidGeneration(generation) || !isValidChallenge(challenge) + || !Number.isSafeInteger(seq) || seq <= 0) { + throw new Error('Codex App control acknowledgement metadata is invalid'); + } + return JSON.stringify({ + version: CONTROL_WIRE_VERSION, + type: 'ack', + sessionId, + generation, + challenge, + seq, + }); +} + +function authUnsigned(sessionId: string, generation: string, challenge: string): Record { + return { sessionId, generation, challenge }; +} + +export function encodeCodexAppControlAuth( + privateKey: KeyObject, + sessionId: string, + generation: string, + challenge: string, +): string { + if (!sessionId || !isValidGeneration(generation) || !isValidChallenge(challenge)) { + throw new Error('Codex App control authentication metadata is invalid'); + } + const unsigned = authUnsigned(sessionId, generation, challenge); + return JSON.stringify({ + version: CONTROL_WIRE_VERSION, + type: 'auth', + ...unsigned, + signature: signature(privateKey, 'auth', unsigned), + }); +} + +function markerUnsigned( + sessionId: string, + generation: string, + challenge: string, + seq: number, + kind: string, + payload: Record, +): Record { + return { sessionId, generation, challenge, seq, kind, payload }; +} + +export function encodeCodexAppSignedControlMarker( + privateKey: KeyObject, + sessionId: string, + generation: string, + challenge: string, + seq: number, + kind: string, + payload: Record, +): string { + if (!sessionId || !isValidGeneration(generation) || !isValidChallenge(challenge) + || !Number.isSafeInteger(seq) || seq <= 0 || !kind) { + throw new Error('Codex App control marker metadata is invalid'); + } + const unsigned = markerUnsigned(sessionId, generation, challenge, seq, kind, payload); + return JSON.stringify({ + version: CONTROL_WIRE_VERSION, + type: 'marker', + ...unsigned, + signature: signature(privateKey, 'marker', unsigned), + }); +} + +export function parseCodexAppControlWireRecord(line: string): CodexAppControlWireRecord | undefined { + try { + const value = JSON.parse(line) as Record; + if (value.version !== CONTROL_WIRE_VERSION || typeof value.sessionId !== 'string' || !value.sessionId) return undefined; + if (value.type === 'challenge' && isValidChallenge(value.challenge)) { + return value as unknown as CodexAppControlChallenge; + } + if (value.type === 'auth' && isValidGeneration(value.generation) + && isValidChallenge(value.challenge) + && typeof value.signature === 'string' && SIGNATURE_RE.test(value.signature)) { + return value as unknown as CodexAppControlAuth; + } + if (value.type === 'accepted' && isValidGeneration(value.generation) + && isValidChallenge(value.challenge) + && (value.endpointEpoch === undefined || isValidGeneration(value.endpointEpoch))) { + return value as unknown as CodexAppControlAccepted; + } + if (value.type === 'ack' && isValidGeneration(value.generation) + && isValidChallenge(value.challenge) + && Number.isSafeInteger(value.seq) && Number(value.seq) > 0) { + return value as unknown as CodexAppControlAck; + } + if (value.type === 'marker' && isValidGeneration(value.generation) + && isValidChallenge(value.challenge) + && Number.isSafeInteger(value.seq) && Number(value.seq) > 0 + && typeof value.kind === 'string' && value.kind.length > 0 + && value.payload && typeof value.payload === 'object' && !Array.isArray(value.payload) + && typeof value.signature === 'string' && SIGNATURE_RE.test(value.signature)) { + return value as unknown as CodexAppSignedControlMarker; + } + return undefined; + } catch { + return undefined; + } +} + +export type CodexAppControlRunnerHandshakeAction = + | { type: 'authenticate'; challenge: string } + | { type: 'accepted'; challenge: string } + | { type: 'ack'; seq: number } + | { type: 'reject' }; + +/** + * Pure runner-side handshake state machine. Keeping the phase checks here + * makes repeated challenges, wrong locator epochs, and out-of-order ACKs + * executable in unit tests instead of relying on source-string assertions. + */ +export class CodexAppControlRunnerHandshake { + private phase: 'challenge' | 'accepted' | 'active' = 'challenge'; + private challengeValue: string | undefined; + + constructor( + private readonly expectedSessionId: string, + private readonly expectedGeneration: string, + private readonly expectedEndpointEpoch?: string, + ) {} + + handle( + record: CodexAppControlWireRecord | undefined, + sentThrough: number, + ): CodexAppControlRunnerHandshakeAction { + if (!record || record.sessionId !== this.expectedSessionId) return { type: 'reject' }; + if (this.phase === 'challenge' && record.type === 'challenge') { + this.challengeValue = record.challenge; + this.phase = 'accepted'; + return { type: 'authenticate', challenge: record.challenge }; + } + if (this.phase === 'accepted' + && record.type === 'accepted' + && record.generation === this.expectedGeneration + && record.challenge === this.challengeValue + && (this.expectedEndpointEpoch === undefined + || record.endpointEpoch === this.expectedEndpointEpoch)) { + this.phase = 'active'; + return { type: 'accepted', challenge: record.challenge }; + } + if (this.phase === 'active' + && record.type === 'ack' + && record.generation === this.expectedGeneration + && record.challenge === this.challengeValue + && record.seq > 0 + && record.seq <= sentThrough) { + return { type: 'ack', seq: record.seq }; + } + return { type: 'reject' }; + } + + get active(): boolean { + return this.phase === 'active'; + } +} + +export function verifyCodexAppControlAuth(auth: CodexAppControlAuth, publicKey: string): boolean { + return signatureValid( + publicKey, + auth.signature, + 'auth', + authUnsigned(auth.sessionId, auth.generation, auth.challenge), + ); +} + +export function authenticateCodexAppControlCandidate(input: { + state: CodexAppControlState | undefined; + auth: CodexAppControlAuth; + sessionId: string; + challenge: string; +}): CodexAppControlIdentity | undefined { + if (!input.state + || input.auth.sessionId !== input.sessionId + || input.auth.challenge !== input.challenge) return undefined; + const identity = input.state.identities.find( + candidate => candidate.generation === input.auth.generation, + ); + return identity && verifyCodexAppControlAuth(input.auth, identity.publicKey) + ? identity + : undefined; +} + +export function verifyCodexAppSignedControlMarker( + marker: CodexAppSignedControlMarker, + publicKey: string, +): boolean { + return signatureValid( + publicKey, + marker.signature, + 'marker', + markerUnsigned( + marker.sessionId, + marker.generation, + marker.challenge, + marker.seq, + marker.kind, + marker.payload, + ), + ); +} + +/** + * Marker sequences may begin above one after a worker replacement, but every + * record on one authenticated connection must then be contiguous. This keeps + * a skipped final fragment from being hidden by a later cumulative ACK. + */ +export class CodexAppControlSequenceFence { + private previous: number | undefined; + + accept(seq: number): boolean { + if (!Number.isSafeInteger(seq) || seq <= 0) return false; + if (this.previous !== undefined && seq !== this.previous + 1) return false; + this.previous = seq; + return true; + } +} + +export type CodexAppFinalAssemblyResult = + | { status: 'not-final' } + | { status: 'accepted' } + | { status: 'complete'; payload: Record } + | { status: 'reject'; reason: string }; + +interface CodexAppFinalAssemblyState { + id: string; + total: number; + metadata: Record; + chunks: Buffer[]; + bytes: number; +} + +const FINAL_ID_MAX_BYTES = 512; +const STRICT_BASE64_RE = /^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/; + +/** + * Per-connection final transaction assembler. Start and chunks deliberately + * remain unacknowledged; only a valid, complete end record is eligible for the + * cumulative ACK. Any gap or interleaving rejects the connection, so the + * runner must re-sign and replay the complete transaction under a new + * challenge. + */ +export class CodexAppControlFinalAssembler { + private active: CodexAppFinalAssemblyState | undefined; + + accept(kind: string, payload: Record): CodexAppFinalAssemblyResult { + const isTransactionKind = kind === 'final-start' || kind === 'final-chunk' || kind === 'final-end'; + if (!isTransactionKind) { + if (kind === 'final' || kind.startsWith('final-')) { + return this.reject(`unsupported final record ${kind}`); + } + return this.active + ? this.reject(`interleaved ${kind} record inside final transaction`) + : { status: 'not-final' }; + } + + const id = typeof payload.id === 'string' ? payload.id : ''; + if (!id || Buffer.byteLength(id, 'utf8') > FINAL_ID_MAX_BYTES) { + return this.reject('invalid final transaction id'); + } + + if (kind === 'final-start') { + const total = typeof payload.total === 'number' && Number.isSafeInteger(payload.total) + ? payload.total + : -1; + if (this.active || total < 0 || total > 2_048) { + return this.reject('invalid or overlapping final-start'); + } + const { id: _id, total: _total, ...metadata } = payload; + this.active = { id, total, metadata, chunks: [], bytes: 0 }; + return { status: 'accepted' }; + } + + const assembly = this.active; + if (!assembly || id !== assembly.id) { + return this.reject('final fragment has no matching start'); + } + + if (kind === 'final-chunk') { + const index = typeof payload.index === 'number' && Number.isSafeInteger(payload.index) + ? payload.index + : -1; + if (index !== assembly.chunks.length || index >= assembly.total + || typeof payload.data !== 'string' || !STRICT_BASE64_RE.test(payload.data)) { + return this.reject('invalid, duplicate, or out-of-order final chunk'); + } + const chunk = Buffer.from(payload.data, 'base64'); + if (chunk.length === 0 || chunk.length > CODEX_APP_CONTROL_FINAL_CHUNK_BYTES) { + return this.reject('final chunk size is invalid'); + } + assembly.bytes += chunk.length; + if (assembly.bytes > CODEX_APP_CONTROL_FINAL_MAX_BYTES) { + return this.reject('final transaction exceeds the byte limit'); + } + assembly.chunks.push(chunk); + return { status: 'accepted' }; + } + + const endTotal = typeof payload.total === 'number' && Number.isSafeInteger(payload.total) + ? payload.total + : -1; + if (endTotal !== assembly.total || assembly.chunks.length !== assembly.total) { + return this.reject('incomplete or inconsistent final-end'); + } + this.active = undefined; + return { + status: 'complete', + payload: { + ...assembly.metadata, + content: Buffer.concat(assembly.chunks, assembly.bytes).toString('utf8'), + }, + }; + } + + clear(): void { + this.active = undefined; + } + + private reject(reason: string): CodexAppFinalAssemblyResult { + this.active = undefined; + return { status: 'reject', reason }; + } +} + +/** + * Per-worker replay window for authenticated runner generations. A runner may + * reconnect after losing an ACK and legitimately re-sign the same sequence + * under the new connection challenge; the worker ACKs that retry without + * applying its lifecycle/final side effects twice. + */ +export class CodexAppControlReplayWindow { + private readonly highWaterByGeneration = new Map(); + + highWater(generation: string): number { + return this.highWaterByGeneration.get(generation) ?? 0; + } + + hasSeen(generation: string, seq: number): boolean { + return seq <= this.highWater(generation); + } + + commit(generation: string, seq: number): void { + if (!isValidGeneration(generation) || !Number.isSafeInteger(seq) || seq <= 0) { + throw new Error('Codex App control replay sequence is invalid'); + } + if (seq > this.highWater(generation)) this.highWaterByGeneration.set(generation, seq); + } + + retainOnly(generation: string): void { + const retained = this.highWaterByGeneration.get(generation); + this.highWaterByGeneration.clear(); + if (retained !== undefined) this.highWaterByGeneration.set(generation, retained); + } +} + +/** Bounded newline decoder; oversized attacker input is discarded until resync. */ +export class CodexAppControlLineDecoder { + private pending = Buffer.alloc(0); + private droppingOversized = false; + + push(chunk: Buffer): { lines: string[]; droppedMalformed: boolean } { + const lines: string[] = []; + let droppedMalformed = false; + let cursor = 0; + while (cursor < chunk.length) { + const nl = chunk.indexOf(0x0a, cursor); + const end = nl >= 0 ? nl : chunk.length; + const piece = chunk.subarray(cursor, end); + cursor = nl >= 0 ? nl + 1 : chunk.length; + + if (this.droppingOversized) { + droppedMalformed = true; + if (nl >= 0) this.droppingOversized = false; + continue; + } + if (this.pending.length + piece.length > CODEX_APP_CONTROL_LINE_MAX_BYTES) { + this.pending = Buffer.alloc(0); + droppedMalformed = true; + if (nl < 0) this.droppingOversized = true; + continue; + } + this.pending = Buffer.concat([this.pending, piece]); + if (nl >= 0) { + const line = this.pending.toString('utf8').trim(); + this.pending = Buffer.alloc(0); + if (line) lines.push(line); + } + } + return { lines, droppedMalformed }; + } + + clear(): void { + this.pending = Buffer.alloc(0); + this.droppingOversized = false; + } +} diff --git a/src/utils/codex-app-dispatch-ledger.ts b/src/utils/codex-app-dispatch-ledger.ts new file mode 100644 index 000000000..2a248f528 --- /dev/null +++ b/src/utils/codex-app-dispatch-ledger.ts @@ -0,0 +1,207 @@ +import type { + CodexAppDispatchLedgerEntry, + CodexAppGenerationCommit, +} from '../types.js'; + +export type CodexAppDispatchIdentity = Pick< + CodexAppDispatchLedgerEntry, + 'dispatchId' | 'turnId' | 'dispatchAttempt' +>; + +function sameIdentity( + left: CodexAppDispatchIdentity, + right: CodexAppDispatchIdentity, +): boolean { + return left.dispatchId === right.dispatchId + && left.turnId === right.turnId + && left.dispatchAttempt === right.dispatchAttempt; +} + +/** Any ledger entry is daemon-owned unfinished work. Lifecycle mutations that + * can replace a worker, pane, cwd, or chat route must reject while this is true; + * explicit session close is the sole abandon operation. */ +export function hasUnsettledCodexAppDispatch( + ledger: readonly CodexAppDispatchLedgerEntry[] | undefined, +): boolean { + return (ledger?.length ?? 0) > 0; +} + +/** + * Authorize a worker-hosted `botmux send` relay against the immutable Codex + * App origin. `requireExact` is true for a live Codex App turn even when the + * ledger has just become empty: settlement winning before watcher admission + * must reject, never downgrade the old capability into an ordinary send. + */ +export function validateCodexAppManagedSendOrigin( + ledger: readonly CodexAppDispatchLedgerEntry[] | undefined, + origin: { turnId?: string; dispatchAttempt?: number }, + requireExact: boolean, +): { ok: true; requiresLedger: boolean } | { ok: false; error: string } { + const entries = ledger ?? []; + if (!requireExact && entries.length === 0) { + return { ok: true, requiresLedger: false }; + } + if (!origin.turnId) { + return { ok: false, error: 'unsettled Codex App output has no live turn identity' }; + } + const sameTurn = entries.filter(entry => entry.turnId === origin.turnId); + const exact = origin.dispatchAttempt === undefined + ? sameTurn + : sameTurn.filter(entry => entry.dispatchAttempt === origin.dispatchAttempt); + if (exact.length !== 1) { + return { + ok: false, + error: `${exact.length} Codex App ledger entries match the live relay origin`, + }; + } + const sink = exact[0]!.deliverySink ?? 'lark'; + if (sink === 'http_wait' || sink === 'http_async' || sink === 'suppressed') { + return { ok: false, error: `Codex App output is bound to ${sink}` }; + } + return { ok: true, requiresLedger: true }; +} + +export function appendAcceptedCodexAppDispatch( + ledger: readonly CodexAppDispatchLedgerEntry[], + entry: Omit, +): CodexAppDispatchLedgerEntry[] { + if (!entry.dispatchId || !entry.turnId) throw new Error('Codex App dispatch identity is incomplete'); + if (ledger.some(candidate => candidate.dispatchId === entry.dispatchId)) { + throw new Error('Codex App dispatch id is already present'); + } + return [...ledger, { ...entry, state: 'accepted' }]; +} + +export function prepareCodexAppDispatch( + ledger: readonly CodexAppDispatchLedgerEntry[], + identity: CodexAppDispatchIdentity, +): { ok: true; ledger: CodexAppDispatchLedgerEntry[] } | { ok: false; error: string } { + const index = ledger.findIndex(entry => entry.dispatchId === identity.dispatchId); + if (index < 0 || !sameIdentity(ledger[index], identity)) { + return { ok: false, error: 'dispatch_not_found' }; + } + if (ledger.slice(0, index).some(entry => entry.state !== 'prepared')) { + return { ok: false, error: 'dispatch_out_of_order' }; + } + if (ledger[index].state === 'prepared') return { ok: true, ledger: [...ledger] }; + const next = ledger.map((entry, candidateIndex) => candidateIndex === index + ? { ...entry, state: 'prepared' as const } + : entry); + return { ok: true, ledger: next }; +} + +/** A worker proved that a prepared write left the runner input untouched (or + * fully flushed invalid), so the exact item may return to accepted and retry in + * the same FIFO without minting a second dispatch. */ +export function retryPreparedCodexAppDispatch( + ledger: readonly CodexAppDispatchLedgerEntry[], + identity: CodexAppDispatchIdentity, +): { ok: true; ledger: CodexAppDispatchLedgerEntry[] } | { ok: false; error: string } { + const index = ledger.findIndex(entry => entry.dispatchId === identity.dispatchId); + if (index < 0 || !sameIdentity(ledger[index], identity)) { + return { ok: false, error: 'dispatch_not_found' }; + } + if (ledger[index].state !== 'prepared') { + return { ok: false, error: 'dispatch_not_prepared' }; + } + if (ledger.slice(index + 1).some(entry => entry.state === 'prepared')) { + return { ok: false, error: 'prepared_successor_exists' }; + } + return { + ok: true, + ledger: ledger.map((entry, candidateIndex) => candidateIndex === index + ? { ...entry, state: 'accepted' as const } + : entry), + }; +} + +export function cancelCodexAppDispatch( + ledger: readonly CodexAppDispatchLedgerEntry[], + identity: CodexAppDispatchIdentity, +): { ok: true; ledger: CodexAppDispatchLedgerEntry[] } | { ok: false; error: string } { + const index = ledger.findIndex(entry => entry.dispatchId === identity.dispatchId); + if (index < 0 || !sameIdentity(ledger[index], identity)) { + return { ok: false, error: 'dispatch_not_found' }; + } + if (ledger.slice(index + 1).some(entry => entry.state === 'prepared')) { + return { ok: false, error: 'prepared_successor_exists' }; + } + return { ok: true, ledger: ledger.filter((_, candidateIndex) => candidateIndex !== index) }; +} + +/** + * Remove one exact VC delivery attempt after its owned CLI backing has been + * authoritatively proved absent. Unlike ordinary cancellation, this does not + * reject a prepared successor: the dead generation can no longer execute the + * fenced attempt, and retaining it would make a replacement restore N beside + * the delivery hub's replayed N+1. + * + * No match is idempotent success because the worker may have durably retired + * the entry before it exited. More than one match is corruption and remains + * fail-closed because the receipt identity does not contain a dispatchId. + */ +export function retireCodexAppDispatchAfterBackingMissing( + ledger: readonly CodexAppDispatchLedgerEntry[], + turnId: string, + dispatchAttempt: number, +): { ok: true; ledger: CodexAppDispatchLedgerEntry[] } | { ok: false; error: string } { + const sameTurnIndexes = ledger.flatMap((entry, index) => entry.turnId === turnId ? [index] : []); + if (sameTurnIndexes.length === 0) return { ok: true, ledger: [...ledger] }; + if (sameTurnIndexes.length !== 1) return { ok: false, error: 'dispatch_identity_ambiguous' }; + const matchIndex = sameTurnIndexes[0]!; + if (ledger[matchIndex]!.dispatchAttempt !== dispatchAttempt) { + return { ok: false, error: 'dispatch_attempt_conflict' }; + } + return { + ok: true, + ledger: ledger.filter((_, index) => index !== matchIndex), + }; +} + +export function settleCodexAppDispatch( + ledger: readonly CodexAppDispatchLedgerEntry[], + commits: readonly CodexAppGenerationCommit[], + identity: CodexAppDispatchIdentity, + generation: string, + seq: number, +): { + ok: true; + ledger: CodexAppDispatchLedgerEntry[]; + commits: CodexAppGenerationCommit[]; + settledEntry: CodexAppDispatchLedgerEntry; + } | { ok: false; error: string } { + const head = ledger[0]; + if (!head || head.state !== 'prepared' || !sameIdentity(head, identity)) { + return { ok: false, error: 'prepared_head_mismatch' }; + } + if (!generation || !Number.isSafeInteger(seq) || seq <= 0) { + return { ok: false, error: 'invalid_control_identity' }; + } + const existing = commits.find(commit => commit.generation === generation)?.committedThrough ?? 0; + const nextCommit = { generation, committedThrough: Math.max(existing, seq) }; + return { + ok: true, + ledger: ledger.slice(1), + settledEntry: { ...head }, + commits: [ + ...commits.filter(commit => commit.generation !== generation), + nextCommit, + ], + }; +} + +export function committedCodexAppSequence( + commits: readonly CodexAppGenerationCommit[], + generation: string, + seq: number, +): boolean { + return commits.some(commit => commit.generation === generation && seq <= commit.committedThrough); +} + +/** A proved fresh runner retires every prior generation and its ACK history. */ +export function retainFreshCodexAppGeneration( + commits: readonly CodexAppGenerationCommit[], + generation: string, +): CodexAppGenerationCommit[] { + return commits.filter(commit => commit.generation === generation); +} diff --git a/src/utils/codex-app-turn-dispatch.ts b/src/utils/codex-app-turn-dispatch.ts new file mode 100644 index 000000000..d8def7803 --- /dev/null +++ b/src/utils/codex-app-turn-dispatch.ts @@ -0,0 +1,196 @@ +/** + * Worker-owned attribution for Codex App's serial runner queue. + * + * The runner can echo a stable clientUserMessageId, but it must never choose + * the daemon delivery attempt. A worker may also finish writing turn N+1 + * before turn N's final transaction arrives, so the mutable "current turn" + * globals are not an attribution boundary. Reserve one immutable entry per + * control-line write and settle final transactions strictly from the head. + */ + +export interface CodexAppTurnDispatchReservation { + handle: number; + dispatchId?: string; + turnId: string; + replyTurnId?: string; + dispatchAttempt?: number; + recovered?: boolean; +} + +export type CodexAppTurnDispatchSettlement = + | { + ok: true; + handle: number; + dispatchId?: string; + turnId: string; + replyTurnId?: string; + dispatchAttempt?: number; + nativeTurnId?: string; + remaining: number; + } + | { + ok: false; + reason: 'no_pending_turn' | 'turn_mismatch' | 'dispatch_attempt_mismatch'; + markerTurnId?: string; + expectedTurnId?: string; + markerDispatchAttempt?: unknown; + expectedDispatchAttempt?: number; + }; + +export class CodexAppTurnDispatchQueue { + private readonly queue: CodexAppTurnDispatchReservation[] = []; + private nextHandle = 1; + + reserve( + turnId: string, + dispatchAttempt?: number, + dispatchId?: string, + recovered = false, + replyTurnId?: string, + ): CodexAppTurnDispatchReservation { + if (!turnId) throw new Error('Codex App dispatch turn id must be non-empty'); + const reservation = { + handle: this.nextHandle++, + ...(dispatchId ? { dispatchId } : {}), + turnId, + ...(replyTurnId ? { replyTurnId } : {}), + ...(dispatchAttempt !== undefined ? { dispatchAttempt } : {}), + ...(recovered ? { recovered: true } : {}), + }; + this.queue.push(reservation); + return { ...reservation }; + } + + /** + * A replacement worker cannot recover the old in-memory queue. It may + * recover exactly the daemon-frozen active identity supplied in `init`, and + * only while no locally submitted entry exists. A runner marker still has + * to assert equality before this entry can settle. + */ + recoverWarmReattach( + turnId: string | undefined, + dispatchAttempt?: number, + dispatchId?: string, + replyTurnId?: string, + ): CodexAppTurnDispatchReservation | undefined { + if (!turnId || this.queue.length > 0) return undefined; + return this.reserve(turnId, dispatchAttempt, dispatchId, true, replyTurnId); + } + + restore(entries: ReadonlyArray<{ + dispatchId: string; + turnId: string; + dispatchAttempt?: number; + replyTurnId?: string; + }>): void { + if (this.queue.length > 0) throw new Error('Codex App dispatch queue is already populated'); + for (const entry of entries) { + this.reserve(entry.turnId, entry.dispatchAttempt, entry.dispatchId, true, entry.replyTurnId); + } + } + + recoveredPrefix(): CodexAppTurnDispatchReservation[] { + const prefix: CodexAppTurnDispatchReservation[] = []; + for (const entry of this.queue) { + if (!entry.recovered) break; + prefix.push({ ...entry }); + } + return prefix; + } + + /** Remove the exact write that threw or reported submitted=false. */ + cancelExact(handle: number): boolean { + const index = this.queue.findIndex(entry => entry.handle === handle); + if (index < 0) return false; + this.queue.splice(index, 1); + return true; + } + + markRecovered(handle: number): boolean { + const entry = this.queue.find(candidate => candidate.handle === handle); + if (!entry) return false; + entry.recovered = true; + return true; + } + + findExact( + turnId: string, + dispatchAttempt?: number, + ): CodexAppTurnDispatchReservation | undefined { + const entry = this.queue.find(candidate => candidate.turnId === turnId + && candidate.dispatchAttempt === dispatchAttempt); + return entry ? { ...entry } : undefined; + } + + /** + * Validate and consume one complete final transaction. The head remains in + * place on every rejection, so a stale/mismatched marker cannot steal the + * next turn or advance the FIFO. + */ + settleFinal(payload: { + turnId?: unknown; + nativeTurnId?: unknown; + dispatchAttempt?: unknown; + }, consume = true): CodexAppTurnDispatchSettlement { + const head = this.queue[0]; + if (!head) return { ok: false, reason: 'no_pending_turn' }; + + const markerTurnId = typeof payload.turnId === 'string' && payload.turnId.length > 0 + ? payload.turnId + : undefined; + if (markerTurnId && markerTurnId !== head.turnId) { + return { + ok: false, + reason: 'turn_mismatch', + markerTurnId, + expectedTurnId: head.turnId, + }; + } + + // Attempt identity is worker-owned. A runner may redundantly assert it, + // but an assertion of any other value (including a malformed one) is a + // rejection and can never select another queue entry. + if (payload.dispatchAttempt !== undefined + && payload.dispatchAttempt !== head.dispatchAttempt) { + return { + ok: false, + reason: 'dispatch_attempt_mismatch', + markerDispatchAttempt: payload.dispatchAttempt, + ...(head.dispatchAttempt !== undefined + ? { expectedDispatchAttempt: head.dispatchAttempt } + : {}), + }; + } + + const nativeTurnId = typeof payload.nativeTurnId === 'string' && payload.nativeTurnId.length > 0 + ? payload.nativeTurnId + : undefined; + if (consume) this.queue.shift(); + return { + ok: true, + handle: head.handle, + ...(head.dispatchId ? { dispatchId: head.dispatchId } : {}), + turnId: head.turnId, + ...(head.replyTurnId ? { replyTurnId: head.replyTurnId } : {}), + ...(head.dispatchAttempt !== undefined + ? { dispatchAttempt: head.dispatchAttempt } + : {}), + ...(nativeTurnId ? { nativeTurnId } : {}), + remaining: this.queue.length - (consume ? 0 : 1), + }; + } + + commitExactHead(handle: number): boolean { + if (this.queue[0]?.handle !== handle) return false; + this.queue.shift(); + return true; + } + + size(): number { + return this.queue.length; + } + + clear(): void { + this.queue.length = 0; + } +} diff --git a/src/utils/codex-app-turn-liveness.ts b/src/utils/codex-app-turn-liveness.ts new file mode 100644 index 000000000..0ca44ce82 --- /dev/null +++ b/src/utils/codex-app-turn-liveness.ts @@ -0,0 +1,364 @@ +import type { BackendType } from '../adapters/backend/types.js'; + +/** + * Codex App turn liveness, driven by the app-server runner's explicit turn + * activity markers. This deliberately reports "no observable progress" + * rather than failure: a long-running tool may recover and emit activity + * later, at which point the stalled projection clears without replaying work. + */ + +export const CODEX_APP_NO_PROGRESS_TIMEOUT_MS = 90_000; + +const PERSISTENT_BACKEND_TYPES: ReadonlySet = new Set(['tmux', 'herdr', 'zellij']); + +/** + * Decide whether an existing persistent pane needs a synthetic observation. + * This deliberately does not depend on pipe mode: Zellij reattaches through + * its own PTY (`isPipeMode=false`) but preserves the same running CLI. + */ +export function shouldBeginCodexAppReattachObservation(input: { + cliId?: string; + backendType: BackendType; + isReattach: boolean; +}): boolean { + return input.cliId === 'codex-app' + && input.isReattach + && PERSISTENT_BACKEND_TYPES.has(input.backendType); +} + +export interface CodexAppLivenessPoll { + active: boolean; + stalled: boolean; + /** True only on the working -> stalled edge. */ + newlyStalled: boolean; + /** True at most once for one submitted turn, even if it later recovers. */ + shouldNotify: boolean; + turnId?: string; +} + +export interface CodexAppActivityApplyResult { + accepted: boolean; + phase?: 'submitted' | 'progress' | 'completed'; + /** A previously rejected inter-turn prompt became authoritative. */ + shouldReplayPrompt?: boolean; +} + +export interface CodexAppStateApplyResult { + accepted: boolean; + busy?: boolean; + tracksTurn?: boolean; + /** Signed idle arrived after the tracker's explicit queue drained. */ + shouldPublishReady?: boolean; + atMs?: number; +} + +/** + * Signed runner state is the Codex App ready authority. Terminal prompt bytes + * remain useful as a recovery hint, but PTY/tmux/Herdr/Zellij delivery can be + * delayed or lost and must never publish idle ahead of the signed final queue. + */ +export class CodexAppReadyAuthority { + private signedIdle = false; + private latePromptRecoveryArmed = false; + + reset(): void { + this.signedIdle = false; + this.latePromptRecoveryArmed = false; + } + + beginWork(): void { + this.signedIdle = false; + this.latePromptRecoveryArmed = false; + } + + /** Returns true only for an authenticated runner's idle boundary. */ + noteSignedState(busy: boolean): boolean { + this.latePromptRecoveryArmed = false; + this.signedIdle = !busy; + return this.signedIdle; + } + + canPublishPromptReady(): boolean { + return this.signedIdle; + } + + /** Arm only after the worker cancels the exact local submit slot. */ + armLatePromptRecovery(): void { + this.signedIdle = false; + this.latePromptRecoveryArmed = true; + } + + /** + * Consume one late terminal prompt after the cancelled slot left no tracked + * work. New work, authenticated activity/state, or reset clears the arm. + */ + consumeLatePromptRecovery(trackerEmpty: boolean): boolean { + if (!trackerEmpty || !this.latePromptRecoveryArmed) return false; + this.latePromptRecoveryArmed = false; + return true; + } +} + +/** Apply a state payload from an already authenticated runner connection. */ +export function applyTrustedCodexAppStateMarker( + tracker: CodexAppTurnLiveness, + authority: CodexAppReadyAuthority, + payload: unknown, + receivedAtMs = Date.now(), +): CodexAppStateApplyResult { + if (!payload || typeof payload !== 'object') return { accepted: false }; + const marker = payload as Record; + if (typeof marker.busy !== 'boolean') return { accepted: false }; + const runnerAtMs = typeof marker.atMs === 'number' && Number.isFinite(marker.atMs) + ? Math.min(marker.atMs, receivedAtMs) + : receivedAtMs; + if (marker.busy) { + authority.noteSignedState(true); + // Goal auto-continuations are native work but do not own a Botmux dispatch + // slot. Tracking them as a synthetic user turn would leave an extra slot + // after a later Lark message steers the same native turn and completes. + const tracksTurn = marker.tracksTurn !== false; + if (tracksTurn) tracker.noteSubmitted(runnerAtMs); + else tracker.discardReattachObservation(); + return { accepted: true, busy: true, tracksTurn, shouldPublishReady: false, atMs: runnerAtMs }; + } + authority.noteSignedState(false); + return { + accepted: true, + busy: false, + shouldPublishReady: tracker.notePrompt(runnerAtMs), + atMs: runnerAtMs, + }; +} + +interface ActiveTurn { + handle: number; + turnId?: string; + lastActivityAtMs: number; + stalled: boolean; + notified: boolean; + reattachObservation: boolean; +} + +export class CodexAppTurnLiveness { + private readonly turns: ActiveTurn[] = []; + private nextHandle = 1; + private promptDeferred = false; + + constructor(private readonly timeoutMs = CODEX_APP_NO_PROGRESS_TIMEOUT_MS) { + if (!Number.isFinite(timeoutMs) || timeoutMs <= 0) { + throw new Error('Codex App liveness timeout must be a positive finite number'); + } + } + + /** + * Queue one Botmux input immediately before its control line is submitted. + * Codex App's runner is serial, so only the head turn owns activity/stall + * state; queued turns get a fresh clock when the head completes. + */ + begin(turnId?: string, nowMs = Date.now()): number { + // A prompt deferred for an earlier queue has no authority over a new queue. + if (this.turns.length === 0) this.promptDeferred = false; + const handle = this.nextHandle++; + this.turns.push({ + handle, + turnId, + lastActivityAtMs: nowMs, + stalled: false, + notified: false, + reattachObservation: false, + }); + return handle; + } + + /** Observe an authenticated runner whose in-memory turn state survived reattach. */ + beginReattachObservation(nowMs = Date.now()): number | undefined { + if (this.turns.length > 0) return undefined; + const handle = this.nextHandle++; + this.turns.push({ + handle, + lastActivityAtMs: nowMs, + stalled: false, + notified: false, + reattachObservation: true, + }); + return handle; + } + + /** Drop only the synthetic reconnect slot when signed state identifies work + * as a native Goal continuation rather than a Botmux-owned turn. */ + discardReattachObservation(): void { + if (!this.turns[0]?.reattachObservation) return; + this.turns.shift(); + this.promptDeferred = false; + } + + /** + * Record runner/app-server activity. Activity after a stall makes the turn + * working again, but the same turn will not notify the user a second time. + */ + noteActivity(nowMs = Date.now()): void { + const active = this.turns[0]; + if (!active) return; + // Real work for the head turn supersedes a prompt rendered in the narrow + // inter-turn gap before its chunked control line finished arriving. + this.promptDeferred = false; + // Signed socket records can be buffered behind other process work. Never + // let an older runner timestamp move the worker's clock backwards. + active.lastActivityAtMs = Math.max(active.lastActivityAtMs, nowMs); + active.stalled = false; + } + + /** + * A signed submitted record can be the first event after worker reattach. + * Recover an explicit slot when no local flush state survived the restart. + */ + noteSubmitted(nowMs = Date.now()): void { + if (this.turns.length === 0) this.begin(undefined, nowMs); + else if (this.turns[0].reattachObservation) this.turns[0].reattachObservation = false; + this.noteActivity(nowMs); + } + + /** Complete only the runner's current turn and activate the next queued one. */ + completeCurrent(nowMs = Date.now()): boolean { + if (this.turns.length === 0) return false; + this.turns.shift(); + const next = this.turns[0]; + if (!next) return this.consumeDeferredPrompt(); + // Time spent waiting behind the previous turn is not lack of progress for + // this turn. Its own timeout begins when it becomes runner-current. + next.lastActivityAtMs = Math.max(next.lastActivityAtMs, nowMs); + next.stalled = false; + return false; + } + + /** Remove the exact control-line submission that failed, preserving peers. */ + cancelExact(handle: number, nowMs = Date.now()): { + cancelled: boolean; + shouldReplayPrompt: boolean; + } { + const index = this.turns.findIndex(turn => turn.handle === handle); + if (index < 0) return { cancelled: false, shouldReplayPrompt: false }; + const wasCurrent = index === 0; + this.turns.splice(index, 1); + if (wasCurrent && this.turns[0]) { + this.turns[0].lastActivityAtMs = Math.max(this.turns[0].lastActivityAtMs, nowMs); + this.turns[0].stalled = false; + } + return { cancelled: true, shouldReplayPrompt: this.consumeDeferredPrompt() }; + } + + /** Backwards-compatible shorthand for callers that only need replay state. */ + cancel(handle: number, nowMs = Date.now()): boolean { + return this.cancelExact(handle, nowMs).shouldReplayPrompt; + } + + /** + * A prompt is authoritative only for the synthetic reattach observation. + * Returns whether no explicit/queued turn remains, so the worker can reject + * a transient inter-turn prompt instead of publishing prompt_ready / idle. + */ + notePrompt(nowMs = Date.now()): boolean { + if (this.turns[0]?.reattachObservation) this.completeCurrent(nowMs); + if (this.turns.length === 0) { + this.promptDeferred = false; + return true; + } + this.promptDeferred = true; + return false; + } + + hasActiveTurn(): boolean { + return this.turns.length > 0; + } + + /** Drop every queued turn on CLI exit, kill, or worker reinitialization. */ + clear(): void { + this.turns.length = 0; + this.promptDeferred = false; + } + + poll(nowMs = Date.now()): CodexAppLivenessPoll { + const active = this.turns[0]; + if (!active) return { active: false, stalled: false, newlyStalled: false, shouldNotify: false }; + + const timedOut = nowMs - active.lastActivityAtMs >= this.timeoutMs; + const newlyStalled = timedOut && !active.stalled; + if (newlyStalled) active.stalled = true; + + const shouldNotify = newlyStalled && !active.notified; + if (shouldNotify) active.notified = true; + + return { + active: true, + stalled: active.stalled, + newlyStalled, + shouldNotify, + turnId: active.turnId, + }; + } + + private consumeDeferredPrompt(): boolean { + if (this.turns.length > 0 || !this.promptDeferred) return false; + this.promptDeferred = false; + return true; + } +} + +/** + * One flush may cancel a queued liveness slot because writeInput throws or + * it returns submitted=false. Preserve the deferred real prompt across that + * async boundary, but replay it only after every peer slot has drained. + */ +export class CodexAppFlushPromptReplay { + private replayRequested = false; + + cancelSubmission( + tracker: CodexAppTurnLiveness, + authority: CodexAppReadyAuthority, + handle: number | undefined, + nowMs = Date.now(), + ): void { + if (handle === undefined) return; + const cancelled = tracker.cancelExact(handle, nowMs); + if (!cancelled.cancelled) return; + authority.armLatePromptRecovery(); + if (cancelled.shouldReplayPrompt) this.replayRequested = true; + } + + consumeAfterFlush(tracker: CodexAppTurnLiveness): boolean { + const shouldReplay = this.replayRequested && !tracker.hasActiveTurn(); + this.replayRequested = false; + return shouldReplay; + } +} + +/** Apply an activity payload whose signed socket connection was authenticated. */ +export function applyTrustedCodexAppActivityMarker( + tracker: CodexAppTurnLiveness, + payload: unknown, + receivedAtMs = Date.now(), +): CodexAppActivityApplyResult { + if (!payload || typeof payload !== 'object') return { accepted: false }; + const marker = payload as Record; + const phase = marker.phase; + if (phase !== 'submitted' && phase !== 'progress' && phase !== 'completed') { + return { accepted: false }; + } + const runnerAtMs = typeof marker.atMs === 'number' && Number.isFinite(marker.atMs) + ? marker.atMs + : receivedAtMs; + // Runner and worker share one host clock. A marker may arrive late, but it + // may never push the activity clock into the future and suppress stalls. + const atMs = Math.min(runnerAtMs, receivedAtMs); + if (phase === 'completed') { + return { + accepted: true, + phase, + shouldReplayPrompt: tracker.completeCurrent(atMs), + }; + } + if (phase === 'submitted') tracker.noteSubmitted(atMs); + else tracker.noteActivity(atMs); + return { accepted: true, phase }; +} diff --git a/src/utils/pending-input-queue.ts b/src/utils/pending-input-queue.ts index 574767381..17a5d9fc3 100644 --- a/src/utils/pending-input-queue.ts +++ b/src/utils/pending-input-queue.ts @@ -7,7 +7,10 @@ export interface PendingCliInput { * receives `content`. */ logicalContent?: string; turnId?: string; + replyTurnId?: string; dispatchAttempt?: number; + codexAppDispatchId?: string; + queuedActivationToken?: string; vcMeetingImTurnOrigin?: VcMeetingImTurnOrigin; codexAppInput?: CodexAppTurnInput; } @@ -24,6 +27,8 @@ export function mergeQueuedCliInput( // per-message attribution/context, so concatenating only their visible text // would drop or mis-attach the sidecar. if (tail.dispatchAttempt !== undefined || next.dispatchAttempt !== undefined + || tail.codexAppDispatchId || next.codexAppDispatchId + || tail.queuedActivationToken || next.queuedActivationToken || tail.vcMeetingImTurnOrigin || next.vcMeetingImTurnOrigin || tail.codexAppInput || next.codexAppInput || tail.logicalContent || next.logicalContent) return false; @@ -55,10 +60,11 @@ export function shouldDeferArgsBakedDurablePrompt(opts: { passesInitialPromptViaArgs: boolean; adoptMode: boolean; dispatchAttempt?: number; + queuedActivationToken?: string; }): boolean { return opts.passesInitialPromptViaArgs && !opts.adoptMode - && opts.dispatchAttempt !== undefined; + && (opts.dispatchAttempt !== undefined || !!opts.queuedActivationToken); } /** Some backends (tmux in particular) reject long launch command strings before @@ -114,6 +120,8 @@ export function shouldStopPendingBatch( ): boolean { return written.dispatchAttempt !== undefined || next?.dispatchAttempt !== undefined + || !!written.queuedActivationToken + || !!next?.queuedActivationToken || !!written.vcMeetingImTurnOrigin || !!next?.vcMeetingImTurnOrigin; } diff --git a/src/utils/runtime-screen-status.ts b/src/utils/runtime-screen-status.ts new file mode 100644 index 000000000..fa36b88ad --- /dev/null +++ b/src/utils/runtime-screen-status.ts @@ -0,0 +1,78 @@ +/** + * Project worker runtime facts into the status consumed by Lark cards and the + * Dashboard. A structured transcript lifecycle outranks screen readiness: + * prompt glyphs and PTY quiescence are UI hints, while a transcript-started + * turn without assistant_final is explicit evidence that work is still running; + * a verified submit also blocks during its bounded transcript-start hand-off. + */ +export function projectRuntimeScreenStatus(state: { + promptReady: boolean; + analyzing: boolean; + structuredTurnBlocking: boolean; +}): 'idle' | 'working' | 'analyzing' { + if (state.analyzing) return 'analyzing'; + if (state.structuredTurnBlocking) return 'working'; + return state.promptReady ? 'idle' : 'working'; +} + +/** Await an asynchronous screen snapshot before reading runtime status. A + * periodic tick can spend measurable time in tmux/observe capture; projecting + * first lets a turn that starts during that await receive a late, stale idle + * update. Keeping the ordering in one helper makes the race deterministic in + * tests and forces callers to read lifecycle state at send time. */ +export async function snapshotWithLatestRuntimeStatus( + capture: () => Promise, + projectStatus: () => ReturnType, +): Promise<{ snapshot: T; status: ReturnType }> { + const snapshot = await capture(); + return { snapshot, status: projectStatus() }; +} + +/** Monotonic evidence for PTY chunks fed to readiness detection. Wall-clock + * milliseconds are not unique: two redraw chunks can land in the same ms, + * making timestamp equality falsely claim that no output followed a rejected + * ready signal. A generation changes for every observed chunk instead. */ +export class PtyOutputGeneration { + private generation = 0; + + observe(): number { + this.generation++; + return this.generation; + } + + snapshot(): number { + return this.generation; + } + + isCurrent(snapshot: number): boolean { + return this.generation === snapshot; + } + + reset(): void { + this.generation = 0; + } +} + +/** Validate a callback captured for one CLI/backend generation. A restart can + * invalidate the generation before async teardown swaps out the backend + * object, so backend identity alone is not a sufficient fence. */ +export function isCliBackendGenerationCurrent( + fence: { generation: number; backend: T }, + current: { generation: number; backend: T; restartInProgress: boolean }, +): boolean { + return fence.generation === current.generation + && fence.backend === current.backend + && !current.restartInProgress; +} + +/** Enter a fresh CLI-write cycle synchronously, before the adapter is allowed + * to yield. This ordering matters for fast turns: transcript final can arrive + * while writeInput is still polling submit history, and IdleDetector must be + * re-armed before that final edge rather than reset after it. */ +export function beginRuntimeWriteCycle(hooks: { + setPromptReady: (ready: boolean) => void; + resetIdleDetector: () => void; +}): void { + hooks.setPromptReady(false); + hooks.resetIdleDetector(); +} diff --git a/src/worker.ts b/src/worker.ts index 3d1883436..40c054fb3 100644 --- a/src/worker.ts +++ b/src/worker.ts @@ -12,11 +12,11 @@ * 6. On 'close', kills CLI and exits * 7. On 'restart', kills CLI and re-spawns with --resume */ -import { randomBytes } from 'node:crypto'; -import { mkdirSync, writeFileSync, unlinkSync, rmdirSync, existsSync, statSync, lstatSync, readdirSync, readlinkSync, readFileSync, realpathSync, copyFileSync, watch as fsWatch, createWriteStream, openSync, closeSync, fstatSync, constants as fsConstants, type FSWatcher, type WriteStream } from 'node:fs'; +import { createHash, randomBytes } from 'node:crypto'; +import { chmodSync, mkdirSync, writeFileSync, unlinkSync, rmdirSync, existsSync, statSync, lstatSync, readdirSync, readlinkSync, readFileSync, realpathSync, copyFileSync, watch as fsWatch, createWriteStream, openSync, closeSync, fstatSync, constants as fsConstants, type FSWatcher, type WriteStream } from 'node:fs'; import { atomicWriteFileSync } from './utils/atomic-write.js'; import { join, basename, dirname } from 'node:path'; -import { homedir, tmpdir } from 'node:os'; +import { homedir, tmpdir, userInfo } from 'node:os'; import { spawnSync } from 'node:child_process'; import { evaluateReadIsolationGate, @@ -26,6 +26,7 @@ import { isCredentialIsolationReservedBasename, buildCredentialIsolationRules, buildSeatbeltProfile, + isolatedPaneOriginChannel, isolatedPaneReattachSafe, sendCredFilePath, botHomePath, @@ -38,9 +39,11 @@ import { buildLinuxReadIsolationMasks, isolationPaneMarkerContent, type IsolationCapability, + isolationPanePolicyDigest, type V2IsolationContext, } from './adapters/cli/read-isolation.js'; import { killPersistentSession, type PersistentBackendType } from './core/persistent-backend.js'; +import { finalizeRawCommandDelivery, writeRawCommandLine } from './core/raw-command-writer.js'; import { readProcessStartIdentity } from './core/session-marker.js'; import { drainTranscript, joinAssistantText, trailingAssistantText, findJsonlContainingFingerprint, findJsonlsContainingExactContent, findLatestJsonl, extractLastAssistantTurn, stringifyUserContent, extractTurnStartText, splitTranscriptEventsByCutoff, type TranscriptEvent } from './services/claude-transcript.js'; import { BridgeTurnQueue, makeFingerprint, normaliseForFingerprint } from './services/bridge-turn-queue.js'; @@ -67,6 +70,7 @@ import { terminalReleasesDurableTurn, type PendingCliInput, } from './utils/pending-input-queue.js'; +import { riffWorkerShutdownInputBlocker } from './core/riff-worker-shutdown-readiness.js'; import { ReadyGate, shouldArmReadyGate } from './utils/ready-gate.js'; import { shouldRunStartupCommandsOnSpawn, shouldDeferInitialPromptForStartup } from './core/startup-commands.js'; import { sanitizePerBotEnv } from './core/per-bot-env.js'; @@ -110,8 +114,11 @@ import { SESSION_ID_FILENAME_RE, type PidFollowResult, } from './services/bridge-rotation-policy.js'; -import { CodexBridgeQueue } from './services/codex-bridge-queue.js'; -import { drainCodexRollout, findCodexRolloutBySessionId, findCodexRolloutByPid, splitCodexEventsByCutoff, extractLastCodexTurn, codexSessionIdFromRolloutPath, type CodexBridgeEvent } from './services/codex-transcript.js'; +import { + CodexBridgeQueue, + pruneExpiredPreStartHeadsAndEmit, +} from './services/codex-bridge-queue.js'; +import { codexSessionIdFromRolloutPath, drainCodexRollout, findCodexRolloutBySessionId, findCodexRolloutByPid, splitCodexEventsByCutoff, extractLastCodexTurn, isStructuredTerminalEvent, type CodexBridgeEvent } from './services/codex-transcript.js'; import { drainTraexRollout, findTraexRolloutBySessionId, findTraexRolloutByPid } from './services/traex-transcript.js'; import { cocoEventsPathForSession, drainCocoEvents, findCocoSessionByPid } from './services/coco-transcript.js'; import { currentHermesStateOffset, drainHermesStateDb, resolveHermesStateDbPath } from './services/hermes-transcript.js'; @@ -128,17 +135,21 @@ import { isStructuredBridgeAdoptIdleCli, isStructuredBridgeAdoptInputCli, isStructuredBridgeFallbackActive, + isStructuredBridgeLifecycleBlockingCli, } from './services/structured-bridge-clis.js'; import { drainCursorTranscript, findCursorChatIdByPid, findCursorTranscriptByChatId, findCursorTranscriptByPid } from './services/cursor-transcript.js'; import { shouldObserveCursorChatId, shouldPersistObservedCursorChatId } from './services/cursor-resume-policy.js'; import { extractKiroSessionIdFromOutput } from './services/kiro-session.js'; import { baselineJsonlCursor } from './services/jsonl-cursor.js'; import { createServer as createHttpServer, type IncomingMessage } from 'node:http'; +import { createServer as createNetServer, type Server as NetServer, type Socket } from 'node:net'; import { WebSocketServer, WebSocket } from 'ws'; import { listenWebTerminalWithFallback } from './utils/web-terminal-listen.js'; import { HerdrWebTerminalBinding } from './utils/herdr-web-terminal-binding.js'; import { TERMINAL_FAVICON_DATA_URI } from './utils/terminal-favicon.js'; import type { + CodexAppDispatchLedgerEntry, + CodexAppGenerationCommit, CodexAppTurnInput, DaemonToWorker, WorkerToDaemon, @@ -163,7 +174,7 @@ import { createCliAdapterSync, locateOnPath } from './adapters/cli/registry.js'; import { buildWrappedLaunch, parseWrapperCli, isTtadkWrapper } from './setup/cli-selection.js'; import { cliUnavailableMessage } from './setup/cli-availability.js'; import { findLaunchedCliPid, scheduleWrapperRealCliPid, readComm, isBareShellComm, bareShellLaunchKind } from './core/session-discovery.js'; -import { codexRpcEligible, paneRunsRemoteTui, orchestrateCodexRpcInit, rolloutUserTurnMatches, decideStartupDialogAction, shouldQueueInitialPrompt, shouldPreMarkFirstTurn, killAndVerifyPersistentPane, type EngageOutcome } from './codex-rpc-lifecycle.js'; +import { CODEX_RPC_TERMINAL_HYDRATION_DELAYS_MS, RpcEngagementFence, codexRpcEligible, paneRunsRemoteTui, orchestrateCodexRpcInit, rolloutUserTurnMatches, decideStartupDialogAction, shouldQueueInitialPrompt, shouldPreMarkFirstTurn, killAndVerifyPersistentPane, rpcTranscriptIngestBlockedByAwaitingActivation, type EngageOutcome } from './codex-rpc-lifecycle.js'; import { delay } from './utils/timing.js'; import { claudeJsonlPathForSession, resolveJsonlFromPid, findOpenClaudeSessionIds, syncClaudeResumeTargetToCwd, DEFAULT_CLAUDE_DATA_DIR } from './adapters/cli/claude-code.js'; import { sessionReadyHookCommand } from './adapters/hook-command.js'; @@ -199,7 +210,12 @@ import { DEVICE_AUTHORITY_DIRECTORY, DEVICE_CREDENTIAL_FILE, } from './platform/device-paths.js'; -import type { BackendType, SessionBackend } from './adapters/backend/types.js'; +import type { + BackendType, + SessionBackend, + SessionDestroyResult, + SessionShutdownDetachResult, +} from './adapters/backend/types.js'; import { tmuxEnv, probeTmuxFunctionalWithRetry } from './setup/ensure-tmux.js'; import { tmuxRestartJitterMs } from './core/tmux-recovery.js'; import { IdleDetector } from './utils/idle-detector.js'; @@ -217,11 +233,20 @@ import { parseWorkerRequestUrl } from './utils/worker-http.js'; import { detectCliUsageLimit, usageLimitStateKey, type CliUsageLimitState } from './utils/cli-usage-limit.js'; import { uploadImageBuffer } from './utils/lark-upload.js'; import { redactChildEnv, scrubSessionCliHomeEnv } from './utils/child-env.js'; -import { decideSubmitConfirmationAction, type SubmitActivityEvidence } from './services/submit-confirmation.js'; +import { + decideSubmitConfirmationAction, + settleDeferredSubmitConfirmation, + settleStaleWriteContinuation, + type SubmitActivityEvidence, +} from './services/submit-confirmation.js'; +import { + runAdoptQueuedWriteSequence, + runAdoptRawInputSequence, + runAdoptSessionRenameSequence, +} from './services/adopt-input-sequence.js'; import { config, resolveChatBotDiscoveryConfig } from './config.js'; import * as sessionStore from './services/session-store.js'; import * as pty from 'node-pty'; -import { createHash } from 'node:crypto'; import { hasInstalledSessionReadyHook, installHook, @@ -234,11 +259,33 @@ import { resolveCodexAppFinalTurnIdentity } from './adapters/cli/codex-app-turn. import { RunnerControlDecoder } from './adapters/cli/runner-control-channel.js'; import { hasMatchingManagedOriginCapability, + ensureManagedOriginAttestationDirectory, + ensureManagedOriginCapabilityLeafSafe, + ensureManagedOriginDataRootProbe, + ensureManagedOriginIsolationSentinel, + ensureManagedOriginRootLocator, + managedOriginLegacyIsolationProbeAccess, + managedOriginDataRootProbeAccess, + managedOriginIsolationSentinelAccess, + managedOriginAttestationDirectory, managedOriginCapabilityPath, + readManagedOriginAuthorityFile, RELAY_ORIGIN_CAPABILITY_BASENAME, replaceManagedOriginCapabilityFile, + sweepManagedOriginAttestationProofs, } from './core/managed-origin-capability.js'; -import { CodexRpcEngine } from './codex-rpc-engine.js'; +import { + CodexRpcEngine, + type CodexRpcTurnIdentity, + type CodexRpcTurnTerminal, +} from './codex-rpc-engine.js'; +import { + beginRuntimeWriteCycle, + isCliBackendGenerationCurrent, + PtyOutputGeneration, + projectRuntimeScreenStatus, +} from './utils/runtime-screen-status.js'; +import { AsyncSerialQueue } from './utils/async-serial-queue.js'; // A worker must never trust an INHERITED session-level CLI home pointer // (CLAUDE_CONFIG_DIR / CODEX_HOME): a stale pm2 dump can resurrect the daemon @@ -255,6 +302,68 @@ import { CodexRpcEngine } from './codex-rpc-engine.js'; // dirs. See SESSION_CLI_HOME_ENV_KEYS for why deleting beats pinning a // default (~/.claude relocates Claude's state file → onboarding rerun) and // why GROK_HOME is exempt. +import { + applyTrustedCodexAppActivityMarker, + applyTrustedCodexAppStateMarker, + CODEX_APP_NO_PROGRESS_TIMEOUT_MS, + CodexAppFlushPromptReplay, + CodexAppReadyAuthority, + CodexAppTurnLiveness, +} from './utils/codex-app-turn-liveness.js'; +import { CodexAppTurnDispatchQueue } from './utils/codex-app-turn-dispatch.js'; +import { + committedCodexAppSequence, + validateCodexAppManagedSendOrigin, +} from './utils/codex-app-dispatch-ledger.js'; +import { + CODEX_APP_CONTROL_BOOTSTRAP_ENV, + CODEX_APP_CONTROL_STARTUP_TIMEOUT_MS, + CodexAppControlFinalAssembler, + CodexAppControlLineDecoder, + CodexAppControlProofDeadline, + CodexAppControlRecordApplicationGate, + CodexAppControlReplayWindow, + CodexAppControlSequenceFence, + activateCodexAppControlIdentity, + acquireCodexAppControlOwnerLease, + acquireCodexAppPosixOwnerLease, + armCodexAppControlStartupTimeout, + authenticateCodexAppControlCandidate, + bindThenPublishCodexAppControlLocator, + cleanupStaleCodexAppControlBootstraps, + codexAppSignedStateReadiness, + codexAppControlLocatorPath, + codexAppPosixControlRoot, + codexAppControlStatePath, + codexAppWindowsOwnerPipeEndpoint, + codexAppWindowsControlRoot, + createCodexAppControlBootstrap, + encodeCodexAppControlAck, + encodeCodexAppControlAccepted, + encodeCodexAppControlChallenge, + ensureCodexAppControlDirectory, + generateCodexAppControlEpoch, + generateCodexAppControlChallenge, + generateCodexAppPosixSocketEndpoint, + generateCodexAppWindowsPipeEndpoint, + hardenWindowsCodexAppControlFile, + mergeCodexAppControlCandidate, + parseCodexAppControlWireRecord, + projectCodexAppControlReadinessStatus, + readCodexAppControlState, + shouldColdStartCodexAppReattach, + shouldFailCodexAppControlChannel, + verifyCodexAppSignedControlMarker, + writeCodexAppControlLocator, + writeCodexAppControlState, + type CodexAppControlState, + type CodexAppControlIdentity, + type CodexAppPosixOwnerLease, + type CodexAppSignedControlMarker, +} from './utils/codex-app-control.js'; + +// Never inherit a session-level CLI home from the daemon's launch environment. +// Each worker re-pins the current bot's home after this scrub. scrubSessionCliHomeEnv(process.env); // ─── State ─────────────────────────────────────────────────────────────────── @@ -295,13 +404,233 @@ function cleanupPiInitialPromptFiles(): void { piInitialPromptEnv = {}; } +const rpcEngagementFence = new RpcEngagementFence(); +/** Native terminal notifications can beat the worker continuation that installs + * rpcActive (response + notifications may share one socket read). Keep those + * exact attempt terminals until the matching bridge entry is active. */ +interface RpcTurnGeneration { + engine: CodexRpcEngine; + cliGeneration: number; +} + +interface PendingRpcTerminal { + terminal: CodexRpcTurnTerminal; + generation: RpcTurnGeneration; +} + +const pendingRpcTurnTerminals = new Map(); +const rpcTurnsAwaitingActivation = new Map(); +const rpcTurnsAwaitingActivationIdentities = new Map(); +/** Local send-start anchor for each exact awaiting owner. A predecessor's + * terminal hydration may ingest this successor's transcript before turn/start + * ACK returns. Reusing the send-start time when the bridge mark is installed + * keeps that already-buffered user/final pair inside its original bounded + * attribution window without widening replay for unrelated turns. */ +const rpcTurnsAwaitingActivationReplayAnchors = new Map(); +/** A bridge mark/activation failure after an accepted RPC submit must never + * degrade to ready. Keep a separate fail-closed gate until that exact native + * turn reaches terminal or the engine is torn down. */ +const rpcLifecycleFailClosedOwners = new Map(); +const rpcLifecycleFailClosedIdentities = new Map(); +const rpcActiveOwners = new Map(); +const rpcTerminalHydrationOwners = new Map(); +const rpcTerminalHydrationTimers = new Map>(); +const rpcTerminalHydrationTerminals = new Map(); +const settlingRpcTerminalOwners = new Map(); +let deferredFreshRpcTurn: { + identity: CodexRpcTurnIdentity; + generation: RpcTurnGeneration; +} | undefined; + +function rpcTurnOwnerKey(identity: CodexRpcTurnIdentity): string { + return `${identity.turnId}\0${identity.dispatchAttempt ?? ''}`; +} + +function sameRpcGeneration( + left: RpcTurnGeneration | undefined, + right: RpcTurnGeneration, +): boolean { + return left?.engine === right.engine + && left.cliGeneration === right.cliGeneration; +} + +function installRpcLifecycleFailClosedOwner( + identity: CodexRpcTurnIdentity, + generation: RpcTurnGeneration, +): void { + const ownerKey = rpcTurnOwnerKey(identity); + rpcLifecycleFailClosedOwners.set(ownerKey, generation); + rpcLifecycleFailClosedIdentities.set(ownerKey, identity); +} + +function installAwaitingRpcActivation( + identity: CodexRpcTurnIdentity, + generation: RpcTurnGeneration, +): void { + const ownerKey = rpcTurnOwnerKey(identity); + rpcTurnsAwaitingActivation.set(ownerKey, generation); + rpcTurnsAwaitingActivationIdentities.set(ownerKey, identity); + rpcTurnsAwaitingActivationReplayAnchors.set(ownerKey, Date.now()); +} + +function awaitingRpcActivationReplayAnchorMs( + identity: CodexRpcTurnIdentity, + generation: RpcTurnGeneration, +): number | undefined { + const ownerKey = rpcTurnOwnerKey(identity); + if (!sameRpcGeneration(rpcTurnsAwaitingActivation.get(ownerKey), generation)) { + return undefined; + } + return rpcTurnsAwaitingActivationReplayAnchors.get(ownerKey); +} + +function clearRpcLifecycleFailClosedOwner( + identity: CodexRpcTurnIdentity, + generation: RpcTurnGeneration, +): boolean { + const ownerKey = rpcTurnOwnerKey(identity); + if (!sameRpcGeneration(rpcLifecycleFailClosedOwners.get(ownerKey), generation)) { + return false; + } + rpcLifecycleFailClosedOwners.delete(ownerKey); + rpcLifecycleFailClosedIdentities.delete(ownerKey); + return true; +} + +function clearAwaitingRpcActivation( + identity: CodexRpcTurnIdentity, + generation: RpcTurnGeneration, +): void { + const ownerKey = rpcTurnOwnerKey(identity); + if (sameRpcGeneration(rpcTurnsAwaitingActivation.get(ownerKey), generation)) { + rpcTurnsAwaitingActivation.delete(ownerKey); + rpcTurnsAwaitingActivationIdentities.delete(ownerKey); + rpcTurnsAwaitingActivationReplayAnchors.delete(ownerKey); + } + const pending = pendingRpcTurnTerminals.get(ownerKey); + if (pending && sameRpcGeneration(pending.generation, generation)) { + pendingRpcTurnTerminals.delete(ownerKey); + } +} + +function notifyRpcTeardownBeforeActivation( + identity: CodexRpcTurnIdentity, + terminalStatus?: CodexRpcTurnTerminal['status'], +): void { + const completedBeforeActivation = terminalStatus === 'completed'; + send({ + type: 'user_notify', + message: completedBeforeActivation + ? 'Codex RPC 消息已执行完成，但会话在本地完成生命周期登记前重启，兜底输出可能未被捕获；原消息未自动重发，如未看到回复请先查看终端结果再人工跟进。' + : 'Codex RPC 消息已写出，但会话在取得 turn/start 归属前重启；为避免重复执行未自动重发，请按需人工确认结果。', + turnId: identity.turnId, + ...(identity.dispatchAttempt !== undefined + ? { dispatchAttempt: identity.dispatchAttempt } + : {}), + }); +} + function stopCodexRpcEngine(): void { + // Invalidate even when the engine has not yet been published: engageCodexRpc + // may be awaiting a local engine/thread/probe while another IPC handler starts + // a restart. That stale continuation must never republish the stopped engine. + rpcEngagementFence.invalidate(); const engine = codexRpcEngine; + const ownedRpcTurns = new Set([ + ...rpcTurnsAwaitingActivation.keys(), + ...rpcLifecycleFailClosedOwners.keys(), + ...rpcActiveOwners.keys(), + ...settlingRpcTerminalOwners.keys(), + ]); + // stop() emits exact 'stopped' terminals for every engine-owned native turn. + // Keep the engine identity installed until those synchronous callbacks have + // retired their matching rpcActive entries. + try { engine?.stop(); } catch { /* best effort */ } + // A native terminal can arrive synchronously from engine.stop() while its + // turn/start continuation is still awaiting activation. handleRpcTurnTerminal + // buffers that exact terminal; settle it now, before teardown clears the + // awaiting/pending maps, so an accepted RPC delivery never loses its only + // terminal at an operator or in-worker restart boundary. + for (const [ownerKey, pending] of [...pendingRpcTurnTerminals]) { + const awaiting = rpcTurnsAwaitingActivation.get(ownerKey); + if (!awaiting || !sameRpcGeneration(awaiting, pending.generation)) continue; + const identity = rpcTurnsAwaitingActivationIdentities.get(ownerKey); + const shouldNotify = !rpcLifecycleFailClosedOwners.has(ownerKey); + settleRpcTurnTerminal(pending.terminal, pending.generation); + if (identity && shouldNotify) { + notifyRpcTeardownBeforeActivation(identity, pending.terminal.status); + } + } + // A follow-up can be accepted by the socket but lose its turn/start response + // while an operator or liveness restart tears down the engine. RPC submissions + // intentionally bypass InflightInputTracker (replay would risk duplicate + // execution), so close every still-unbound logical owner explicitly instead + // of letting its stale continuation assume an ordinary carryover exists. + for (const [ownerKey, identity] of [...rpcTurnsAwaitingActivationIdentities]) { + const awaiting = rpcTurnsAwaitingActivation.get(ownerKey); + if (!awaiting) continue; + clearAwaitingRpcActivation(identity, awaiting); + emitTurnTerminal( + identity.turnId, + 'ambiguous', + 'rpc_engine_teardown_before_turn_start_ack', + identity.dispatchAttempt, + ); + if (!rpcLifecycleFailClosedOwners.has(ownerKey)) { + notifyRpcTeardownBeforeActivation(identity); + } + } + for (const pending of [...rpcTerminalHydrationTerminals.values()]) { + finalizeRpcTurnTerminal(pending.terminal, pending.generation, true); + } + // A dispatched delivery for which the app-server never yielded a native turn + // id cannot receive engine.stop()'s native terminal callback. Publish one + // exact logical terminal before clearing the owner so the daemon's durable + // ledger never waits forever on a teardown-only attempt. + for (const [ownerKey, identity] of rpcLifecycleFailClosedIdentities) { + if (!rpcLifecycleFailClosedOwners.has(ownerKey)) continue; + emitTurnTerminal( + identity.turnId, + 'ambiguous', + 'rpc_engine_teardown_without_native_terminal', + identity.dispatchAttempt, + ); + } codexRpcEngine = undefined; remoteWsUrl = undefined; remoteThreadId = undefined; clearRpcEnginePidMarker(); - try { engine?.stop(); } catch { /* best effort */ } + // Defensive cleanup for a protocol/transport failure that never yielded a + // native id. No RPC turn may survive an engine teardown holding the lifecycle + // gate open. + for (const turn of codexBridgeQueue.peek()) { + if (turn.finalText === undefined + && (turn.rpcActive + || ownedRpcTurns.has(rpcTurnOwnerKey({ + turnId: turn.turnId, + ...(turn.dispatchAttempt !== undefined + ? { dispatchAttempt: turn.dispatchAttempt } + : {}), + })))) { + codexBridgeQueue.stopRpcActive(turn.turnId, turn.dispatchAttempt); + codexBridgeQueue.dropPendingTurn(turn.turnId, turn.dispatchAttempt, true); + } + } + pendingRpcTurnTerminals.clear(); + rpcTurnsAwaitingActivation.clear(); + rpcTurnsAwaitingActivationIdentities.clear(); + rpcTurnsAwaitingActivationReplayAnchors.clear(); + rpcLifecycleFailClosedOwners.clear(); + rpcLifecycleFailClosedIdentities.clear(); + rpcActiveOwners.clear(); + for (const timer of rpcTerminalHydrationTimers.values()) clearTimeout(timer); + rpcTerminalHydrationTimers.clear(); + rpcTerminalHydrationTerminals.clear(); + rpcTerminalHydrationOwners.clear(); + settlingRpcTerminalOwners.clear(); + deferredFreshRpcTurn = undefined; + stopStructuredStartGraceRecheck(); + structuredRejectedReadyEvidenceGeneration = undefined; } /** Resolve the persistent pane name + liveness for a backend type, mirroring @@ -383,8 +712,20 @@ async function engageCodexRpc(cfg: Extract): P if (!codexRpcEligible(cfg, { sandboxForced: sandboxEnabled() })) return 'not-engaged'; const wantResume = cfg.resume === true && !!cfg.cliSessionId; stopCodexRpcEngine(); + const engagementLease = rpcEngagementFence.begin(); let engine: CodexRpcEngine | undefined; + let enginePidMarker: string | null = null; + const assertRpcEngagementCurrent = (): void => { + if (!rpcEngagementFence.isCurrent(engagementLease)) { + throw new CliSpawnSupersededError(); + } + }; try { + // engage runs immediately before spawnCli, which advances the viewer + // generation once. Bind every callback from this app-server to that target + // generation so a late terminal from an old server can never retire a + // same-(turnId,attempt) entry installed by its replacement. + const engineCliGeneration = cliSpawnGeneration + 1; const cliBin = createCliAdapterSync(cfg.cliId as CliId, cfg.cliPathOverride).resolvedBin; const engineEnv: NodeJS.ProcessEnv = { ...redactChildEnv(process.env) }; engineEnv.PATH = `${join(homedir(), '.botmux', 'bin')}:${engineEnv.PATH ?? ''}`; @@ -407,6 +748,13 @@ async function engageCodexRpc(cfg: Extract): P engine = new CodexRpcEngine({ cliBin, cwd: cfg.workingDir, env: engineEnv, sessionId: cfg.sessionId, model: cfg.model, log: (m: string) => log(m), + onTurnTerminal: (terminal) => { + if (!engine) return; + handleRpcTurnTerminal(terminal, { + engine, + cliGeneration: engineCliGeneration, + }); + }, onDead: () => { if (codexRpcEngine === engine) { log('Codex RPC app-server died; replacing the tmux session and re-engaging the thread'); @@ -424,19 +772,40 @@ async function engageCodexRpc(cfg: Extract): P }, }); await engine.start(); + assertRpcEngagementCurrent(); // Shell tools run under the app-server process tree, not the viewer TUI. // Mark its pid before the first turn so `botmux send` resolves the current // per-turn identity instead of falling back to a stale/session-only env. - registerRpcEnginePidMarker(engine.appServerPid); + enginePidMarker = registerRpcEnginePidMarker(engine.appServerPid); const threadId = wantResume ? await engine.resumeThread(cfg.cliSessionId!) : await engine.startThread(); + assertRpcEngagementCurrent(); let outcome: EngageOutcome = wantResume ? 'resumed' : 'accepted'; if (!wantResume && cfg.prompt) { + const firstIdentity: CodexRpcTurnIdentity = { + turnId: cfg.turnId ?? `codex-rpc-${randomBytes(8).toString('hex')}`, + ...(cfg.dispatchAttempt !== undefined + ? { dispatchAttempt: cfg.dispatchAttempt } + : {}), + }; + const firstGeneration: RpcTurnGeneration = { + engine, + cliGeneration: engineCliGeneration, + }; + installAwaitingRpcActivation( + firstIdentity, + firstGeneration, + ); // Three-state delivery (P1-1, exactly-once priority): 'accepted' (ack or // rollout evidence), 'not-sent' (frame never dispatched → safe paste), or // 'ambiguous' (dispatched, unconfirmed → engaged but NEVER resend). - const first = await engine.sendFirstTurn(cfg.prompt, cfg.turnId, + const first = await engine.sendFirstTurn(cfg.prompt, firstIdentity, (tid) => codexRolloutProbe(cfg.cliId, tid, cfg.prompt, 12_000)); - if (first === 'not-sent') { + // restart/close may have settled this exact attempt as ambiguous while the + // rollout probe was pending. Fence before ANY durable/bridge/global engine + // mutation; a superseded delivery is never pasted or re-queued. + assertRpcEngagementCurrent(); + if (first.outcome === 'not-sent') { + clearAwaitingRpcActivation(firstIdentity, firstGeneration); // The turn/start frame never left → the turn cannot have run → tear the // engine down and fall back to paste. flushPending marks the bridge once // on the paste path — we must NOT pre-mark here or that would double-mark @@ -446,15 +815,93 @@ async function engageCodexRpc(cfg: Extract): P try { engine.stop(); } catch { /* best effort */ } return 'not-engaged'; } + // Fresh RPC delivery bypasses flushPending(), which is the normal owner + // of this durable head-of-line gate. Claim it here only after the frame is + // known dispatched (accepted or ambiguous); the not-sent branch above + // safely falls back to paste and lets flushPending claim it once. + if (firstIdentity.dispatchAttempt !== undefined) { + durableTurnInFlight = true; + } // Bridge mark ONLY for a confirmed-accepted turn — so the structured // fallback can attribute the reply even if the model skips `botmux send`. // Marked here (after the outcome, before persistCliSessionId/attach — late - // attach from offset 0 still matches, timing-safe). 'ambiguous' does NOT - // mark: there is no positive evidence the turn ran, and an unstarted head - // would permanently block every later turn's drain — the notify + Web - // terminal are the authoritative recovery surface (Codex P1). - if (shouldPreMarkFirstTurn(first)) codexBridgeMarkPendingTurn(cfg.prompt, cfg.turnId); - outcome = first; // 'accepted' | 'ambiguous' — both stay engaged, prompt never re-queued + // attach from offset 0 still matches, timing-safe). An ambiguous outcome + // gets an attribution-only mark plus a separate fail-closed owner below: + // it cannot publish false idle or head-of-line a successor because no + // successor is admitted until terminal/teardown retires that owner. + if (shouldPreMarkFirstTurn(first.outcome)) { + if (!first.nativeTurnId) { + // Positive rollout evidence proves delivery, but without a native id + // no terminal can be associated precisely. Preserve exactly-once + // (never paste/requeue) and hold the explicit fail-closed lifecycle + // gate until a structured terminal or engine teardown instead of + // guessing a latest turn. + codexBridgeMarkPendingTurn( + cfg.prompt, + firstIdentity.turnId, + firstIdentity.dispatchAttempt, + awaitingRpcActivationReplayAnchorMs(firstIdentity, firstGeneration), + ); + installRpcLifecycleFailClosedOwner(firstIdentity, firstGeneration); + deferredFreshRpcTurn = { + identity: firstIdentity, + generation: firstGeneration, + }; + log(`Codex RPC fresh accepted turn has no native id; lifecycle failed closed for ${firstIdentity.turnId}`); + send({ + type: 'user_notify', + message: 'Codex RPC 首条消息已确认落盘，但无法取得原生 turn id；为避免重复执行未重发，并保持忙碌直到检测到终态或会话重启。', + turnId: firstIdentity.turnId, + ...(firstIdentity.dispatchAttempt !== undefined + ? { dispatchAttempt: firstIdentity.dispatchAttempt } + : {}), + }); + } else { + // Keep terminal delivery deferred until spawnCli has attached and + // baselined the rollout; otherwise an ultra-fast first completion can + // retire the bridge mark before fallback output is harvested. + activateRpcTurnLifecycle( + firstIdentity, + cfg.prompt, + false, + firstGeneration, + true, + ); + deferredFreshRpcTurn = { + identity: firstIdentity, + generation: firstGeneration, + }; + } + } else { + // A dispatched-but-unconfirmed first turn is never re-sent. Keep an + // exact attribution mark plus an explicit fail-closed gate even without + // a native id. A structured transcript terminal can retire it; a later + // native terminal without an owner is intentionally ignored instead of + // guessed. If no structured terminal becomes visible, exact engine + // teardown is the expected fallback and drops this attempt's mark. + codexBridgeMarkPendingTurn( + cfg.prompt, + firstIdentity.turnId, + firstIdentity.dispatchAttempt, + awaitingRpcActivationReplayAnchorMs(firstIdentity, firstGeneration), + ); + installRpcLifecycleFailClosedOwner(firstIdentity, firstGeneration); + deferredFreshRpcTurn = { + identity: firstIdentity, + generation: firstGeneration, + }; + } + if (first.outcome === 'accepted' && cfg.queuedActivationToken) { + // Fresh RPC input bypasses pendingMessages/flushPending entirely. The + // app-server's accepted turn/start (or positive rollout proof) is the + // durable submission boundary for the daemon's queued-opening journal. + send({ + type: 'queued_activation_submitted', + sessionId: cfg.sessionId, + activationToken: cfg.queuedActivationToken, + }); + } + outcome = first.outcome; // accepted | ambiguous — both stay engaged, prompt never re-queued } persistCliSessionId(threadId); codexRpcEngine = engine; @@ -463,10 +910,22 @@ async function engageCodexRpc(cfg: Extract): P log(`Codex RPC input engaged (${outcome}${wantResume ? '/resume' : '/fresh'}): app-server ${engine.wsUrl} thread ${threadId}`); return outcome; } catch (err: any) { + if (err instanceof CliSpawnSupersededError + || !rpcEngagementFence.isCurrent(engagementLease)) { + log('Codex RPC engagement was superseded; stopping only its local app-server and preserving the replacement generation'); + try { engine?.stop(); } catch { /* best effort */ } + if (enginePidMarker) clearRpcEnginePidMarker(enginePidMarker); + throw err instanceof CliSpawnSupersededError + ? err + : new CliSpawnSupersededError(); + } log(`Codex RPC input failed to start (${err?.message ?? err}); falling back to paste mode`); clearRpcEnginePidMarker(); try { engine?.stop(); } catch { /* best effort */ } // P1-3a: stop the LOCAL ref (codexRpcEngine may be unassigned) - codexRpcEngine = undefined; remoteWsUrl = undefined; remoteThreadId = undefined; + // The local engine may not yet have been published into codexRpcEngine. + // Still clear any exact lifecycle state its callbacks installed before the + // failure; otherwise paste fallback could inherit a stale fail-closed mark. + stopCodexRpcEngine(); return 'not-engaged'; } } @@ -515,6 +974,18 @@ function armRpcStartupDialogDismiss(): void { }; rpcDialogDismissTimer = setTimeout(tick, 2500); } + +/** Monotonic identity for the owned/adopted CLI backend generation. Deferred + * callbacks capture it and must re-check after every await before touching + * transcript lifecycle or user-visible state. */ +let cliSpawnGeneration = 0; + +class CliSpawnSupersededError extends Error { + constructor() { + super('CLI spawn was superseded by a newer lifecycle operation'); + this.name = 'CliSpawnSupersededError'; + } +} let cliPidMarker: string | null = null; // path to .botmux-cli-pids/ let seatbeltProfilePath: string | null = null; // per-session Seatbelt .sb profile to rm at exit (external-wrapper read isolation) let sandboxStopWatcher: (() => void) | null = null; // stop fn for the sandbox outbox watcher @@ -522,6 +993,7 @@ let sandboxCleanup: (() => void) | null = null; // unmount overlays + rm th let sandboxRelayOutbox: string | null = null; let sandboxRelayCapability: { token: string; turnId?: string; dispatchAttempt?: number } | null = null; let readIsolationOriginCapabilityFile: string | null = null; +let readIsolationOriginChannelId: string | null = null; let sandboxTeardownDone = false; // guards the exit-time best-effort teardown from double-running / running on suspend-for-resume let sessionMcpGatewayHost: SessionMcpGatewayHost | null = null; /** Counts consecutive in-worker restart cycles (see case 'restart'). Used by @@ -776,6 +1248,10 @@ const IDLE_PROBE_INTERVAL_MS = 3_500; const IDLE_PROBE_MAX_ATTEMPTS = 24; let busyPatternIdleProbeTimer: ReturnType | null = null; let reattachIdleProbeTimer: ReturnType | null = null; +let structuredStartGraceRecheckTimer: ReturnType | null = null; +let structuredRejectedReadyEvidenceGeneration: number | undefined; +const ptyOutputGeneration = new PtyOutputGeneration(); +const adoptWriteQueue = new AsyncSerialQueue(); /** The effectiveResume flag used by the most recent spawnCli call. Written * immediately after the two-tier fallback check so late-attach timers * (hermes, cursor, etc.) can read THE SAME semantics the spawn used, @@ -810,6 +1286,23 @@ let effectiveBackendType: BackendType = 'pty'; * generation. The daemon receives this over private IPC; child-writable PID * marker files remain diagnostics only. */ let currentCliCredentialIsolated = false; +/** Successful Riff prepare-close request awaiting the daemon's durable close + * commit. Until commit the worker stays alive but the backend closing fence + * rejects new input, preventing an exit/restart race before row publication. */ +let preparedCloseRequestId: string | null = null; +/** Riff close prepare currently awaiting remote cancellation. Kept separate + * from `preparedCloseRequestId` so racing turns are rejected throughout the + * await, not only after the successful ACK. */ +let closeRequestInFlightId: string | null = null; +/** Failed close prepares restore the backend before publishing close_result. + * Remember that exact request until the daemon's close_abort handshake ACKs + * the already-completed restoration. */ +let lastAbortedCloseRequestId: string | null = null; +/** Graceful daemon-shutdown Riff detach transaction. The worker does not exit + * after prepare: the daemon must first durably ACK the exact final lineage, + * then send commit. */ +let shutdownDetachRequestId: string | null = null; +let shutdownDetachPhase: 'preparing' | 'prepared' | null = null; /** pty-under-zellij backend (BACKEND_TYPE=zellij). Behaves like the non-tmux * pty path for the worker (renderer screenshots, relay web terminal) but owns * a persistent zellij session that survives daemon restart. */ @@ -934,7 +1427,7 @@ const READY_SIGNAL_TIMEOUT_MS = 45_000; const FIRST_PROMPT_TIMEOUT_MS = 15_000; /** Hard cap for startup screens that outlive the soft fallback. Prevents a * changed/missing readyPattern from trapping the first queued input forever. */ -const FIRST_PROMPT_HARD_TIMEOUT_MS = 90_000; +const FIRST_PROMPT_HARD_TIMEOUT_MS = CODEX_APP_CONTROL_STARTUP_TIMEOUT_MS; /** Epoch ms of the most recent PTY output — used to settle for quiescence * before the first flush (see settleThenFlush). */ let lastPtyOutputAtMs = 0; @@ -1035,46 +1528,12 @@ const COCO_SLASH_TYPE_THROTTLE_MS = 40; * the match before submit) → a separate Enter. Non-TUI backends fall back to a * single write + CR. Shared by the `raw_input` IPC handler and runStartupCommands * so both stay in lockstep. */ -async function sendRawCommandLine(be: NonNullable, content: string): Promise { - if ('sendText' in be && 'sendSpecialKeys' in be) { - if (lastInitConfig?.cliId === 'coco') { - // CoCo (Trae CLI, Ink TUI) detects "several bytes in one PTY read = paste", - // so a one-shot sendText('/model') lands as PASTED text: command mode + the - // slash picker never activate and the trailing Enter submits `/model` to the - // model (the "/model 多一个换行" bug). Fix: type char-by-char (throttled) so - // each char is a distinct keystroke that opens command mode, and append ONE - // trailing space to a bare command so the suggestion popup is dismissed. - // Without that dismissal the popup captures the first Enter (CoCo then needs - // two), and for an interactive command like /model — which opens a model - // selector — a stray second Enter would confirm whatever model is highlighted. - // Popup gone ⇒ exactly one Enter executes (/model just opens the selector and - // waits). trim() first so a trailing newline carried from the Lark message - // isn't typed as a literal newline that re-breaks command detection. - const cmd = content.trim(); - const typed = cmd.includes(' ') ? cmd : `${cmd} `; - for (const ch of typed) { - (be as any).sendText(ch); - await new Promise(r => setTimeout(r, COCO_SLASH_TYPE_THROTTLE_MS)); - } - await new Promise(r => setTimeout(r, 200)); - (be as any).sendSpecialKeys('Enter'); - return; - } - (be as any).sendText(content); - await new Promise(r => setTimeout(r, 200)); - (be as any).sendSpecialKeys('Enter'); - } else { - // PtyBackend has no sendText/sendSpecialKeys, so write the keystrokes - // directly — but still beat between the text and the Enter. Writing - // `content + '\r'` in one chunk submits before the CLI's slash-command - // parser has registered the `/cmd` match, so the command is left - // unsent in the input box (observed with `/goal ` on a pty - // workflow worker: typed but never executed). Mirror the tmux path's - // 200ms beat. - be.write(content); - await new Promise(r => setTimeout(r, 200)); - be.write('\r'); - } +async function sendRawCommandLine(be: NonNullable, content: string): Promise { + return writeRawCommandLine(be, content, { + coco: lastInitConfig?.cliId === 'coco', + cocoThrottleMs: COCO_SLASH_TYPE_THROTTLE_MS, + submitBeatMs: 200, + }); } /** Serialize only the literal command-line write window (text -> beat -> Enter). @@ -1083,14 +1542,14 @@ async function sendRawCommandLine(be: NonNullable, content: stri * from splicing keystrokes into one another. */ let commandLineWriteTail: Promise = Promise.resolve(); let commandLineWritesPending = 0; -async function sendRawCommandLineSerially(be: NonNullable, content: string): Promise { +async function sendRawCommandLineSerially(be: NonNullable, content: string): Promise { const previous = commandLineWriteTail; let release!: () => void; commandLineWriteTail = new Promise(resolve => { release = resolve; }); commandLineWritesPending += 1; await previous; try { - await sendRawCommandLine(be, content); + return await sendRawCommandLine(be, content); } finally { commandLineWritesPending -= 1; release(); @@ -1134,7 +1593,11 @@ async function runStartupCommands(): Promise { for (const cmd of cmds) { if (!backend) break; try { - await sendRawCommandLineSerially(backend, cmd); + const accepted = await sendRawCommandLineSerially(backend, cmd); + if (!accepted) { + log(`Startup command skipped (backend rejected write): ${cmd}`); + continue; + } await awaitPtyQuiescence(STARTUP_CMD_QUIET_MS, STARTUP_CMD_CAP_MS); log(`Startup command sent: ${cmd}`); } catch (e: any) { @@ -1147,26 +1610,95 @@ async function runStartupCommands(): Promise { } const pendingMessages: PendingCliInput[] = []; +/** RiffAdapter.writeInput returns when it has only appended an async backend + * write. A queued activation is accepted later, at the exact NEW non-null task + * id. One-per-task-boundary flushing guarantees at most one armed token. */ +let pendingRiffQueuedActivation: { token: string; turnId?: string; dispatchAttempt?: number } | null = null; + +function failPendingRiffQueuedActivation(reason: string): void { + const pending = pendingRiffQueuedActivation; + if (!pending) return; + pendingRiffQueuedActivation = null; + log( + `Riff queued activation ${pending.token.substring(0, 8)} failed before task-id acceptance: ${reason}`, + ); + // The daemon still owns the exact activation journal because no submitted + // ACK was emitted. A real worker exit makes it re-park/replay N; continuing + // this generation could bind the retained token to a later unrelated task. + void sendFatalWorkerErrorAndExit( + new Error(`Riff queued activation failed before remote acceptance: ${reason}`), + pending.turnId, + pending.dispatchAttempt, + ); +} +/** The async init handler must materialize its exact opening item before a + * concurrently delivered `message` IPC may flush. */ +let initPromptMaterialized = false; +/** Adopt messages accepted while native /rename owns the TUI. They cannot use + * pendingMessages because adopt writes need writeAdoptMessage's transcript + * baseline + complete adapter lifecycle. flushPending drains them only after + * deferred raw commands and the latest native rename have settled. */ +const pendingAdoptMessages: PendingCliInput[] = []; /** Literal commands that arrived while native /rename owned the TUI or while * an owned CLI restart was fenced. Normal raw_input commands are still * delivered immediately (including while busy). */ const pendingRawInputs: Array> = []; + +interface AdoptWriteFence { + generation: number; + backend: SessionBackend; +} + +function captureAdoptWriteFence(): AdoptWriteFence | undefined { + if (!backend || cliRestartInProgress || rawInputRestartGate) return undefined; + return { generation: cliSpawnGeneration, backend }; +} + +/** A queued adopt task may outlive its CLI. Never let process-lifetime queue + * work run against backend=null or a replacement backend generation. Tasks + * that fail this check have not written anything and are safe to requeue. */ +function adoptWriteFenceIsCurrent(fence: AdoptWriteFence): boolean { + return cliSpawnGeneration === fence.generation + && backend === fence.backend + && !cliRestartInProgress + && !rawInputRestartGate; +} /** Latest requested canonical session title. Unlike a normal prompt this is an * administrative TUI command: never type-ahead while the agent is busy, never * open a model turn, and latest-wins if several renames arrive before idle. */ let pendingSessionRename: string | null = null; -/** True after the rename command's Enter lands and until the TUI returns to its - * prompt. Blocks type-ahead user messages from racing into the command UI. */ -let sessionRenameInFlight = false; +/** Native rename lifecycle. `reserved` covers time spent waiting behind an + * earlier adopt composer write; `writing` covers literal text→beat→Enter; + * `sent` begins only after Enter has landed and lasts until the next genuine + * prompt. Every non-idle phase blocks raw/adopt input, but only `sent` can be + * released by prompt-ready. */ +type SessionRenamePhase = 'idle' | 'reserved' | 'writing' | 'sent'; +let sessionRenamePhase: SessionRenamePhase = 'idle'; const SESSION_RENAME_IDLE_TIMEOUT_MS = 5_000; let sessionRenameIdleTimer: ReturnType | null = null; -function clearSessionRenameInFlight(): void { +function sessionRenameInFlight(): boolean { + return sessionRenamePhase !== 'idle'; +} + +function forceClearSessionRenameInFlight(): void { if (sessionRenameIdleTimer) { clearTimeout(sessionRenameIdleTimer); sessionRenameIdleTimer = null; } - sessionRenameInFlight = false; + sessionRenamePhase = 'idle'; +} + +/** A prompt can release rename only after the command Enter was actually sent. + * A fast final from an older adopt write may arrive while rename is merely + * reserved behind the queue; keep that reservation and let the queued helper + * consume the newly-ready prompt. */ +function settleSessionRenameOnPrompt(): void { + if (sessionRenamePhase === 'sent') forceClearSessionRenameInFlight(); +} + +function settleFailedSessionRenameWrite(): void { + if (sessionRenamePhase === 'writing') sessionRenamePhase = 'sent'; } /** Fail open if a CLI executes /rename without emitting enough redraw output @@ -1174,12 +1706,12 @@ function clearSessionRenameInFlight(): void { * normally settle immediately; this cap prevents one administrative command * from wedging all later Lark messages forever. */ function armSessionRenameIdleTimeout(): void { - if (!sessionRenameInFlight) return; + if (sessionRenamePhase !== 'sent') return; if (sessionRenameIdleTimer) clearTimeout(sessionRenameIdleTimer); sessionRenameIdleTimer = setTimeout(() => { sessionRenameIdleTimer = null; - if (!sessionRenameInFlight) return; - sessionRenameInFlight = false; + if (sessionRenamePhase !== 'sent') return; + sessionRenamePhase = 'idle'; // The timeout specifically means prompt redraw detection failed. Fail open // by restoring the gate explicitly; otherwise a deferred raw_input is the // only queued work and flushPending would keep waiting for the very signal @@ -1196,43 +1728,147 @@ function armSessionRenameIdleTimeout(): void { * Commands received while /rename owns the TUI are deferred by the IPC handler * and come through this same function after the prompt returns. */ async function deliverRawInput(msg: Extract): Promise { - renderer?.markNewTurn(); - usageLimitTracker.beginTurn(currentUsageLimitSnapshot()); - if (tmuxScrolledHalfPages > 0) exitTmuxScrollMode(); - const targetBackend = backend; - if (!targetBackend) return; - - // A passthrough is still an accepted input boundary. Rotate (or revoke) the - // marker immediately before the literal command reaches the CLI so it can - // never inherit a previous human turn. System-generated raw commands omit - // turnId and therefore publish an explicitly unattributed marker. - currentBotmuxTurnId = msg.turnId; - currentBotmuxDispatchAttempt = undefined; - currentVcMeetingImTurnOrigin = undefined; - writeCliPidMarker(); - publishSandboxRelayCapability(); + const writeRawInput = async ( + targetBackend: SessionBackend, + fence?: AdoptWriteFence, + onFollowUp?: () => void, + ): Promise => { + renderer?.markNewTurn(); + usageLimitTracker.beginTurn(currentUsageLimitSnapshot()); + if (tmuxScrolledHalfPages > 0) exitTmuxScrollMode(); + + // A passthrough is still an accepted input boundary. Rotate (or revoke) the + // marker immediately before the literal command reaches the CLI so it can + // never inherit a previous human turn. System-generated raw commands omit + // turnId and therefore publish an explicitly unattributed marker. + currentBotmuxTurnId = msg.turnId; + currentBotmuxDispatchAttempt = undefined; + currentVcMeetingImTurnOrigin = undefined; + writeCliPidMarker(); + publishSandboxRelayCapability(); - let sent = false; - try { - await sendRawCommandLineSerially(targetBackend, msg.content); - sent = true; - isPromptReady = false; - idleDetector?.reset(); - log(`Passthrough slash command: ${msg.content}`); - } catch (err: any) { - // Do not send another queued command against a backend whose write failed. - isPromptReady = false; - log(`Passthrough slash command failed (${msg.content}): ${err?.message ?? err}`); - } + let sent = false; + try { + sent = await sendRawCommandLineSerially(targetBackend, msg.content); + if (fence && !adoptWriteFenceIsCurrent(fence)) { + send({ + type: 'user_notify', + message: `Passthrough command ${msg.content} became ambiguous because the adopted CLI backend changed while it was being written. Its dependent follow-up was withheld; please verify the terminal and retry the bundle explicitly.`, + }); + sent = false; + } + } catch (err: any) { + // Do not send a bundled follow-up or another queued command against a + // backend whose literal command write failed. + log(`Passthrough slash command failed (${msg.content}): ${err?.message ?? err}`); + } + + const accepted = finalizeRawCommandDelivery({ + accepted: sent, + durableActivation: !!msg.queuedActivationToken, + acknowledgeActivation: !!msg.queuedActivationToken && effectiveBackendType !== 'riff', + hasFollowUp: !!onFollowUp, + onAccepted: () => { + isPromptReady = false; + idleDetector?.reset(); + log(`Passthrough slash command: ${msg.content}`); + }, + // Non-adopt follows the normal busy queue. Adopt keeps the complete + // adapter lifecycle in runAdoptRawInputSequence below. + onFollowUp: () => onFollowUp?.(), + // Pending-repo raw openings are durable too. ACK only after text + Enter. + onActivationAck: () => send({ + type: 'queued_activation_submitted', + sessionId, + activationToken: msg.queuedActivationToken!, + }), + onDurableFailure: () => { + isPromptReady = false; + log('Durable raw activation write rejected; retiring worker generation'); + void sendFatalWorkerErrorAndExit( + new Error('durable raw activation was not accepted by the backend'), + msg.turnId, + ); + }, + }); + if (!accepted && !msg.queuedActivationToken) { + log(`Passthrough slash command was not accepted by the backend: ${msg.content}`); + } + return accepted; + }; - // Follow-up rides on the same IPC and is enqueued only after this command's - // Enter lands. sendToPty also observes commandLineWritesPending, so another - // raw command's text -> Enter window cannot be interrupted by the follow-up. - if (sent && msg.followUpContent) { - sendToPty(msg.followUpContent, msg.followUpTurnId, { - codexAppInput: msg.followUpCodexAppInput, + if (lastInitConfig?.adoptMode) { + const fence = captureAdoptWriteFence(); + if (!fence) { + pendingRawInputs.push(msg); + log(`Deferred adopt passthrough until a stable CLI generation is ready: ${msg.content}`); + return; + } + const followUpContent = msg.followUpContent; + // Adopt raw commands, their bundled follow-up and ordinary adopt messages + // all share one queue. Keep the queue until writeAdoptMessage has completed + // transcript marking, adapter/history verification and lifecycle settling. + await runAdoptRawInputSequence({ + queue: adoptWriteQueue, + isCurrent: () => adoptWriteFenceIsCurrent(fence), + onStaleBeforeWrite: () => { + pendingRawInputs.push(msg); + log(`Re-queued stale adopt passthrough for the replacement CLI generation: ${msg.content}`); + }, + onStaleBeforeFollowUp: () => { + if (!followUpContent) return; + // The raw command already landed on the previous backend. Replaying + // only its dependent prompt into the replacement could run in the + // wrong repo/session (for example after `/cd ...`). Hold the bundle + // for an explicit user retry instead of splitting its atomic meaning. + send({ + type: 'user_notify', + message: `Passthrough command ${msg.content} completed on the previous CLI, but the backend changed before its dependent follow-up. The follow-up was withheld; verify the terminal and retry the bundle explicitly.`, + }); + log(`Withheld bundled adopt follow-up after CLI generation changed (${followUpContent.length} chars)`); + }, + writeRawInput: () => writeRawInput(fence.backend, fence), + ...(followUpContent + ? { + writeFollowUp: async () => { + const result = await writeAdoptMessage( + followUpContent, + msg.followUpTurnId, + undefined, + undefined, + fence, + ); + if (result === 'stale-before-write') { + send({ + type: 'user_notify', + message: `The CLI backend changed after passthrough command ${msg.content} but before its dependent follow-up. The follow-up was withheld; verify the terminal and retry the bundle explicitly.`, + }); + log(`Withheld stale bundled adopt follow-up after raw command (${followUpContent.length} chars)`); + } else if (result === 'completed') { + log(`Completed adopt follow-up after raw input (${followUpContent.length} chars)`); + } + }, + } + : {}), }); - log(`Enqueued follow-up after raw input (${msg.followUpContent.length} chars)`); + } else { + const targetBackend = backend; + if (!targetBackend) { + pendingRawInputs.push(msg); + return; + } + await writeRawInput( + targetBackend, + undefined, + msg.followUpContent + ? () => { + sendToPty(msg.followUpContent!, msg.followUpTurnId, { + codexAppInput: msg.followUpCodexAppInput, + }); + log(`Enqueued follow-up after raw input (${msg.followUpContent!.length} chars)`); + } + : undefined, + ); } // A pending /rename may have been held by the command-write mutex. It still // waits for a genuine prompt because isPromptReady was cleared above. @@ -1263,19 +1899,6 @@ function publishSandboxRelayCapability(opts: { failClosed?: boolean } = {}): boo body: JSON.stringify({ token: capability.token }), }] : []), - ...(readIsolationOriginCapabilityFile && sessionId - ? [{ - path: readIsolationOriginCapabilityFile, - body: JSON.stringify({ - sessionId, - capability: capability.token, - ...(capability.turnId ? { turnId: capability.turnId } : {}), - ...(capability.dispatchAttempt !== undefined - ? { dispatchAttempt: capability.dispatchAttempt } - : {}), - }), - }] - : []), ]; let publishError: unknown; for (const file of files) { @@ -1309,6 +1932,9 @@ function publishSandboxRelayCapability(opts: { failClosed?: boolean } = {}): boo type: 'managed_turn_origin', sessionId, capability: capability.token, + ...(readIsolationOriginChannelId + ? { originChannelId: readIsolationOriginChannelId } + : {}), ...(capability.turnId ? { turnId: capability.turnId } : {}), ...(capability.dispatchAttempt !== undefined ? { dispatchAttempt: capability.dispatchAttempt } @@ -1319,34 +1945,21 @@ function publishSandboxRelayCapability(opts: { failClosed?: boolean } = {}): boo } function unlinkManagedOriginCapabilityFiles(): void { + // The Linux outbox belongs to this worker generation and can be removed. + // The macOS capability belongs to the persistent pane channel. A warm + // Node-worker reattach reuses that path, so stale worker teardown must never + // unlink the successor worker's freshly published token. Leaving old bytes + // is fail-closed against the daemon's exact live origin tuple. const files = [ sandboxRelayOutbox ? join(sandboxRelayOutbox, RELAY_ORIGIN_CAPABILITY_BASENAME) : undefined, - readIsolationOriginCapabilityFile ?? undefined, ]; for (const file of new Set(files.filter((p): p is string => !!p))) { try { unlinkSync(file); } catch { /* absent or teardown racing */ } } } -/** Read a host-owned isolation marker without following a child-planted - * symlink between lookup and open. */ -function readRegularHostFileNoFollow(filePath: string): string | null { - let fd: number | undefined; - try { - fd = openSync(filePath, fsConstants.O_RDONLY | (fsConstants.O_NOFOLLOW ?? 0)); - if (!fstatSync(fd).isFile()) return null; - return readFileSync(fd, 'utf8'); - } catch { - return null; - } finally { - if (fd !== undefined) { - try { closeSync(fd); } catch { /* best-effort */ } - } - } -} - function completeManagedTurnOriginRevocation( revoked: typeof sandboxRelayCapability, turnId: string | undefined, @@ -1362,6 +1975,9 @@ function completeManagedTurnOriginRevocation( type: 'managed_turn_origin_revoked', sessionId, ...(revoked ? { capability: revoked.token } : {}), + ...(readIsolationOriginChannelId + ? { originChannelId: readIsolationOriginChannelId } + : {}), ...(turnId ? { turnId } : {}), ...(dispatchAttempt !== undefined ? { dispatchAttempt } : {}), }); @@ -1402,7 +2018,14 @@ function revokeManagedTurnOriginForTerminal( } function authorizeManagedSend( claim: { capability?: string }, -): { ok: true; origin: { turnId?: string; dispatchAttempt?: number } } | { ok: false; error: string } { +): { + ok: true; + origin: { + turnId?: string; + dispatchAttempt?: number; + requiresCodexAppLedger?: boolean; + }; +} | { ok: false; error: string } { if (!sessionId) return { ok: false, error: 'VC policy cannot resolve session id' }; const dataDir = process.env.SESSION_DATA_DIR; if (!dataDir) return { ok: false, error: 'VC policy cannot resolve session data' }; @@ -1414,7 +2037,22 @@ function authorizeManagedSend( if (!capability || !claim.capability || claim.capability !== capability.token) { return { ok: false, error: 'origin_mismatch: relay capability is stale or missing' }; } - const session = sessionStore.getSession(sessionId); + // The daemon owns and rotates this ledger in another process. Never consult + // SessionStore's worker-local cache here: it was loaded at init and would + // stale-authorize turn N or reject turn N+1 after daemon settlement. + const session = sessionStore.getSessionFresh(sessionId); + const ledger = session?.codexAppDispatchLedger ?? []; + const codexAppManagedOrigin = lastInitConfig?.cliId === 'codex-app' + || session?.cliId === 'codex-app'; + const codexLedgerDecision = validateCodexAppManagedSendOrigin( + ledger, + capability, + codexAppManagedOrigin, + ); + if (!codexLedgerDecision.ok) { + return { ok: false, error: `origin_mismatch: ${codexLedgerDecision.error}` }; + } + const requiresCodexAppLedger = codexLedgerDecision.requiresLedger; const currentImOrigin = currentVcMeetingImTurnOrigin; const imOrigin = currentImOrigin?.larkMessageId === capability.turnId && currentImOrigin?.receiverSessionId === sessionId @@ -1435,29 +2073,33 @@ function authorizeManagedSend( ...(capability.dispatchAttempt !== undefined ? { dispatchAttempt: capability.dispatchAttempt } : {}), + ...(requiresCodexAppLedger ? { requiresCodexAppLedger: true } : {}), }, } : { ok: false, error: `${decision.errorCode}: ${decision.error}` }; } -function clearRpcEnginePidMarker(): void { +function clearRpcEnginePidMarker(expectedMarker?: string): void { + if (expectedMarker !== undefined && rpcEnginePidMarker !== expectedMarker) return; if (!rpcEnginePidMarker) return; try { unlinkSync(rpcEnginePidMarker); } catch { /* already gone */ } rpcEnginePidMarker = null; } -function registerRpcEnginePidMarker(pid: number | undefined): void { +function registerRpcEnginePidMarker(pid: number | undefined): string | null { clearRpcEnginePidMarker(); - if (!pid || !process.env.SESSION_DATA_DIR) return; + if (!pid || !process.env.SESSION_DATA_DIR) return null; try { const markersDir = join(process.env.SESSION_DATA_DIR, '.botmux-cli-pids'); mkdirSync(markersDir, { recursive: true }); rpcEnginePidMarker = join(markersDir, String(pid)); writeCliPidMarker(); log(`Codex RPC app-server PID marker written: ${pid}`); + return rpcEnginePidMarker; } catch (err: any) { rpcEnginePidMarker = null; log(`Failed to write Codex RPC app-server PID marker: ${err?.message ?? err}`); + return null; } } @@ -1483,77 +2125,1039 @@ function writeCliPidMarker(): void { } } let lastStructuredBridgeActivityAtMs = 0; +const codexAppTurnLiveness = new CodexAppTurnLiveness(); +const codexAppReadyAuthority = new CodexAppReadyAuthority(); +const codexAppTurnDispatchQueue = new CodexAppTurnDispatchQueue(); +let codexAppFallbackTurnSequence = 0; +let codexAppRecoveredDispatches: CodexAppDispatchLedgerEntry[] = []; +let codexAppGenerationCommits: CodexAppGenerationCommit[] = []; +const codexAppPendingDaemonAcks = new Map void; + timer: NodeJS.Timeout; +}>(); +let codexAppCompletionAwaitingFinal = false; +let codexAppControlBootstrapPathForSpawn: string | undefined; +let codexAppControlStateValue: CodexAppControlState | undefined; +let codexAppControlProven = false; +/** Authentication proves identity, not app-server readiness. These become true + * only after the active generation publishes a valid signed state marker. */ +let codexAppSignedStateObserved = false; +let codexAppInputReady = false; +let codexAppControlFatal = false; +let codexAppControlPersistentGeneration = false; +let codexAppControlDirectoryForSpawn: string | undefined; +let codexAppControlSocketPathValue: string | undefined; +let codexAppControlSocketDirectory: string | undefined; +let codexAppControlLocatorPathValue: string | undefined; +let codexAppControlEndpointEpoch: string | undefined; +let codexAppControlServer: NetServer | undefined; +let codexAppWindowsOwnerLeaseServer: NetServer | undefined; +let codexAppWindowsOwnerLeaseSessionId: string | undefined; +let codexAppWindowsOwnerLeasePromise: Promise | undefined; +let codexAppPosixOwnerLease: CodexAppPosixOwnerLease | undefined; +let codexAppPosixOwnerLeaseSessionId: string | undefined; +let codexAppPosixOwnerLeasePromise: Promise | undefined; +let codexAppControlActiveSocket: Socket | undefined; +let codexAppControlActiveIdentity: CodexAppControlIdentity | undefined; +let codexAppFreshCandidateGeneration: string | undefined; +let codexAppUnprovenPromptDeferred = false; +let codexAppRejectedControlLogged = false; +let codexAppMalformedControlLogged = false; +let codexAppBootstrapCleanupTimer: NodeJS.Timeout | undefined; +const codexAppProofDeadline = new CodexAppControlProofDeadline(); +let codexAppControlStopping = false; +let codexAppControlChannelId = 0; +let codexAppControlRotation: Promise | undefined; +const CODEX_APP_CONTROL_ENDPOINT_RETRY_MS = 5_000; + +interface CodexAppControlConnection { + socket: Socket; + endpoint: string; + epoch: string; + channelId: number; + challenge: string; + decoder: CodexAppControlLineDecoder; + sequenceFence: CodexAppControlSequenceFence; + finalAssembler: CodexAppControlFinalAssembler; + authenticated: boolean; + pendingLines: string[]; + processingLines: boolean; + identity?: CodexAppControlIdentity; + authTimer: NodeJS.Timeout; +} +const codexAppControlConnections = new Map(); +const codexAppControlReplayWindow = new CodexAppControlReplayWindow(); +const codexAppControlRecordApplicationGate = new CodexAppControlRecordApplicationGate(); + +function cleanupCodexAppControlBootstrap(): void { + if (codexAppBootstrapCleanupTimer) { + clearTimeout(codexAppBootstrapCleanupTimer); + codexAppBootstrapCleanupTimer = undefined; + } + const path = codexAppControlBootstrapPathForSpawn; + codexAppControlBootstrapPathForSpawn = undefined; + if (!path) return; + try { unlinkSync(path); } catch { /* runner normally unlinks it first */ } +} -type RuntimeScreenStatus = Exclude; - -// Per-turn usage-limit state machine. Owns the turn counter plus the -// "did this turn hit a limit" / "suppress a stale retry-ready banner" flags, so -// classify()'s state writes are explicit method calls rather than hidden -// mutations of module globals from a function that otherwise reads as a pure -// mapper. -function createUsageLimitTracker() { - let turnSeq = 0; - let detectedTurn: number | undefined; - let suppressedRetryReadyKey: string | undefined; - - return { - currentTurn(): number { - return turnSeq; - }, - // Open a new turn; remember any stale retry-ready banner still on screen so - // classify() doesn't re-flag it as a fresh limit this turn. - beginTurn(snapshot: string): number { - turnSeq++; - detectedTurn = undefined; - const current = detectCliUsageLimit(snapshot); - suppressedRetryReadyKey = current.limited && current.retryReady - ? usageLimitStateKey(current) - : undefined; - return turnSeq; - }, - // Map a runtime status to a usage-limit-aware status, recording whether this - // turn hit a limit (read back via detectedThisTurn). - classify( - content: string, - status: RuntimeScreenStatus, - ): { status: RuntimeScreenStatus | 'limited'; usageLimit?: CliUsageLimitState } { - const detected = detectCliUsageLimit(content); - if (!detected.limited) return { status }; - - const key = usageLimitStateKey(detected); - if (detected.retryReady && key === suppressedRetryReadyKey) { - return { status }; - } - - suppressedRetryReadyKey = undefined; - detectedTurn = turnSeq; - return { status: 'limited', usageLimit: detected }; - }, - detectedThisTurn(seq: number): boolean { - return detectedTurn === seq; - }, - }; +function readPersistedCodexAppControlState( + cfg: Extract, +): CodexAppControlState | undefined { + const dataDir = process.env.SESSION_DATA_DIR; + if (!dataDir && process.platform !== 'win32') return undefined; + const statePath = codexAppControlStatePath(dataDir ?? '', cfg.sessionId); + if (process.platform === 'win32') { + ensureCodexAppControlDirectory(codexAppWindowsControlRoot()); + ensureCodexAppControlDirectory(dirname(statePath)); + if (existsSync(statePath)) hardenWindowsCodexAppControlFile(statePath); + } + return readCodexAppControlState(statePath); } -const usageLimitTracker = createUsageLimitTracker(); +function persistCodexAppControlState( + cfg: Extract, + state: CodexAppControlState, +): void { + const dataDir = process.env.SESSION_DATA_DIR; + if (!dataDir && process.platform !== 'win32') { + throw new Error('SESSION_DATA_DIR is required for persistent Codex App control state'); + } + const statePath = codexAppControlStatePath(dataDir ?? '', cfg.sessionId); + if (process.platform === 'win32') { + ensureCodexAppControlDirectory(codexAppWindowsControlRoot()); + ensureCodexAppControlDirectory(dirname(statePath)); + } + writeCodexAppControlState(statePath, state); +} -function currentUsageLimitSnapshot(): string { - return lastAnalyzerSnapshot || renderer?.rawSnapshot() || ''; +function stopCodexAppControlChannel( + opts: { preserveDispatchRecovery?: boolean } = {}, +): void { + codexAppControlStopping = true; + // Attribution belongs to exactly one worker/control generation. A fresh or + // replacement worker may recover only the daemon-frozen active identity + // after the old runner proves a warm reattach; stale queued entries must not + // cross a stop/restart boundary. + if (!opts.preserveDispatchRecovery) { + codexAppTurnDispatchQueue.clear(); + codexAppRecoveredDispatches = []; + } + for (const pending of codexAppPendingDaemonAcks.values()) { + clearTimeout(pending.timer); + pending.resolve(false); + } + codexAppPendingDaemonAcks.clear(); + codexAppControlChannelId++; + codexAppControlRotation = undefined; + codexAppProofDeadline.clear(); + codexAppSignedStateObserved = false; + codexAppInputReady = false; + for (const connection of codexAppControlConnections.values()) { + clearTimeout(connection.authTimer); + connection.socket.destroy(); + } + codexAppControlConnections.clear(); + codexAppControlActiveSocket = undefined; + codexAppControlActiveIdentity = undefined; + const server = codexAppControlServer; + codexAppControlServer = undefined; + try { server?.close(); } catch { /* worker exit/restart */ } + const socketPath = codexAppControlSocketPathValue; + // Named pipes are kernel objects, not filesystem entries. Never lstat, + // chmod, or unlink them on Windows; closing the server retires the endpoint. + if (socketPath && process.platform !== 'win32') { + try { + const stat = lstatSync(socketPath); + const uid = process.geteuid?.() ?? process.getuid?.(); + if (stat.isSocket() && !stat.isSymbolicLink() && (uid === undefined || stat.uid === uid)) { + unlinkSync(socketPath); + } + } catch { /* absent or already removed */ } + } + // Do not read-then-unlink the fixed locator here. That is not an + // atomic compare-and-delete: a replacement process could publish a new epoch + // between those operations. A stale locator is harmless (the random pipe is + // closed and the runner validates its independent epoch) and the next owner + // overwrites it atomically after binding a fresh endpoint. + codexAppControlEndpointEpoch = undefined; +} + +function failCodexAppControlGeneration(reason: string): void { + if (codexAppControlFatal) return; + codexAppControlFatal = true; + log(`Codex App control generation failed closed: ${reason}`); + codexAppControlProven = false; + codexAppSignedStateObserved = false; + codexAppInputReady = false; + codexAppTurnLiveness.clear(); + codexAppReadyAuthority.reset(); + cleanupCodexAppControlBootstrap(); + stopCodexAppControlChannel(); + const cfg = lastInitConfig; + if (cfg && effectiveBackendType !== 'pty') { + const name = effectiveBackendType === 'tmux' + ? TmuxBackend.sessionName(cfg.sessionId) + : effectiveBackendType === 'zellij' + ? ZellijBackend.sessionName(cfg.sessionId) + : effectiveBackendType === 'herdr' + ? HerdrBackend.sessionName(cfg.sessionId) + : undefined; + if (name) { + try { killPersistentSession(effectiveBackendType as PersistentBackendType, name); } + catch (err: any) { log(`Failed to kill rejected Codex App generation: ${err?.message ?? err}`); } + } + } + try { backend?.kill(); } catch { /* process exit is the final fail-close */ } + backend = null; + queueMicrotask(() => { + void sendFatalWorkerErrorAndExit( + new Error(reason), + undefined, + undefined, + { hardExit: true }, + ); + }); } -// ─── Adopt-bridge state (Claude Code only) ───────────────────────────────── -// -// In bridge mode the daemon adopted an existing CLI session that we do NOT -// own; the model never sees botmux. We harvest assistant turns by tailing -// Claude Code's transcript JSONL and forward only the bytes appended after -// each Lark-driven user turn — never the historical content present at -// attach time, never local-terminal-driven turns. -// -// Attribution lives in BridgeTurnQueue; this file only manages the -// fs.watch wakeup, byte-offset bookkeeping, lazy baseline, and IPC emit. -let bridgeJsonlPath: string | undefined; -/** Directory enclosing bridgeJsonlPath. We poll this dir for newer jsonl - * files so the bridge follows `/clear` / `/resume` in the user's CLI — +function activateCodexAppControlConnection( + connection: CodexAppControlConnection, + identity: CodexAppControlIdentity, +): void { + const cfg = lastInitConfig; + const state = codexAppControlStateValue; + if (!cfg || !state) { + connection.socket.destroy(); + return; + } + const proofKind = identity.generation === codexAppFreshCandidateGeneration + ? 'fresh runner' + : 'warm reattach'; + if (proofKind === 'fresh runner' + && codexAppRecoveredDispatches.some(entry => entry.state === 'prepared')) { + // A fresh process cannot prove whether the prior generation buffered a + // prepared frame. Fail before persisting/announcing this generation or + // ACKing authentication; otherwise the daemon may trim recovery state and + // observers can briefly treat an unusable runner as active. + connection.socket.destroy(); + failCodexAppControlGeneration( + 'Fresh Codex App runner cannot adopt a recovered prepared dispatch', + ); + return; + } + let active: CodexAppControlState; + try { + active = activateCodexAppControlIdentity(state, identity.generation); + if (codexAppControlPersistentGeneration) persistCodexAppControlState(cfg, active); + } catch (err: any) { + failCodexAppControlGeneration(`Could not persist authenticated Codex App generation: ${err?.message ?? err}`); + return; + } + codexAppControlStateValue = active; + codexAppControlProven = true; + codexAppSignedStateObserved = false; + codexAppInputReady = false; + if (!codexAppProofDeadline.armed) { + codexAppProofDeadline.arm(() => { + failCodexAppControlGeneration( + `Authenticated Codex App runner did not publish signed state within ${CODEX_APP_CONTROL_STARTUP_TIMEOUT_MS / 1000} seconds`, + ); + }); + } + codexAppControlActiveIdentity = identity; + codexAppControlReplayWindow.retainOnly(identity.generation); + connection.authenticated = true; + connection.identity = identity; + clearTimeout(connection.authTimer); + const previous = codexAppControlActiveSocket; + codexAppControlActiveSocket = connection.socket; + if (previous && previous !== connection.socket) previous.destroy(); + cleanupCodexAppControlBootstrap(); + connection.socket.write(`${encodeCodexAppControlAccepted( + cfg.sessionId, + identity.generation, + connection.challenge, + connection.epoch, + )}\n`); + log(`Authenticated Codex App ${proofKind} by Ed25519 challenge proof (generation=${identity.generation.slice(0, 8)})`); + send({ + type: 'codex_app_generation_active', + sessionId: cfg.sessionId, + generation: identity.generation, + fresh: proofKind === 'fresh runner', + }); + if (proofKind === 'fresh runner') { + codexAppGenerationCommits = codexAppGenerationCommits.filter( + commit => commit.generation === identity.generation, + ); + } + if (proofKind === 'warm reattach') { + codexAppTurnLiveness.beginReattachObservation(); + } + // Authentication is not an input-ready barrier. A warm runner may still + // replay old final/state records immediately after auth. Only the later + // signed idle record, after recovered-prefix reconciliation, may flush. +} + +async function handleCodexAppControlLine( + connection: CodexAppControlConnection, + line: string, +): Promise { + const record = parseCodexAppControlWireRecord(line); + const cfg = lastInitConfig; + if (!record || !cfg || record.sessionId !== cfg.sessionId + || connection.channelId !== codexAppControlChannelId + || connection.epoch !== codexAppControlEndpointEpoch) { + rejectCodexAppControlMarker('malformed socket'); + connection.socket.destroy(); + return; + } + if (!connection.authenticated) { + if (record.type !== 'auth' || record.challenge !== connection.challenge) { + rejectCodexAppControlMarker(record.type); + connection.socket.destroy(); + return; + } + const identity = authenticateCodexAppControlCandidate({ + state: codexAppControlStateValue, + auth: record, + sessionId: cfg.sessionId, + challenge: connection.challenge, + }); + if (!identity) { + rejectCodexAppControlMarker('challenge response'); + connection.socket.destroy(); + return; + } + activateCodexAppControlConnection(connection, identity); + return; + } + const identity = connection.identity; + if (record.type !== 'marker' || !identity + || connection.socket !== codexAppControlActiveSocket + || record.generation !== identity.generation + || record.challenge !== connection.challenge + || !verifyCodexAppSignedControlMarker(record, identity.publicKey)) { + rejectCodexAppControlMarker(record.type === 'marker' ? record.kind : record.type); + connection.socket.destroy(); + return; + } + if (!connection.sequenceFence.accept(record.seq)) { + rejectCodexAppControlMarker('non-contiguous signed sequence'); + connection.socket.destroy(); + return; + } + // A replacement worker may see the runner replay a complete final whose + // daemon-side delivery and FIFO pop were committed just before the old + // worker died. The per-generation high-water mark is a durable cumulative + // ACK boundary: acknowledge each replayed prefix record without assembling + // or re-emitting the final. + if (committedCodexAppSequence(codexAppGenerationCommits, identity.generation, record.seq)) { + connection.socket.write(`${encodeCodexAppControlAck( + cfg.sessionId, + identity.generation, + connection.challenge, + record.seq, + )}\n`); + return; + } + // ACK loss is expected around endpoint replacement. The runner re-signs its + // unacknowledged record against the fresh challenge; acknowledge that retry + // without applying completed/final side effects twice. + if (codexAppControlReplayWindow.hasSeen(identity.generation, record.seq)) { + connection.socket.write(`${encodeCodexAppControlAck( + cfg.sessionId, + identity.generation, + connection.challenge, + record.seq, + )}\n`); + return; + } + const finalResult = connection.finalAssembler.accept(record.kind, record.payload); + if (finalResult.status === 'reject') { + rejectCodexAppControlMarker(finalResult.reason); + connection.socket.destroy(); + return; + } + // start/chunk records intentionally remain uncommitted so a replacement + // connection can rebuild the complete final. Every record that does apply + // worker effects is single-flight by signed generation/sequence until the + // replay window below is committed. + if (finalResult.status === 'accepted') return; + const application = codexAppControlRecordApplicationGate.run( + identity.generation, + record.seq, + () => finalResult.status === 'not-final' + ? handleTrustedCodexAppMarker( + record.kind, + record.payload, + { generation: identity.generation, seq: record.seq }, + ) + : handleTrustedCodexAppMarker( + 'final', + finalResult.payload, + { generation: identity.generation, seq: record.seq }, + ), + ); + let applied: boolean; + try { + applied = await application; + } catch (err) { + codexAppControlRecordApplicationGate.release( + identity.generation, + record.seq, + application, + ); + throw err; + } + if (!applied) { + codexAppControlRecordApplicationGate.release( + identity.generation, + record.seq, + application, + ); + // Do not commit or cumulatively ACK a semantically rejected signed marker. + // In particular, a final for the wrong FIFO head must remain replayable + // rather than becoming a permanently lost answer. + connection.socket.destroy(); + return; + } + codexAppControlReplayWindow.commit(identity.generation, record.seq); + codexAppControlRecordApplicationGate.release( + identity.generation, + record.seq, + application, + ); + if (!connection.socket.destroyed) { + connection.socket.write(`${encodeCodexAppControlAck( + cfg.sessionId, + identity.generation, + connection.challenge, + record.seq, + )}\n`); + } +} + +const CODEX_APP_CONTROL_PENDING_LINE_LIMIT = 4_096; + +async function drainCodexAppControlLines(connection: CodexAppControlConnection): Promise { + if (connection.processingLines) return; + connection.processingLines = true; + connection.socket.pause(); + try { + while (!connection.socket.destroyed && connection.pendingLines.length > 0) { + const line = connection.pendingLines.shift()!; + await handleCodexAppControlLine(connection, line); + } + } catch (err: any) { + log(`Codex App control record processing failed: ${err?.message ?? err}`); + connection.socket.destroy(); + } finally { + connection.processingLines = false; + if (!connection.socket.destroyed) connection.socket.resume(); + } +} + +function acceptCodexAppControlSocket( + socket: Socket, + endpoint: string, + epoch: string, + channelId: number, +): void { + if (codexAppControlFatal || codexAppControlStopping || !lastInitConfig + || channelId !== codexAppControlChannelId) { + socket.destroy(); + return; + } + // Never let a stalled unauthenticated client monopolize the endpoint. Every + // connection gets its own challenge and timeout; new accepts continue while + // bad same-UID clients are rejected independently. + const unauthenticated = [...codexAppControlConnections.values()] + .filter(connection => !connection.authenticated); + if (unauthenticated.length >= 16) { + socket.destroy(); + return; + } + const challenge = generateCodexAppControlChallenge(); + const connection: CodexAppControlConnection = { + socket, + endpoint, + epoch, + channelId, + challenge, + decoder: new CodexAppControlLineDecoder(), + sequenceFence: new CodexAppControlSequenceFence(), + finalAssembler: new CodexAppControlFinalAssembler(), + authenticated: false, + pendingLines: [], + processingLines: false, + authTimer: setTimeout(() => socket.destroy(), CODEX_APP_CONTROL_ENDPOINT_RETRY_MS), + }; + connection.authTimer.unref?.(); + codexAppControlConnections.set(socket, connection); + socket.setNoDelay(true); + socket.on('data', chunk => { + const decoded = connection.decoder.push(chunk); + if (decoded.droppedMalformed) { + if (!codexAppMalformedControlLogged) { + codexAppMalformedControlLogged = true; + log('Dropped malformed/oversized Codex App socket control line'); + } + socket.destroy(); + return; + } + if (connection.pendingLines.length + decoded.lines.length > CODEX_APP_CONTROL_PENDING_LINE_LIMIT) { + log('Dropped Codex App control connection whose pending line queue exceeded the bound'); + socket.destroy(); + return; + } + connection.pendingLines.push(...decoded.lines); + void drainCodexAppControlLines(connection); + }); + socket.on('error', () => { /* close performs local cleanup */ }); + socket.on('close', () => { + clearTimeout(connection.authTimer); + codexAppControlConnections.delete(socket); + connection.pendingLines.length = 0; + const wasActive = codexAppControlActiveSocket === socket; + if (wasActive) { + codexAppControlActiveSocket = undefined; + codexAppControlActiveIdentity = undefined; + codexAppControlProven = false; + codexAppSignedStateObserved = false; + codexAppInputReady = false; + codexAppReadyAuthority.beginWork(); + codexAppTurnLiveness.beginReattachObservation(); + if (!codexAppControlStopping + && !codexAppControlFatal + && connection.channelId === codexAppControlChannelId) { + codexAppProofDeadline.arm(() => { + failCodexAppControlGeneration( + `Codex App runner did not re-authenticate within ${CODEX_APP_CONTROL_STARTUP_TIMEOUT_MS / 1000} seconds after its control socket closed`, + ); + }); + } + log('Codex App authenticated control socket closed; waiting for a fresh challenge proof'); + } + // Rotate only after the authenticated runner closes. An unauthenticated + // close cannot prove that the legitimate runner has no pending connect to + // this name; retiring it could let another local process rebind that name + // under the pending attempt. Pre-auth failures therefore remain bound until + // the shared 90-second fail-close deadline (availability-only boundary). + if (wasActive + && !codexAppControlStopping + && !codexAppControlFatal + && connection.channelId === codexAppControlChannelId + && connection.epoch === codexAppControlEndpointEpoch) { + void rotateCodexAppControlEndpoint('active socket closed'); + } + }); + socket.write(`${encodeCodexAppControlChallenge(lastInitConfig.sessionId, challenge)}\n`); +} + +function removeStaleCodexAppSocket(path: string): void { + if (process.platform === 'win32') return; + try { + const stat = lstatSync(path); + const uid = process.geteuid?.() ?? process.getuid?.(); + if (!stat.isSocket() || stat.isSymbolicLink() || (uid !== undefined && stat.uid !== uid)) { + throw new Error('existing Codex App control socket path is not an owned socket'); + } + unlinkSync(path); + } catch (err: any) { + if (err?.code !== 'ENOENT') throw err; + } +} + +function listenCodexAppControlServer(server: NetServer, endpoint: string): Promise { + return new Promise((resolve, reject) => { + const onError = (err: Error) => { + server.off('listening', onListening); + reject(err); + }; + const onListening = () => { + server.off('error', onError); + resolve(); + }; + server.once('error', onError); + server.once('listening', onListening); + server.listen(endpoint); + }); +} + +async function ensureCodexAppWindowsOwnerLease(sessionId: string): Promise { + if (process.platform !== 'win32') return; + if (codexAppWindowsOwnerLeaseServer) { + if (codexAppWindowsOwnerLeaseSessionId !== sessionId) { + throw new Error('Windows Codex App owner lease belongs to another session'); + } + return; + } + if (codexAppWindowsOwnerLeasePromise) return codexAppWindowsOwnerLeasePromise; + const endpoint = codexAppWindowsOwnerPipeEndpoint(sessionId); + let acquisition!: Promise; + acquisition = (async () => { + const server = await acquireCodexAppControlOwnerLease({ + bind: async () => { + const candidate = createNetServer(socket => socket.destroy()); + try { + await listenCodexAppControlServer(candidate, endpoint); + return candidate; + } catch (err) { + try { candidate.close(); } catch { /* bind never completed */ } + throw err; + } + }, + }); + if (codexAppWindowsOwnerLeaseServer) { + try { server.close(); } catch { /* another local acquire won */ } + if (codexAppWindowsOwnerLeaseSessionId !== sessionId) { + throw new Error('Windows Codex App owner lease changed session during acquisition'); + } + return; + } + codexAppWindowsOwnerLeaseServer = server; + codexAppWindowsOwnerLeaseSessionId = sessionId; + server.on('error', err => { + if (codexAppWindowsOwnerLeaseServer === server && !codexAppControlStopping) { + failCodexAppControlGeneration(`Windows Codex App owner lease failed: ${err.message}`); + } + }); + })().finally(() => { + if (codexAppWindowsOwnerLeasePromise === acquisition) { + codexAppWindowsOwnerLeasePromise = undefined; + } + }); + codexAppWindowsOwnerLeasePromise = acquisition; + return acquisition; +} + +async function ensureCodexAppPosixOwnerLease( + controlRoot: string, + ownerSessionId: string, +): Promise { + if (process.platform === 'win32') return; + if (codexAppPosixOwnerLease) { + if (codexAppPosixOwnerLeaseSessionId !== ownerSessionId || !codexAppPosixOwnerLease.isOwned()) { + throw new Error('POSIX Codex App owner lease is not owned by this worker/session'); + } + return; + } + if (codexAppPosixOwnerLeasePromise) return codexAppPosixOwnerLeasePromise; + let acquisition!: Promise; + acquisition = (async () => { + const lease = await acquireCodexAppPosixOwnerLease({ + controlRoot, + sessionId: ownerSessionId, + }); + if (codexAppPosixOwnerLease) { + lease.release(); + if (codexAppPosixOwnerLeaseSessionId !== ownerSessionId) { + throw new Error('POSIX Codex App owner lease changed session during acquisition'); + } + return; + } + codexAppPosixOwnerLease = lease; + codexAppPosixOwnerLeaseSessionId = ownerSessionId; + })().finally(() => { + if (codexAppPosixOwnerLeasePromise === acquisition) { + codexAppPosixOwnerLeasePromise = undefined; + } + }); + codexAppPosixOwnerLeasePromise = acquisition; + return acquisition; +} + +function releaseCodexAppPosixOwnerLease(): void { + codexAppPosixOwnerLease?.release(); + codexAppPosixOwnerLease = undefined; + codexAppPosixOwnerLeaseSessionId = undefined; +} + +interface StartedCodexAppControlEndpoint { + server: NetServer; + endpoint: string; + epoch: string; +} + +async function startCodexAppControlEndpoint( + cfg: Extract, + channelId: number, +): Promise { + const epoch = generateCodexAppControlEpoch(); + const endpoint = process.platform === 'win32' + ? generateCodexAppWindowsPipeEndpoint() + : codexAppControlSocketDirectory + ? generateCodexAppPosixSocketEndpoint(codexAppControlSocketDirectory) + : undefined; + if (!endpoint) throw new Error('Codex App control endpoint path was not prepared'); + const locatorPath = codexAppControlLocatorPathValue; + if (!locatorPath) throw new Error('Codex App control locator path was not prepared'); + const server = createNetServer(socket => { + acceptCodexAppControlSocket(socket, endpoint, epoch, channelId); + }); + const priorServer = codexAppControlServer; + const priorEndpoint = codexAppControlSocketPathValue; + const priorEpoch = codexAppControlEndpointEpoch; + try { + const publisherLeaseOwned = (): boolean => process.platform === 'win32' + ? !!codexAppWindowsOwnerLeaseServer + : codexAppPosixOwnerLease?.isOwned() === true; + const published = await bindThenPublishCodexAppControlLocator({ + sessionId: cfg.sessionId, + epoch, + endpoint, + platform: process.platform, + locatorPath, + expectedControlRoot: process.platform === 'win32' + ? undefined + : codexAppPosixControlRoot(), + listen: async () => { + await listenCodexAppControlServer(server, endpoint); + if (process.platform !== 'win32') { + chmodSync(endpoint, 0o600); + const stat = lstatSync(endpoint); + const uid = process.geteuid?.() ?? process.getuid?.(); + if (!stat.isSocket() || stat.isSymbolicLink() || (uid !== undefined && stat.uid !== uid)) { + throw new Error('listening path is not the owned Codex App socket'); + } + } + }, + isCurrent: () => !codexAppControlStopping + && !codexAppControlFatal + && channelId === codexAppControlChannelId + && publisherLeaseOwned(), + retire: () => { try { server.close(); } catch { /* already retired */ } }, + publish: locator => { + if (!publisherLeaseOwned()) throw new Error('Codex App locator publisher lease was lost'); + // Install the bound epoch before making the locator visible. Any + // immediate connection is therefore checked against this exact epoch. + codexAppControlServer = server; + codexAppControlSocketPathValue = endpoint; + codexAppControlEndpointEpoch = epoch; + writeCodexAppControlLocator(locatorPath, locator); + }, + }); + if (!published) throw new Error('Codex App control endpoint was retired before locator publication'); + } catch (err) { + if (codexAppControlServer === server) { + codexAppControlServer = priorServer; + codexAppControlSocketPathValue = priorEndpoint; + codexAppControlEndpointEpoch = priorEpoch; + } + try { server.close(); } catch { /* bind may not have completed */ } + if (process.platform !== 'win32') { + try { removeStaleCodexAppSocket(endpoint); } catch { /* preserve original error */ } + } + throw err; + } + server.on('error', err => { + if (codexAppControlServer === server + && channelId === codexAppControlChannelId + && !codexAppControlStopping) { + failCodexAppControlGeneration(`Codex App control endpoint failed: ${err.message}`); + } + }); + return { server, endpoint, epoch }; +} + +function installCodexAppControlEndpoint(started: StartedCodexAppControlEndpoint): NetServer | undefined { + const previous = codexAppControlServer; + codexAppControlServer = started.server; + codexAppControlSocketPathValue = started.endpoint; + codexAppControlEndpointEpoch = started.epoch; + return previous; +} + +function closeRetiredCodexAppControlEndpoint( + server: NetServer | undefined, + endpoint: string | undefined, +): void { + try { server?.close(); } catch { /* already closed */ } + if (endpoint && process.platform !== 'win32') { + try { removeStaleCodexAppSocket(endpoint); } catch { /* next bind will verify */ } + } +} + +function rotateCodexAppControlEndpoint(reason: string): Promise { + if (codexAppControlStopping || codexAppControlFatal) { + return Promise.resolve(); + } + if (codexAppControlRotation) return codexAppControlRotation; + const cfg = lastInitConfig; + const channelId = codexAppControlChannelId; + if (!cfg) return Promise.resolve(); + const oldServer = codexAppControlServer; + const oldEndpoint = codexAppControlSocketPathValue; + let rotation!: Promise; + rotation = (async () => { + try { + const started = await startCodexAppControlEndpoint(cfg, channelId); + if (codexAppControlStopping || codexAppControlFatal + || channelId !== codexAppControlChannelId) { + closeRetiredCodexAppControlEndpoint(started.server, started.endpoint); + return; + } + installCodexAppControlEndpoint(started); + // Bind + publish the fresh random endpoint before retiring the old one. + closeRetiredCodexAppControlEndpoint(oldServer, oldEndpoint); + for (const connection of [...codexAppControlConnections.values()]) { + if (connection.epoch !== started.epoch) connection.socket.destroy(); + } + log(`Rotated Codex App control endpoint (${reason}, epoch=${started.epoch.slice(0, 8)})`); + } catch (err: any) { + if (shouldFailCodexAppControlChannel({ + channelId, + currentChannelId: codexAppControlChannelId, + stopping: codexAppControlStopping, + })) { + failCodexAppControlGeneration(`Could not rotate Codex App control endpoint: ${err?.message ?? err}`); + } + } finally { + if (codexAppControlRotation === rotation) codexAppControlRotation = undefined; + } + })(); + codexAppControlRotation = rotation; + return rotation; +} + +async function prepareCodexAppControlGeneration( + cfg: Extract, + _willReattachPersistent: boolean, + persistentGeneration: boolean, +): Promise { + cleanupCodexAppControlBootstrap(); + // init already restored the daemon-owned FIFO before spawn. Retiring a + // previous/empty socket endpoint must not erase that recovery snapshot just + // before a warm runner replays its unacked final. + stopCodexAppControlChannel({ preserveDispatchRecovery: true }); + codexAppControlStopping = false; + const channelId = codexAppControlChannelId; + codexAppControlStateValue = undefined; + codexAppControlProven = false; + codexAppSignedStateObserved = false; + codexAppInputReady = false; + codexAppReadyAuthority.reset(); + codexAppControlFatal = false; + codexAppControlPersistentGeneration = false; + codexAppControlDirectoryForSpawn = undefined; + codexAppControlSocketPathValue = undefined; + codexAppControlSocketDirectory = undefined; + codexAppControlLocatorPathValue = undefined; + codexAppControlEndpointEpoch = undefined; + codexAppFreshCandidateGeneration = undefined; + codexAppUnprovenPromptDeferred = false; + codexAppRejectedControlLogged = false; + codexAppMalformedControlLogged = false; + if (cfg.cliId !== 'codex-app') return; + + const dataDir = process.env.SESSION_DATA_DIR; + const windowsRoot = process.platform === 'win32' ? codexAppWindowsControlRoot() : undefined; + const controlRoot = windowsRoot ?? codexAppPosixControlRoot(); + // The daemon's kill-then-fork replacement path can briefly overlap worker + // processes. Hold a process-lifetime per-session publisher lease before + // touching the fixed locator so an old process cannot publish after its + // replacement. The lease is never exposed as a runner control endpoint. + if (windowsRoot) await ensureCodexAppWindowsOwnerLease(cfg.sessionId); + else await ensureCodexAppPosixOwnerLease(controlRoot, cfg.sessionId); + const controlDirectory = windowsRoot + ? join(windowsRoot, 'bootstraps') + : dataDir + ? join(botHomePath(dirname(dataDir), cfg.larkAppId), 'codex-app-control-bootstrap') + : join(tmpdir(), `botmux-codex-app-control-${process.getuid?.() ?? 'unknown'}`); + // AF_UNIX paths are capped at roughly 104 bytes on macOS. Keep random POSIX + // endpoints in the short private control root; both platforms publish them + // through a fixed protected locator while the process-lifetime owner lease + // serializes replacement workers. + const socketDirectory = join(controlRoot, 'sockets'); + ensureCodexAppControlDirectory(socketDirectory); + ensureCodexAppControlDirectory(controlDirectory); + cleanupStaleCodexAppControlBootstraps(controlDirectory, cfg.sessionId); + codexAppControlDirectoryForSpawn = controlDirectory; + codexAppControlSocketDirectory = socketDirectory; + codexAppControlLocatorPathValue = codexAppControlLocatorPath(controlRoot, cfg.sessionId); + ensureCodexAppControlDirectory(dirname(codexAppControlLocatorPathValue)); + // A crashed worker may leave a stale locator. Keep it until the new random + // endpoint has bound, then atomically overwrite it. Deleting first would + // introduce a cross-process read/unlink race and is unnecessary because + // accepted carries the protected, independently random locator epoch. + // Preserve old public identities for every persistent spawn attempt, even + // when the existence probe predicted fresh. The backend may race between + // probe and spawn; the socket proof, not that prediction, selects old reuse + // versus the fresh candidate. + if (persistentGeneration) codexAppControlStateValue = readPersistedCodexAppControlState(cfg); + const started = await startCodexAppControlEndpoint(cfg, channelId); + if (codexAppControlStopping || channelId !== codexAppControlChannelId) { + closeRetiredCodexAppControlEndpoint(started.server, started.endpoint); + throw new Error('Codex App control endpoint was retired during startup'); + } + installCodexAppControlEndpoint(started); +} + +/** Late-create the only secret-bearing file immediately before backend.spawn. */ +function prepareFreshCodexAppControlBootstrap( + cfg: Extract, + persistentGeneration: boolean, +): void { + if (cfg.cliId !== 'codex-app') return; + if (!codexAppControlDirectoryForSpawn || !codexAppControlLocatorPathValue) { + throw new Error('Codex App control channel was not prepared'); + } + const bootstrap = createCodexAppControlBootstrap( + codexAppControlDirectoryForSpawn, + cfg.sessionId, + { kind: 'locator', locatorPath: codexAppControlLocatorPathValue }, + ); + codexAppControlBootstrapPathForSpawn = bootstrap.path; + codexAppFreshCandidateGeneration = bootstrap.identity.generation; + codexAppControlStateValue = mergeCodexAppControlCandidate( + codexAppControlStateValue, + bootstrap.identity, + ); + codexAppControlPersistentGeneration = persistentGeneration; + if (persistentGeneration) persistCodexAppControlState(cfg, codexAppControlStateValue); + codexAppBootstrapCleanupTimer = armCodexAppControlStartupTimeout(cleanupCodexAppControlBootstrap); + codexAppBootstrapCleanupTimer.unref?.(); +} + +function finalizeCodexAppControlGeneration( + cfg: Extract, + _actuallyReattached: boolean, + persistentGeneration: boolean, +): void { + if (cfg.cliId !== 'codex-app') return; + codexAppControlPersistentGeneration = persistentGeneration; + // A live runner can answer while synchronous backend.spawn is still + // returning. Authentication already persisted the active public identity; + // do not mistake that success for a missing pending bootstrap. + if (codexAppControlProven && codexAppControlStateValue?.status === 'active') { + cleanupCodexAppControlBootstrap(); + if (!codexAppSignedStateObserved && !codexAppProofDeadline.armed) { + codexAppProofDeadline.arm(() => { + failCodexAppControlGeneration( + `Authenticated Codex App runner did not publish signed state within ${CODEX_APP_CONTROL_STARTUP_TIMEOUT_MS / 1000} seconds`, + ); + }); + } + return; + } + if (codexAppControlStateValue?.status !== 'pending' + || !codexAppControlBootstrapPathForSpawn) { + cleanupCodexAppControlBootstrap(); + throw new Error('Codex App pending asymmetric control generation was not prepared'); + } + codexAppProofDeadline.arm(() => { + failCodexAppControlGeneration( + `Codex App runner did not authenticate and publish signed state within ${CODEX_APP_CONTROL_STARTUP_TIMEOUT_MS / 1000} seconds`, + ); + }); +} + +function rejectCodexAppControlMarker(kind: string): void { + if (codexAppRejectedControlLogged) return; + codexAppRejectedControlLogged = true; + log(`Ignored unauthenticated Codex App ${kind} control record`); +} + +type RuntimeScreenStatus = Exclude; + +/** + * Project an explicit Codex App no-progress state above the screen heuristic. + * The warning is once-per-turn and intentionally does not restart or replay + * anything: both actions could duplicate model/tool side effects. + */ +function codexAppLivenessStatus(base: RuntimeScreenStatus, nowMs = Date.now()): RuntimeScreenStatus { + if (lastInitConfig?.cliId !== 'codex-app') return base; + base = projectCodexAppControlReadinessStatus(base, { + controlProven: codexAppControlProven, + signedStateObserved: codexAppSignedStateObserved, + inputReady: codexAppInputReady, + }); + const liveness = codexAppTurnLiveness.poll(nowMs); + if (liveness.shouldNotify) { + send({ + type: 'user_notify', + turnId: liveness.turnId ?? currentBotmuxTurnId, + ...(liveness.turnId === currentBotmuxTurnId + && currentBotmuxDispatchAttempt !== undefined + ? { dispatchAttempt: currentBotmuxDispatchAttempt } + : {}), + message: t('worker.codex_app.no_progress', { + seconds: Math.round(CODEX_APP_NO_PROGRESS_TIMEOUT_MS / 1000), + }), + }); + } + if (liveness.stalled) return 'stalled'; + return liveness.active && base === 'idle' ? 'working' : base; +} + +// Per-turn usage-limit state machine. Owns the turn counter plus the +// "did this turn hit a limit" / "suppress a stale retry-ready banner" flags, so +// classify()'s state writes are explicit method calls rather than hidden +// mutations of module globals from a function that otherwise reads as a pure +// mapper. +function createUsageLimitTracker() { + let turnSeq = 0; + let detectedTurn: number | undefined; + let suppressedRetryReadyKey: string | undefined; + + return { + currentTurn(): number { + return turnSeq; + }, + // Open a new turn; remember any stale retry-ready banner still on screen so + // classify() doesn't re-flag it as a fresh limit this turn. + beginTurn(snapshot: string): number { + turnSeq++; + detectedTurn = undefined; + const current = detectCliUsageLimit(snapshot); + suppressedRetryReadyKey = current.limited && current.retryReady + ? usageLimitStateKey(current) + : undefined; + return turnSeq; + }, + // Map a runtime status to a usage-limit-aware status, recording whether this + // turn hit a limit (read back via detectedThisTurn). + classify( + content: string, + status: RuntimeScreenStatus, + ): { status: RuntimeScreenStatus | 'limited'; usageLimit?: CliUsageLimitState } { + const detected = detectCliUsageLimit(content); + if (!detected.limited) return { status }; + + const key = usageLimitStateKey(detected); + if (detected.retryReady && key === suppressedRetryReadyKey) { + return { status }; + } + + suppressedRetryReadyKey = undefined; + detectedTurn = turnSeq; + return { status: 'limited', usageLimit: detected }; + }, + detectedThisTurn(seq: number): boolean { + return detectedTurn === seq; + }, + }; +} + +const usageLimitTracker = createUsageLimitTracker(); + +function currentUsageLimitSnapshot(): string { + return lastAnalyzerSnapshot || renderer?.rawSnapshot() || ''; +} + +// ─── Adopt-bridge state (Claude Code only) ───────────────────────────────── +// +// In bridge mode the daemon adopted an existing CLI session that we do NOT +// own; the model never sees botmux. We harvest assistant turns by tailing +// Claude Code's transcript JSONL and forward only the bytes appended after +// each Lark-driven user turn — never the historical content present at +// attach time, never local-terminal-driven turns. +// +// Attribution lives in BridgeTurnQueue; this file only manages the +// fs.watch wakeup, byte-offset bookkeeping, lazy baseline, and IPC emit. +let bridgeJsonlPath: string | undefined; +/** Directory enclosing bridgeJsonlPath. We poll this dir for newer jsonl + * files so the bridge follows `/clear` / `/resume` in the user's CLI — * those create a brand-new sessionId.jsonl, and a watcher pinned to the * original path would silently stop receiving events. */ let bridgeJsonlDir: string | undefined; @@ -1788,7 +3392,7 @@ function maybeEmitAdoptPreamble(events: TranscriptEvent[]): void { * 对齐 maybeEmitAdoptPreamble；区别只在事件取出方式（codex/coco 是结构化 * event，不需要走 claude 那套 jsonl turn assembly）。 */ function maybeEmitCodexAdoptPreamble( - history: readonly { kind: 'user' | 'assistant_final'; text: string }[], + history: readonly CodexBridgeEvent[], ): void { if (!lastInitConfig?.adoptMode) return; if (lastInitConfig?.adoptRestoredFromMetadata) return; @@ -2833,6 +4437,22 @@ function codexBridgeFallbackActive(): boolean { return isStructuredBridgeFallbackActive(lastInitConfig?.cliId, lastInitConfig?.adoptMode === true); } +/** Only drivers with a complete normal-final + interrupted-terminal contract + * may let a transcript-started turn override the screen-ready heuristic. */ +function hasStructuredLifecycleBlock(): boolean { + if (rpcTurnsAwaitingActivation.size > 0) return true; + if (rpcLifecycleFailClosedOwners.size > 0) return true; + if (rpcTerminalHydrationOwners.size > 0) return true; + // rpcActive is an explicit native app-server ownership proof and is valid for + // every RPC-capable Codex-family adapter (currently codex + traex), including + // drivers intentionally excluded from transcript-only lifecycle blocking. + if (codexBridgeQueue.peek().some(turn => + turn.rpcActive === true && turn.finalText === undefined, + )) return true; + return isStructuredBridgeLifecycleBlockingCli(lastInitConfig?.cliId) + && codexBridgeQueue.hasBlockingTurn(); +} + function structuredBridgeIsCodex(): boolean { return lastInitConfig?.cliId === 'codex'; } @@ -2890,7 +4510,10 @@ function codexBridgeStartTimer(): void { // up. Emitting here when isPromptReady=true closes that window. // Codex's queue only releases turns on `assistant_final` (the model's // declared end-of-turn), so a tick-driven emit can't accidentally - // publish a half-streamed response. + // publish a half-streamed response. The finally-path also expires stale + // attribution heads after every tick even when no transcript exists yet or + // its offset produced no events; otherwise an adapter returning undefined + // before late-attach could leave a bare fingerprint at the head forever. codexBridgeTimer = setInterval(() => { try { if (structuredBridgeIsHermes()) { @@ -2980,6 +4603,16 @@ function codexBridgeStartTimer(): void { if (isPromptReady) emitReadyCodexTurns(); } catch (err: any) { log(`Codex bridge tick error: ${err.message}`); + } finally { + // All branch returns above still pass through here. Ingest always gets + // first chance to claim a boundary-time user event; only afterwards do + // we expire an unstarted head, replay buffered successors, and emit any + // completion created by that replay in the same call stack. + try { + pruneExpiredStructuredHeadsAndEmit('structured bridge tick'); + } catch (err: any) { + log(`Codex bridge tick expiry error: ${err.message}`); + } } }, 1000); } @@ -3015,7 +4648,8 @@ function hermesBridgeIngest(): void { } if (filtered.events.length > 0) lastStructuredBridgeActivityAtMs = Date.now(); codexBridgeQueue.ingest(filtered.events); - if (filtered.events.some(e => e.kind === 'assistant_final')) { + pruneExpiredStructuredHeadsAndEmit('Hermes ingest'); + if (filtered.events.some(isStructuredTerminalEvent)) { idleDetector?.fireIdle(); } } @@ -3028,6 +4662,7 @@ function mtrBridgeAttach(source: MtrTranscriptSource, mode: 'baseline-existing' const { history, live } = splitCodexEventsByCutoff(result.events, cutoff); codexBridgeQueue.absorb(history); codexBridgeQueue.ingest(live); + pruneExpiredStructuredHeadsAndEmit('MTR split-live attach'); mtrBridgeOffset = result.newOffset; mtrBridgeBaselineDone = true; log(`MTR bridge split-live: ${source.dbPath}#${source.sessionId} (history=${history.length}, live=${live.length}, cutoff=${cutoff}, offset=${mtrBridgeOffset})`); @@ -3053,7 +4688,8 @@ function mtrBridgeIngest(): void { mtrBridgeOffset = result.newOffset; if (result.events.length > 0) lastStructuredBridgeActivityAtMs = Date.now(); codexBridgeQueue.ingest(result.events); - if (result.events.some(e => e.kind === 'assistant_final')) { + pruneExpiredStructuredHeadsAndEmit('MTR ingest'); + if (result.events.some(isStructuredTerminalEvent)) { idleDetector?.fireIdle(); } } @@ -3083,6 +4719,14 @@ function codexBridgeAttach(rolloutPath: string, mode: 'baseline-existing' | 'bas const { history, live } = splitCodexEventsByCutoff(result.events, cutoff); codexBridgeQueue.absorb(history); codexBridgeQueue.ingest(live); + pruneExpiredStructuredHeadsAndEmit('structured split-live attach'); + // Late attach can discover an already-completed live turn in the same + // drain. Re-drive prompt readiness from that terminal event immediately; + // otherwise a ready edge rejected by the lifecycle gate waits for the + // 20-30s lease timer even though the transcript has already ended. + if (live.some(isStructuredTerminalEvent)) { + idleDetector?.fireIdle(); + } codexBridgeOffset = result.newOffset; codexBridgePendingTail = result.pendingTail; codexBridgeBaselineDone = true; @@ -3335,7 +4979,19 @@ function maybeFollowTraexSessionRotationViaPid(): void { codexBridgeNotifyCliSessionId(observed.cliSessionId); } -function codexBridgeIngest(opts: { signalIdle?: boolean } = {}): void { +function codexBridgeIngest(opts: { + signalIdle?: boolean; + hydrationOwnerKey?: string; +} = {}): void { + // Follow-up RPC turns install their exact bridge mark only after the + // turn/start ACK passes the generation fence. Ordinary ingest must not + // advance the rollout cursor in that window. Terminal hydration for an + // older owner may continue: successor events reached by the same drain stay + // buffered until the exact successor mark is installed. + if (rpcTranscriptIngestBlockedByAwaitingActivation( + rpcTurnsAwaitingActivation.keys(), + opts.hydrationOwnerKey, + )) return; if (structuredBridgeIsHermes()) { hermesBridgeIngest(); return; @@ -3350,13 +5006,14 @@ function codexBridgeIngest(opts: { signalIdle?: boolean } = {}): void { codexBridgePendingTail = result.pendingTail; if (result.events.length > 0) lastStructuredBridgeActivityAtMs = Date.now(); codexBridgeQueue.ingest(result.events); - // Transcript-driven idle: an `assistant_final` event is the CLI declaring - // end-of-turn, far more reliable than the screen-pattern heuristic + pruneExpiredStructuredHeadsAndEmit('structured ingest'); + // Transcript-driven idle: a normal `assistant_final` or no-output + // `turn_aborted` is Codex declaring end-of-turn, far more reliable than the screen-pattern heuristic // (CoCo's status bar varies by --yolo flag, version, theme; codex has // its own moving targets). Pushing idle here lets the bridge emit // immediately instead of waiting for readyPattern + quiescence to // converge. Idempotent — IdleDetector.fireIdle no-ops while already idle. - if (opts.signalIdle !== false && result.events.some(e => e.kind === 'assistant_final')) { + if (opts.signalIdle !== false && result.events.some(isStructuredTerminalEvent)) { idleDetector?.fireIdle(); } } @@ -3368,14 +5025,375 @@ function codexBridgeMarkPendingTurn( messageText: string, preferredTurnId?: string, dispatchAttempt?: number, -): boolean { - if (!codexBridgeFallbackActive()) return false; + markTimeMs: number = Date.now(), +): string | undefined { + if (!codexBridgeFallbackActive()) return undefined; const turnId = preferredTurnId ?? `codex-${randomBytes(8).toString('hex')}`; - codexBridgeQueue.mark(turnId, messageText, Date.now(), dispatchAttempt); + codexBridgeQueue.mark(turnId, messageText, markTimeMs, dispatchAttempt); + return turnId; +} + +function finalizeRpcTurnTerminal( + terminal: CodexRpcTurnTerminal, + generation: RpcTurnGeneration, + dropPending: boolean, +): void { + const { identity } = terminal; + const ownerKey = rpcTurnOwnerKey(identity); + if (!sameRpcGeneration( + settlingRpcTerminalOwners.get(ownerKey), + generation, + )) return; + const timer = rpcTerminalHydrationTimers.get(ownerKey); + if (timer) clearTimeout(timer); + rpcTerminalHydrationTimers.delete(ownerKey); + if (sameRpcGeneration(rpcTerminalHydrationOwners.get(ownerKey), generation)) { + rpcTerminalHydrationOwners.delete(ownerKey); + rpcTerminalHydrationTerminals.delete(ownerKey); + } + settlingRpcTerminalOwners.delete(ownerKey); + clearAwaitingRpcActivation(identity, generation); + clearRpcLifecycleFailClosedOwner(identity, generation); + if (sameRpcGeneration(rpcActiveOwners.get(ownerKey), generation)) { + rpcActiveOwners.delete(ownerKey); + } + codexBridgeQueue.stopRpcActive(identity.turnId, identity.dispatchAttempt); + if (dropPending) { + codexBridgeQueue.dropPendingTurn(identity.turnId, identity.dispatchAttempt, true); + } + + const status = terminal.status === 'completed' ? 'completed' + : terminal.status === 'failed' || terminal.status === 'aborted' ? 'failed' + : 'ambiguous'; + const errorCode = terminal.errorCode + ?? (terminal.status === 'engine-dead' ? 'rpc_engine_dead' + : terminal.status === 'stopped' ? 'rpc_engine_stopped' + : terminal.status === 'aborted' ? 'rpc_turn_aborted' + : terminal.status === 'failed' ? 'rpc_turn_failed' + : undefined); + emitTurnTerminal(identity.turnId, status, errorCode, identity.dispatchAttempt); + log( + `Codex RPC terminal ${terminal.status}: native=${terminal.nativeTurnId.slice(0, 12)} ` + + `turn=${identity.turnId.slice(0, 12)} attempt=${identity.dispatchAttempt ?? '-'}`, + ); + redriveRejectedStructuredReady(); + idleDetector?.fireIdle(); +} + +/** turn/completed can race ahead of rollout fs visibility even though it is an + * authoritative execution terminal. Release rpcActive immediately, but retain + * a short non-ready hydration gate while polling the structured transcript. + * This preserves fallback final output without allowing the stale fingerprint + * to head-of-line block a new RPC turn indefinitely. */ +function hydrateCompletedRpcTurn( + terminal: CodexRpcTurnTerminal, + generation: RpcTurnGeneration, + attempt = 0, +): void { + const { identity } = terminal; + const ownerKey = rpcTurnOwnerKey(identity); + if (!sameRpcGeneration( + settlingRpcTerminalOwners.get(ownerKey), + generation, + )) return; + // restartCliProcess advances the generation before its asynchronous teardown. + // Do not let an old hydration timer touch a bridge queue that may already be + // owned by the replacement. stopCodexRpcEngine will synchronously retire the + // old generation when teardown reaches the engine. + if (cliSpawnGeneration !== generation.cliGeneration + || codexRpcEngine !== generation.engine) return; + try { + codexBridgeDrainAndMaybeEmit({ + signalIdle: false, + hydrationOwnerKey: ownerKey, + }); + } catch { /* retry */ } + if (!codexBridgeQueue.hasPendingTurn(identity.turnId, identity.dispatchAttempt)) { + // Structured ingest already emitted final_output + terminal in the canonical + // order. The explicit native terminal below is idempotently suppressed. + finalizeRpcTurnTerminal(terminal, generation, false); + return; + } + // The native terminal can beat rollout persistence by seconds on a cold or + // busy filesystem. Keep this bounded (~11.6s), aligned with the fresh-turn + // 12s positive-evidence probe, rather than dropping fallback output after the + // previous 1.55s window. + const delays = CODEX_RPC_TERMINAL_HYDRATION_DELAYS_MS; + if (attempt >= delays.length) { + finalizeRpcTurnTerminal(terminal, generation, true); + return; + } + const timer = setTimeout(() => { + rpcTerminalHydrationTimers.delete(ownerKey); + hydrateCompletedRpcTurn(terminal, generation, attempt + 1); + }, delays[attempt]); + timer.unref?.(); + rpcTerminalHydrationTimers.set(ownerKey, timer); +} + +function settleRpcTurnTerminal( + terminal: CodexRpcTurnTerminal, + generation: RpcTurnGeneration, +): void { + const ownerKey = rpcTurnOwnerKey(terminal.identity); + const existingSettlement = settlingRpcTerminalOwners.get(ownerKey); + if (existingSettlement) { + if (!sameRpcGeneration(existingSettlement, generation)) { + log(`Ignored stale Codex RPC terminal settlement for ${terminal.identity.turnId}`); + } + return; + } + settlingRpcTerminalOwners.set(ownerKey, generation); + clearAwaitingRpcActivation(terminal.identity, generation); + clearRpcLifecycleFailClosedOwner(terminal.identity, generation); + if (sameRpcGeneration(rpcActiveOwners.get(ownerKey), generation)) { + rpcActiveOwners.delete(ownerKey); + } + codexBridgeQueue.stopRpcActive( + terminal.identity.turnId, + terminal.identity.dispatchAttempt, + ); + if (terminal.status === 'completed' + && codexBridgeQueue.hasPendingTurn( + terminal.identity.turnId, + terminal.identity.dispatchAttempt, + )) { + rpcTerminalHydrationOwners.set(ownerKey, generation); + rpcTerminalHydrationTerminals.set(ownerKey, { terminal, generation }); + hydrateCompletedRpcTurn(terminal, generation); + return; + } + finalizeRpcTurnTerminal(terminal, generation, true); +} + +function handleRpcTurnTerminal( + terminal: CodexRpcTurnTerminal, + generation: RpcTurnGeneration, +): void { + const ownerKey = rpcTurnOwnerKey(terminal.identity); + const awaiting = rpcTurnsAwaitingActivation.get(ownerKey); + if (awaiting && sameRpcGeneration(awaiting, generation)) { + // Response + turn/completed can be decoded before the await continuation + // installs rpcActive. The exact terminal is replayed immediately after that + // activation, never against a guessed/latest queue entry. + pendingRpcTurnTerminals.set(ownerKey, { terminal, generation }); + return; + } + if (awaiting && !sameRpcGeneration(awaiting, generation)) { + log(`Ignored stale Codex RPC terminal from replaced engine for ${terminal.identity.turnId}`); + return; + } + const activeGeneration = rpcActiveOwners.get(ownerKey); + const failedClosedGeneration = rpcLifecycleFailClosedOwners.get(ownerKey); + if (!sameRpcGeneration(activeGeneration, generation) + && !sameRpcGeneration(failedClosedGeneration, generation)) { + // The structured bridge may already have consumed the same turn and emitted + // its terminal, or a replacement generation may own the same logical + // delivery key. In either case this callback has no queue state to mutate. + log( + `Codex RPC terminal already retired: native=${terminal.nativeTurnId.slice(0, 12)} ` + + `turn=${terminal.identity.turnId.slice(0, 12)}`, + ); + return; + } + settleRpcTurnTerminal(terminal, generation); +} + +/** Install the exact bridge mark + rpcActive hand-off after turn/start is known + * accepted. Failure is fail-closed: a separate lifecycle gate remains asserted + * for this owner until its native terminal/engine teardown, so a bookkeeping + * bug cannot publish false idle. */ +function activateRpcTurnLifecycle( + identity: CodexRpcTurnIdentity, + messageText: string, + alreadyMarked: boolean, + generation: RpcTurnGeneration, + deferTerminal = false, +): boolean { + const ownerKey = rpcTurnOwnerKey(identity); + if (!sameRpcGeneration( + rpcTurnsAwaitingActivation.get(ownerKey), + generation, + )) { + log(`Refused stale Codex RPC lifecycle activation for ${identity.turnId}`); + return false; + } + const replayAnchorMs = awaitingRpcActivationReplayAnchorMs(identity, generation); + let marked = alreadyMarked + && (codexBridgeQueue.hasPendingTurn(identity.turnId, identity.dispatchAttempt) + || codexBridgeQueue.hasTerminalTurn(identity.turnId, identity.dispatchAttempt)); + if (!alreadyMarked) { + const bridgeTurnId = codexBridgeMarkPendingTurn( + messageText, + identity.turnId, + identity.dispatchAttempt, + replayAnchorMs, + ); + marked = bridgeTurnId === identity.turnId; + } + // An older turn's terminal hydration may have drained and buffered this + // successor's complete user/final pair while turn/start ACK was still + // pending. mark() replays that pair synchronously. Treat the exact transcript + // terminal as an already-retired activation instead of failing closed merely + // because markRpcActive correctly refuses a completed queue entry. + const transcriptTerminalObserved = marked + && codexBridgeQueue.hasTerminalTurn(identity.turnId, identity.dispatchAttempt); + const activated = transcriptTerminalObserved + || (marked && codexBridgeQueue.markRpcActive( + identity.turnId, + identity.dispatchAttempt, + )); + if (activated && !transcriptTerminalObserved) { + rpcActiveOwners.set(ownerKey, generation); + } + if (transcriptTerminalObserved) emitReadyCodexTurns(); + if (!deferTerminal + && sameRpcGeneration(rpcTurnsAwaitingActivation.get(ownerKey), generation)) { + rpcTurnsAwaitingActivation.delete(ownerKey); + rpcTurnsAwaitingActivationIdentities.delete(ownerKey); + rpcTurnsAwaitingActivationReplayAnchors.delete(ownerKey); + } + if (!activated) { + if (marked) codexBridgeQueue.dropPendingTurn(identity.turnId, identity.dispatchAttempt, true); + installRpcLifecycleFailClosedOwner(identity, generation); + log( + `Codex RPC lifecycle failed closed: mark=${marked} active=${activated} ` + + `turn=${identity.turnId.slice(0, 12)} attempt=${identity.dispatchAttempt ?? '-'}`, + ); + send({ + type: 'user_notify', + message: 'Codex RPC 已接收消息，但本地生命周期登记失败；在原生终态到达前保持忙碌，未自动重发。', + turnId: identity.turnId, + ...(identity.dispatchAttempt !== undefined + ? { dispatchAttempt: identity.dispatchAttempt } + : {}), + }); + } + const pendingTerminal = pendingRpcTurnTerminals.get(ownerKey); + if (pendingTerminal + && sameRpcGeneration(pendingTerminal.generation, generation) + && !deferTerminal) { + settleRpcTurnTerminal(pendingTerminal.terminal, generation); + } + return activated; +} + +function releaseRpcTurnTerminalDeferral( + identity: CodexRpcTurnIdentity, + generation: RpcTurnGeneration, +): void { + const ownerKey = rpcTurnOwnerKey(identity); + if (sameRpcGeneration(rpcTurnsAwaitingActivation.get(ownerKey), generation)) { + rpcTurnsAwaitingActivation.delete(ownerKey); + rpcTurnsAwaitingActivationIdentities.delete(ownerKey); + rpcTurnsAwaitingActivationReplayAnchors.delete(ownerKey); + } + const pendingTerminal = pendingRpcTurnTerminals.get(ownerKey); + if (pendingTerminal && sameRpcGeneration(pendingTerminal.generation, generation)) { + settleRpcTurnTerminal(pendingTerminal.terminal, generation); + } +} + +function retireRpcLifecycleFromStructuredTerminal( + identity: CodexRpcTurnIdentity, +): void { + const ownerKey = rpcTurnOwnerKey(identity); + // Native completion hydration owns its own finalization after this canonical + // drain returns. Do not tear down that generation's settlement underneath it. + if (settlingRpcTerminalOwners.has(ownerKey)) return; + const generation = rpcActiveOwners.get(ownerKey) + ?? rpcLifecycleFailClosedOwners.get(ownerKey) + ?? rpcTurnsAwaitingActivation.get(ownerKey); + if (!generation) return; + clearAwaitingRpcActivation(identity, generation); + if (sameRpcGeneration(rpcActiveOwners.get(ownerKey), generation)) { + rpcActiveOwners.delete(ownerKey); + } + clearRpcLifecycleFailClosedOwner(identity, generation); + codexBridgeQueue.stopRpcActive(identity.turnId, identity.dispatchAttempt); +} + +/** Expire confirmed or attribution-only unstarted queue heads at an explicit worker + * lifecycle boundary. Pruning can replay a buffered successor user+final and + * thereby produce an immediately emittable completion, so query/projection + * methods must never do this mutation invisibly: every removal funnels + * through this helper and drains ready output in the same call stack. */ +function pruneExpiredStructuredHeadsAndEmit(source: string): boolean { + let fenceCurrentRpcEngine = false; + const dropped = pruneExpiredPreStartHeadsAndEmit( + codexBridgeQueue, + emitReadyCodexTurns, + undefined, + turns => { + // A bounded pre-start lease is also the terminal boundary for a durable + // delivery attempt that the CLI accepted but never wrote to transcript. + // Settle N before the prune replay drains successor N+1; ordinary IM + // turns have no dispatchAttempt and remain log-only as before. + for (const turn of turns) { + const identity: CodexRpcTurnIdentity = { + turnId: turn.turnId, + ...(turn.dispatchAttempt !== undefined + ? { dispatchAttempt: turn.dispatchAttempt } + : {}), + }; + const ownerKey = rpcTurnOwnerKey(identity); + const failedClosedGeneration = rpcLifecycleFailClosedOwners.get(ownerKey); + const retiredRpcOwner = failedClosedGeneration + ? clearRpcLifecycleFailClosedOwner(identity, failedClosedGeneration) + : false; + let shouldFenceRpcEngine = false; + if (retiredRpcOwner && failedClosedGeneration) { + clearAwaitingRpcActivation(identity, failedClosedGeneration); + if (deferredFreshRpcTurn + && rpcTurnOwnerKey(deferredFreshRpcTurn.identity) === ownerKey + && sameRpcGeneration(deferredFreshRpcTurn.generation, failedClosedGeneration)) { + deferredFreshRpcTurn = undefined; + } + if (codexRpcEngine === failedClosedGeneration.engine + && cliSpawnGeneration === failedClosedGeneration.cliGeneration + && !cliRestartInProgress) { + fenceCurrentRpcEngine = true; + shouldFenceRpcEngine = true; + } + } + if (retiredRpcOwner || turn.dispatchAttempt !== undefined) { + emitTurnTerminal( + turn.turnId, + 'ambiguous', + retiredRpcOwner + ? 'rpc_delivery_ambiguous_timeout' + : 'structured_start_timeout', + turn.dispatchAttempt, + ); + } + // Publish/retire N before a synchronous immediate restart can clear the + // current exact identity in killCli(). The callback still installs the + // restart fence before it returns, so the helper cannot emit buffered + // successor N+1 against this ambiguous engine generation. + if (shouldFenceRpcEngine) { + void restartCliProcess( + 'Codex RPC ambiguous delivery exceeded its bounded attribution lease', + { immediate: true, preservePending: true }, + ); + } + } + }, + ); + if (dropped.length === 0) return false; + log(`${source}: expired ${dropped.length} structured head(s) without transcript start (${dropped.map(turn => turn.turnId).join(', ')})`); + if (fenceCurrentRpcEngine) log('Fenced ambiguous Codex RPC generation before structured successor replay'); + // A rejected prompt-ready signal may be waiting on this exact pre-start + // lease. Re-evaluate it after both the queue mark and its separate RPC + // fail-closed owner have retired; otherwise the projector can stay busy + // forever even though there is no remaining terminal source. + redriveRejectedStructuredReady(); return true; } -function codexBridgeDrainAndMaybeEmit(opts: { signalIdle?: boolean } = {}): void { +function codexBridgeDrainAndMaybeEmit(opts: { + signalIdle?: boolean; + hydrationOwnerKey?: string; +} = {}): void { if (!codexBridgeFallbackActive()) return; if (structuredBridgeIsHermes() || structuredBridgeIsMtr() || (codexBridgeRolloutPath && codexBridgeBaselineDone)) { try { codexBridgeIngest(opts); } catch (err: any) { log(`Codex bridge ingest error: ${err.message}`); } @@ -3441,6 +5459,12 @@ function emitReadyCodexTurns(): void { }); } for (const turn of ready) { + retireRpcLifecycleFromStructuredTerminal({ + turnId: turn.turnId, + ...(turn.dispatchAttempt !== undefined + ? { dispatchAttempt: turn.dispatchAttempt } + : {}), + }); emitTurnTerminal( turn.turnId, turn.terminalStatus ?? 'completed', @@ -3747,6 +5771,201 @@ let screenAnalyzer: ScreenAnalyzer | null = null; /** When true, user messages are queued because a TUI prompt is active */ let tuiPromptBlocking = false; +/** One composed status projection shared by the immediate screen path, + * periodic text updates, and screenshot uploads. */ +function projectedRuntimeScreenStatus(): RuntimeScreenStatus { + return codexAppLivenessStatus(projectRuntimeScreenStatus({ + promptReady: isPromptReady, + analyzing: screenAnalyzer?.isAnalyzing === true, + structuredTurnBlocking: hasStructuredLifecycleBlock(), + })); +} + +/** Worker-local widening of the status snapshot helper. Codex App composes a + * `stalled` state on top of the base screen/structured projection, while the + * shared utility intentionally exposes only the base three-state union. */ +async function snapshotWithLatestRuntimeStatus( + capture: () => Promise, + projectStatus: () => RuntimeScreenStatus, +): Promise<{ snapshot: T; status: RuntimeScreenStatus }> { + const snapshot = await capture(); + return { snapshot, status: projectStatus() }; +} + +/** Re-arm readiness before every individual CLI write. A whole flush can span + * several type-ahead items, and adopt writeInput can await history polling; + * either path may observe an assistant_final before the await returns. Reset + * here (never after the await) so that final remains a usable ready edge. */ +function beginCliWriteCycle(): void { + beginRuntimeWriteCycle({ + setPromptReady: ready => { isPromptReady = ready; }, + resetIdleDetector: () => { idleDetector?.reset(); }, + }); +} + +/** Serialize one adopted-pane message from transcript baseline/mark through + * paste, Enter/history verification and lifecycle settlement. Node's process + * message listener does not await a prior async invocation, so without the + * outer AsyncSerialQueue two CoCo/Codex writes can overwrite the same composer + * while the first is sleeping before Enter or polling history. */ +type AdoptWriteResult = 'completed' | 'stale-before-write' | 'stale-after-write'; + +async function writeAdoptMessage( + content: string, + turnId: string | undefined, + dispatchAttempt?: number, + vcMeetingImTurnOrigin?: VcMeetingImTurnOrigin, + fence?: AdoptWriteFence, +): Promise { + const executionFence = fence ?? captureAdoptWriteFence(); + if (!executionFence || !adoptWriteFenceIsCurrent(executionFence)) { + return 'stale-before-write'; + } + const adoptBackend = executionFence.backend; + + renderer?.markNewTurn(); + const turnSeq = usageLimitTracker.beginTurn(currentUsageLimitSnapshot()); + currentBotmuxTurnId = turnId; + currentBotmuxDispatchAttempt = dispatchAttempt; + currentVcMeetingImTurnOrigin = vcMeetingImTurnOrigin; + if (dispatchAttempt !== undefined) durableTurnInFlight = true; + writeCliPidMarker(); + publishSandboxRelayCapability(); + let adoptStructuredBridgeTurnId: string | undefined; + + // Capture transcript baseline before writing so subsequent assistant events + // are attributed to this Lark turn, not prior local pane activity. + if (bridgeJsonlPath) { + try { bridgeIngest(); } catch { /* best effort */ } + bridgeMarkPendingTurn(content, turnId, dispatchAttempt); + } else if (codexBridgeFallbackActive()) { + if (codexBridgeIsCursor()) { + // Cursor may append the current line before IPC handling; mark first so + // the pre-existing line can fingerprint-match instead of becoming seen. + adoptStructuredBridgeTurnId = codexBridgeMarkPendingTurn(content, turnId, dispatchAttempt); + try { codexBridgeIngest(); } catch { /* best effort */ } + } else { + try { codexBridgeIngest(); } catch { /* best effort */ } + adoptStructuredBridgeTurnId = codexBridgeMarkPendingTurn(content, turnId, dispatchAttempt); + } + } + + const settleStaleAfterWrite = (errorCode: string): AdoptWriteResult => { + if (adoptStructuredBridgeTurnId) { + codexBridgeQueue.finishSubmitVerification( + adoptStructuredBridgeTurnId, + undefined, + dispatchAttempt, + ); + } + dropFailedBridgeMark(adoptStructuredBridgeTurnId, dispatchAttempt); + if (turnId && dispatchAttempt !== undefined) { + emitTurnTerminal(turnId, 'ambiguous', errorCode, dispatchAttempt); + } else { + send({ + type: 'user_notify', + turnId, + message: 'Adopt input could not be reconciled because the CLI backend changed while it was being written. Please verify the terminal before retrying.', + }); + } + return 'stale-after-write'; + }; + + // Re-arm before the adapter can yield. An assistant_final may arrive while + // writeInput is still polling history. + beginCliWriteCycle(); + if (isStructuredBridgeAdoptInputCli(lastInitConfig?.cliId) && cliAdapter) { + try { + if (adoptStructuredBridgeTurnId) { + codexBridgeQueue.beginSubmitVerification(adoptStructuredBridgeTurnId, undefined, dispatchAttempt); + } + const result = await cliAdapter.writeInput(adoptBackend as unknown as PtyHandle, content); + if (!adoptWriteFenceIsCurrent(executionFence)) { + return settleStaleAfterWrite('adopt_generation_changed'); + } + if (result?.cliSessionId) { + persistCliSessionId(result.cliSessionId); + codexBridgeNotifyCliSessionId(result.cliSessionId); + } + if (result?.submitted === true && adoptStructuredBridgeTurnId) { + codexBridgeQueue.confirmPendingTurn(adoptStructuredBridgeTurnId, undefined, dispatchAttempt); + } else if (adoptStructuredBridgeTurnId && !(result?.submitted === false && result.recheck && !result.failureReason)) { + codexBridgeQueue.finishSubmitVerification(adoptStructuredBridgeTurnId, undefined, dispatchAttempt); + } + redriveRejectedStructuredReady(); + if (result && result.submitted === false) { + scheduleSubmitFailureNotify( + content, + result.recheck, + 'submit history', + adoptStructuredBridgeTurnId, + result.failureReason, + turnSeq, + { turnId, dispatchAttempt }, + true, + ); + } + } catch (err: any) { + log(`Adopt writeInput error (${lastInitConfig?.cliId}): ${err.message}`); + if (!adoptWriteFenceIsCurrent(executionFence)) { + return settleStaleAfterWrite('adopt_generation_changed'); + } + if (adoptStructuredBridgeTurnId) { + codexBridgeQueue.finishSubmitVerification(adoptStructuredBridgeTurnId, undefined, dispatchAttempt); + } + dropFailedBridgeMark(adoptStructuredBridgeTurnId, dispatchAttempt); + if (turnId && dispatchAttempt !== undefined) { + emitTurnTerminal(turnId, 'ambiguous', 'adopt_write_input_threw', dispatchAttempt); + } + redriveRejectedStructuredReady(); + } + } else if ('sendText' in adoptBackend && 'sendSpecialKeys' in adoptBackend) { + (adoptBackend as any).sendText(content); + // Beat between text and Enter so Ink-based TUIs register pasted text before + // submit. The serial queue holds across this await. + await new Promise(r => setTimeout(r, 200)); + if (!adoptWriteFenceIsCurrent(executionFence)) { + return settleStaleAfterWrite('adopt_generation_changed_before_enter'); + } + (adoptBackend as any).sendSpecialKeys('Enter'); + } else { + adoptBackend.write(content + '\r'); + } + return 'completed'; +} + +async function runAdoptMessageForCapturedGeneration( + item: PendingCliInput, + requeue: () => void, +): Promise { + const fence = captureAdoptWriteFence(); + if (!fence) { + requeue(); + return 'stale-before-write'; + } + let requeued = false; + const requeueOnce = () => { + if (requeued) return; + requeued = true; + requeue(); + }; + const queued = await runAdoptQueuedWriteSequence({ + queue: adoptWriteQueue, + isCurrent: () => adoptWriteFenceIsCurrent(fence), + onStale: requeueOnce, + write: () => writeAdoptMessage( + item.content, + item.turnId, + item.dispatchAttempt, + item.vcMeetingImTurnOrigin, + fence, + ), + }); + if (queued.status === 'stale-before-write') return 'stale-before-write'; + if (queued.value === 'stale-before-write') requeueOnce(); + return queued.value; +} + function isWorkflowWorker(): boolean { return process.env.BOTMUX_WORKFLOW === '1'; } @@ -3972,8 +6191,7 @@ async function captureAndUpload(): Promise { return; } - let status: RuntimeScreenStatus = isPromptReady ? 'idle' : 'working'; - if (screenAnalyzer?.isAnalyzing) status = 'analyzing'; + const status = projectedRuntimeScreenStatus(); send({ type: 'screenshot_uploaded', imageKey, @@ -4139,14 +6357,14 @@ async function flushPendingInjections(): Promise { if (!canStartInjectionFlush({ injectionFlushing, userFlushing: isFlushing, - sessionRenameInFlight, + sessionRenameInFlight: sessionRenameInFlight(), commandLineWritesPending, bareShellLaunchBlocked, })) return; injectionFlushing = true; try { while (pendingInjections.length > 0 && backend && isPromptReady && !bareShellLaunchBlocked - && !sessionRenameInFlight) { + && !sessionRenameInFlight()) { // Mirror flushPending's one-shot launch-failure guard: when an injection is // the FIRST writer after a (re)spawn, flushPending may never have run // (pendingMessages empty ⇒ early return) so the bare-shell check must also @@ -4333,99 +6551,417 @@ function dismissAidenCodexUpdateDialog(data: string): boolean { idleDetector?.reset(); if (action === 'suppress') return true; - log('Codex startup update dialog detected behind Aiden, selecting the non-upgrade option...'); - if (backend && 'sendSpecialKeys' in backend) { - (backend as any).sendSpecialKeys('Down', 'Enter'); - } else { - backend?.write('\x1b[B\r'); + log('Codex startup update dialog detected behind Aiden, selecting the non-upgrade option...'); + if (backend && 'sendSpecialKeys' in backend) { + (backend as any).sendSpecialKeys('Down', 'Enter'); + } else { + backend?.write('\x1b[B\r'); + } + return true; +} + +// Mira/Mir still send terminal OSC control messages. Codex App deliberately +// does not: its signed Unix-socket channel is independent of terminal/backend +// rendering (including Herdr and Zellij). +const APP_RUNNER_OSC_CLI_IDS = new Set(['mira', 'mir']); +const appRunnerControlDecoder = new RunnerControlDecoder(); +let kiroSessionIdCaptureArmed = false; +let kiroSessionIdCaptureBuffer = ''; + +function decodeCodexAppPayload(payload: string): any | undefined { + try { + return JSON.parse(Buffer.from(payload, 'base64').toString('utf8')); + } catch { + return undefined; + } +} + +const CODEX_APP_DAEMON_PERSIST_TIMEOUT_MS = 30_000; + +function waitForCodexAppDaemonPersistence( + requestId: string, + publish: () => void, +): Promise { + return new Promise(resolve => { + const timer = setTimeout(() => { + const pending = codexAppPendingDaemonAcks.get(requestId); + if (!pending) return; + codexAppPendingDaemonAcks.delete(requestId); + resolve(false); + }, CODEX_APP_DAEMON_PERSIST_TIMEOUT_MS); + timer.unref?.(); + codexAppPendingDaemonAcks.set(requestId, { resolve, timer }); + try { + publish(); + } catch { + clearTimeout(timer); + codexAppPendingDaemonAcks.delete(requestId); + resolve(false); + } + }); +} + +function requestCodexAppDispatchTransition( + operation: 'submit' | 'cancel' | 'retry', + entries: Array<{ dispatchId: string; turnId: string; dispatchAttempt?: number }>, +): Promise { + if (!sessionId || entries.length === 0) return Promise.resolve(entries.length === 0); + const requestId = randomBytes(16).toString('hex'); + return waitForCodexAppDaemonPersistence(requestId, () => send({ + type: 'codex_app_dispatch_transition', + sessionId, + requestId, + operation, + entries, + })); +} + +/** Retire one exact durable dispatch before its receipt/lease owner is allowed + * to replay it. The daemon ledger is the authority; local queue removal is + * deliberately after the durable ACK so a failed session-file write cannot + * create a replay window while the old worker still owns the turn. */ +async function retireCodexAppDispatchForDurableReplay( + turnId: string, + dispatchAttempt: number, +): Promise { + if (lastInitConfig?.cliId !== 'codex-app') { + for (let index = pendingMessages.length - 1; index >= 0; index--) { + const item = pendingMessages[index]; + if (item.turnId === turnId && item.dispatchAttempt === dispatchAttempt) { + pendingMessages.splice(index, 1); + } + } + return true; + } + const reservation = codexAppTurnDispatchQueue.findExact(turnId, dispatchAttempt); + const pending = pendingMessages.find(item => item.turnId === turnId + && item.dispatchAttempt === dispatchAttempt); + const recovered = codexAppRecoveredDispatches.find(entry => entry.turnId === turnId + && entry.dispatchAttempt === dispatchAttempt); + const dispatchIds = new Set([ + reservation?.dispatchId, + pending?.codexAppDispatchId, + recovered?.dispatchId, + ].filter((value): value is string => !!value)); + if (dispatchIds.size !== 1) { + log( + `Cannot retire Codex App durable dispatch turn=${turnId.slice(0, 12)} ` + + `attempt=${dispatchAttempt}: exact dispatch id is ${dispatchIds.size === 0 ? 'missing' : 'ambiguous'}`, + ); + return false; + } + const dispatchId = [...dispatchIds][0]!; + if (!await requestCodexAppDispatchTransition('cancel', [{ + dispatchId, + turnId, + dispatchAttempt, + }])) return false; + + if (reservation && !codexAppTurnDispatchQueue.cancelExact(reservation.handle)) return false; + for (let index = pendingMessages.length - 1; index >= 0; index--) { + const item = pendingMessages[index]; + if (item.turnId === turnId && item.dispatchAttempt === dispatchAttempt) { + pendingMessages.splice(index, 1); + } } + codexAppRecoveredDispatches = codexAppRecoveredDispatches.filter( + entry => entry.dispatchId !== dispatchId, + ); return true; } -// Codex App runner sends botmux control messages as OSC sequences so they do -// not pollute the visible terminal. Strip them before xterm rendering and -// translate them back into worker IPC. -const APP_RUNNER_OSC_CLI_IDS = new Set(['codex-app', 'mira', 'mir']); -const appRunnerControlDecoder = new RunnerControlDecoder(); -let kiroSessionIdCaptureArmed = false; -let kiroSessionIdCaptureBuffer = ''; - -function decodeCodexAppPayload(payload: string): any | undefined { - try { - return JSON.parse(Buffer.from(payload, 'base64').toString('utf8')); - } catch { - return undefined; +async function handleTrustedCodexAppMarker( + kind: string, + payload: Record, + control?: { generation: string; seq: number }, +): Promise { + if (kind === 'thread' && typeof payload.threadId === 'string') { + persistCliSessionId(payload.threadId); + return true; } -} -function handleCodexAppMarker(body: string): void { - const sep = body.indexOf(':'); - if (sep < 0) return; - const kind = body.slice(0, sep); - const payload = decodeCodexAppPayload(body.slice(sep + 1)); - if (!payload || typeof payload !== 'object') return; + if (kind === 'state' && lastInitConfig?.cliId === 'codex-app') { + const readiness = codexAppSignedStateReadiness(payload); + if (readiness === 'invalid') { + rejectCodexAppControlMarker('signed state missing boolean acceptingInput'); + failCodexAppControlGeneration( + 'Codex App runner published signed state without boolean acceptingInput', + ); + return false; + } + if (readiness === 'waiting') { + if (typeof payload.busy !== 'boolean') { + rejectCodexAppControlMarker('invalid signed state'); + return false; + } + codexAppSignedStateObserved = false; + codexAppInputReady = false; + codexAppReadyAuthority.beginWork(); + isPromptReady = false; + idleDetector?.reset(); + if (!codexAppProofDeadline.armed) { + codexAppProofDeadline.arm(() => { + failCodexAppControlGeneration( + `Codex App runner did not become input-ready within ${CODEX_APP_CONTROL_STARTUP_TIMEOUT_MS / 1000} seconds`, + ); + }); + } + log('Authenticated Codex App runner is not yet accepting input; readiness deadline remains armed'); + return true; + } + const state = applyTrustedCodexAppStateMarker( + codexAppTurnLiveness, + codexAppReadyAuthority, + payload, + ); + if (!state.accepted) { + rejectCodexAppControlMarker('invalid signed state'); + return false; + } + // This is the first actual readiness proof. Authentication alone cannot + // clear startup/reconnect failure detection or release queued input. + codexAppSignedStateObserved = true; + codexAppInputReady = true; + codexAppProofDeadline.clear(); + cleanupCodexAppControlBootstrap(); + if (codexAppInputReady && awaitingFirstPrompt) { + awaitingFirstPrompt = false; + renderer?.markNewTurn(); + } + if (state.busy) { + isPromptReady = false; + idleDetector?.reset(); + if (codexAppInputReady) queueMicrotask(() => { void flushPending(); }); + } else { + if (codexAppCompletionAwaitingFinal) { + // Every runner generation that can authenticate this new signed + // channel emits an explicit final transaction, including zero chunks + // for an empty answer. Guessing an empty final here would let a + // mismatched/rejected final advance the FIFO and be ACKed as success. + failCodexAppControlGeneration( + 'Codex App runner published idle before the required final transaction', + ); + return false; + } + // Signed idle proves only that the runner has no queued/active turn. It + // does NOT prove its raw stdin inputBuffer is empty: the old worker can + // die after writing the complete frame but before Enter, then this warm + // runner reconnects and reports idle. Resetting prepared→accepted would + // let the replacement pre-flush submit that old frame and then replay it + // a second time. A recovered prepared entry therefore advances only by + // replaying its final/high-water ACK; otherwise fail closed for explicit + // operator abandon. + if (codexAppTurnDispatchQueue.recoveredPrefix().length > 0) { + failCodexAppControlGeneration( + 'Codex App signed idle cannot prove the recovered prepared frame was never buffered', + ); + return false; + } + if (state.shouldPublishReady) { + codexAppUnprovenPromptDeferred = false; + queueMicrotask(() => markPromptReady()); + } + } + return true; + } - if (kind === 'thread' && typeof payload.threadId === 'string') { - persistCliSessionId(payload.threadId); - return; + if (kind === 'diagnostic' && lastInitConfig?.cliId === 'codex-app') { + if (payload.code !== 'native_turn_identity_conflict' + || typeof payload.message !== 'string') { + rejectCodexAppControlMarker('invalid signed diagnostic'); + return false; + } + log(`Codex App fail-closed diagnostic: ${payload.message}`); + send({ + type: 'user_notify', + message: payload.message, + ...(typeof payload.turnId === 'string' ? { turnId: payload.turnId } : {}), + }); + return true; } - if (kind === 'final' && typeof payload.content === 'string') { - const startedAtMs = typeof payload.startedAtMs === 'number' ? payload.startedAtMs : undefined; - const completedAtMs = typeof payload.completedAtMs === 'number' ? payload.completedAtMs : Date.now(); - // Codex App keeps the app-server-generated id separately for diagnostics. - // Routing must use its stable clientUserMessageId marker; legacy envelopes - // intentionally omit it and fall back to the worker's frozen botmux turn. - const identity = resolveCodexAppFinalTurnIdentity( + if (kind === 'activity' && lastInitConfig?.cliId === 'codex-app') { + const activity = applyTrustedCodexAppActivityMarker( + codexAppTurnLiveness, payload, - currentBotmuxTurnId, - `${lastInitConfig?.cliId ?? 'app'}-${Date.now()}`, ); - if (!identity.ok) { - log( - `${cliName()} rejected final marker with mismatched turn ` - + `(marker=${identity.markerTurnId.substring(0, 12)}, ` - + `current=${identity.currentBotmuxTurnId?.substring(0, 12) ?? '-'})`, + if (!activity.accepted) { + rejectCodexAppControlMarker('invalid signed activity'); + return false; + } + cleanupCodexAppControlBootstrap(); + if (activity.phase === 'completed') { + codexAppCompletionAwaitingFinal = true; + } else { + if (activity.phase === 'submitted' && codexAppCompletionAwaitingFinal) { + failCodexAppControlGeneration( + 'Codex App runner submitted the next turn before the required final transaction', + ); + return false; + } + codexAppReadyAuthority.beginWork(); + isPromptReady = false; + idleDetector?.reset(); + } + return true; + } + + if (kind === 'final' && typeof payload.content === 'string') { + const finalContent = payload.content; + const startedAtMs = typeof payload.startedAtMs === 'number' && Number.isFinite(payload.startedAtMs) + ? payload.startedAtMs + : undefined; + const receivedAtMs = Date.now(); + const completedAtMs = typeof payload.completedAtMs === 'number' && Number.isFinite(payload.completedAtMs) + ? Math.min(payload.completedAtMs, receivedAtMs) + : receivedAtMs; + let turnId: string; + let nativeTurnId: string | undefined; + let replyTurnId: string | undefined; + let dispatchAttempt: number | undefined; + let codexAppDispatchId: string | undefined; + let codexAppDispatchHandle: number | undefined; + if (lastInitConfig?.cliId === 'codex-app') { + // flushPending may finish writing N+1 before final N arrives. Attribute + // only from the immutable worker-owned FIFO head; currentBotmuxTurnId is + // intentionally not consulted here. + const settlement = codexAppTurnDispatchQueue.settleFinal(payload, false); + if (!settlement.ok) { + log( + `${cliName()} rejected final marker (${settlement.reason}; ` + + `marker=${settlement.markerTurnId?.substring(0, 12) ?? '-'}, ` + + `expected=${settlement.expectedTurnId?.substring(0, 12) ?? '-'})`, + ); + return false; + } + ({ + turnId, + replyTurnId, + nativeTurnId, + dispatchAttempt, + dispatchId: codexAppDispatchId, + handle: codexAppDispatchHandle, + } = settlement); + if (!codexAppCompletionAwaitingFinal) { + // turn/start failures emit a final transaction without an app-server + // completed activity edge. Close exactly one liveness slot here. + codexAppTurnLiveness.completeCurrent(completedAtMs); + } + codexAppCompletionAwaitingFinal = false; + // A terminal prompt may already have been observed, but ready waits for + // the later signed state{busy:false}. The runner sequences that state + // after this complete final transaction. + } else { + // Mira/Mir retain their terminal OSC control path and do not use the + // Codex App serial dispatch FIFO. + const identity = resolveCodexAppFinalTurnIdentity( + payload, + currentBotmuxTurnId, + `${lastInitConfig?.cliId ?? 'app'}-${Date.now()}`, ); - return; + if (!identity.ok) { + log( + `${cliName()} rejected final marker with mismatched turn ` + + `(marker=${identity.markerTurnId.substring(0, 12)}, ` + + `current=${identity.currentBotmuxTurnId?.substring(0, 12) ?? '-'})`, + ); + return false; + } + ({ turnId, nativeTurnId } = identity); + if (payload.dispatchAttempt !== undefined + && payload.dispatchAttempt !== currentBotmuxDispatchAttempt) { + log( + `${cliName()} rejected final marker with mismatched dispatch attempt ` + + `(marker=${String(payload.dispatchAttempt)}, current=${currentBotmuxDispatchAttempt ?? '-'})`, + ); + return false; + } + dispatchAttempt = currentBotmuxDispatchAttempt; } - const { turnId, nativeTurnId } = identity; if (nativeTurnId && nativeTurnId !== turnId) { log(`${cliName()} native turn ${nativeTurnId.substring(0, 12)} mapped to botmux turn ${turnId.substring(0, 12)}`); } - if (payload.dispatchAttempt !== undefined - && payload.dispatchAttempt !== currentBotmuxDispatchAttempt) { - log( - `${cliName()} rejected final marker with mismatched dispatch attempt ` - + `(marker=${String(payload.dispatchAttempt)}, current=${currentBotmuxDispatchAttempt ?? '-'})`, - ); - return; - } - // Attempt authority is worker-owned. A runner marker may redundantly assert - // equality for compatibility, but can never select another attempt. - const dispatchAttempt = currentBotmuxDispatchAttempt; - if (startedAtMs !== undefined) { - const sentByModel = shouldSuppressBridgeEmit( - { markTimeMs: startedAtMs, isLocal: false, finalText: payload.content }, + let suppressDelivery = false; + if (finalContent && startedAtMs !== undefined) { + suppressDelivery = shouldSuppressBridgeEmit( + { markTimeMs: startedAtMs, isLocal: false, finalText: finalContent }, completedAtMs + 5_001, readSendMarkers(), false, ); - if (sentByModel) { + if (suppressDelivery) { log(`${cliName()} final_output suppressed (model already called botmux send)`); - emitTurnTerminal(turnId, 'completed', undefined, dispatchAttempt); - return; } } - send({ - type: 'final_output', - content: payload.content, - lastUuid: turnId, - turnId, - ...(dispatchAttempt !== undefined ? { dispatchAttempt } : {}), - }); + + if (codexAppDispatchId) { + if (!control || codexAppDispatchHandle === undefined) return false; + const requestId = randomBytes(16).toString('hex'); + const persisted = await waitForCodexAppDaemonPersistence(requestId, () => send({ + type: 'final_output', + content: suppressDelivery ? '' : finalContent, + lastUuid: turnId, + turnId, + ...(replyTurnId ? { replyTurnId } : {}), + ...(dispatchAttempt !== undefined ? { dispatchAttempt } : {}), + ...(suppressDelivery ? { suppressDelivery: true } : {}), + codexAppSettlement: { + requestId, + generation: control.generation, + seq: control.seq, + dispatchId: codexAppDispatchId!, + }, + })); + if (!persisted || !codexAppTurnDispatchQueue.commitExactHead(codexAppDispatchHandle)) { + return false; + } + codexAppGenerationCommits = [ + ...codexAppGenerationCommits.filter(commit => commit.generation !== control.generation), + { + generation: control.generation, + committedThrough: Math.max( + control.seq, + codexAppGenerationCommits.find( + commit => commit.generation === control.generation, + )?.committedThrough ?? 0, + ), + }, + ]; + codexAppRecoveredDispatches = codexAppRecoveredDispatches.filter( + entry => entry.dispatchId !== codexAppDispatchId, + ); + } else { + if (lastInitConfig?.cliId === 'codex-app' + && codexAppDispatchHandle !== undefined + && !codexAppTurnDispatchQueue.commitExactHead(codexAppDispatchHandle)) { + return false; + } + if (finalContent && !suppressDelivery) { + send({ + type: 'final_output', + content: finalContent, + lastUuid: turnId, + turnId, + ...(dispatchAttempt !== undefined ? { dispatchAttempt } : {}), + }); + } else if (!finalContent) { + log(`${cliName()} empty final settled for botmux turn ${turnId.substring(0, 12)}`); + } + } emitTurnTerminal(turnId, 'completed', undefined, dispatchAttempt); + return true; } + rejectCodexAppControlMarker(`unsupported signed ${kind}`); + return false; +} + +function handleAppRunnerOscMarker(body: string): void { + const sep = body.indexOf(':'); + if (sep < 0) return; + const kind = body.slice(0, sep); + const payload = decodeCodexAppPayload(body.slice(sep + 1)); + if (!payload || typeof payload !== 'object') return; + void handleTrustedCodexAppMarker(kind, payload); } function maybeCaptureKiroSessionId(data: string): void { @@ -4442,7 +6978,7 @@ function splitCodexAppControl(data: string): string { return appRunnerControlDecoder.push( data, APP_RUNNER_OSC_CLI_IDS.has(lastInitConfig?.cliId ?? ''), - handleCodexAppMarker, + handleAppRunnerOscMarker, ); } @@ -4556,8 +7092,11 @@ function onPtyData(data: string): void { } // Track last PTY output time for the ready-gate quiescence settle (see - // settleThenFlush) and delegate idle detection to IdleDetector. + // settleThenFlush) and a monotonic generation for rejected-ready evidence. + // Generation (not timestamp equality) distinguishes two redraw chunks that + // happen inside the same millisecond. lastPtyOutputAtMs = Date.now(); + ptyOutputGeneration.observe(); idleDetector?.feed(data); } @@ -4576,8 +7115,78 @@ function markPromptReadyFromPty(): void { } } +function stopStructuredStartGraceRecheck(): void { + if (!structuredStartGraceRecheckTimer) return; + clearTimeout(structuredStartGraceRecheckTimer); + structuredStartGraceRecheckTimer = null; +} + +function redriveRejectedStructuredReady(): void { + const readyEvidenceGeneration = structuredRejectedReadyEvidenceGeneration; + if (readyEvidenceGeneration === undefined) return; + stopStructuredStartGraceRecheck(); + pruneExpiredStructuredHeadsAndEmit('structured pre-start gate'); + if (!backend || isPromptReady) { + structuredRejectedReadyEvidenceGeneration = undefined; + return; + } + if (hasStructuredLifecycleBlock()) { + const nextRemainingMs = codexBridgeQueue.preStartLeaseRemainingMs(); + if (nextRemainingMs !== undefined) scheduleStructuredStartGraceRecheck(nextRemainingMs); + return; + } + + structuredRejectedReadyEvidenceGeneration = undefined; + if (ptyOutputGeneration.isCurrent(readyEvidenceGeneration)) { + log('Structured pre-start gate settled with no newer PTY output — re-driving prior prompt-ready signal'); + idleDetector?.fireIdle(); + return; + } + if (cliAdapter?.busyPattern && (backend.captureCurrentScreen || backend.captureViewport)) { + probeBusyPatternIdle('structured pre-start gate', backend); + return; + } + try { + const currentScreen = captureBackendScreen(backend); + if (currentScreen) idleDetector?.feed(currentScreen); + } catch (err: any) { + log(`Structured pre-start screen recheck failed: ${err.message}`); + } +} + +/** Re-drive a prompt signal rejected only because an explicitly confirmed + * submit or its in-flight verification had not reached the structured + * transcript yet. The bounded lease prevents permanent false-busy. */ +function scheduleStructuredStartGraceRecheck(remainingMs: number): void { + stopStructuredStartGraceRecheck(); + const backendAtSchedule = backend; + const cliGenerationAtSchedule = cliSpawnGeneration; + structuredStartGraceRecheckTimer = setTimeout(() => { + structuredStartGraceRecheckTimer = null; + if (!isCliBackendGenerationCurrent( + { generation: cliGenerationAtSchedule, backend: backendAtSchedule }, + { generation: cliSpawnGeneration, backend, restartInProgress: cliRestartInProgress }, + )) { + return; + } + redriveRejectedStructuredReady(); + }, Math.max(1, remainingMs + 1)); + structuredStartGraceRecheckTimer.unref?.(); +} + function markPromptReady(): void { - if (isPromptReady) return; // guard against duplicate calls + if (isPromptReady) { + stopStructuredStartGraceRecheck(); + return; // guard against duplicate calls + } + stopStructuredStartGraceRecheck(); + if (lastInitConfig?.cliId === 'codex-app' && !codexAppControlProven) { + if (!codexAppUnprovenPromptDeferred) { + log('Ignoring Codex App prompt until this worker verifies a fresh Ed25519 challenge response'); + } + codexAppUnprovenPromptDeferred = true; + return; + } stopBusyPatternIdleProbe(); // Ready-gate: a startup selector's ❯ (cjadk et al.) falsely matches // readyPattern → the IdleDetector fires idle while the CLI is NOT actually at @@ -4607,8 +7216,42 @@ function markPromptReady(): void { log('Idle detected during ready-gate settle; deferring prompt-ready until settle completes'); return; } + // Authenticated runners advance their explicit control queue on signed + // completed/final records. + // A prompt is authoritative only for the synthetic observation installed + // after a verified warm reattach; clearing normal queued turns here could + // lose follow-up inputs written by one flushPending() call. + if (lastInitConfig?.cliId === 'codex-app' && !codexAppTurnLiveness.notePrompt()) { + log('Ignoring transient Codex App prompt while an explicit runner turn remains queued'); + return; + } + if (lastInitConfig?.cliId === 'codex-app' && !codexAppReadyAuthority.canPublishPromptReady()) { + if (!codexAppReadyAuthority.consumeLatePromptRecovery(!codexAppTurnLiveness.hasActiveTurn())) { + log('Deferring Codex App terminal prompt until signed runner state confirms busy=false'); + return; + } + log('Accepted one late Codex App prompt after exact local submit cancellation'); + } + // Screen prompt/quiescence is only a UI heuristic. Structured transcript + // bridges have the stronger lifecycle signal: a transcript-started turn + // without assistant_final is still running even if the TUI redraw exposes a + // prompt. An adapter-confirmed type-ahead submit also blocks during a bounded + // hand-off lease while the CLI waits to write its transcript user event. A + // bare mark is never authoritative, and the lease is explicitly re-driven, + // so a dropped Enter cannot create permanent false-busy. + // Reject the heuristic and re-arm IdleDetector so the later transcript-final + // fireIdle() can drive the real ready edge. + if (hasStructuredLifecycleBlock()) { + const remainingMs = codexBridgeQueue.preStartLeaseRemainingMs(); + structuredRejectedReadyEvidenceGeneration = ptyOutputGeneration.snapshot(); + log('Ignoring prompt-ready heuristic while a structured turn is unfinished or submit verification/start is pending'); + idleDetector?.reset(); + if (remainingMs !== undefined) scheduleStructuredStartGraceRecheck(remainingMs); + return; + } + structuredRejectedReadyEvidenceGeneration = undefined; isPromptReady = true; - clearSessionRenameInFlight(); + settleSessionRenameOnPrompt(); // An old backend can still report idle while its async teardown is running. // Only a prompt observed after the general restart fence drops may release // slash commands to the replacement generation. @@ -4640,9 +7283,15 @@ function markPromptReady(): void { // make the CLI busy, so the idle state is transient and shouldn't appear // in the card. This avoids a false "就绪" flash on daemon restart // (where the initial prompt is queued before the CLI becomes idle). - if (renderer && pendingMessages.length === 0 && pendingRawInputs.length === 0 && pendingSessionRename === null && !isFlushing) { + if (renderer && pendingMessages.length === 0 && pendingAdoptMessages.length === 0 && pendingRawInputs.length === 0 && pendingSessionRename === null && !isFlushing) { const { content } = renderer.snapshot(); - send({ type: 'screen_update', content, ...usageLimitTracker.classify(content, 'idle'), turnId: currentBotmuxTurnId, dispatchAttempt: currentBotmuxDispatchAttempt }); + send({ + type: 'screen_update', + content, + ...usageLimitTracker.classify(content, projectedRuntimeScreenStatus()), + turnId: currentBotmuxTurnId, + dispatchAttempt: currentBotmuxDispatchAttempt, + }); } // cwd-move 注入（barrier=true，如 /cd）必须先于本次 pending 用户消息落地： // 用户在 bot 发完「已切换角色」确认后立刻发下一条消息是最常见路径——若 @@ -4668,15 +7317,11 @@ function persistCliSessionId(cliSessionId: string): void { turnId: currentBotmuxTurnId, dispatchAttempt: currentBotmuxDispatchAttempt, }); - try { - const session = sessionStore.getSession(sessionId); - if (!session || session.cliSessionId === cliSessionId) return; - session.cliSessionId = cliSessionId; - sessionStore.updateSession(session); - log(`Persisted CLI session id: ${cliSessionId}`); - } catch (err: any) { - log(`Failed to persist CLI session id: ${err.message}`); - } + // The daemon is the sole sessions-file writer for Codex App durability. + // Writing the worker's stale in-process snapshot here could roll a freshly + // persisted accepted→prepared transition back to accepted. The ordered IPC + // above already has an authoritative daemon handler that persists this id. + log(`Published CLI session id for daemon persistence: ${cliSessionId}`); } function observeCursorCliSessionId(pid: number, label = 'spawn'): void { @@ -4728,18 +7373,40 @@ function observeCursorCliSessionId(pid: number, label = 'spawn'): void { * both without being so long that a true failure goes unsurfaced. */ const SUBMIT_DEFERRED_RECHECK_MS = 20_000; +/** Remove a transcript mark after the submit path has conclusively failed. + * Both bridge implementations mark before writing so they cannot miss a + * fast transcript append; a failed write must undo only an UNSTARTED mark. + * Cleanup itself does not synthesize ready: an unverified mark never blocks, + * and any verified pre-start lease is bounded and owns its expiry re-drive. */ +function dropFailedBridgeMark(bridgeTurnId?: string, dispatchAttempt?: number): void { + if (!bridgeTurnId) return; + const dropped = bridgeQueue.dropPendingTurn(bridgeTurnId, dispatchAttempt); + const droppedStructured = codexBridgeQueue.dropPendingTurn(bridgeTurnId, dispatchAttempt); + if (dropped) { + if (dropped.contentFingerprint) bridgeFingerprintScanLastMs.delete(dropped.contentFingerprint); + log(`Bridge mark dropped after submit failure (turnId=${bridgeTurnId}) — rotation-fallback scan will stop spinning on this fingerprint.`); + } + if (droppedStructured) { + log(`Structured bridge mark dropped after submit failure (turnId=${bridgeTurnId}) — later buffered turns were rechecked.`); + // dropPendingTurn can replay a buffered successor user+final pair. Drain + // it in the same explicit mutation path; leaving completion production to + // a later transcript event can strand fallback output indefinitely. + emitReadyCodexTurns(); + } +} + /** Worker-side handler for `submitted: false`. Defers the user-facing * warning and runs the adapter-supplied `recheck` closure first; if the * message has shown up in the transcript by then (slow path, hook delay), * suppresses the warning entirely. Adapters without a recheck still fall * through to the warning after the same delay so the UX is uniform. * - * `bridgeTurnId` is the BridgeTurnQueue mark created right before the + * `bridgeTurnId` is the transcript queue mark created right before the * failing writeInput. When the deferred recheck conclusively fails (= no - * jsonl line will ever match this fingerprint), we drop that exact dispatch - * attempt's mark — leaving it would keep `maybeSwitchBridgeJsonl` doing - * full-directory scans every poll tick for a fingerprint that's permanently - * dead, the 99% CPU bug this whole patch series is fixing. */ + * transcript event will ever match this fingerprint), we drop that exact + * dispatch attempt's mark. Leaving a Claude mark would keep rotation-fallback + * scans spinning; leaving a structured mark would block later fingerprint + * attribution even after its bounded status lease expires. */ function scheduleSubmitFailureNotify( msg: string, recheck: (() => SubmitRecheckResult | Promise) | undefined, @@ -4748,6 +7415,7 @@ function scheduleSubmitFailureNotify( failureReason?: string, turnSeq = usageLimitTracker.currentTurn(), turnIdentity?: Pick, + structuredTarget = false, ): void { const preview = msg.length > 60 ? msg.slice(0, 60) + '…' : msg; const emitDurableFailure = (errorCode: string): void => { @@ -4755,21 +7423,14 @@ function scheduleSubmitFailureNotify( emitTurnTerminal(turnIdentity.turnId, 'failed', errorCode, turnIdentity.dispatchAttempt); } }; - const dropBridgeMark = (): void => { - if (!bridgeTurnId) return; - const dropped = bridgeQueue.dropPendingTurn(bridgeTurnId, turnIdentity?.dispatchAttempt); - if (dropped) { - if (dropped.contentFingerprint) bridgeFingerprintScanLastMs.delete(dropped.contentFingerprint); - log(`Bridge mark dropped after submit failure (turnId=${bridgeTurnId}, attempt=${turnIdentity?.dispatchAttempt ?? '-'}) — rotation-fallback scan will stop spinning on this fingerprint.`); - } - }; if (failureReason) { const action = decideSubmitConfirmationAction({ failureReason, recheckSubmitted: false, usageLimitDetected: false, }); - dropBridgeMark(); + dropFailedBridgeMark(bridgeTurnId, turnIdentity?.dispatchAttempt); + redriveRejectedStructuredReady(); const reason = action.kind === 'notify-hard-failure' ? action.reason : failureReason; log(`writeInput: submit impossible — notifying user immediately. reason="${reason}" preview="${preview}"`); emitDurableFailure(`submit_impossible:${reason}`); @@ -4783,29 +7444,38 @@ function scheduleSubmitFailureNotify( return; } const activityBaselineMs = Date.now(); + const cliGenerationAtSchedule = cliSpawnGeneration; + const backendAtSchedule = backend; + const deferredAttemptIsCurrent = (): boolean => ( + cliSpawnGeneration === cliGenerationAtSchedule + && backendAtSchedule !== null + && backend === backendAtSchedule + && !cliRestartInProgress + ); log(`writeInput: submit not confirmed after retries — deferred ${SUBMIT_DEFERRED_RECHECK_MS}ms recheck queued. preview="${preview}"`); setTimeout(async () => { - let recheckSubmitted = false; - let cliSessionId: string | undefined; - if (recheck) { - try { - const recheckResult = await recheck(); - recheckSubmitted = typeof recheckResult === 'boolean' - ? recheckResult - : recheckResult.submitted === true; - cliSessionId = typeof recheckResult === 'object' && recheckResult && typeof recheckResult.cliSessionId === 'string' - ? recheckResult.cliSessionId - : undefined; - } catch (err: any) { - log(`Deferred recheck threw (${err?.message ?? err}); falling through to warning.`); - } - } - - const action = decideSubmitConfirmationAction({ - recheckSubmitted, - usageLimitDetected: usageLimitTracker.detectedThisTurn(turnSeq), - activityEvidence: submitActivityEvidenceSince(activityBaselineMs), + const settlement = await settleDeferredSubmitConfirmation(codexBridgeQueue, { + turnId: bridgeTurnId, + dispatchAttempt: turnIdentity?.dispatchAttempt, + structuredTarget, + recheck, + usageLimitDetected: () => usageLimitTracker.detectedThisTurn(turnSeq), + activityEvidence: () => submitActivityEvidenceSince(activityBaselineMs), + isCurrent: deferredAttemptIsCurrent, }); + // Restart/exit or exact-attempt expiry can happen during either the 20s + // delay or the awaited adapter recheck. A stale callback must perform no + // side effects at all: no old cliSessionId persistence, ready redrive, + // recursive timer, terminal, warning, or mutation of replay attempt N+1. + if (settlement.stale) { + log(`Discarded stale deferred submit recheck (${settlement.staleReason}) turn=${bridgeTurnId ?? '-'} attempt=${turnIdentity?.dispatchAttempt ?? '-'}`); + return; + } + if (settlement.recheckError !== undefined) { + const err = settlement.recheckError as any; + log(`Deferred recheck threw (${err?.message ?? err}); falling through to warning.`); + } + const { action, cliSessionId } = settlement; switch (action.kind) { case 'suppress-confirmed': @@ -4814,13 +7484,16 @@ function scheduleSubmitFailureNotify( if (codexBridgeFallbackActive()) codexBridgeNotifyCliSessionId(cliSessionId); } log(`Deferred recheck found submit in ${transcriptLabel} — suppressing warning. preview="${preview}"`); + redriveRejectedStructuredReady(); return; case 'suppress-usage-limit': - dropBridgeMark(); + dropFailedBridgeMark(bridgeTurnId, turnIdentity?.dispatchAttempt); + redriveRejectedStructuredReady(); log(`Deferred recheck missing but usage limit was detected for this turn — suppressing submit warning. preview="${preview}"`); emitDurableFailure('submit_usage_limit'); return; case 'suppress-active': + redriveRejectedStructuredReady(); log(`Deferred recheck missing but later ${action.evidence} shows ${cliName()} is active — suppressing submit warning. preview="${preview}"`); // For a durable turn, activity is evidence of possible submission, not // a terminal result. Keep the arbiter closed and re-check until either @@ -4834,6 +7507,7 @@ function scheduleSubmitFailureNotify( undefined, turnSeq, turnIdentity, + structuredTarget, ); } return; @@ -4844,7 +7518,8 @@ function scheduleSubmitFailureNotify( break; } - dropBridgeMark(); + dropFailedBridgeMark(bridgeTurnId, turnIdentity?.dispatchAttempt); + redriveRejectedStructuredReady(); log(`Deferred recheck still missing — notifying user. preview="${preview}"`); emitDurableFailure('submit_unconfirmed'); if (turnIdentity?.dispatchAttempt === undefined) { @@ -4929,15 +7604,36 @@ function detectBareShellLaunch(): boolean { * sends the next message (type-ahead) without waiting for idle detection. * Messages pushed during a flush are picked up by the while loop. */ +function requeueUnsubmittedQueuedActivation(item: PendingCliInput): void { + if (!item.queuedActivationToken) return; + // backend.onExit already moved the same object into carryOver; spawnCli will + // restore it after the fenced restart, so do not create a second owner here. + if (!backend) return; + inflightInputs.forget(item); + if (!pendingMessages.some(candidate => + candidate.queuedActivationToken === item.queuedActivationToken)) { + pendingMessages.unshift(item); + } + log(`Retained queued activation ${item.queuedActivationToken.substring(0, 8)} for retry`); +} + +function codexAppRuntimeTypeAheadReady(): boolean { + return lastInitConfig?.cliId === 'codex-app' + && codexAppControlProven + && codexAppSignedStateObserved + && codexAppInputReady; +} + async function flushPending(): Promise { // destroySession() may be asynchronous while `backend` still references the // old CLI. Never let a new flush (including one triggered by the old // backend's idle/task-done callback) write across that restart boundary. if (cliRestartInProgress) return; + if (shutdownDetachRequestId || closeRequestInFlightId || preparedCloseRequestId) return; if (isFlushing) return; // while loop in active flush will pick up new messages if (!backend || !cliAdapter) return; - if (pendingMessages.length === 0 && pendingRawInputs.length === 0 && pendingSessionRename === null) return; // nothing to flush — keep isPromptReady - if (sessionRenameInFlight) return; // wait for /rename to finish before any user input + if (pendingMessages.length === 0 && pendingAdoptMessages.length === 0 && pendingRawInputs.length === 0 && pendingSessionRename === null) return; // nothing to flush — keep isPromptReady + if (sessionRenameInFlight()) return; // wait for /rename to finish before any user input if (commandLineWritesPending > 0) return; // do not splice into text -> Enter // 注入进行中不得并发写 PTY（用户消息留在 pendingMessages，注入完成后的下一次 // markPromptReady 自然排空）——防止 type-ahead 插进注入的 text→Enter 窗口。 @@ -4954,6 +7650,16 @@ async function flushPending(): Promise { // meeting turn may cross this boundary until an explicit terminal releases // the active attempt. if (!pendingInputMayFlush(durableTurnInFlight)) return; + // A no-native or locally-unregistered RPC delivery is deliberately + // fail-closed: its exact execution state is unknown. Codex normally permits + // type-ahead/steer, but admitting a successor here could merge it into the + // unknown turn or overlap a durable replay. Keep every input class queued + // until structured evidence, exact terminal, bounded generation restart, or + // teardown retires the owner. + if (rpcLifecycleFailClosedOwners.size > 0) { + log(`Holding pending input behind ${rpcLifecycleFailClosedOwners.size} unresolved Codex RPC delivery owner(s)`); + return; + } // Ready-gate: hold the FIRST prompt until the SessionStart hook fires a true- // ready signal. A cjadk-style startup selector's ❯ falsely matches readyPattern // and would otherwise eat this message. releaseReadyGate() re-invokes us once @@ -4999,7 +7705,7 @@ async function flushPending(): Promise { const claudeBridgeActive = !!bridgeJsonlPath && !lastInitConfig?.adoptMode; const codexBridgeActive = codexBridgeFallbackActive(); const typeAheadAllowed = pendingInputAllowsTypeAhead( - cliAdapter.supportsTypeAhead === true, + cliAdapter.supportsTypeAhead === true || codexAppRuntimeTypeAheadReady(), durableTurnInFlight, pendingMessages[0], ); @@ -5008,20 +7714,25 @@ async function flushPending(): Promise { // pending messages can still drain while busy; the rename stays queued. const sessionRenameReady = isPromptReady && pendingSessionRename !== null; const rawInputReady = isPromptReady && pendingRawInputs.length > 0; + const adoptInputReady = isPromptReady && lastInitConfig?.adoptMode === true && pendingAdoptMessages.length > 0; let supportedSessionRenameReady = sessionRenameReady; if (sessionRenameReady && (!cliAdapter.buildSessionRenameCommand || effectiveBackendType === 'riff')) { pendingSessionRename = null; supportedSessionRenameReady = false; log(`Ignoring native session rename — unsupported by ${cliName()}${effectiveBackendType === 'riff' ? ' on riff backend' : ''}`); - if (pendingMessages.length === 0 && pendingRawInputs.length === 0) return; + if (pendingMessages.length === 0 && pendingAdoptMessages.length === 0 && pendingRawInputs.length === 0) return; } if (!isPromptReady && pendingMessages.length === 0) return; if (!isPromptReady && !typeAheadAllowed) return; isFlushing = true; - if (isPromptReady) { - isPromptReady = false; - idleDetector?.reset(); + const codexAppPromptReplay = new CodexAppFlushPromptReplay(); + // Raw input and native rename own their explicit command-line/session gates; + // pending adopt writes re-arm inside writeAdoptMessage. Clearing readiness + // here would make an adopt rename's in-queue readiness recheck always see a + // synthetic false. Normal prompt/startup flushes keep the original re-arm. + if (!rawInputReady && !supportedSessionRenameReady && !adoptInputReady) { + beginCliWriteCycle(); } try { @@ -5061,10 +7772,48 @@ async function flushPending(): Promise { if (supportedSessionRenameReady && pendingSessionRename !== null && backend && cliAdapter) { const title = pendingSessionRename; const buildRename = cliAdapter.buildSessionRenameCommand!; + const renameBackend = backend; + const renameGeneration = cliSpawnGeneration; pendingSessionRename = null; - sessionRenameInFlight = true; + sessionRenamePhase = 'reserved'; try { - await sendRawCommandLineSerially(backend, buildRename(title)); + const writeRename = async () => { + // Transition to busy inside the adopt queue, after the readiness + // recheck. Keep `writing` through text→beat→Enter: a terminal from + // the preceding turn in that window must not release the rename gate. + beginCliWriteCycle(); + sessionRenamePhase = 'writing'; + await sendRawCommandLineSerially(renameBackend, buildRename(title)); + if (cliSpawnGeneration !== renameGeneration || backend !== renameBackend || cliRestartInProgress) { + throw new Error('rename backend generation changed before Enter settlement'); + } + sessionRenamePhase = 'sent'; + // A previous turn's terminal may have raised ready while the rename + // was still writing. Re-arm after Enter so only rename's own prompt + // can settle `sent` (with the bounded timeout as fail-open fallback). + beginCliWriteCycle(); + }; + const sent = lastInitConfig?.adoptMode + ? await runAdoptSessionRenameSequence({ + queue: adoptWriteQueue, + // An adopt write queued immediately before rename can make the + // outer readiness decision stale while this task waits. + isPromptReady: () => isPromptReady + && backend === renameBackend + && cliSpawnGeneration === renameGeneration + && !cliRestartInProgress + && !rawInputRestartGate, + writeRename, + }) + : (await writeRename(), true); + if (!sent) { + // Keep a newer title if one arrived while this rename waited. Retry + // this title only when it is still the latest requested canonical title. + if (pendingSessionRename === null) pendingSessionRename = title; + forceClearSessionRenameInFlight(); + log(`Deferred native session rename until the next prompt (${cliName()}): ${title}`); + return; + } armSessionRenameIdleTimeout(); idleDetector?.reset(); log(`Native session rename command sent (${cliName()}): ${title}`); @@ -5074,15 +7823,40 @@ async function flushPending(): Promise { // Keep the rename gate for the same bounded fail-open window: the write // may have stopped after typing only part of the command, so immediately // appending a deferred raw_input could corrupt both commands. - armSessionRenameIdleTimeout(); + if (cliSpawnGeneration === renameGeneration && backend === renameBackend && !cliRestartInProgress) { + settleFailedSessionRenameWrite(); + armSessionRenameIdleTimeout(); + } else { + forceClearSessionRenameInFlight(); + } log(`Native session rename command failed (${cliName()}): ${err?.message ?? err}; waiting for prompt or fail-open timeout`); } // Wait for the command to finish and the TUI to become idle again before // sending queued user prompts; otherwise they can land in its picker. return; } + if (adoptInputReady && pendingAdoptMessages.length > 0) { + const item = pendingAdoptMessages.shift()!; + await runAdoptMessageForCapturedGeneration(item, () => { + pendingAdoptMessages.unshift(item); + log(`Re-queued adopt message at queue head for the replacement CLI generation (${item.content.length} chars)`); + }); + return; + } while (pendingMessages.length > 0 && backend && cliAdapter) { const item = pendingMessages.shift()!; + beginCliWriteCycle(); + const writeGeneration = cliSpawnGeneration; + const writeBackend = backend; + const writeAdapter = cliAdapter; + const writeRpcEngine = codexRpcEngine; + const writeContinuationIsCurrent = (): boolean => ( + cliSpawnGeneration === writeGeneration + && backend === writeBackend + && cliAdapter === writeAdapter + && codexRpcEngine === writeRpcEngine + && !cliRestartInProgress + ); const durableWrite = item.dispatchAttempt !== undefined; if (durableWrite) durableTurnInFlight = true; // Track as in-flight until the CLI returns to idle (markPromptReady). @@ -5114,12 +7888,13 @@ async function flushPending(): Promise { if (claudeBridgeActive) { try { bridgeIngest(); } catch { /* best-effort */ } bridgeTurnId = bridgeMarkPendingTurn(logicalMsg, item.turnId, item.dispatchAttempt); - } else if (codexBridgeActive) { + } else if (codexBridgeActive && !writeRpcEngine) { // Codex mark works even before the rollout path is known: the // queue is path-agnostic, and the late-attach below will start // ingest from offset 0 so the user_message that lands shortly // after still fingerprint-matches this turn. - codexBridgeMarkPendingTurn(logicalMsg, item.turnId, item.dispatchAttempt); + bridgeTurnId = codexBridgeMarkPendingTurn(logicalMsg, item.turnId, item.dispatchAttempt); + if (bridgeTurnId) codexBridgeQueue.beginSubmitVerification(bridgeTurnId, undefined, item.dispatchAttempt); } if (durableWrite && cliAdapter.reliableTurnTerminal === true @@ -5139,6 +7914,50 @@ async function flushPending(): Promise { log('Refused durable Claude submit: transcript terminal bridge is unavailable'); break; } + // Unlike the transcript bridge, Codex App has an explicit app-server + // runner. Start its liveness clock before the control line is written so + // an accepted-but-never-dequeued input is diagnosed too; authenticated + // activity records can arrive while writeInput awaits chunk delivery. + const tracksCodexAppLiveness = lastInitConfig?.cliId === 'codex-app'; + const codexAppFrozenTurnId = tracksCodexAppLiveness + ? item.turnId + || item.codexAppInput?.clientUserMessageId + || `codex-app-${sessionId || 'unknown'}-${Date.now().toString(36)}-${++codexAppFallbackTurnSequence}` + : undefined; + // Reserve attribution before the first chunk is written. The runner can + // dequeue and finish a small turn while writeRunnerInput is still + // awaiting later chunk throttles, and this flush may then write N+1. + const codexAppDispatchReservation = codexAppFrozenTurnId + ? codexAppTurnDispatchQueue.reserve( + codexAppFrozenTurnId, + item.dispatchAttempt, + item.codexAppDispatchId, + false, + item.replyTurnId, + ) + : undefined; + if (tracksCodexAppLiveness && item.codexAppDispatchId && codexAppFrozenTurnId) { + const prepared = await requestCodexAppDispatchTransition('submit', [{ + dispatchId: item.codexAppDispatchId, + turnId: codexAppFrozenTurnId, + ...(item.dispatchAttempt !== undefined + ? { dispatchAttempt: item.dispatchAttempt } + : {}), + }]); + if (!prepared) { + if (codexAppDispatchReservation) { + codexAppTurnDispatchQueue.cancelExact(codexAppDispatchReservation.handle); + } + failCodexAppControlGeneration( + 'Daemon could not persist the Codex App prepared dispatch before runner write', + ); + break; + } + } + const codexAppLivenessHandle = tracksCodexAppLiveness + ? codexAppTurnLiveness.begin(codexAppFrozenTurnId) + : undefined; + if (tracksCodexAppLiveness) codexAppReadyAuthority.beginWork(); log(`Writing to PTY (flush): "${msg.substring(0, 80)}"`); // Defense in depth: TmuxPipeBackend's send methods no longer throw on a // dead pane (they fire onExit instead), but writeInput can still throw @@ -5146,25 +7965,215 @@ async function flushPending(): Promise { // backend regression). flushPending is invoked fire-and-forget, so an // escaping rejection would become an unhandledRejection and crash the // worker — exactly the failure mode this change is closing. Contain it. - let result: Awaited> | undefined; + if (effectiveBackendType === 'riff' && item.queuedActivationToken) { + if (pendingRiffQueuedActivation) { + failPendingRiffQueuedActivation('a second activation reached the single-task Riff boundary'); + break; + } + pendingRiffQueuedActivation = { + token: item.queuedActivationToken, + turnId: item.turnId, + dispatchAttempt: item.dispatchAttempt, + }; + log( + `Riff queued activation ${item.queuedActivationToken.substring(0, 8)} armed; ` + + 'awaiting exact remote task id', + ); + } + const handleStaleWriteContinuation = (errorCode: string): void => { + const disposition = settleStaleWriteContinuation( + item, + errorCode, + (turnId, code, dispatchAttempt) => { + emitTurnTerminal(turnId, 'ambiguous', code, dispatchAttempt); + }, + ); + // Generation change has already handed ordinary input to + // InflightInputTracker carryover. Do not touch global bridge queues: + // replacement may already have marked the same turnId/undefined key. + log(`Discarded stale writeInput continuation turn=${item.turnId ?? '-'} attempt=${item.dispatchAttempt ?? '-'} generation=${writeGeneration} disposition=${disposition}`); + }; + let result: Awaited> | undefined; + let rpcTurnIdentity: CodexRpcTurnIdentity | undefined; + let rpcTurnGeneration: RpcTurnGeneration | undefined; try { - if (codexRpcEngine) { + if (writeRpcEngine) { + rpcTurnIdentity = { + turnId: item.turnId ?? `codex-rpc-${randomBytes(8).toString('hex')}`, + ...(item.dispatchAttempt !== undefined + ? { dispatchAttempt: item.dispatchAttempt } + : {}), + }; + rpcTurnGeneration = { + engine: writeRpcEngine, + cliGeneration: writeGeneration, + }; + installAwaitingRpcActivation( + rpcTurnIdentity, + rpcTurnGeneration, + ); // RPC input mode: deliver via JSON-RPC turn/start (its ack IS the // submit confirmation), which the attached `codex --remote` TUI // renders. No tmux paste → the history.jsonl verify/retry/recover // machinery is bypassed. A throw here falls into the catch below and // surfaces as a normal submit-failure notice. - await codexRpcEngine.sendTurn(msg, item.turnId); + await writeRpcEngine.sendTurn(msg, rpcTurnIdentity); + // The await may overlap an engine/pane replacement. Fence the captured + // generation BEFORE touching the global bridge queue; a stale ack must + // never activate a mark owned by the replacement generation. + if (!writeContinuationIsCurrent()) { + // RPC writes bypass InflightInputTracker because replay could + // duplicate an already-accepted server turn. Keep this exact owner + // registered until stopCodexRpcEngine settles a native terminal or + // publishes one ambiguous teardown terminal + notice. + log( + `Deferred stale Codex RPC continuation to engine teardown ` + + `turn=${rpcTurnIdentity.turnId} generation=${writeGeneration}`, + ); + break; + } result = { submitted: true }; - } else if (item.codexAppInput && cliAdapter.writeStructuredInput) { - result = await cliAdapter.writeStructuredInput(backend, msg, item.codexAppInput); + // Only the ACKed, still-current generation may create bridge state. + // While turn/start was pending, codexBridgeIngest left its cursor + // untouched; marking now therefore still precedes replay of any + // already-persisted user/final events. + bridgeTurnId = rpcTurnIdentity.turnId; + // The app-server ack confirms execution has begun, but no local + // transcript event will follow to flip started. Mark the turn active + // so the lifecycle gate stays asserted for the full server-side run + // instead of relying on the bounded 20s confirmation lease (which + // would expire mid-turn on a long-running or approval-pending turn). + const activated = activateRpcTurnLifecycle( + rpcTurnIdentity, + msg, + false, + rpcTurnGeneration, + ); + if (activated) { + codexBridgeDrainAndMaybeEmit({ signalIdle: false }); + } } else { - result = await cliAdapter.writeInput(backend, msg); + result = item.codexAppInput && writeAdapter.writeStructuredInput + ? await writeAdapter.writeStructuredInput(writeBackend, msg, item.codexAppInput) + : await writeAdapter.writeInput(writeBackend, msg); + } + if (!writeContinuationIsCurrent()) { + handleStaleWriteContinuation('write_generation_changed'); + break; + } + scheduleBusyPatternIdleProbe(`${cliName()} post-submit`); + } catch (err: any) { + // A replacement generation may already have installed an equal + // turn/attempt key. Fence before touching any process-global queue or + // liveness state owned by that successor. + if (!writeContinuationIsCurrent()) { + if (rpcTurnIdentity && rpcTurnGeneration) { + // The frame may have crossed the socket before the error. Never hand + // RPC input to ordinary carryover/replay; exact teardown owns the + // terminal and user notice for this still-awaiting identity. + log( + `Deferred stale failed Codex RPC continuation to engine teardown ` + + `turn=${rpcTurnIdentity.turnId} generation=${writeGeneration}`, + ); + } else { + handleStaleWriteContinuation('write_generation_changed_after_error'); + } + break; + } + if (rpcTurnIdentity && rpcTurnGeneration && writeRpcEngine) { + const replayAnchorMs = awaitingRpcActivationReplayAnchorMs( + rpcTurnIdentity, + rpcTurnGeneration, + ); + clearAwaitingRpcActivation(rpcTurnIdentity, rpcTurnGeneration); + bridgeTurnId = codexBridgeMarkPendingTurn( + msg, + rpcTurnIdentity.turnId, + rpcTurnIdentity.dispatchAttempt, + replayAnchorMs, + ); + installRpcLifecycleFailClosedOwner(rpcTurnIdentity, rpcTurnGeneration); + log( + `Codex RPC turn/start became ambiguous (${err?.message ?? err}); ` + + `held exact owner ${rpcTurnIdentity.turnId} and blocked successors`, + ); + send({ + type: 'user_notify', + message: 'Codex RPC 消息已发出但未取得可验证的 turn/start 响应；为避免重复执行，未自动重发，并暂时阻塞后续消息直到结构化终态、边界重启或会话重启。', + turnId: rpcTurnIdentity.turnId, + ...(rpcTurnIdentity.dispatchAttempt !== undefined + ? { dispatchAttempt: rpcTurnIdentity.dispatchAttempt } + : {}), + }); + if (bridgeTurnId !== rpcTurnIdentity.turnId && !cliRestartInProgress) { + void restartCliProcess( + 'Codex RPC ambiguous delivery could not establish an attribution lease', + { immediate: true, preservePending: true }, + ); + } + break; + } + codexAppPromptReplay.cancelSubmission( + codexAppTurnLiveness, + codexAppReadyAuthority, + codexAppLivenessHandle, + ); + if (effectiveBackendType === 'riff' + && item.queuedActivationToken + && pendingRiffQueuedActivation?.token === item.queuedActivationToken) { + failPendingRiffQueuedActivation(`adapter write threw: ${err?.message ?? err}`); + break; + } + if (tracksCodexAppLiveness) { + // A throwing framed write has unknown side effects: the runner may + // hold a complete valid line without its newline. Never cancel and + // continue in this generation. Ordinary IM ownership stays in the + // daemon's prepared ledger for explicit crash recovery. Durable + // ownership also stays intact: the worker-exit path atomically marks + // its receipt ambiguous and arms the runtime fence before replay. + log(`writeInput threw with unknown Codex App runner state: ${err?.message ?? err}`); + failCodexAppControlGeneration( + 'Codex App input write became ambiguous; fenced the runner before any successor could submit', + ); + break; + } + // Legacy/non-control adapters keep their existing submit-failure path. + // A durable receiver attempt transfers replay ownership to the + // receipt/lease reconciler on the ambiguous terminal below, so remove + // any exact local reservation first to avoid two independent replayers. + let dispatchStillPending = true; + if (codexAppDispatchReservation) { + if (item.codexAppDispatchId && durableWrite && codexAppFrozenTurnId) { + const cancelled = codexAppTurnDispatchQueue.cancelExact( + codexAppDispatchReservation.handle, + ) && await requestCodexAppDispatchTransition('cancel', [{ + dispatchId: item.codexAppDispatchId, + turnId: codexAppFrozenTurnId, + dispatchAttempt: item.dispatchAttempt!, + }]); + if (!cancelled) { + failCodexAppControlGeneration( + 'Could not transfer an ambiguous Codex App delivery to durable recovery', + ); + break; + } + } else if (!item.codexAppDispatchId) { + dispatchStillPending = codexAppTurnDispatchQueue.cancelExact( + codexAppDispatchReservation.handle, + ); + } + } + if (bridgeTurnId) { + codexBridgeQueue.stopRpcActive(bridgeTurnId, item.dispatchAttempt); + codexBridgeQueue.finishSubmitVerification( + bridgeTurnId, + undefined, + item.dispatchAttempt, + ); } - scheduleBusyPatternIdleProbe(`${cliName()} post-submit`); - } catch (err: any) { log(`writeInput threw: ${err?.message ?? err}`); - if (durableWrite && item.turnId) { + requeueUnsubmittedQueuedActivation(item); + if (dispatchStillPending && durableWrite && item.turnId) { // A throwing backend cannot prove whether bytes reached the CLI. // Reconcile as ambiguous (not a definitive failure) so the receiver // can replay the same frozen delivery behind the action gate. @@ -5174,7 +8183,7 @@ async function flushPending(): Promise { // nulled `backend` and told the user the CLI exited) — nothing more to // do. Otherwise surface it as a submit failure so the message isn't // silently lost. - if (backend) scheduleSubmitFailureNotify( + if (backend && dispatchStillPending) scheduleSubmitFailureNotify( logicalMsg, undefined, '会话 JSONL', @@ -5182,7 +8191,9 @@ async function flushPending(): Promise { undefined, turnSeq, item, + codexBridgeActive, ); + redriveRejectedStructuredReady(); break; } // Persist any sessionId the adapter observed via authoritative sources @@ -5196,19 +8207,89 @@ async function flushPending(): Promise { // attributed to this turn. if (codexBridgeActive) codexBridgeNotifyCliSessionId(result.cliSessionId); } + if (result?.submitted === true && bridgeTurnId) { + codexBridgeQueue.confirmPendingTurn(bridgeTurnId, undefined, item.dispatchAttempt); + } else if (bridgeTurnId && !(result?.submitted === false && result.recheck && !result.failureReason)) { + // Keep verification pending only while an adapter-supplied deferred + // recheck can still produce authoritative submit evidence. + codexBridgeQueue.finishSubmitVerification(bridgeTurnId, undefined, item.dispatchAttempt); + } + redriveRejectedStructuredReady(); // `&& backend`: if the CLI exited during this write (pane gone → onExit // nulled backend) the user already got a "CLI exited" notice; don't also // nag that the submit wasn't confirmed. - if (result && result.submitted === false && backend) { - scheduleSubmitFailureNotify( - logicalMsg, - result.recheck, - '会话 JSONL', - bridgeTurnId, - result.failureReason, - turnSeq, - item, + if (result && result.submitted === false) { + codexAppPromptReplay.cancelSubmission( + codexAppTurnLiveness, + codexAppReadyAuthority, + codexAppLivenessHandle, ); + const codexAppSafeNonSubmission = result.submissionDisposition === 'untouched' + || result.submissionDisposition === 'flushed_invalid'; + if (tracksCodexAppLiveness && !codexAppSafeNonSubmission) { + // final-Enter exhaustion, cleanup failure, and thrown/unknown writes + // can leave a complete valid frame buffered. Retain an ordinary + // prepared ledger entry and fence the entire generation. For a + // durable attempt, worker exit owns the ambiguous receipt + runtime + // fence transition; publishing an earlier terminal would open a + // replay window before the old runner teardown is proven. + failCodexAppControlGeneration( + 'Codex App runner input buffer is not provably clean; fenced before any successor could submit', + ); + break; + } + const dispatchStillPending = codexAppDispatchReservation + ? codexAppTurnDispatchQueue.cancelExact(codexAppDispatchReservation.handle) + : true; + if (dispatchStillPending && item.codexAppDispatchId && codexAppFrozenTurnId) { + const retryQueuedActivation = tracksCodexAppLiveness + && !!item.queuedActivationToken + && codexAppSafeNonSubmission; + const transitioned = await requestCodexAppDispatchTransition( + retryQueuedActivation ? 'retry' : 'cancel', [{ + dispatchId: item.codexAppDispatchId, + turnId: codexAppFrozenTurnId, + ...(item.dispatchAttempt !== undefined + ? { dispatchAttempt: item.dispatchAttempt } + : {}), + }], + ); + if (!transitioned) { + failCodexAppControlGeneration( + retryQueuedActivation + ? 'Daemon could not return a safely untouched queued activation to accepted' + : 'Daemon could not cancel a Codex App dispatch rejected before submission', + ); + break; + } + if (!retryQueuedActivation) { + codexAppRecoveredDispatches = codexAppRecoveredDispatches.filter( + entry => entry.dispatchId !== item.codexAppDispatchId, + ); + } + } + if (backend && dispatchStillPending) { + scheduleSubmitFailureNotify( + logicalMsg, + result.recheck, + '会话 JSONL', + bridgeTurnId, + result.failureReason, + turnSeq, + item, + codexBridgeActive, + ); + } + requeueUnsubmittedQueuedActivation(item); + } else if (item.queuedActivationToken && effectiveBackendType !== 'riff') { + // The daemon keeps the exact journal and route reservation until this + // adapter-level boundary. IPC loss may replay at-least-once; an early + // worker/daemon crash can never silently consume the opening turn. + send({ + type: 'queued_activation_submitted', + sessionId, + activationToken: item.queuedActivationToken, + }); } // All structured bridges now drain every pending message in one flush: // Claude's BridgeTurnQueue handles `attachment(queued_command)` events @@ -5219,10 +8300,28 @@ async function flushPending(): Promise { // that correctly. Durable receiver attempts are the exception: they and // adjacent IM turns wait for separate idle edges so neither can be // HOL-dropped or steered into the other. + // + // Riff is also a hard one-per-boundary adapter. writeInput() only appends + // an async HTTP operation to RiffBackend.writeChain and returns + // immediately; draining this while-loop would enqueue N independent 10s + // create/follow-up requests before the first remote task reaches done. + // Stop after one and let onTaskDone -> markPromptReady() advance the next + // item. Besides preserving turn semantics, this gives graceful shutdown + // a proven single-request drain bound. + if (rpcLifecycleFailClosedOwners.size > 0) break; + if (effectiveBackendType === 'riff') break; if (shouldStopPendingBatch(item, pendingMessages[0])) break; } } finally { isFlushing = false; + // A prompt can arrive after turn N completes but before turn N+1's chunked + // control line finishes. We reject it while N+1 has a liveness slot. If that + // write then fails, the prompt is still the runner's real ready boundary; + // replay it after releasing the flush mutex so queued peers can drain. + if (codexAppPromptReplay.consumeAfterFlush(codexAppTurnLiveness)) { + log('Replaying deferred Codex App prompt after a queued submission was cancelled'); + markPromptReady(); + } } } @@ -5232,19 +8331,30 @@ function sendToPty( opts: { codexAppInput?: CodexAppTurnInput; dispatchAttempt?: number; + codexAppDispatchId?: string; + queuedActivationToken?: string; + replyTurnId?: string; vcMeetingImTurnOrigin?: VcMeetingImTurnOrigin; } = {}, ): void { - if (!cliAdapter) return; const next: PendingCliInput = { content, turnId, + ...(opts.replyTurnId ? { replyTurnId: opts.replyTurnId } : {}), + ...(opts.codexAppDispatchId ? { codexAppDispatchId: opts.codexAppDispatchId } : {}), + ...(opts.queuedActivationToken ? { queuedActivationToken: opts.queuedActivationToken } : {}), ...(opts.codexAppInput ? { codexAppInput: opts.codexAppInput } : {}), ...(opts.dispatchAttempt !== undefined ? { dispatchAttempt: opts.dispatchAttempt } : {}), ...(opts.vcMeetingImTurnOrigin ? { vcMeetingImTurnOrigin: opts.vcMeetingImTurnOrigin } : {}), }; + if (!initPromptMaterialized) { + pendingMessages.push(next); + log(`Queued message behind async init materialization (${pendingMessages.length} pending)`); + return; + } + if (!cliAdapter) return; // During an exact lease-fenced CLI restart the worker stays alive while the // backend is rebuilt. Preserve incoming attempt N+1 in the worker queue; the // old early-return silently dropped it after receiver had already persisted @@ -5255,7 +8365,7 @@ function sendToPty( return; } const supportsTypeAhead = pendingInputAllowsTypeAhead( - cliAdapter.supportsTypeAhead === true, + cliAdapter.supportsTypeAhead === true || codexAppRuntimeTypeAheadReady(), durableTurnInFlight, next, ); @@ -5300,7 +8410,7 @@ function sendToPty( // type-ahead write is dropped (no input box yet) — markPromptReady()'s flush // delivers queued messages instead. See input-gate.ts; this fixes dispatch's // brief reaching Codex before its first idle and never landing. - if (!sessionRenameInFlight && commandLineWritesPending === 0 && shouldWriteNow({ + if (!sessionRenameInFlight() && commandLineWritesPending === 0 && shouldWriteNow({ isPromptReady, isFlushing, supportsTypeAhead, awaitingFirstPrompt, })) { if (!mergedQueued) log(`Writing to PTY: "${content.substring(0, 80)}"`); @@ -5333,56 +8443,65 @@ function startScreenUpdates(): void { let lastSnapshotPtyActivity = -1; screenUpdateTimer = setInterval(() => { if (awaitingFirstPrompt) return; - let status: RuntimeScreenStatus = isPromptReady ? 'idle' : 'working'; - if (screenAnalyzer?.isAnalyzing) status = 'analyzing'; void (async () => { - let content = lastContent; - let changed = false; - - // Capture only when the pane has emitted output since our last snapshot. - // During idle (the steady state for a parked session) this skips a tmux - // capture-pane + a throwaway xterm-headless instantiation every tick — - // the dominant per-session background cost — while the status-transition - // send below still fires off the cached content. The exception is a - // self-driven screen (observe backend with a live web-attach): the - // watermark can't be trusted there, so capture every tick. - const ptyActivity = lastPtyActivityAtMs; - if (shouldCaptureScreen({ - ptyActivity, - lastCapturedPtyActivity: lastSnapshotPtyActivity, - screenSelfDriven: isScreenSelfDriven(backend), - })) { - lastSnapshotPtyActivity = ptyActivity; - // Preferred path: pipe-pane backends pull a fresh viewport snapshot - // from tmux every tick. This eliminates the accumulated-buffer drift - // that produced duplicated/staircase text in 'text' display mode. - const pipeText = await snapshotToText(backend, renderCols, renderRows, { filter: true }); - if (pipeText) { - content = pipeText.content; - const hash = pipeText.ansi; - changed = hash !== lastTextSnapshotHash; - lastTextSnapshotHash = hash; - // Refresh the unfiltered cache that ScreenAnalyzer reads from. Same - // tmux call would otherwise need to fire twice per tick. - if (changed) { - const rawSnap = await snapshotToText(backend, renderCols, renderRows, { filter: false }); - if (rawSnap) lastAnalyzerSnapshot = rawSnap.content; + const { snapshot, status } = await snapshotWithLatestRuntimeStatus(async () => { + let content = lastContent; + let changed = false; + + // Capture only when the pane has emitted output since our last snapshot. + // During idle (the steady state for a parked session) this skips a tmux + // capture-pane + a throwaway xterm-headless instantiation every tick — + // the dominant per-session background cost — while the status-transition + // send below still fires off the cached content. The exception is a + // self-driven screen (observe backend with a live web-attach): the + // watermark can't be trusted there, so capture every tick. + const ptyActivity = lastPtyActivityAtMs; + if (shouldCaptureScreen({ + ptyActivity, + lastCapturedPtyActivity: lastSnapshotPtyActivity, + screenSelfDriven: isScreenSelfDriven(backend), + })) { + lastSnapshotPtyActivity = ptyActivity; + // Preferred path: pipe-pane backends pull a fresh viewport snapshot + // from tmux every tick. This eliminates the accumulated-buffer drift + // that produced duplicated/staircase text in 'text' display mode. + const pipeText = await snapshotToText(backend, renderCols, renderRows, { filter: true }); + if (pipeText) { + content = pipeText.content; + const hash = pipeText.ansi; + changed = hash !== lastTextSnapshotHash; + lastTextSnapshotHash = hash; + // Refresh the unfiltered cache that ScreenAnalyzer reads from. Same + // tmux call would otherwise need to fire twice per tick. + if (changed) { + const rawSnap = await snapshotToText(backend, renderCols, renderRows, { filter: false }); + if (rawSnap) lastAnalyzerSnapshot = rawSnap.content; + } + } else if (renderer) { + const snap = renderer.snapshot(); + content = snap.content; + changed = snap.changed; + } else { + return null; } - } else if (renderer) { - const snap = renderer.snapshot(); - content = snap.content; - changed = snap.changed; - } else { - return; + lastContent = content; } - lastContent = content; - } - const usageAware = usageLimitTracker.classify(content, status); - if (changed || usageAware.status !== lastSentStatus) { + return { content, changed }; + }, projectedRuntimeScreenStatus); + if (!snapshot) return; + + const usageAware = usageLimitTracker.classify(snapshot.content, status); + if (snapshot.changed || usageAware.status !== lastSentStatus) { lastSentStatus = usageAware.status; - send({ type: 'screen_update', content, ...usageAware, turnId: currentBotmuxTurnId, dispatchAttempt: currentBotmuxDispatchAttempt }); + send({ + type: 'screen_update', + content: snapshot.content, + ...usageAware, + turnId: currentBotmuxTurnId, + dispatchAttempt: currentBotmuxDispatchAttempt, + }); } })(); }, SCREEN_UPDATE_INTERVAL_MS); @@ -5655,7 +8774,9 @@ async function spawnCli( cfg: Extract, opts: { pluginGenerationPrepared?: boolean } = {}, ): Promise { - clearSessionRenameInFlight(); + const spawnGeneration = ++cliSpawnGeneration; + // Prefer force-clear so a half-finished rename cannot block the new generation. + forceClearSessionRenameInFlight(); currentCliCredentialIsolated = false; // Enrollment writes the fixed marker before any device credential appears. // From that instant onward every NEW local CLI must carry a credential @@ -6105,6 +9226,30 @@ async function spawnCli( currentCliCredentialIsolated = appliedIsolationCapabilities.includes('credential'); const isolationRuntimeDataDir = process.env.SESSION_DATA_DIR ?? join(defaultBotmuxHome, 'data'); + const canonicalPolicyPath = (value: string | undefined): string => { + if (!value) return ''; + try { return realpathSync(value); } catch { return value; } + }; + const policySessionDataDir = canonicalPolicyPath(process.env.SESSION_DATA_DIR); + const darwinWriteExtraPaths = process.env.TMPDIR + ? [canonicalPolicyPath(process.env.TMPDIR)] + : []; + const darwinIsolationPolicyDigest = process.platform === 'darwin' + ? isolationPanePolicyDigest({ + readIsolation: willReadIsolate, + writeSandbox: willWriteSandbox, + readDenyExtraPaths: (cfg.readDenyExtraPaths ?? []).map(canonicalPolicyPath), + writeAllowExtraPaths: darwinWriteExtraPaths, + workingDir: canonicalPolicyPath(cfg.workingDir), + homeDir: canonicalPolicyPath(homedir()), + osUserHomeDir: canonicalPolicyPath(userInfo().homedir), + botmuxHome: policySessionDataDir ? dirname(policySessionDataDir) : '', + sessionDataDir: policySessionDataDir, + currentAppId: cfg.larkAppId, + cliId: cfg.cliId, + resolvedBin: canonicalPolicyPath(cliAdapter.resolvedBin), + }) + : undefined; // Every bot — isolated OR not — gets its own BOT_HOME dir as a ready-made private- // storage slot. An isolated sibling denies this path regardless of whether the owner // is isolated (deny uses the full bots.json), so a non-isolated bot can drop private @@ -6183,18 +9328,40 @@ async function spawnCli( // so the probe below sees no pane and we cold-spawn fresh isolated. A current- // policy pane survives daemon restarts and suspend→resume safely because the // confinement remains attached to the live process. - if (appliedIsolationCapabilities.length > 0 && persistentSessionName && effectiveBackendType !== 'pty') { + let persistentPaneOriginChannelId: string | undefined; + if (persistentSessionName && effectiveBackendType !== 'pty') { const paneLive = effectiveBackendType === 'tmux' ? TmuxBackend.hasSession(persistentSessionName) : effectiveBackendType === 'zellij' ? ZellijBackend.hasSession(persistentSessionName) : HerdrBackend.hasSession(persistentSessionName); if (paneLive) { - let marker: string | null = null; - marker = readRegularHostFileNoFollow( - join(isolationRuntimeDataDir, 'read-isolation', `${cfg.sessionId}.boot`), + const markerPath = join( + isolationRuntimeDataDir, 'read-isolation', `${cfg.sessionId}.boot`, ); - if (isolatedPaneReattachSafe(marker, appliedIsolationCapabilities)) { + const marker = readManagedOriginAuthorityFile(markerPath); + const darwinPolicyExpected = process.platform === 'darwin' + && (willReadIsolate || willWriteSandbox); + // A stamped pane must match even when the new policy is OFF. Otherwise a + // disable followed by restart could reattach the still-confined process + // without rebuilding its authority/profile. An unsafe planted marker + // leaf is treated as stamped/unknown by the no-follow existence check. + const policyMatches = appliedIsolationCapabilities.length > 0 + ? isolatedPaneReattachSafe(marker, { + requiredCapabilities: appliedIsolationCapabilities, + exactCapabilities: true, + ...(darwinPolicyExpected ? { + readIsolation: willReadIsolate, + writeSandbox: willWriteSandbox, + requireOriginChannel: true, + policyDigest: darwinIsolationPolicyDigest!, + } : {}), + }) + : marker === null && !hostEntryExistsNoFollow(markerPath); + if (policyMatches) { + if (darwinPolicyExpected) { + persistentPaneOriginChannelId = isolatedPaneOriginChannel(marker); + } // Pane was spawned under the current isolation policy → still confined // on the running process across daemon restarts; warm reattach preserves // resume/context + tmux idle-suspend. @@ -6229,7 +9396,7 @@ async function spawnCli( killPersistentSession(effectiveBackendType as PersistentBackendType, persistentSessionName); } } - const willReattachPersistent = persistentSessionName + let willReattachPersistent = persistentSessionName ? effectiveBackendType === 'tmux' ? TmuxBackend.hasSession(persistentSessionName) : effectiveBackendType === 'zellij' @@ -6237,6 +9404,38 @@ async function spawnCli( : HerdrBackend.hasSession(persistentSessionName) : false; + // A pane created before asymmetric control framing has no persisted public + // identity capable of answering this worker's fresh challenge. Never fall + // back to terminal OSC trust: terminate it and cold-spawn a signed runner. + if (persistentSessionName && shouldColdStartCodexAppReattach({ + cliId: cfg.cliId, + backendType: effectiveBackendType, + isReattach: willReattachPersistent, + persistedState: readPersistedCodexAppControlState(cfg), + })) { + log(`Codex App persistent pane ${persistentSessionName} has no valid public control identity — killing + cold-spawning authenticated runner`); + try { + killPersistentSession(effectiveBackendType as PersistentBackendType, persistentSessionName); + } catch (err: any) { + throw new Error(`Refusing unauthenticated Codex App reattach: could not kill stale pane (${err?.message ?? err})`); + } + willReattachPersistent = false; + } + + // The worker establishes trust before any runner output can be parsed. A + // fresh runner gets a new capability; a persistent reattach reloads the + // capability created by that same runner generation. + // The control endpoint is an authenticated lifecycle prerequisite. Await + // bind + protected locator publication before backend.spawn so + // a runner can never observe a not-yet-owned or stale endpoint. + try { + await prepareCodexAppControlGeneration(cfg, willReattachPersistent, !!persistentSessionName); + } catch (err) { + if (spawnGeneration !== cliSpawnGeneration) throw new CliSpawnSupersededError(); + throw err; + } + if (spawnGeneration !== cliSpawnGeneration) throw new CliSpawnSupersededError(); + // The plugin set is stable only for the lifetime of one real CLI process. // A warm worker reattach keeps the existing Gateway and catalog untouched; // every fresh/resumed CLI spawn atomically refreshes both from current Bot config. @@ -6245,6 +9444,7 @@ async function spawnCli( ? readSessionMcpRuntimeManifest(cfg.sessionId, config.session.dataDir) : await prepareCliPluginGenerationAndGateway(cfg, cliAdapter); } + if (spawnGeneration !== cliSpawnGeneration) throw new CliSpawnSupersededError(); // Re-arm the startup-commands one-shot ONLY for a genuinely fresh CLI process. // A reattach to a LIVE persistent pane (daemon-restart recovery) is the SAME @@ -6408,6 +9608,7 @@ async function spawnCli( passesInitialPromptViaArgs: cliAdapter.passesInitialPromptViaArgs === true, adoptMode: cfg.adoptMode === true, dispatchAttempt: cfg.dispatchAttempt, + queuedActivationToken: cfg.queuedActivationToken, }) || (effectiveResume && cliAdapter.initialPromptArgsIgnoredOnResume === true) || (!promptArgPreparationChanged && shouldDeferInitialPromptForArgLimit({ passesInitialPromptViaArgs: cliAdapter.passesInitialPromptViaArgs === true, @@ -6440,23 +9641,71 @@ async function spawnCli( let readIsolationCtx: V2IsolationContext | undefined; if (willReadIsolate) { const sessionDataDir = process.env.SESSION_DATA_DIR!; + const osUserHomeDir = userInfo().homedir; + if (!osUserHomeDir) { + throw new Error('[read-isolation] OS account home is unavailable; refusing to create sentinel'); + } + readIsolationOriginChannelId = process.platform === 'darwin' + ? (persistentPaneOriginChannelId ?? randomBytes(32).toString('hex')) + : null; + if (readIsolationOriginChannelId) { + persistentPaneOriginChannelId = readIsolationOriginChannelId; + } readIsolationCtx = { homeDir: homedir(), + osUserHomeDir, botmuxHome: dirname(sessionDataDir), - defaultBotmuxHome: join(homedir(), '.botmux'), + defaultBotmuxHome: join(osUserHomeDir, '.botmux'), sessionDataDir, currentAppId: cfg.larkAppId, currentSessionId: cfg.sessionId, + ...(readIsolationOriginChannelId + ? { currentOriginChannelId: readIsolationOriginChannelId } + : {}), extraDenyPaths: cfg.readDenyExtraPaths, }; readIsolationOriginCapabilityFile = process.platform === 'darwin' - ? managedOriginCapabilityPath(sessionDataDir, cfg.sessionId) + ? managedOriginCapabilityPath( + sessionDataDir, + cfg.sessionId, + readIsolationOriginChannelId!, + ) : null; // Replace any legacy child-planted symlink before Seatbelt canonicalizes // carve-outs. A failure is fatal: spawning with a missing/ambiguous // capability transport would either break all IPC or reopen an attacker- // selected realpath in the generated profile. if (readIsolationOriginCapabilityFile) { + // Keep both probes. The legacy dashboard-secret inode catches panes + // created before the fixed sentinel existed; the OS-home sentinel covers + // custom HOME/BOTMUX roots where the daemon secret lives elsewhere. + ensureManagedOriginIsolationSentinel(osUserHomeDir); + ensureManagedOriginRootLocator(osUserHomeDir, cfg.sessionId, sessionDataDir); + const canonicalSessionDataDir = realpathSync(sessionDataDir); + ensureManagedOriginDataRootProbe(canonicalSessionDataDir, cfg.sessionId); + const legacyProbe = managedOriginLegacyIsolationProbeAccess(osUserHomeDir); + const fixedProbe = managedOriginIsolationSentinelAccess(osUserHomeDir); + const dataRootProbe = managedOriginDataRootProbeAccess( + canonicalSessionDataDir, + cfg.sessionId, + ); + if (legacyProbe !== 'host_accessible' && fixedProbe !== 'host_accessible') { + throw new Error('[read-isolation] kernel isolation probes are unavailable or unsafe'); + } + if (dataRootProbe !== 'host_accessible') { + throw new Error('[read-isolation] locator-selected data-root probe is unavailable or unsafe'); + } + ensureManagedOriginAttestationDirectory( + sessionDataDir, + cfg.sessionId, + readIsolationOriginChannelId!, + ); + sweepManagedOriginAttestationProofs( + sessionDataDir, + cfg.sessionId, + readIsolationOriginChannelId!, + ); + ensureManagedOriginCapabilityLeafSafe(readIsolationOriginCapabilityFile); publishSandboxRelayCapability({ failClosed: true }); } // Write this bot's OWN send-credential into its BOT_HOME (the same per-bot @@ -6476,6 +9725,9 @@ async function spawnCli( log(`[read-isolation] WARN could not write send-cred file: ${(e as Error).message}`); } } + else { + readIsolationOriginChannelId = null; + } // macOS write-sandbox rules (pure): the writable-zone allow-list + crown-jewel // re-denies. Realpath'd + emitted into the Seatbelt profile at the spawn site. let writeSandboxRules: { @@ -6501,7 +9753,7 @@ async function spawnCli( currentAppId: cfg.larkAppId, // TMPDIR on macOS resolves under /private/var/folders (already allowed), but // pass it explicitly so a customized TMPDIR is covered too. - extraWritePaths: process.env.TMPDIR ? [process.env.TMPDIR] : [], + extraWritePaths: darwinWriteExtraPaths, }); } const args = cliAdapter.buildArgs({ @@ -6608,6 +9860,9 @@ async function spawnCli( delete childEnv[MCP_GATEWAY_SOCKET_ENV]; delete childEnv[MCP_GATEWAY_REQUIRED_ENV]; } + // Never inherit an ambient/stale bootstrap path from the daemon's own launch + // environment. The raw capability is never placed in any environment. + delete childEnv[CODEX_APP_CONTROL_BOOTSTRAP_ENV]; // Put the daemon-written wrapper dir (~/.botmux/bin/botmux = THIS build) ahead of any // stale npm-global botmux in PATH, so the agent's `botmux` is always this build. Matters // most under read isolation: only this build has the send-cred reader — a shadowing stale @@ -6670,6 +9925,16 @@ async function spawnCli( // already guarantee the pane starts clean, and the CLI's built-in default // preserves stock semantics. if (isolationBotHome) { + // Classification hint only. A confined process can mutate its env, so + // cmdSend pairs this with a host-owned static marker and still requires a + // live daemon challenge; this bit never grants send authority. + childEnv.BOTMUX_READ_ISOLATED = '1'; + if (process.platform === 'darwin' && !readIsolationOriginChannelId) { + throw new Error('[read-isolation] origin channel is unavailable'); + } + if (readIsolationOriginChannelId) { + childEnv.BOTMUX_ORIGIN_CHANNEL_ID = readIsolationOriginChannelId; + } if (claudeDataDir) childEnv.CLAUDE_CONFIG_DIR = claudeDataDir; // = /claude else childEnv.CODEX_HOME = isolatedCodexHome!; } @@ -6729,6 +9994,7 @@ async function spawnCli( const profileCtx: V2IsolationContext = { ...readIsolationCtx, homeDir: canonical(readIsolationCtx.homeDir), + osUserHomeDir: canonical(readIsolationCtx.osUserHomeDir ?? readIsolationCtx.homeDir), botmuxHome: canonical(readIsolationCtx.botmuxHome), defaultBotmuxHome: canonical( readIsolationCtx.defaultBotmuxHome ?? join(readIsolationCtx.homeDir, '.botmux'), @@ -6741,13 +10007,22 @@ async function spawnCli( const capabilityCarvePath = managedOriginCapabilityPath( profileCtx.sessionDataDir, profileCtx.currentSessionId, + profileCtx.currentOriginChannelId!, + ); + const attestationCarvePath = managedOriginAttestationDirectory( + profileCtx.sessionDataDir, + profileCtx.currentSessionId, + profileCtx.currentOriginChannelId!, ); allowPaths = [ // Never realpath the capability leaf: another legacy sandbox can race // a symlink into that shared host directory while multiple persistent // sessions recover. Keeping the exact path may fail closed on a raced // symlink, but can never turn its target into a Seatbelt allow rule. - ...carve.allowPaths.map(path => path === capabilityCarvePath ? path : canonical(path)), + ...carve.allowPaths.map(path => + path === capabilityCarvePath || path === attestationCarvePath + ? path + : canonical(path)), ...buildCliExecutableReadCarveOuts({ homeDir: profileCtx.homeDir, cliId: cliAdapter.id, @@ -6855,9 +10130,25 @@ async function spawnCli( try { const markerDir = join(isolationRuntimeDataDir, 'read-isolation'); mkdirSync(markerDir, { recursive: true }); + if (process.platform === 'darwin' + && (willReadIsolate || willWriteSandbox) + && !persistentPaneOriginChannelId) { + persistentPaneOriginChannelId = randomBytes(32).toString('hex'); + } replaceManagedOriginCapabilityFile( join(markerDir, `${cfg.sessionId}.boot`), - isolationPaneMarkerContent(cfg.daemonBootId ?? '', appliedIsolationCapabilities), + isolationPaneMarkerContent( + cfg.daemonBootId ?? '', + appliedIsolationCapabilities, + process.platform === 'darwin' && (willReadIsolate || willWriteSandbox) + ? { + originChannelId: persistentPaneOriginChannelId!, + readIsolation: willReadIsolate, + writeSandbox: willWriteSandbox, + policyDigest: darwinIsolationPolicyDigest!, + } + : undefined, + ), ); } catch { /* non-fatal: worst case a same-lifetime reattach cold-spawns instead */ } } @@ -6984,7 +10275,18 @@ async function spawnCli( // bot's daemon-derived private CLI home after that parent mask, then // re-mask its send credential so an untrusted receiver cannot bypass // the host-authorized relay with a direct Lark API call. - trustedWritablePaths: readIsoLinuxMasks?.ownReadWritePaths ?? [], + trustedWritablePaths: [ + ...(readIsoLinuxMasks?.ownReadWritePaths ?? []), + ...(codexAppControlDirectoryForSpawn + ? [codexAppControlDirectoryForSpawn] + : []), + ...(codexAppControlSocketPathValue + ? [dirname(codexAppControlSocketPathValue)] + : []), + ...(codexAppControlLocatorPathValue + ? [dirname(codexAppControlLocatorPathValue)] + : []), + ], finalHidePaths: readIsoLinuxMasks ? [sendCredFilePath(dataDir, cfg.larkAppId)] : [], @@ -7210,14 +10512,54 @@ async function spawnCli( ); } - backend.spawn(spawnBin, spawnArgs, { - cwd: spawnCwd, - cols: PTY_COLS, - rows: PTY_ROWS, - env: childEnv as Record, - injectEnv: perBotInjectKeys.length ? perBotInjectEnv : undefined, - launchShell: lastInitConfig?.launchShell, - }); + // Create the asymmetric candidate as late as possible. Persistent backends + // receive it even on a predicted reattach: an actual reuse ignores the + // launch env and proves the old key, while an actual fresh start consumes + // the candidate and proves the new key. The worker trusts cryptographic + // proof, not a backend's prediction flag. + try { + if (spawnGeneration !== cliSpawnGeneration) throw new CliSpawnSupersededError(); + prepareFreshCodexAppControlBootstrap(cfg, !!persistentSessionName); + if (codexAppControlBootstrapPathForSpawn) { + childEnv[CODEX_APP_CONTROL_BOOTSTRAP_ENV] = codexAppControlBootstrapPathForSpawn; + } + backend.spawn(spawnBin, spawnArgs, { + cwd: spawnCwd, + cols: PTY_COLS, + rows: PTY_ROWS, + env: childEnv as Record, + injectEnv: perBotInjectKeys.length ? perBotInjectEnv : undefined, + launchShell: lastInitConfig?.launchShell, + }); + } catch (err) { + cleanupCodexAppControlBootstrap(); + throw err; + } finally { + delete childEnv[CODEX_APP_CONTROL_BOOTSTRAP_ENV]; + } + const actuallyReattachedPersistent = 'isReattach' in backend + && backend.isReattach === true; + try { + finalizeCodexAppControlGeneration( + cfg, + actuallyReattachedPersistent, + !!persistentSessionName, + ); + } catch (err) { + cleanupCodexAppControlBootstrap(); + // A runner whose generation cannot be authenticated durably must not keep + // executing behind an apparently failed worker. This covers both a + // fresh-persist failure or a generation that cannot complete challenge + // setup. + if (persistentSessionName) { + try { + killPersistentSession(effectiveBackendType as PersistentBackendType, persistentSessionName); + } catch (killErr: any) { + log(`Failed to kill unauthenticated Codex App persistent generation: ${killErr?.message ?? killErr}`); + } + } + throw err; + } // Write CLI PID marker so agent-facing subcommands (`botmux send`, etc.) // can verify they were spawned inside a botmux session by walking the @@ -7373,7 +10715,10 @@ async function spawnCli( if (effectiveCliSessionId) { const rolloutPath = findCodexRolloutBySessionId(effectiveCliSessionId); if (rolloutPath) { - codexBridgeAttach(rolloutPath, 'baseline-existing'); + codexBridgeAttach( + rolloutPath, + effectiveResume ? 'baseline-existing' : 'fresh-empty', + ); } else { codexBridgePendingSessionId = effectiveCliSessionId; codexBridgeStartTimer(); @@ -7389,7 +10734,10 @@ async function spawnCli( if (effectiveCliSessionId) { const rolloutPath = findTraexRolloutBySessionId(effectiveCliSessionId); if (rolloutPath) { - codexBridgeAttach(rolloutPath, 'baseline-existing'); + codexBridgeAttach( + rolloutPath, + effectiveResume ? 'baseline-existing' : 'fresh-empty', + ); } else { codexBridgePendingSessionId = effectiveCliSessionId; codexBridgeStartTimer(); @@ -7466,6 +10814,7 @@ async function spawnCli( cfg.sessionId, sandboxRelayCapability?.token, sandboxRelayOutbox ?? undefined, + readIsolationOriginChannelId ?? undefined, ); const readySignalAvailable = readyHookAvailable && readyPortAvailable && readyCapabilityAvailable; @@ -7481,6 +10830,7 @@ async function spawnCli( ]; log(`Ready gate skipped — preflight failed: ${reasons.join('; ')}`); } + ptyOutputGeneration.reset(); if (shouldArmReadyGate({ injectsReadyHook: cliAdapter.injectsReadyHook === true, readySignalAvailable, @@ -7537,12 +10887,39 @@ async function spawnCli( // prompt-ready — without this hook a follow-up arriving mid-task would sit // in pendingMessages forever once the task finishes. backend.onTaskDone?.(() => { + if (pendingRiffQueuedActivation) { + // create/follow-up failure can signal only taskDone (create has no id to + // clear). Never leave the token armed for a later unrelated task id. + failPendingRiffQueuedActivation('task ended without a new task id'); + return; + } + if (fatalWorkerErrorPending) return; log(`${cliName()} task finished — re-arming prompt-ready for queued follow-ups`); markPromptReady(); }); // riff：任务 id 变更同步给 daemon 持久化，daemon 重启后 follow-up 血缘不断。 backend.onTaskId?.((taskId) => { + // IPC is ordered per worker. Publish lineage first; the activation ACK + // follows only for the exact new non-null id and carries that id so the + // daemon can commit lineage+journal-clear atomically. send({ type: 'riff_task_id', taskId }); + const pending = pendingRiffQueuedActivation; + if (!pending) return; // includes the spawn-time resumeParentTaskId callback + if (taskId === null) { + failPendingRiffQueuedActivation('backend cleared lineage before acceptance'); + return; + } + pendingRiffQueuedActivation = null; + send({ + type: 'queued_activation_submitted', + sessionId, + activationToken: pending.token, + riffTaskId: taskId, + }); + log( + `Riff queued activation ${pending.token.substring(0, 8)} accepted as task ${taskId}; ` + + 'lineage IPC sent before activation ACK', + ); }); if (backend instanceof HerdrBackend) { wireHerdrWebTerminalRelays(backend); @@ -7571,6 +10948,32 @@ async function spawnCli( exitedDispatchAttempt, ); log(`${cliName()} exited (code: ${code}, signal: ${signal})`); + if (lastInitConfig?.cliId === 'codex-app' && codexAppControlFatal) { + // An earlier control-path failure already made this whole worker + // generation terminal and scheduled its hard OS exit. That failure may + // synchronously kill the backend, invoking this callback after + // stopCodexAppControlChannel() cleared the local FIFO. Never reinterpret + // that cleared snapshot as safe ownership: publishing ambiguous or + // claude_exit here would let the daemon admit N+1 before the Node-worker + // exit arms the durable recovery fence. + log('Suppressed Codex App backend-exit signals for a fatal worker generation'); + return; + } + const codexAppPreparedAtExit = !intentionalRestart + && lastInitConfig?.cliId === 'codex-app' + && (codexAppTurnDispatchQueue.size() > 0 + || codexAppRecoveredDispatches.some(entry => entry.state === 'prepared')); + if (codexAppPreparedAtExit) { + // Do not publish a standalone ambiguous terminal or claude_exit first. + // Either signal can let the durable receiver admit N+1 before the later + // Node-worker exit arms its persistent-pane fence. Exit this worker + // generation directly; daemon onWorkerExit performs the ambiguous + // transition and recovery-arm as one ordering boundary. + failCodexAppControlGeneration( + 'Codex App runner exited with a prepared dispatch; worker replacement requires exact recovery', + ); + return; + } if (cliAdapter?.reliableTurnTerminal === true && exitedTurnId && exitedDispatchAttempt !== undefined) { @@ -7605,6 +11008,22 @@ async function spawnCli( stopCodexRpcEngine(); if (rpcDialogDismissTimer) { clearTimeout(rpcDialogDismissTimer); rpcDialogDismissTimer = null; } } + cleanupCodexAppControlBootstrap(); + // A natural Codex App runner exit is only the CLI-generation terminal, + // not proof that any prepared frame was unconsumed. Keep the exact local + // FIFO/recovery snapshot until the daemon answers claude_exit with its + // cli_crash decision. The restart handler then sees prepared ownership and + // exits this Node worker too, which is what drives the daemon's + // onWorkerExit durable-receipt fence. Intentional in-worker teardown has + // already established its own replay/cancel boundary and may clear here. + stopCodexAppControlChannel({ + preserveDispatchRecovery: lastInitConfig?.cliId === 'codex-app' && !intentionalRestart, + }); + codexAppControlStateValue = undefined; + codexAppControlProven = false; + codexAppTurnLiveness.clear(); + codexAppReadyAuthority.reset(); + codexAppCompletionAwaitingFinal = false; const logTail = recentTerminalLogTail(); // Don't park a diagnostic shell here: most exits are immediately // auto-restarted by the daemon, so an inline park would just be torn down @@ -7638,8 +11057,16 @@ async function spawnCli( } }); - if (isPipeMode && backend && 'isReattach' in backend && backend.isReattach) { + const isPersistentBackendReattach = actuallyReattachedPersistent; + // Codex App warm observation is armed only after the old generation proves + // its private key over this worker's fresh socket challenge. Backend flags + // remain useful for screen seeding but are not authentication evidence. + + if (isPipeMode && backend && isPersistentBackendReattach) { log(`Re-attached to existing ${effectiveBackendType} session via pipe backend: ${persistentSessionName}`); + // Pipe backends expose an authoritative snapshot + busy-pattern probe. + // Keep those operations pipe-only; the liveness slot above is backend- + // independent and Zellij receives its screen through normal PTY output. seedBackendScreen(`${effectiveBackendType} reattach`, backend); scheduleReattachIdleProbe(`${effectiveBackendType} reattach`, backend); } @@ -7711,16 +11138,30 @@ async function spawnCli( } } +// Invalidate an in-flight async spawn before any other teardown hook runs. The +// remaining cleanup is synchronous today, but generation ownership must not +// depend on that implementation detail. function killCli(opts: { preservePending?: boolean } = {}): void { + cliSpawnGeneration++; currentCliCredentialIsolated = false; stopSessionMcpGatewayHost(); stopCodexRpcEngine(); + cleanupCodexAppControlBootstrap(); + stopCodexAppControlChannel(); + codexAppControlStateValue = undefined; + codexAppControlProven = false; + codexAppTurnLiveness.clear(); + codexAppReadyAuthority.reset(); + codexAppCompletionAwaitingFinal = false; if (!opts.preservePending) cleanupPiInitialPromptFiles(); destroyCrashDiagnosticTerminal('killCli'); idleDetector?.dispose(); idleDetector = null; stopReattachIdleProbe(); stopBusyPatternIdleProbe(); + stopStructuredStartGraceRecheck(); + structuredRejectedReadyEvidenceGeneration = undefined; + ptyOutputGeneration.reset(); // Cancel any pending ready-gate fallback / settle timers; spawnCli re-arms on respawn. if (readySignalTimer) { clearTimeout(readySignalTimer); readySignalTimer = null; } if (readyFlushSettleTimer) { clearTimeout(readyFlushSettleTimer); readyFlushSettleTimer = null; } @@ -7757,6 +11198,7 @@ function killCli(opts: { preservePending?: boolean } = {}): void { } sandboxRelayOutbox = null; readIsolationOriginCapabilityFile = null; + readIsolationOriginChannelId = null; currentBotmuxTurnId = undefined; currentBotmuxDispatchAttempt = undefined; currentVcMeetingImTurnOrigin = undefined; @@ -7765,8 +11207,11 @@ function killCli(opts: { preservePending?: boolean } = {}): void { sandboxCleanup = null; } isPromptReady = false; - clearSessionRenameInFlight(); - if (!opts.preservePending) pendingMessages.length = 0; + forceClearSessionRenameInFlight(); + if (!opts.preservePending) { + pendingMessages.length = 0; + pendingAdoptMessages.length = 0; + } // pendingRawInputs contains only commands that were accepted but never typed // because /rename owned the TUI or an owned CLI restart was already fenced. // Preserve them across restart; unlike an in-flight raw command, replaying @@ -7794,6 +11239,18 @@ async function restartCliProcess( log(`Restart ignored in adopt mode (${reason})`); return; } + if (effectiveBackendType === 'riff') { + // Direct-call defense for lease/reset paths: only the IPC restart handler + // should normally reach here for an operator restart, but no caller may + // replace a Riff generation without a daemon persistence handshake for a + // task id that might still be materializing. + log(`Restart ignored for Riff backend (${reason})`); + return; + } + // Invalidate queued/deferred callbacks before async teardown begins. Waiting + // until killCli/spawnCli would leave a window where an old timer can mutate + // the next durable attempt while destroySession is still awaiting. + cliSpawnGeneration += 1; // Set before touching destroySession(): remote teardown can await for many // seconds while the old backend object is still non-null and still capable // of firing idle/task-done callbacks. Inputs accepted in that interval must @@ -7820,12 +11277,6 @@ async function restartCliProcess( const rpcThreadId = remoteThreadId; const restartingBackend = backend; if (restartingBackend) intentionalRestartBackend = restartingBackend; - // Riff teardown cancels a remote task asynchronously. Wait for the cancel - // (bounded) before respawning, otherwise the old and replacement tasks can - // overlap and both emit into the same Lark thread. Local backends normally - // return void here and continue synchronously. A synchronous throw is - // handled by the same fatal restart path below, so it cannot leave the - // input gate permanently armed. const teardown = restartingBackend?.destroySession?.(); if (teardown && typeof (teardown as Promise).then === 'function') { try { @@ -7857,6 +11308,7 @@ async function restartCliProcess( if (codexRpcEngine) armRpcStartupDialogDismiss(); } catch (err) { cliRestartInProgress = false; + if (err instanceof CliSpawnSupersededError) return; await sendFatalWorkerErrorAndExit(err); return; } @@ -8963,7 +12415,7 @@ function emitTurnTerminal( if (!sessionId || !turnId) return; if (!emittedTurnTerminals.claim(sessionId, turnId, dispatchAttempt)) return; if (status !== 'completed') { - const dropped = codexBridgeQueue.dropPendingTurn(turnId, dispatchAttempt); + const dropped = codexBridgeQueue.dropPendingTurn(turnId, dispatchAttempt, true); if (dropped) { log(`Structured bridge retired terminal attempt turn=${turnId.slice(0, 12)} attempt=${dispatchAttempt ?? '-'} status=${status}`); } @@ -9031,10 +12483,21 @@ function sendAndFlush(msg: WorkerToDaemon): Promise { resolve(); return; } + let settled = false; + const finish = (): void => { + if (settled) return; + settled = true; + clearTimeout(timer); + resolve(); + }; + // IPC callbacks are best-effort transport evidence, not an exit gate. A + // parent that has stopped draining (or a runtime callback edge) must not + // keep a fail-closed worker and its stale ownership alive indefinitely. + const timer = setTimeout(finish, 1_000); try { - process.send(payload, () => resolve()); + process.send(payload, finish); } catch { - resolve(); + finish(); } }); } @@ -9049,6 +12512,7 @@ async function sendFatalWorkerErrorAndExit( err: unknown, turnId = currentBotmuxTurnId, dispatchAttempt = currentBotmuxDispatchAttempt, + opts: { hardExit?: boolean } = {}, ): Promise { if (fatalWorkerErrorPending) return; fatalWorkerErrorPending = true; @@ -9060,6 +12524,19 @@ async function sendFatalWorkerErrorAndExit( // failure to the receipt/lease recovery path instead of replying out-of-band. ...(dispatchAttempt !== undefined ? { dispatchAttempt } : {}), }); + log('Fatal worker error delivered; exiting process'); + if (opts.hardExit) { + // A fail-closed Codex App generation must produce a real OS-level worker + // exit so the daemon can arm its worker-generation receipt fence. On some + // runtimes process.exit() can hang joining a native/libuv fs worker (the + // control/terminal stack uses long-lived reads). Perform the synchronous + // cleanup we control, then use an uncatchable self-signal for this one + // replacement path. Persistent-session and SIGKILL residue both have + // daemon/next-worker reconcilers as additional backstops. + try { teardownSandboxBestEffort(); } catch { /* best effort */ } + try { cleanup(); } catch { /* best effort */ } + try { process.kill(process.pid, 'SIGKILL'); } catch { /* fall through */ } + } process.exit(1); } @@ -9094,6 +12571,23 @@ process.on('message', async (raw: unknown) => { if (msg.larkAppId && process.env.BOTMUX_WORKFLOW !== '1') { sessionStore.init(msg.larkAppId); } + if (msg.cliId === 'codex-app') { + codexAppRecoveredDispatches = (msg.codexAppRecoveredDispatches ?? []).map(entry => ({ + ...entry, + codexAppInput: entry.codexAppInput ? { ...entry.codexAppInput } : undefined, + })); + codexAppGenerationCommits = [...(msg.codexAppGenerationCommits ?? [])]; + codexAppTurnDispatchQueue.restore( + codexAppRecoveredDispatches + .filter(entry => entry.state === 'prepared') + .map(entry => ({ + dispatchId: entry.dispatchId, + turnId: entry.turnId, + replyTurnId: entry.replyTurnId, + dispatchAttempt: entry.dispatchAttempt, + })), + ); + } // Capture credentials for direct image upload from worker larkAppIdForUpload = msg.larkAppId; larkAppSecretForUpload = msg.larkAppSecret; @@ -9164,6 +12658,14 @@ process.on('message', async (raw: unknown) => { } await spawnCli(msg, { pluginGenerationPrepared: rpcPluginGenerationPrepared }); if (codexRpcEngine) armRpcStartupDialogDismiss(); // boundary #4: keep the --remote pane from freezing on a startup dialog + if (deferredFreshRpcTurn) { + const deferred = deferredFreshRpcTurn; + deferredFreshRpcTurn = undefined; + releaseRpcTurnTerminalDeferral( + deferred.identity, + deferred.generation, + ); + } // Queue the initial prompt — flushed when CLI shows idle. // Adapters with passesInitialPromptViaArgs (e.g. Gemini -i) bake the @@ -9185,7 +12687,8 @@ process.on('message', async (raw: unknown) => { // Mark it here before the CLI starts processing; late-attach is fine // because CodexBridgeQueue is path-agnostic until ingest discovers the // transcript file. - codexBridgeMarkPendingTurn(msg.prompt, msg.turnId, msg.dispatchAttempt); + const bridgeTurnId = codexBridgeMarkPendingTurn(msg.prompt, msg.turnId, msg.dispatchAttempt); + if (bridgeTurnId) codexBridgeQueue.confirmPendingTurn(bridgeTurnId, undefined, msg.dispatchAttempt); } // Queue the initial prompt for flushPending (pure decision, unit-tested): // - paste (no engine): normal queue. @@ -9194,6 +12697,22 @@ process.on('message', async (raw: unknown) => { // execution — exactly-once, Codex P1-1). Ambiguous never reaches here. // - RPC RESUME: queuePrompt=true → queue for post-ready flush (bridge // marked at flush time, P0-1). + const recoveredAcceptedEntries = codexAppRecoveredDispatches + .filter(entry => entry.state === 'accepted'); + const recoveredAcceptedInputs = recoveredAcceptedEntries + .map(entry => ({ + content: entry.content, + turnId: entry.turnId, + replyTurnId: entry.replyTurnId, + dispatchAttempt: entry.dispatchAttempt, + codexAppDispatchId: entry.dispatchId, + queuedActivationToken: entry.queuedActivationToken + ?? (recoveredAcceptedEntries.length === 1 + ? msg.queuedActivationToken + : undefined), + codexAppInput: entry.codexAppInput, + vcMeetingImTurnOrigin: entry.vcMeetingImTurnOrigin, + })); if (shouldQueueInitialPrompt({ hasPrompt: !!msg.prompt, rpcEngineActive: !!codexRpcEngine, @@ -9201,26 +12720,36 @@ process.on('message', async (raw: unknown) => { passesInitialPromptViaArgs: cliAdapter?.passesInitialPromptViaArgs === true, deferInitialPrompt, })) { - pendingMessages.push({ + // Accepted entries that never crossed the old worker's prepared/write + // boundary are replayable payload, not runner attribution. Queue + // them before this fork's new prompt while prepared predecessors stay + // preloaded in the exact final FIFO above. + pendingMessages.unshift(...recoveredAcceptedInputs, { content: lastSpawnQueuedInitialPrompt ?? msg.prompt, ...(lastSpawnQueuedInitialPromptLogicalContent ? { logicalContent: lastSpawnQueuedInitialPromptLogicalContent } : {}), turnId: msg.turnId, + replyTurnId: msg.replyTurnId, dispatchAttempt: msg.dispatchAttempt, + codexAppDispatchId: msg.codexAppDispatchId, + queuedActivationToken: msg.queuedActivationToken, vcMeetingImTurnOrigin: msg.vcMeetingImTurnOrigin, codexAppInput: msg.promptCodexAppInput, }); + } else if (msg.cliId === 'codex-app') { + pendingMessages.unshift(...recoveredAcceptedInputs); } - // Riff (remote HTTP backends): spawnCli already marked the prompt ready - // (no local boot → isPromptReady=true immediately), but the initial prompt - // was queued AFTER spawnCli returned, so flushPending() ran on an empty - // queue. Flush now that the prompt is enqueued. isPromptReady is already - // true so the flush gates all pass. - if (effectiveBackendType === 'riff' && pendingMessages.length > 0) { - flushPending(); - } + // Any ordinary message IPC that arrived while spawnCli awaited is now + // behind the exact activation/recovery head and may safely flush. + initPromptMaterialized = true; + + // The runner may authenticate and publish signed idle while spawnCli is + // still resolving, before the init prompt/recovered accepted entries are + // materialized above. Re-run the normal flush unconditionally after + // queuing; every backend/readiness gate remains authoritative inside it. + if (pendingMessages.length > 0) void flushPending(); send({ type: 'ready', @@ -9231,16 +12760,47 @@ process.on('message', async (raw: unknown) => { dispatchAttempt: currentBotmuxDispatchAttempt, }); } catch (err: any) { + if (err instanceof CliSpawnSupersededError) return; await sendFatalWorkerErrorAndExit(err); return; } break; } + case 'codex_app_dispatch_persisted': { + const pending = codexAppPendingDaemonAcks.get(msg.requestId); + if (!pending) break; + codexAppPendingDaemonAcks.delete(msg.requestId); + clearTimeout(pending.timer); + pending.resolve(msg.ok); + if (!msg.ok) { + log(`Daemon rejected Codex App dispatch persistence: ${msg.error ?? 'unknown error'}`); + } + break; + } + case 'message': { - // Mark new turn baseline so the streaming card only shows this turn's content - renderer?.markNewTurn(); - const turnSeq = usageLimitTracker.beginTurn(currentUsageLimitSnapshot()); + if (effectiveBackendType === 'riff' + && (closeRequestInFlightId || preparedCloseRequestId || shutdownDetachRequestId)) { + log( + `Rejected turn ${msg.turnId ?? '?'} while Riff retirement fence is ` + + `${shutdownDetachRequestId ? `shutdown-${shutdownDetachPhase}` : preparedCloseRequestId ? 'close-prepared' : 'close-preparing'}`, + ); + send({ + type: 'user_notify', + message: t('worker.riff_close_in_progress'), + turnId: msg.turnId, + dispatchAttempt: msg.dispatchAttempt, + }); + break; + } + // Adopt handlers are async and serialized below, so their renderer/usage + // turn begins inside writeAdoptMessage. Non-adopt keeps the immediate + // baseline while the message waits for normal flush scheduling. + if (!lastInitConfig?.adoptMode) { + renderer?.markNewTurn(); + usageLimitTracker.beginTurn(currentUsageLimitSnapshot()); + } // Cancel any active tmux copy-mode scroll so user input reaches the CLI. if (tmuxScrolledHalfPages > 0) exitTmuxScrollMode(); let content = msg.content; @@ -9272,91 +12832,30 @@ process.on('message', async (raw: unknown) => { // Pass the message's own attempt (not the stale currentBotmux* from a // prior IM turn) so a durable delivery relaunch failure carries the // right attribution for the daemon's receipt/lease gate. + if (err instanceof CliSpawnSupersededError) return; await sendFatalWorkerErrorAndExit(err, msg.turnId, msg.dispatchAttempt); return; } } if (lastInitConfig?.adoptMode) { - // Adopt writes immediately below, so this turn really becomes current - // now. Non-adopt messages update the marker only when flushPending - // dequeues/writes them; doing it at IPC arrival lets a queued IM turn - // steal the identity of a still-running durable delivery. - currentBotmuxTurnId = msg.turnId; - currentBotmuxDispatchAttempt = msg.dispatchAttempt; - currentVcMeetingImTurnOrigin = msg.vcMeetingImTurnOrigin; - writeCliPidMarker(); - publishSandboxRelayCapability(); - // Bridge mode: capture transcript baseline BEFORE writing to the pane, - // so any assistant uuids appended after this point are attributed to - // *this* Lark turn (not local user activity in the pane). Mark may - // return false (baseline not ready) — we still write to the pane; - // user just won't get a final_output for this message. - if (bridgeJsonlPath) { - try { bridgeIngest(); } catch { /* best effort */ } - bridgeMarkPendingTurn(content, msg.turnId, msg.dispatchAttempt); - } else if (codexBridgeFallbackActive()) { - // Codex adopt: same idea, different bridge. ingest first so any - // in-flight events from a local-typed prior turn close before - // this Lark turn's fingerprint window opens. Mark works even - // pre-attach (queue is path-agnostic). - if (codexBridgeIsCursor()) { - // Cursor may append the current Lark/user line to its transcript - // before this IPC message is handled. Mark first so that preexisting - // current-line can still fingerprint-match instead of being marked - // seen as an unmatched event. - codexBridgeMarkPendingTurn(content, msg.turnId, msg.dispatchAttempt); - try { codexBridgeIngest(); } catch { /* best effort */ } - } else { - try { codexBridgeIngest(); } catch { /* best effort */ } - codexBridgeMarkPendingTurn(content, msg.turnId, msg.dispatchAttempt); - } - } - // Adopt mode write: - // - Structured-bridge adopt-input CLIs (codex/traex/pi/grok/mtr) - // route through cliAdapter.writeInput so paste+Enter-retry + - // transcript/history verify (and grok's prompt_history → - // cliSessionId re-attach after /new) all run. Hardcoding only - // codex/traex left grok on raw sendText, so /new rotation never - // re-attached the bridge. - // - everything else keeps the simple raw sendText+Enter — the - // claude-code adopt bridge has its own dual-write recovery - // path, and the other CLIs' adopt flows haven't surfaced - // this submit-detection issue. - if (backend) { - if (isStructuredBridgeAdoptInputCli(lastInitConfig?.cliId) && cliAdapter) { - // writeInput is async but we're already inside an async - // message handler. Errors are best-effort logged; the bridge - // ingest path is unaffected because mark already happened - // above (codexBridgeMarkPendingTurn / bridgeMarkPendingTurn). - try { - const result = await cliAdapter.writeInput(backend as unknown as PtyHandle, content); - if (result?.cliSessionId) { - persistCliSessionId(result.cliSessionId); - codexBridgeNotifyCliSessionId(result.cliSessionId); - } - if (result && result.submitted === false) { - scheduleSubmitFailureNotify(content, result.recheck, 'submit history', undefined, result.failureReason, turnSeq); - } - } catch (err: any) { - log(`Adopt writeInput error (${lastInitConfig?.cliId}): ${err.message}`); - } - } else if ('sendText' in backend && 'sendSpecialKeys' in backend) { - (backend as any).sendText(content); - // Beat between text and Enter so the adopted CLI's input layer - // has time to register the typed chars before submit. Without - // this, Ink-based TUIs (CoCo, Claude Code) flag the rapid - // input+Enter as paste continuation and treat the trailing - // Enter as a soft-newline, leaving the message stranded in the - // input box. 200ms mirrors the per-adapter writeInput delay - // that fresh-spawn mode goes through and matches the slash- - // command (raw_input) fix. - await new Promise(r => setTimeout(r, 200)); - (backend as any).sendSpecialKeys('Enter'); - } else { - backend.write(content + '\r'); - } - isPromptReady = false; - idleDetector?.reset(); + const item: PendingCliInput = { + content, + turnId: msg.turnId, + dispatchAttempt: msg.dispatchAttempt, + vcMeetingImTurnOrigin: msg.vcMeetingImTurnOrigin, + codexAppInput, + }; + // process.on('message') does not serialize async listeners. Hold the + // per-worker queue across transcript mark + complete adapter write so + // two CoCo/Codex paste→wait→Enter/history cycles cannot overlap. + if (cliRestartInProgress || rawInputRestartGate || !backend || sessionRenameInFlight()) { + pendingAdoptMessages.push(item); + log(`Deferred adopt message until the CLI generation/input gate settles (${content.length} chars)`); + } else { + await runAdoptMessageForCapturedGeneration(item, () => { + pendingAdoptMessages.push(item); + log(`Re-queued stale adopt message for the replacement CLI generation (${content.length} chars)`); + }); } } else { // Non-adopt: enqueue only. Bridge mark is deferred to flushPending @@ -9367,6 +12866,9 @@ process.on('message', async (raw: unknown) => { sendToPty(content, msg.turnId, { codexAppInput, dispatchAttempt: msg.dispatchAttempt, + codexAppDispatchId: msg.codexAppDispatchId, + queuedActivationToken: msg.queuedActivationToken, + replyTurnId: msg.replyTurnId, vcMeetingImTurnOrigin: msg.vcMeetingImTurnOrigin, }); } @@ -9374,6 +12876,10 @@ process.on('message', async (raw: unknown) => { } case 'raw_input': { + if (shutdownDetachRequestId || closeRequestInFlightId || preparedCloseRequestId) { + log(`Rejected passthrough slash command during Riff retirement fence: ${msg.content}`); + break; + } // Preserve legacy busy delivery (/btw and other steering commands). A // native /rename and an owned CLI restart are the exceptions: never splice // into the rename UI, write through an old backend during async teardown, @@ -9383,7 +12889,7 @@ process.on('message', async (raw: unknown) => { // barrier（shouldDeferUserFlush——/cd 未落地前任何用户输入都不得写入， // 否则 passthrough 会执行在旧 cwd 的 CLI 里）时入队。注入排空后由 // flushPendingInjections 的 finally 补踢 flushPending 送达。 - if (cliRestartInProgress || rawInputRestartGate || sessionRenameInFlight + if (cliRestartInProgress || rawInputRestartGate || sessionRenameInFlight() || injectionFlushing || shouldDeferUserFlush(pendingInjections)) { pendingRawInputs.push(msg); log(`Deferred passthrough slash command until CLI input gate settles: ${msg.content}`); @@ -9415,7 +12921,53 @@ process.on('message', async (raw: unknown) => { } case 'restart': { - await restartCliProcess('daemon request', { preservePending: true }); + if (effectiveBackendType === 'riff') { + // Riff replacement cannot be made safe inside this worker alone: a + // task id may still be materializing, and lastInitConfig carries only + // the lineage known at init time. Preserve this generation so its + // onTaskId IPC remains authoritative. Explicit close is the only + // supported Riff retirement boundary. + log('Refused Riff generation restart; live worker and remote task retained'); + send({ + type: 'user_notify', + turnId: currentBotmuxTurnId, + dispatchAttempt: currentBotmuxDispatchAttempt, + message: t('worker.riff_restart_unsupported'), + }); + break; + } + const codexAppHasPreparedOwnership = codexAppTurnDispatchQueue.size() > 0 + || codexAppRecoveredDispatches.some(entry => entry.state === 'prepared'); + const codexAppHasAnyOwnership = codexAppHasPreparedOwnership + || pendingMessages.some(item => !!item.codexAppDispatchId) + || codexAppRecoveredDispatches.length > 0; + if (lastInitConfig?.cliId === 'codex-app' + && msg.reason === 'cli_crash' + && codexAppHasPreparedOwnership) { + // The crashed runner cannot prove whether a prepared frame executed. + // Exit the Node worker too: daemon onWorkerExit atomically arms durable + // fencing, while ordinary prepared input remains explicitly ambiguous. + failCodexAppControlGeneration( + 'Codex App runner exited with a prepared dispatch; worker replacement requires exact recovery', + ); + break; + } + if (lastInitConfig?.cliId === 'codex-app' + && msg.reason !== 'cli_crash' + && codexAppHasAnyOwnership) { + log('Refused in-worker Codex App restart while durable dispatch ledger ownership is non-empty'); + send({ + type: 'user_notify', + turnId: currentBotmuxTurnId, + dispatchAttempt: currentBotmuxDispatchAttempt, + message: 'Codex App 当前仍有未结算消息，暂不能重启；请等待本轮完成或关闭会话。', + }); + break; + } + await restartCliProcess( + msg.reason === 'cli_crash' ? 'CLI crash recovery' : 'daemon request', + { preservePending: true }, + ); break; } @@ -9435,6 +12987,10 @@ process.on('message', async (raw: unknown) => { log('Refused durable expiry ACK for adopt-mode session'); break; } + if (!await retireCodexAppDispatchForDurableReplay(msg.turnId, msg.dispatchAttempt)) { + log('Withholding durable expiry ACK because the daemon dispatch ledger could not retire exactly'); + break; + } log( `Expiring active durable turn=${msg.turnId.slice(0, 12)} attempt=${msg.dispatchAttempt}; ` + 'restarting CLI with queued follow-ups preserved', @@ -9453,15 +13009,10 @@ process.on('message', async (raw: unknown) => { // ordinary IM turn. Remove that exact queued generation and ACK without // restarting the unrelated active turn; otherwise attempt N survives in // the queue and executes after the hub has already replayed N+1. - let removedPending = 0; - for (let i = pendingMessages.length - 1; i >= 0; i--) { - const item = pendingMessages[i]; - if (item.turnId === msg.turnId && item.dispatchAttempt === msg.dispatchAttempt) { - pendingMessages.splice(i, 1); - removedPending++; - } - } - if (removedPending > 0) { + const removedPending = pendingMessages.filter(item => item.turnId === msg.turnId + && item.dispatchAttempt === msg.dispatchAttempt).length; + if (removedPending > 0 + && await retireCodexAppDispatchForDurableReplay(msg.turnId, msg.dispatchAttempt)) { log( `Expired ${removedPending} queued durable input(s) turn=${msg.turnId.slice(0, 12)} ` + `attempt=${msg.dispatchAttempt}`, @@ -9494,6 +13045,10 @@ process.on('message', async (raw: unknown) => { `Boot recovery fencing ambiguous receiver turn=${msg.turnId.slice(0, 12)} ` + `attempt=${msg.dispatchAttempt}`, ); + if (!await retireCodexAppDispatchForDurableReplay(msg.turnId, msg.dispatchAttempt)) { + log('Withholding receiver reset ACK because the daemon dispatch ledger could not retire exactly'); + break; + } durableTurnInFlight = false; inflightInputs.onTurnComplete(); await restartCliProcess('ambiguous receiver boot recovery', { immediate: true, preservePending: true }); @@ -9607,16 +13162,75 @@ process.on('message', async (raw: unknown) => { case 'close': { log('Close requested'); + if (effectiveBackendType === 'riff') { + if (!msg.requestId) { + // Fail closed against old/new callsites accidentally treating a + // generation retirement as explicit abandon. Exiting after an + // unacknowledged cancellation would orphan a credential-bearing task. + log('Refused unsafe request-less Riff close; explicit close requires prepare/commit'); + break; + } + // Two-phase explicit close. The Riff backend first fences new writes, + // drains any in-flight create/follow-up, and returns the exact outcome + // of cancelling the final (including late-created) task. The worker + // stays alive after a successful ACK until the daemon durably closes + // the row and sends close_commit. + let result: SessionDestroyResult; + let attemptedPrepare = false; + if (shutdownDetachRequestId) { + result = { + ok: false, + error: `shutdown detach already ${shutdownDetachPhase ?? 'active'} as ${shutdownDetachRequestId}`, + }; + } else if (preparedCloseRequestId) { + result = { + ok: false, + error: `close already prepared as ${preparedCloseRequestId}`, + }; + } else if (closeRequestInFlightId) { + result = { + ok: false, + error: `close prepare already in flight as ${closeRequestInFlightId}`, + }; + } else { + attemptedPrepare = true; + lastAbortedCloseRequestId = null; + closeRequestInFlightId = msg.requestId; + try { + const raw = await backend?.destroySession?.(); + result = raw && typeof raw === 'object' && 'ok' in raw + ? raw as SessionDestroyResult + : { ok: true }; + } catch (err) { + result = { ok: false, error: err instanceof Error ? err.message : String(err) }; + } + } + if (result.ok) { + preparedCloseRequestId = msg.requestId; + } else if (attemptedPrepare) { + await backend?.abortDestroySession?.(); + lastAbortedCloseRequestId = msg.requestId; + } + if (closeRequestInFlightId === msg.requestId) closeRequestInFlightId = null; + send({ + type: 'close_result', + requestId: msg.requestId, + ok: result.ok, + ...(result.taskId ? { taskId: result.taskId } : {}), + ...(result.error ? { error: result.error } : {}), + }); + if (!result.ok) { + log(`Riff close prepare failed (${result.error ?? 'cancel failed'}); session stays active for retry`); + } + break; + } + stopScreenshotLoop(); - // destroySession kills tmux session permanently; kill() only detaches. - // riff 的 destroySession 是异步远端取消——必须有界 await：紧跟着的 - // process.exit 会掐断未发出的 fetch，让已关闭话题的远端 agent 继续跑。 + // Local close: destroySession kills persistent owned sessions. Riff has + // already been handled above and can never enter this request-less path. const closeTeardown = backend?.destroySession?.(); if (closeTeardown && typeof (closeTeardown as Promise).then === 'function') { - try { - // 预算层级见 RiffBackend.destroySession（总 deadline 20s）——这里 22s 只作兜底。 - await Promise.race([closeTeardown, new Promise((r) => setTimeout(r, 22_000))]); - } catch { /* logged inside destroySession */ } + try { await Promise.race([closeTeardown, new Promise((r) => setTimeout(r, 22_000))]); } catch { /* logged by backend */ } } killCli(); // Bridge marker file outlives a single CLI process (we keep it across @@ -9628,7 +13242,195 @@ process.on('message', async (raw: unknown) => { process.exit(0); } + case 'riff_shutdown_prepare': { + if (effectiveBackendType !== 'riff') { + send({ + type: 'riff_shutdown_result', + requestId: msg.requestId, + phase: 'prepare', + ok: false, + taskId: null, + error: 'not_riff_backend', + }); + break; + } + let result: SessionShutdownDetachResult; + const inputBlocker = riffWorkerShutdownInputBlocker({ + initPromptMaterialized, + isFlushing, + pendingMessages: pendingMessages.length, + pendingRawInputs: pendingRawInputs.length, + pendingSessionRename: pendingSessionRename !== null, + sessionRenameInFlight: sessionRenameInFlight(), + commandLineWritesPending, + }); + if (preparedCloseRequestId || closeRequestInFlightId) { + result = { ok: false, taskId: null, error: 'explicit_close_in_progress' }; + } else if (shutdownDetachRequestId) { + result = { + ok: false, + taskId: null, + error: `shutdown detach already ${shutdownDetachPhase ?? 'active'} as ${shutdownDetachRequestId}`, + }; + } else if (inputBlocker) { + // These inputs have been accepted by the worker but are not proven to + // be in RiffBackend.writeChain. Do not install a backend fence: the + // daemon can retain this live generation and retry shutdown after the + // current task boundary flushes the queue. + result = { + ok: false, + taskId: null, + error: `worker_inputs_not_drained:${inputBlocker}`, + }; + } else { + shutdownDetachRequestId = msg.requestId; + shutdownDetachPhase = 'preparing'; + try { + result = await backend?.prepareShutdownDetach?.() + ?? { ok: false, taskId: null, error: 'shutdown_detach_unsupported' }; + } catch (err) { + result = { + ok: false, + taskId: null, + error: err instanceof Error ? err.message : String(err), + }; + } + if (shutdownDetachRequestId === msg.requestId) { + if (result.ok) shutdownDetachPhase = 'prepared'; + // On failure the backend may still be fenced or its pre-fence drain + // may still be settling. Keep request ownership until the daemon's + // explicit abort receives an admission-restored ACK. + } + } + send({ + type: 'riff_shutdown_result', + requestId: msg.requestId, + phase: 'prepare', + ok: result.ok, + taskId: result.taskId, + ...(result.error ? { error: result.error } : {}), + }); + break; + } + + case 'riff_shutdown_commit': { + if (effectiveBackendType !== 'riff' + || shutdownDetachRequestId !== msg.requestId + || shutdownDetachPhase !== 'prepared') { + log(`Ignoring stale Riff shutdown commit ${msg.requestId}`); + break; + } + log(`Riff shutdown detach committed (${msg.requestId})`); + shutdownDetachRequestId = null; + shutdownDetachPhase = null; + backend?.commitShutdownDetach?.(); + // backend.kill() is planned process retirement, not a CLI crash. Mark it + // intentional before killCli so onExit never emits claude_exit/restart. + intentionalRestartBackend = backend; + stopScreenshotLoop(); + killCli(); + cleanup(); + process.exit(0); + } + + case 'riff_shutdown_abort': { + if (shutdownDetachRequestId !== msg.requestId) { + log(`Ignoring stale Riff shutdown abort ${msg.requestId}`); + send({ + type: 'riff_shutdown_result', + requestId: msg.requestId, + phase: 'abort', + ok: false, + taskId: null, + error: 'shutdown_detach_not_active', + }); + break; + } + log(`Riff shutdown detach aborted (${msg.requestId})`); + let result: SessionShutdownDetachResult; + try { + result = await backend?.abortShutdownDetach?.() + ?? { ok: false, taskId: null, error: 'shutdown_abort_unsupported' }; + } catch (err) { + result = { + ok: false, + taskId: null, + error: err instanceof Error ? err.message : String(err), + }; + } + if (result.ok && shutdownDetachRequestId === msg.requestId) { + shutdownDetachRequestId = null; + shutdownDetachPhase = null; + } + send({ + type: 'riff_shutdown_result', + requestId: msg.requestId, + phase: 'abort', + ok: result.ok, + taskId: result.taskId, + ...(result.error ? { error: result.error } : {}), + }); + break; + } + + case 'close_commit': { + if (!preparedCloseRequestId || preparedCloseRequestId !== msg.requestId) { + log(`Ignoring stale close_commit ${msg.requestId}`); + break; + } + log(`Close committed (${msg.requestId})`); + preparedCloseRequestId = null; + closeRequestInFlightId = null; + lastAbortedCloseRequestId = null; + backend?.commitDestroySession?.(); + stopScreenshotLoop(); + killCli(); + clearSendMarkers(); + cleanup(); + process.exit(0); + } + + case 'close_abort': { + const alreadyRestored = lastAbortedCloseRequestId === msg.requestId; + if (!alreadyRestored + && preparedCloseRequestId !== msg.requestId + && closeRequestInFlightId !== msg.requestId) { + log(`Ignoring stale close_abort ${msg.requestId}`); + send({ + type: 'close_abort_result', + requestId: msg.requestId, + ok: false, + error: 'close_abort_not_active', + }); + break; + } + log(`Close aborted (${msg.requestId}); Riff admission restored`); + let abortError: string | null = null; + if (!alreadyRestored) { + try { await backend?.abortDestroySession?.(); } + catch (err) { abortError = err instanceof Error ? err.message : String(err); } + } + if (!abortError) { + preparedCloseRequestId = null; + closeRequestInFlightId = null; + lastAbortedCloseRequestId = null; + } + send({ + type: 'close_abort_result', + requestId: msg.requestId, + ok: abortError === null, + ...(abortError ? { error: abortError } : {}), + }); + break; + } + case 'suspend': { + if (effectiveBackendType === 'riff') { + // Request-less suspend has no daemon persistence ACK. Exiting here can + // beat adoptLateTask's task-id IPC and lose the only retry handle. + log('Refused unsafe Riff suspend; explicit close requires prepare/commit'); + break; + } log('Suspend requested'); stopScreenshotLoop(); stopBridgeWatcher(); @@ -9647,10 +13449,7 @@ process.on('message', async (raw: unknown) => { // uses to recover sessions after a reboot kills the tmux server). revokeManagedTurnOriginForRestart(); try { - // riff：suspend 语义是「休眠待续」——绝不能 cancel 远端任务（血缘已持久化， - // 恢复时 follow-up 续上）；只断流 detach。 - if (effectiveBackendType === 'riff') backend?.kill(); - else (backend?.destroySession ?? backend?.kill)?.call(backend); + (backend?.destroySession ?? backend?.kill)?.call(backend); } catch { /* best-effort */ } backend = null; isPromptReady = false; @@ -9692,6 +13491,10 @@ function cleanup(): void { try { workflowPtyLogStream.end(); } catch { /* already closed */ } workflowPtyLogStream = undefined; } + // Publisher ownership spans every CLI generation in this worker. Releasing + // from killCli/restart would reopen a stale-publish window; only process-level + // cleanup retires the lease. SIGKILL residue is reclaimed by the next worker. + releaseCodexAppPosixOwnerLease(); } process.on('SIGTERM', () => { stopScreenshotLoop(); killCli(); cleanup(); process.exit(0); }); diff --git a/src/workflows/v3/session-relay-client.ts b/src/workflows/v3/session-relay-client.ts index 92c4aa469..6a96d6d7e 100644 --- a/src/workflows/v3/session-relay-client.ts +++ b/src/workflows/v3/session-relay-client.ts @@ -24,6 +24,7 @@ import { V3_SESSION_RUN_MUTATION_ROUTE_PREFIX } from './session-relay.js'; export interface WorkflowSessionRelayContext { sessionId: string; capability: string; + originChannelId?: string; turnId?: string; dispatchAttempt?: number; /** Routing hints only — never identity. */ @@ -58,14 +59,25 @@ export function readWorkflowSessionRelayContext(options: { if (marker?.sessionId) return null; const readClaim = options.readClaim ?? readManagedOriginCapability; const relayDir = options.env.BOTMUX_SEND_RELAY?.trim(); - const claim = readClaim(options.dataDir, sessionId, relayDir || undefined); + const originChannelId = options.env.BOTMUX_ORIGIN_CHANNEL_ID?.trim(); + const claim = readClaim( + options.dataDir, + sessionId, + relayDir || undefined, + originChannelId || undefined, + ); if (!claim) return null; const portRaw = Number(options.env.BOTMUX_DAEMON_IPC_PORT); - const ipcPortFallback = Number.isSafeInteger(portRaw) && portRaw > 0 ? portRaw : undefined; + // Persistent panes can retain a spawn-time daemon port across daemon + // restarts. Prefer the current worker-published port from the rotating, + // protected claim; the inherited env remains a compatibility fallback. + const ipcPortFallback = claim.ipcPort + ?? (Number.isSafeInteger(portRaw) && portRaw > 0 ? portRaw : undefined); const larkAppId = options.env.BOTMUX_LARK_APP_ID?.trim(); return { sessionId, capability: claim.capability, + ...(claim.channelId ? { originChannelId: claim.channelId } : {}), ...(claim.turnId ? { turnId: claim.turnId } : {}), ...(claim.dispatchAttempt !== undefined ? { dispatchAttempt: claim.dispatchAttempt } : {}), ...(larkAppId ? { larkAppId } : {}), @@ -88,7 +100,7 @@ export async function postWorkflowSessionRunMutation(input: { fetchImpl?: typeof fetch; }): Promise { const discovered = input.resolveIpcPort?.(input.context.larkAppId); - const ipcPort = discovered ?? input.context.ipcPortFallback; + const ipcPort = input.context.ipcPortFallback ?? discovered; if (!ipcPort) { throw new WorkflowDaemonMutationTransportError( '找不到目标 daemon 端口（daemon 发现目录不可见且缺少 BOTMUX_DAEMON_IPC_PORT）；请确认 daemon 在线', @@ -99,6 +111,9 @@ export async function postWorkflowSessionRunMutation(input: { ...(input.body ?? {}), sessionId: input.context.sessionId, originCapability: input.context.capability, + ...(input.context.originChannelId + ? { originChannelId: input.context.originChannelId } + : {}), ...(input.context.turnId ? { originTurnId: input.context.turnId } : {}), ...(input.context.dispatchAttempt !== undefined ? { originDispatchAttempt: input.context.dispatchAttempt } diff --git a/test/anchor-serializer.test.ts b/test/anchor-serializer.test.ts index a9f23372c..63cff9bf4 100644 --- a/test/anchor-serializer.test.ts +++ b/test/anchor-serializer.test.ts @@ -10,7 +10,7 @@ * * Run: pnpm vitest run test/anchor-serializer.test.ts */ -import { describe, it, expect, beforeEach } from 'vitest'; +import { describe, it, expect, beforeEach, vi } from 'vitest'; import { serializeByAnchor, __resetAnchorQueues } from '../src/utils/anchor-serializer.js'; const delayPush = (order: string[], label: string, ms: number) => () => @@ -84,4 +84,38 @@ describe('serializeByAnchor — wait cap (head-of-line-blocking guard)', () => { await Promise.all([p1, p2]); expect(order).toEqual(['s:1', 'e:1', 's:2', 'e:2']); }); + + it('keeps strict admission ordered past five seconds when cap=0', async () => { + vi.useFakeTimers(); + try { + const order: string[] = []; + let releaseFirst!: () => void; + const first = serializeByAnchor('strict', () => new Promise(resolve => { + order.push('start:1'); + releaseFirst = () => { + order.push('end:1'); + resolve(); + }; + }), 0); + const second = serializeByAnchor('strict', async () => { + order.push('start:2'); + }, 0); + + await Promise.resolve(); + await Promise.resolve(); + expect(order).toEqual(['start:1']); + + // The ordinary serializer deliberately escapes after 5s. A raw inbound + // admission lane must not: concurrent cold-session handlers can reorder + // or drop accepted turns even when the first await is merely slow. + await vi.advanceTimersByTimeAsync(5_100); + expect(order).toEqual(['start:1']); + + releaseFirst(); + await Promise.all([first, second]); + expect(order).toEqual(['start:1', 'end:1', 'start:2']); + } finally { + vi.useRealTimers(); + } + }); }); diff --git a/test/async-serial-queue.test.ts b/test/async-serial-queue.test.ts new file mode 100644 index 000000000..471dc19a6 --- /dev/null +++ b/test/async-serial-queue.test.ts @@ -0,0 +1,290 @@ +import { describe, expect, it } from 'vitest'; +import { + runAdoptQueuedWriteSequence, + runAdoptRawInputSequence, + runAdoptSessionRenameSequence, +} from '../src/services/adopt-input-sequence.js'; +import { AsyncSerialQueue } from '../src/utils/async-serial-queue.js'; + +function deferred() { + let resolve!: (value: T) => void; + let reject!: (reason?: unknown) => void; + const promise = new Promise((res, rej) => { + resolve = res; + reject = rej; + }); + return { promise, resolve, reject }; +} + +describe('AsyncSerialQueue', () => { + it('does not start a second adopt write until the first paste/verify cycle settles', async () => { + const queue = new AsyncSerialQueue(); + const firstGate = deferred(); + const order: string[] = []; + let composer = ''; + const submitted: string[] = []; + + const first = queue.run(async () => { + order.push('first:paste'); + composer = 'first prompt'; + await firstGate.promise; + submitted.push(composer); + order.push('first:verified'); + return 'first-result'; + }); + const second = queue.run(async () => { + order.push('second:paste'); + composer = 'second prompt'; + submitted.push(composer); + order.push('second:verified'); + return 'second-result'; + }); + + await Promise.resolve(); + expect(order).toEqual(['first:paste']); + firstGate.resolve(); + + await expect(first).resolves.toBe('first-result'); + await expect(second).resolves.toBe('second-result'); + expect(order).toEqual([ + 'first:paste', + 'first:verified', + 'second:paste', + 'second:verified', + ]); + expect(submitted).toEqual(['first prompt', 'second prompt']); + }); + + it('releases the next write after a prior write rejects', async () => { + const queue = new AsyncSerialQueue(); + const gate = deferred(); + const order: string[] = []; + const first = queue.run(async () => { + order.push('first:start'); + await gate.promise; + throw new Error('write failed'); + }); + const second = queue.run(async () => { + order.push('second:start'); + return 'recovered'; + }); + + await Promise.resolve(); + gate.resolve(); + await expect(first).rejects.toThrow('write failed'); + await expect(second).resolves.toBe('recovered'); + expect(order).toEqual(['first:start', 'second:start']); + }); + + it('holds a bundled raw follow-up through adapter settlement before the next adopt write', async () => { + const queue = new AsyncSerialQueue(); + const followUpStarted = deferred(); + const followUpVerification = deferred(); + const order: string[] = []; + const submitted: string[] = []; + let composer = ''; + + const rawAndFollowUp = runAdoptRawInputSequence({ + queue, + writeRawInput: async () => { + composer = '/model'; + order.push('raw:enter'); + submitted.push(composer); + composer = ''; + return true; + }, + writeFollowUp: async () => { + composer = 'bundled follow-up'; + order.push('follow-up:paste'); + followUpStarted.resolve(); + await followUpVerification.promise; + submitted.push(composer); + order.push('follow-up:settled'); + }, + }); + const nextMessage = queue.run(async () => { + composer = 'next adopt message'; + order.push('next:paste'); + submitted.push(composer); + order.push('next:settled'); + }); + + await followUpStarted.promise; + expect(order).toEqual(['raw:enter', 'follow-up:paste']); + expect(composer).toBe('bundled follow-up'); + + followUpVerification.resolve(); + await Promise.all([rawAndFollowUp, nextMessage]); + expect(order).toEqual([ + 'raw:enter', + 'follow-up:paste', + 'follow-up:settled', + 'next:paste', + 'next:settled', + ]); + expect(submitted).toEqual(['/model', 'bundled follow-up', 'next adopt message']); + }); + + it('does not write the bundled follow-up when the raw command was not sent', async () => { + const queue = new AsyncSerialQueue(); + const order: string[] = []; + + await runAdoptRawInputSequence({ + queue, + writeRawInput: async () => { + order.push('raw:failed'); + return false; + }, + writeFollowUp: async () => { + order.push('follow-up:must-not-run'); + }, + }); + + expect(order).toEqual(['raw:failed']); + }); + + it('requeues an untouched adopt write when its captured generation is stale at dequeue', async () => { + const queue = new AsyncSerialQueue(); + const priorGate = deferred(); + let generation = 1; + const order: string[] = []; + const prior = queue.run(async () => { + order.push('prior:start'); + await priorGate.promise; + order.push('prior:end'); + }); + const queued = runAdoptQueuedWriteSequence({ + queue, + isCurrent: () => generation === 1, + write: async () => { + order.push('stale:must-not-write'); + }, + onStale: () => { + order.push('stale:requeued'); + }, + }); + + await Promise.resolve(); + generation = 2; + priorGate.resolve(); + await prior; + await expect(queued).resolves.toEqual({ status: 'stale-before-write' }); + expect(order).toEqual(['prior:start', 'prior:end', 'stale:requeued']); + }); + + it('withholds a dependent bundled follow-up when generation changes after raw Enter', async () => { + const queue = new AsyncSerialQueue(); + let current = true; + const order: string[] = []; + + await runAdoptRawInputSequence({ + queue, + isCurrent: () => current, + writeRawInput: async () => { + order.push('raw:enter'); + current = false; + return true; + }, + writeFollowUp: async () => { + order.push('follow-up:must-not-write'); + }, + onStaleBeforeFollowUp: () => { + order.push('follow-up:withheld-notified'); + }, + }); + + expect(order).toEqual(['raw:enter', 'follow-up:withheld-notified']); + }); + + it('rechecks prompt readiness after prior adopt writes before native rename', async () => { + const queue = new AsyncSerialQueue(); + const priorWriteGate = deferred(); + const order: string[] = []; + let promptReady = true; + + const priorWrite = queue.run(async () => { + order.push('message:start'); + await priorWriteGate.promise; + promptReady = false; + order.push('message:submitted'); + }); + const staleRenameAttempt = runAdoptSessionRenameSequence({ + queue, + isPromptReady: () => promptReady, + writeRename: async () => { + order.push('rename:must-not-run'); + }, + }); + + await Promise.resolve(); + expect(order).toEqual(['message:start']); + priorWriteGate.resolve(); + await priorWrite; + await expect(staleRenameAttempt).resolves.toBe(false); + expect(order).toEqual(['message:start', 'message:submitted']); + + promptReady = true; + await expect(runAdoptSessionRenameSequence({ + queue, + isPromptReady: () => promptReady, + writeRename: async () => { + order.push('rename:enter'); + }, + })).resolves.toBe(true); + expect(order).toEqual(['message:start', 'message:submitted', 'rename:enter']); + }); + + it('starts rename only after a queued fast-final write has restored readiness', async () => { + const queue = new AsyncSerialQueue(); + const priorAdapterGate = deferred(); + const renameWriteGate = deferred(); + const order: string[] = []; + let promptReady = false; + let renamePhase: 'reserved' | 'writing' | 'sent' | 'idle' = 'reserved'; + + const priorWrite = queue.run(async () => { + order.push('message:adapter-wait'); + await priorAdapterGate.promise; + // Models assistant_final/prompt-ready arriving before writeInput returns. + promptReady = true; + order.push('message:fast-final'); + }); + const rename = runAdoptSessionRenameSequence({ + queue, + isPromptReady: () => promptReady, + writeRename: async () => { + promptReady = false; + renamePhase = 'writing'; + order.push('rename:text-to-enter-window'); + await renameWriteGate.promise; + renamePhase = 'sent'; + order.push('rename:enter-landed'); + }, + }); + + await Promise.resolve(); + expect(renamePhase).toBe('reserved'); + priorAdapterGate.resolve(); + await priorWrite; + await Promise.resolve(); + expect(order).toEqual(['message:adapter-wait', 'message:fast-final', 'rename:text-to-enter-window']); + expect(promptReady).toBe(false); + expect(renamePhase).toBe('writing'); + + // A terminal belonging to the previous model turn may arrive during the + // rename's 200ms text→Enter beat. Only `sent` may release the gate. + if (renamePhase === 'sent') renamePhase = 'idle'; + expect(renamePhase).toBe('writing'); + + renameWriteGate.resolve(); + await expect(rename).resolves.toBe(true); + // Command write completion alone does not represent the second prompt. + expect(renamePhase).toBe('sent'); + expect(order).toEqual([ + 'message:adapter-wait', + 'message:fast-final', + 'rename:text-to-enter-window', + 'rename:enter-landed', + ]); + }); +}); diff --git a/test/bot-registry.test.ts b/test/bot-registry.test.ts index 6e6f00a48..dafec5bb7 100644 --- a/test/bot-registry.test.ts +++ b/test/bot-registry.test.ts @@ -72,6 +72,12 @@ describe('registerBot', () => { expect(client.opts.appSecret).toBe('secret_001'); }); + it('bounds the SDK HTTP transport timeout when the client exposes axios defaults', () => { + const client = { httpInstance: { defaults: { timeout: 0 } } }; + mod.configureLarkClientHttpTimeout(client); + expect(client.httpInstance.defaults.timeout).toBe(mod.LARK_REQUEST_TIMEOUT_MS); + }); + it('should default the SDK Client domain to feishu when brand is unset', () => { const state = mod.registerBot(makeCfg()); const client = state.client as unknown as { opts: Record }; diff --git a/test/bot-routing.test.ts b/test/bot-routing.test.ts index 30b4d5457..90fda0b29 100644 --- a/test/bot-routing.test.ts +++ b/test/bot-routing.test.ts @@ -188,6 +188,13 @@ describe('buildFooterAddressing', () => { )).toEqual({ sendTo: 'ou_owner', cc: [] }); }); + it('honors a frozen bot-sender bit even before cross-ref learns that bot id', () => { + expect(buildFooterAddressing( + { ownerOpenId: 'ou_owner', lastCallerOpenId: 'ou_new_bot', lastCallerIsBot: true }, + { isOncall: true, knownBotOpenIds }, + )).toEqual({ sendTo: 'ou_owner', cc: [] }); + }); + it('drops addressing when the resolved recipient would be a bot', () => { expect(buildFooterAddressing( { ownerOpenId: 'ou_codex_bot', lastCallerOpenId: 'ou_claude_bot' }, diff --git a/test/bot-turn-mutation-gate.test.ts b/test/bot-turn-mutation-gate.test.ts new file mode 100644 index 000000000..5dff838e9 --- /dev/null +++ b/test/bot-turn-mutation-gate.test.ts @@ -0,0 +1,400 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { + __testOnly_resetBotTurnMutationGates, + runDetachedBotTurnAdmission, + runDetachedBotTurnMutation, + tryWithBotTurnMutation, + withBotTurnAdmission, + withBotTurnMutation, +} from '../src/core/bot-turn-mutation-gate.js'; + +function deferred(): { promise: Promise; resolve: () => void } { + let resolve!: () => void; + const promise = new Promise(done => { resolve = done; }); + return { promise, resolve }; +} + +describe('per-bot turn mutation gate', () => { + afterEach(() => { + vi.useRealTimers(); + __testOnly_resetBotTurnMutationGates(); + }); + + it('drains an in-flight admission and blocks new turns through the mutation', async () => { + const finishFirst = deferred(); + const finishMutation = deferred(); + const firstEntered = vi.fn(); + const mutationEntered = vi.fn(); + const secondEntered = vi.fn(); + + const first = withBotTurnAdmission('app-a', async () => { + firstEntered(); + await finishFirst.promise; + }); + await vi.waitFor(() => expect(firstEntered).toHaveBeenCalledOnce()); + + const mutation = withBotTurnMutation('app-a', async () => { + mutationEntered(); + await finishMutation.promise; + }); + await Promise.resolve(); + expect(mutationEntered).not.toHaveBeenCalled(); + + const second = withBotTurnAdmission('app-a', () => { secondEntered(); }); + await Promise.resolve(); + expect(secondEntered).not.toHaveBeenCalled(); + + finishFirst.resolve(); + await vi.waitFor(() => expect(mutationEntered).toHaveBeenCalledOnce()); + expect(secondEntered).not.toHaveBeenCalled(); + + finishMutation.resolve(); + await Promise.all([first, mutation, second]); + expect(secondEntered).toHaveBeenCalledOnce(); + }); + + it('does not serialize an unrelated bot', async () => { + const finishMutation = deferred(); + const mutation = withBotTurnMutation('app-a', () => finishMutation.promise); + const other = vi.fn(); + await withBotTurnAdmission('app-b', () => other()); + expect(other).toHaveBeenCalledOnce(); + finishMutation.resolve(); + await mutation; + }); + + it('keeps woken waiters on the canonical state for the next mutation', async () => { + const finishFirstMutation = deferred(); + const finishQueuedAdmission = deferred(); + const firstMutation = withBotTurnMutation('app-a', () => finishFirstMutation.promise); + const queuedEntered = vi.fn(); + const queuedAdmission = withBotTurnAdmission('app-a', async () => { + queuedEntered(); + await finishQueuedAdmission.promise; + }); + await Promise.resolve(); + expect(queuedEntered).not.toHaveBeenCalled(); + + finishFirstMutation.resolve(); + await firstMutation; + await vi.waitFor(() => expect(queuedEntered).toHaveBeenCalledOnce()); + + const nextMutationEntered = vi.fn(); + const nextMutation = withBotTurnMutation('app-a', () => nextMutationEntered()); + await Promise.resolve(); + expect(nextMutationEntered).not.toHaveBeenCalled(); + + finishQueuedAdmission.resolve(); + await Promise.all([queuedAdmission, nextMutation]); + expect(nextMutationEntered).toHaveBeenCalledOnce(); + }); + + it('lets a draining admitted handler enter a nested same-bot boundary', async () => { + const enterNested = deferred(); + const outerEntered = vi.fn(); + const nestedEntered = vi.fn(); + const mutationEntered = vi.fn(); + const outer = withBotTurnAdmission('app-a', async () => { + outerEntered(); + await enterNested.promise; + await withBotTurnAdmission('app-a', () => nestedEntered()); + }); + await vi.waitFor(() => expect(outerEntered).toHaveBeenCalledOnce()); + + const mutation = withBotTurnMutation('app-a', () => mutationEntered()); + await Promise.resolve(); + expect(mutationEntered).not.toHaveBeenCalled(); + + enterNested.resolve(); + await Promise.all([outer, mutation]); + expect(nestedEntered).toHaveBeenCalledOnce(); + expect(mutationEntered).toHaveBeenCalledOnce(); + }); + + it('revokes inherited reentrancy for a detached descendant after the outer lease ends', async () => { + const releaseDetached = deferred(); + const finishMutation = deferred(); + const detachedEntered = vi.fn(); + let detached!: Promise; + + await withBotTurnAdmission('app-a', async () => { + detached = (async () => { + await releaseDetached.promise; + await withBotTurnAdmission('app-a', () => detachedEntered()); + })(); + }); + + const mutation = withBotTurnMutation('app-a', () => finishMutation.promise); + await Promise.resolve(); + releaseDetached.resolve(); + await Promise.resolve(); + await Promise.resolve(); + expect(detachedEntered).not.toHaveBeenCalled(); + + finishMutation.resolve(); + await Promise.all([mutation, detached]); + expect(detachedEntered).toHaveBeenCalledOnce(); + }); + + it('upgrades an admitted handler to a mutation without self-deadlock', async () => { + const order: string[] = []; + await withBotTurnAdmission('app-a', async () => { + order.push('admission'); + await withBotTurnMutation('app-a', async () => { + order.push('mutation'); + await withBotTurnAdmission('app-a', () => order.push('nested-admission')); + await withBotTurnMutation('app-a', () => order.push('nested-mutation')); + }); + order.push('after'); + }); + expect(order).toEqual([ + 'admission', + 'mutation', + 'nested-admission', + 'nested-mutation', + 'after', + ]); + }); + + it('upgrades through an awaited nested admission without retaining an ancestor count', async () => { + const events: string[] = []; + await withBotTurnAdmission('app-a', async () => { + await withBotTurnAdmission('app-a', async () => { + await withBotTurnMutation('app-a', () => events.push('mutated')); + }); + events.push('outer-finished'); + }); + expect(events).toEqual(['mutated', 'outer-finished']); + }); + + it('an upgraded mutation drains peer admissions and blocks later turns', async () => { + const letPeerFinish = deferred(); + const letCloseFinish = deferred(); + const startUpgrade = deferred(); + const events: string[] = []; + + const closer = withBotTurnAdmission('app-a', async () => { + events.push('close-admitted'); + await startUpgrade.promise; + await withBotTurnMutation('app-a', async () => { + events.push('close-mutating'); + await letCloseFinish.promise; + }); + events.push('close-after'); + }); + const peer = withBotTurnAdmission('app-a', async () => { + events.push('peer-admitted'); + await letPeerFinish.promise; + events.push('peer-finished'); + }); + await vi.waitFor(() => expect(events).toContain('peer-admitted')); + + startUpgrade.resolve(); + await Promise.resolve(); + expect(events).not.toContain('close-mutating'); + + letPeerFinish.resolve(); + await vi.waitFor(() => expect(events).toContain('close-mutating')); + + const later = withBotTurnAdmission('app-a', () => events.push('later-admitted')); + await Promise.resolve(); + expect(events).not.toContain('later-admitted'); + + letCloseFinish.resolve(); + await Promise.all([closer, peer, later]); + expect(events.indexOf('peer-finished')).toBeLessThan(events.indexOf('close-mutating')); + expect(events.indexOf('close-mutating')).toBeLessThan(events.indexOf('close-after')); + }); + + it('revokes inherited mutation ownership for detached descendants', async () => { + const releaseDetached = deferred(); + const holdNextMutation = deferred(); + const detachedEntered = vi.fn(); + let detached!: Promise; + + await withBotTurnMutation('app-a', async () => { + detached = (async () => { + await releaseDetached.promise; + await withBotTurnAdmission('app-a', () => detachedEntered()); + })(); + }); + + const nextMutation = withBotTurnMutation('app-a', () => holdNextMutation.promise); + await Promise.resolve(); + releaseDetached.resolve(); + await Promise.resolve(); + await Promise.resolve(); + expect(detachedEntered).not.toHaveBeenCalled(); + + holdNextMutation.resolve(); + await Promise.all([nextMutation, detached]); + expect(detachedEntered).toHaveBeenCalledOnce(); + }); + + it('runs a detached mutation only after its admission parent drains', async () => { + const releaseDetachedMutation = deferred(); + const detachedEntered = vi.fn(); + let detached!: Promise; + + await withBotTurnAdmission('app-a', () => { + detached = runDetachedBotTurnMutation('app-a', async () => { + detachedEntered(); + await releaseDetachedMutation.promise; + }); + expect(detachedEntered).not.toHaveBeenCalled(); + }); + await vi.waitFor(() => expect(detachedEntered).toHaveBeenCalledOnce()); + + releaseDetachedMutation.resolve(); + await detached; + + const laterMutation = vi.fn(); + const laterAdmission = vi.fn(); + await withBotTurnMutation('app-a', () => laterMutation()); + await withBotTurnAdmission('app-a', () => laterAdmission()); + expect(laterMutation).toHaveBeenCalledOnce(); + expect(laterAdmission).toHaveBeenCalledOnce(); + }); + + it('counts an explicit detached admission begun before its parent returns', async () => { + const releaseChild = deferred(); + const childEntered = vi.fn(); + let child!: Promise; + + await withBotTurnAdmission('app-a', () => { + child = runDetachedBotTurnAdmission('app-a', async () => { + childEntered(); + await releaseChild.promise; + }); + }); + expect(childEntered).toHaveBeenCalledOnce(); + + const mutationEntered = vi.fn(); + const mutation = withBotTurnMutation('app-a', () => mutationEntered()); + await Promise.resolve(); + expect(mutationEntered).not.toHaveBeenCalled(); + + releaseChild.resolve(); + await Promise.all([child, mutation]); + expect(mutationEntered).toHaveBeenCalledOnce(); + }); + + it('clears mutation context for an explicit detached mutation', async () => { + const releaseChild = deferred(); + const childEntered = vi.fn(); + let child!: Promise; + + const parent = withBotTurnMutation('app-a', () => { + child = runDetachedBotTurnMutation('app-a', async () => { + childEntered(); + await releaseChild.promise; + }); + }); + await parent; + await vi.waitFor(() => expect(childEntered).toHaveBeenCalledOnce()); + const admissionEntered = vi.fn(); + const admission = withBotTurnAdmission('app-a', () => admissionEntered()); + await Promise.resolve(); + expect(admissionEntered).not.toHaveBeenCalled(); + + releaseChild.resolve(); + await Promise.all([child, admission]); + expect(admissionEntered).toHaveBeenCalledOnce(); + }); + + it('keeps an explicit detached admission behind its mutation parent', async () => { + const holdParent = deferred(); + const detachedEntered = vi.fn(); + let detached!: Promise; + const parent = withBotTurnMutation('app-a', async () => { + detached = runDetachedBotTurnAdmission('app-a', () => detachedEntered()); + await Promise.resolve(); + expect(detachedEntered).not.toHaveBeenCalled(); + await holdParent.promise; + }); + await Promise.resolve(); + holdParent.resolve(); + await parent; + await detached; + expect(detachedEntered).toHaveBeenCalledOnce(); + }); + + it('removes a bounded waiter on timeout so its action can never run later', async () => { + const releaseOwner = deferred(); + const ownerEntered = vi.fn(); + const owner = withBotTurnMutation('app-a', async () => { + ownerEntered(); + await releaseOwner.promise; + }); + await vi.waitFor(() => expect(ownerEntered).toHaveBeenCalledOnce()); + + const staleAction = vi.fn(); + await expect(tryWithBotTurnMutation('app-a', 10, staleAction)).resolves.toEqual({ + acquired: false, + reason: 'timeout', + }); + releaseOwner.resolve(); + await owner; + await new Promise(resolve => setTimeout(resolve, 20)); + expect(staleAction).not.toHaveBeenCalled(); + + const later = vi.fn(); + await expect(tryWithBotTurnMutation('app-a', 50, later)).resolves.toEqual({ + acquired: true, + value: undefined, + }); + expect(later).toHaveBeenCalledOnce(); + }); + + it('rejects an overdue wake even when owner release runs before the timer callback', async () => { + vi.useFakeTimers(); + vi.setSystemTime(1_000); + const releaseOwner = deferred(); + const ownerEntered = vi.fn(); + const owner = withBotTurnMutation('app-a', async () => { + ownerEntered(); + await releaseOwner.promise; + }); + await Promise.resolve(); + expect(ownerEntered).toHaveBeenCalledOnce(); + + const staleAction = vi.fn(); + const bounded = tryWithBotTurnMutation('app-a', 10, staleAction); + await Promise.resolve(); + // Advance wall time without running the overdue timer. Releasing the owner + // wakes the waiter first; the wake path itself must enforce the deadline. + vi.setSystemTime(1_020); + releaseOwner.resolve(); + await owner; + + await expect(bounded).resolves.toEqual({ acquired: false, reason: 'timeout' }); + expect(staleAction).not.toHaveBeenCalled(); + await vi.runAllTimersAsync(); + expect(staleAction).not.toHaveBeenCalled(); + }); + + it('rolls back a closed gate when admissions do not drain before the deadline', async () => { + const releaseAdmission = deferred(); + const heldEntered = vi.fn(); + const held = withBotTurnAdmission('app-a', async () => { + heldEntered(); + await releaseAdmission.promise; + }); + await vi.waitFor(() => expect(heldEntered).toHaveBeenCalledOnce()); + + const mutationAction = vi.fn(); + const bounded = tryWithBotTurnMutation('app-a', 10, mutationAction); + const admittedAfterRollback = vi.fn(); + const queuedAdmission = withBotTurnAdmission('app-a', admittedAfterRollback); + + await expect(bounded).resolves.toEqual({ acquired: false, reason: 'timeout' }); + await queuedAdmission; + expect(admittedAfterRollback).toHaveBeenCalledOnce(); + expect(mutationAction).not.toHaveBeenCalled(); + + releaseAdmission.resolve(); + await held; + await new Promise(resolve => setTimeout(resolve, 20)); + expect(mutationAction).not.toHaveBeenCalled(); + }); +}); diff --git a/test/bridge-final-output-retry.test.ts b/test/bridge-final-output-retry.test.ts index 7126496a8..d8893c6f3 100644 --- a/test/bridge-final-output-retry.test.ts +++ b/test/bridge-final-output-retry.test.ts @@ -238,6 +238,223 @@ describe('Bridge final_output delivery (P2 retry)', () => { expect(ds.lastBridgeEmittedUuid).toBe(SCOPED_DEDUPE_KEY); }); + it('routes synthetic Codex App identities through their frozen reply turn and uses dispatch-stable Lark UUIDs', async () => { + const sessionReply = vi.fn(async () => 'om_reply'); + initWorkerPool({ + sessionReply, + getSessionWorkingDir: () => '/tmp', + getActiveCount: () => 1, + closeSession: vi.fn(), + }); + const ds = makeDs(); + const { __testOnly_deliverFinalOutput } = await import('../src/core/worker-pool.js') as any; + + __testOnly_deliverFinalOutput(ds, { + type: 'final_output', + sessionId: ds.session.sessionId, + content: 'scheduled answer one', + lastUuid: 'synthetic-1', + turnId: 'codex-app-dispatch-synthetic-1', + replyTurnId: 'om_shared_route', + codexAppSettlement: { + requestId: 'request-1', generation: 'generation-1', seq: 1, dispatchId: 'dispatch-1', + }, + }, 'tag', 0); + __testOnly_deliverFinalOutput(ds, { + type: 'final_output', + sessionId: ds.session.sessionId, + content: 'scheduled answer two', + lastUuid: 'synthetic-2', + turnId: 'codex-app-dispatch-synthetic-2', + replyTurnId: 'om_shared_route', + codexAppSettlement: { + requestId: 'request-2', generation: 'generation-1', seq: 2, dispatchId: 'dispatch-2', + }, + }, 'tag', 0); + await vi.advanceTimersByTimeAsync(10); + + expect(sessionReply).toHaveBeenCalledTimes(2); + expect(sessionReply.mock.calls.map(call => call[4])).toEqual([ + 'om_shared_route', + 'om_shared_route', + ]); + expect(sessionReply.mock.calls.map(call => call[5]?.uuid)).toEqual([ + 'ca_dispatch-1', + 'ca_dispatch-2', + ]); + expect(sessionReply.mock.calls[0][5]).not.toHaveProperty('suppressHook'); + }); + + it('routes sequential Codex App settlements through each ledger-frozen shared-chat root', async () => { + const sessionReply = vi.fn(async () => 'om_reply'); + initWorkerPool({ + sessionReply, + getSessionWorkingDir: () => '/tmp', + getActiveCount: () => 1, + closeSession: vi.fn(), + }); + const ds = makeDs(); + ds.adoptedFrom = undefined; + ds.scope = 'chat'; + ds.session.scope = 'chat'; + ds.session.cliId = 'codex-app'; + ds.currentReplyTarget = { + rootMessageId: 'om_topic_b', turnId: 'turn-b', updatedAt: new Date().toISOString(), + }; + ds.session.currentReplyTarget = ds.currentReplyTarget; + ds.session.codexAppDispatchLedger = [ + { + dispatchId: 'dispatch-a', turnId: 'turn-a', state: 'prepared', content: 'A', + replyTarget: { mode: 'thread', rootMessageId: 'om_topic_a' }, + }, + { + dispatchId: 'dispatch-b', turnId: 'turn-b', state: 'accepted', content: 'B', + replyTarget: { mode: 'thread', rootMessageId: 'om_topic_b' }, + }, + ]; + __testOnly_setupWorkerHandlers(ds, ds.worker as any); + + (ds.worker as any).emit('message', { + type: 'final_output', sessionId: ds.session.sessionId, + content: 'answer A', lastUuid: 'uuid-a', turnId: 'turn-a', + codexAppSettlement: { + requestId: 'settle-a', generation: 'generation-1', seq: 1, dispatchId: 'dispatch-a', + }, + } satisfies Extract); + await vi.advanceTimersByTimeAsync(10); + await vi.waitFor(() => expect(sessionReply).toHaveBeenCalledTimes(1)); + expect(sessionReply.mock.calls[0][5]?.replyTarget) + .toEqual({ mode: 'thread', rootMessageId: 'om_topic_a' }); + + (ds.worker as any).emit('message', { + type: 'codex_app_dispatch_transition', + sessionId: ds.session.sessionId, + requestId: 'prepare-b', + operation: 'submit', + entries: [{ dispatchId: 'dispatch-b', turnId: 'turn-b' }], + } satisfies Extract); + await vi.waitFor(() => expect(ds.session.codexAppDispatchLedger?.[0]?.state).toBe('prepared')); + (ds.worker as any).emit('message', { + type: 'final_output', sessionId: ds.session.sessionId, + content: 'answer B', lastUuid: 'uuid-b', turnId: 'turn-b', + codexAppSettlement: { + requestId: 'settle-b', generation: 'generation-1', seq: 2, dispatchId: 'dispatch-b', + }, + } satisfies Extract); + await vi.advanceTimersByTimeAsync(10); + await vi.waitFor(() => expect(sessionReply).toHaveBeenCalledTimes(2)); + expect(sessionReply.mock.calls[1][5]?.replyTarget) + .toEqual({ mode: 'thread', rootMessageId: 'om_topic_b' }); + }); + + it.each(['doc_comment', 'http_wait', 'http_async', 'suppressed'] as const)( + 'fails closed instead of leaking a recovered %s settlement into Lark', + async deliverySink => { + const sessionReply = vi.fn(async () => 'om_forbidden'); + initWorkerPool({ + sessionReply, + getSessionWorkingDir: () => '/tmp', + getActiveCount: () => 1, + closeSession: vi.fn(), + }); + const ds = makeDs(); + ds.adoptedFrom = undefined; + ds.scope = 'chat'; + ds.session.scope = 'chat'; + ds.session.cliId = 'codex-app'; + ds.session.codexAppDispatchLedger = [{ + dispatchId: `dispatch-${deliverySink}`, + turnId: `turn-${deliverySink}`, + state: 'prepared', + content: 'recovered payload', + deliverySink, + }]; + __testOnly_setupWorkerHandlers(ds, ds.worker as any); + + (ds.worker as any).emit('message', { + type: 'final_output', + sessionId: ds.session.sessionId, + content: 'must not cross channels', + lastUuid: `uuid-${deliverySink}`, + turnId: `turn-${deliverySink}`, + codexAppSettlement: { + requestId: `settle-${deliverySink}`, + generation: 'generation-recovered', + seq: 1, + dispatchId: `dispatch-${deliverySink}`, + }, + } satisfies Extract); + + await vi.waitFor(() => expect(ds.session.codexAppDispatchLedger).toEqual([])); + expect(sessionReply).not.toHaveBeenCalled(); + await vi.waitFor(() => expect((ds.worker as any).send).toHaveBeenCalledWith( + expect.objectContaining({ + type: 'codex_app_dispatch_persisted', + requestId: `settle-${deliverySink}`, + ok: true, + }), + )); + }, + ); + + it('still resolves a live HTTP wait sink without posting to Lark', async () => { + const sessionReply = vi.fn(async () => 'om_forbidden'); + const resolveWait = vi.fn(); + initWorkerPool({ + sessionReply, + getSessionWorkingDir: () => '/tmp', + getActiveCount: () => 1, + closeSession: vi.fn(), + }); + const ds = makeDs(); + ds.adoptedFrom = undefined; + ds.session.cliId = 'codex-app'; + ds.pendingWaitPromises = new Map([['turn-wait-live', { resolve: resolveWait }]]); + ds.session.codexAppDispatchLedger = [{ + dispatchId: 'dispatch-wait-live', + turnId: 'turn-wait-live', + state: 'prepared', + content: 'live payload', + deliverySink: 'http_wait', + }]; + __testOnly_setupWorkerHandlers(ds, ds.worker as any); + + (ds.worker as any).emit('message', { + type: 'final_output', sessionId: ds.session.sessionId, + content: 'HTTP answer', lastUuid: 'uuid-wait-live', turnId: 'turn-wait-live', + codexAppSettlement: { + requestId: 'settle-wait-live', generation: 'generation-live', seq: 1, + dispatchId: 'dispatch-wait-live', + }, + } satisfies Extract); + + await vi.waitFor(() => expect(resolveWait).toHaveBeenCalledWith('HTTP answer')); + await vi.waitFor(() => expect(ds.session.codexAppDispatchLedger).toEqual([])); + expect(sessionReply).not.toHaveBeenCalled(); + }); + + it('abandons a delayed final before external delivery when worker/session ownership changes', async () => { + const sessionReply = vi.fn(async () => 'om_reply'); + initWorkerPool({ + sessionReply, + getSessionWorkingDir: () => '/tmp', + getActiveCount: () => 1, + closeSession: vi.fn(), + }); + const ds = makeDs(); + const { __testOnly_deliverFinalOutput } = await import('../src/core/worker-pool.js') as any; + let owned = true; + const complete = vi.fn(); + + __testOnly_deliverFinalOutput(ds, finalOutputMsg(), 'tag', 0, complete, () => owned); + owned = false; + await vi.advanceTimersByTimeAsync(10); + + expect(sessionReply).not.toHaveBeenCalled(); + expect(complete).toHaveBeenCalledWith(false); + expect(ds.lastBridgeEmittedUuid).toBeUndefined(); + }); + it('drops final_output whose worker sessionId does not match the daemon session', async () => { const sessionReply = vi.fn(async () => 'om_reply'); initWorkerPool({ @@ -1197,6 +1414,7 @@ describe('Bridge final_output delivery (P2 retry)', () => { }); const ds = makeDs(); + const worker = ds.worker as any; const { __testOnly_deliverFinalOutput } = await import('../src/core/worker-pool.js') as any; __testOnly_deliverFinalOutput(ds, finalOutputMsg(), 'tag', 0); @@ -1206,6 +1424,72 @@ describe('Bridge final_output delivery (P2 retry)', () => { expect(sessionReply).toHaveBeenCalledTimes(1); expect(ds.lastBridgeEmittedUuid).toBe(SCOPED_DEDUPE_KEY); expect(closeSession).toHaveBeenCalledWith(ds); + expect(worker.send).not.toHaveBeenCalledWith({ type: 'close' }); + expect(worker.kill).not.toHaveBeenCalled(); + }); + + it('does not commit dedup or completion when withdrawn-root Riff close is retryable', async () => { + const sessionReply = vi.fn().mockRejectedValue(new MessageWithdrawnError('om_root')); + // Production returns false when closeSessionHelper reports a failed Riff + // prepare/cancel; the active row, worker, and task id remain authoritative. + const closeSession = vi.fn(async () => false); + const complete = vi.fn(); + initWorkerPool({ + sessionReply, + getSessionWorkingDir: () => '/tmp', + getActiveCount: () => 1, + closeSession, + }); + + const ds = makeDs(); + ds.session.backendType = 'riff'; + ds.session.cliId = 'riff'; + ds.session.riffParentTaskId = 'task-riff-withdraw-retry'; + ds.initConfig = { backendType: 'riff' } as any; + const worker = ds.worker; + const { __testOnly_deliverFinalOutput } = await import('../src/core/worker-pool.js') as any; + __testOnly_deliverFinalOutput(ds, finalOutputMsg(), 'tag', 0, complete); + + await vi.advanceTimersByTimeAsync(0); + + expect(sessionReply).toHaveBeenCalledOnce(); + expect(closeSession).toHaveBeenCalledWith(ds); + expect(ds.lastBridgeEmittedUuid).toBeUndefined(); + expect(complete).toHaveBeenCalledWith(false); + expect(ds.worker).toBe(worker); + expect(ds.session).toMatchObject({ + status: 'active', + riffParentTaskId: 'task-riff-withdraw-retry', + }); + }); + + it('preserves an unsettled Codex FIFO owner when the root is withdrawn', async () => { + const sessionReply = vi.fn().mockRejectedValue(new MessageWithdrawnError('om_root')); + const closeSession = vi.fn(); + const complete = vi.fn(); + initWorkerPool({ + sessionReply, + getSessionWorkingDir: () => '/tmp', + getActiveCount: () => 1, + closeSession, + }); + const ds = makeDs(); + ds.session.codexAppDispatchLedger = [{ + dispatchId: 'dispatch-pending', + turnId: 'turn-1', + state: 'prepared', + content: 'pending answer', + }]; + const { __testOnly_deliverFinalOutput } = await import('../src/core/worker-pool.js') as any; + __testOnly_deliverFinalOutput(ds, finalOutputMsg(), 'tag', 0, complete); + await vi.advanceTimersByTimeAsync(0); + + expect(sessionReply).toHaveBeenCalledOnce(); + expect(closeSession).not.toHaveBeenCalled(); + expect((ds.worker as any).kill).not.toHaveBeenCalled(); + expect(ds.session.codexAppDispatchLedger).toHaveLength(1); + expect(ds.lastBridgeEmittedUuid).toBeUndefined(); + expect(complete).toHaveBeenCalledWith(false); }); it('aborts pending retry if the session was closed in the meantime', async () => { diff --git a/test/card-builder.test.ts b/test/card-builder.test.ts index ffe4aad00..30aebbf6e 100644 --- a/test/card-builder.test.ts +++ b/test/card-builder.test.ts @@ -502,6 +502,19 @@ describe('buildStreamingCard', () => { expect(card.header.title.content).toContain('工作中'); }); + it('shows a red, neutral no-progress label for stalled turns', () => { + const zh = parse(buildStreamingCard(SID, ROOT, URL, TITLE, '', 'stalled')); + expect(zh.header.template).toBe('red'); + expect(zh.header.title.content).toContain('长时间无进展'); + + const en = parse(buildStreamingCard( + SID, ROOT, URL, TITLE, '', 'stalled', undefined, 'hidden', + undefined, undefined, false, false, 'en', + )); + expect(en.header.template).toBe('red'); + expect(en.header.title.content).toContain('No recent progress'); + }); + it('should show green template and "等待输入" for idle status', () => { const card = parse(buildStreamingCard(SID, ROOT, URL, TITLE, '', 'idle')); expect(card.header.template).toBe('green'); diff --git a/test/card-handler-relay-pickup.test.ts b/test/card-handler-relay-pickup.test.ts index ee4895ccc..d7b983772 100644 --- a/test/card-handler-relay-pickup.test.ts +++ b/test/card-handler-relay-pickup.test.ts @@ -178,6 +178,27 @@ describe('relay_confirm button click', () => { expect(transferSessionMock).not.toHaveBeenCalled(); }); + it('refuses a Riff relay before posting M1 or invoking transferSession', async () => { + const ds = makeDs({ + cliId: 'riff', + backendType: 'riff', + riffParentTaskId: 'task-riff-card', + }); + const map = new Map(); + map.set(sessionKey('om_source_root', LARK_APP_ID), ds); + + const r = await handleCardAction(actionData({ sessionId: 'sess-source-1' }), deps(map), LARK_APP_ID); + + expect(r?.toast?.type).toBe('error'); + expect(r?.toast?.content).toContain('Riff'); + expect(getChatNameMock).not.toHaveBeenCalled(); + expect(sendMessageMock).not.toHaveBeenCalled(); + expect(replyMessageMock).not.toHaveBeenCalled(); + expect(transferSessionMock).not.toHaveBeenCalled(); + expect(deleteMessageMock).not.toHaveBeenCalled(); + expect(ds.session.riffParentTaskId).toBe('task-riff-card'); + }); + it('refuses to relay a session onto its own anchor (same_anchor)', async () => { // Thread-scope source anchored at om_source_root; targeting that same 话题 // root → same anchor → refuse (relaying onto itself). A different chat / diff --git a/test/card-handler-repo-select.test.ts b/test/card-handler-repo-select.test.ts index 45814d256..943d1665f 100644 --- a/test/card-handler-repo-select.test.ts +++ b/test/card-handler-repo-select.test.ts @@ -61,7 +61,27 @@ vi.mock('../src/services/session-store.js', () => ({ getSession: vi.fn(), })); -vi.mock('../src/core/worker-pool.js', () => ({ +vi.mock('../src/core/worker-pool.js', () => { + const lockTails = new WeakMap, Map>>(); + const withActiveSessionKeyLock = vi.fn(async (map: Map, key: string, action: () => T | Promise) => { + let tails = lockTails.get(map); + if (!tails) { + tails = new Map(); + lockTails.set(map, tails); + } + const previous = tails.get(key) ?? Promise.resolve(); + let release!: () => void; + const hold = new Promise(resolve => { release = resolve; }); + const tail = previous.catch(() => {}).then(() => hold); + tails.set(key, tail); + await previous.catch(() => {}); + try { return await action(); } + finally { + release(); + if (tails.get(key) === tail) tails.delete(key); + } + }); + return { forkWorker: vi.fn(), killWorker: vi.fn(), scheduleCardPatch: vi.fn(), @@ -72,8 +92,11 @@ vi.mock('../src/core/worker-pool.js', () => ({ resolvePrivateCardAudience: vi.fn(() => []), deliverWriteLinkCard: vi.fn(), deliverEphemeralOrReply: vi.fn(), + closeSession: vi.fn(async () => ({ ok: true, alreadyClosed: false })), + withActiveSessionKeyLock, CARD_POSTING_SENTINEL: '__posting__', -})); + }; +}); vi.mock('../src/core/session-manager.js', () => ({ getSessionWorkingDir: vi.fn(() => '/tmp'), @@ -93,6 +116,11 @@ vi.mock('../src/im/lark/event-dispatcher.js', () => ({ vi.mock('../src/core/session-activity.js', () => ({ publishAttentionPatch: vi.fn(), + announcePendingRepoSession: vi.fn(), +})); + +vi.mock('../src/services/default-worktree.js', () => ({ + maybeCreateDefaultWorktree: vi.fn(), })); vi.mock('../src/services/frozen-card-store.js', () => ({ @@ -123,11 +151,11 @@ vi.mock('@larksuiteoapi/node-sdk', () => ({ // ─── Imports ────────────────────────────────────────────────────────────── -import { handleCardAction, type CardHandlerDeps } from '../src/im/lark/card-handler.js'; -import { forkWorker, killWorker, deliverEphemeralOrReply, deliverWriteLinkCard } from '../src/core/worker-pool.js'; -import { buildNewTopicCliInput, getAvailableBots, getSessionWorkingDir } from '../src/core/session-manager.js'; +import { handleCardAction, runAutoWorktreeCommit, type CardHandlerDeps } from '../src/im/lark/card-handler.js'; +import { forkWorker, killWorker, deliverEphemeralOrReply, deliverWriteLinkCard, closeSession as closeWorkerPoolSession, withActiveSessionKeyLock } from '../src/core/worker-pool.js'; +import { buildNewTopicCliInput, getAvailableBots } from '../src/core/session-manager.js'; import { getBot } from '../src/bot-registry.js'; -import { createSession, closeSession, updateSession } from '../src/services/session-store.js'; +import { createSession, updateSession } from '../src/services/session-store.js'; import { createRepoWorktree, pushWorktreeBranch, removeRepoWorktree } from '../src/services/git-worktree.js'; import { applyConfigField } from '../src/services/bot-config-store.js'; import { deleteMessage } from '../src/im/lark/client.js'; @@ -138,6 +166,8 @@ import type { ProjectInfo } from '../src/services/project-scanner.js'; import { mkdtempSync, rmSync } from 'node:fs'; import { homedir, tmpdir } from 'node:os'; import { basename, join } from 'node:path'; +import { maybeCreateDefaultWorktree } from '../src/services/default-worktree.js'; +import { withBotTurnMutation } from '../src/core/bot-turn-mutation-gate.js'; // ─── Helpers ────────────────────────────────────────────────────────────── @@ -172,8 +202,9 @@ function makeDs(overrides?: Partial): DaemonSession { cliVersion: '1.0.0', lastMessageAt: Date.now(), hasHistory: true, - // Match makeSelectEvent / makeSkipEvent / makeManualEvent open_message_id — - // only the live posted card may drive selection. + // Every card callback below carries context.open_message_id=om_card. The + // production handler now requires that capability to match the currently + // published picker exactly, so the fixture must model a real live card. repoCardMessageId: 'om_card', ...overrides, } as unknown as DaemonSession; @@ -258,6 +289,42 @@ beforeEach(() => { // ─── Tests ──────────────────────────────────────────────────────────────── describe('repo select card — plain switch', () => { + it('rejects a callback from any card id other than the currently published picker', async () => { + const ds = makeDs({ + pendingRepo: true, + pendingPrompt: 'OPENING_N', + worker: null, + repoCardMessageId: 'om_current_picker', + }); + const { deps, sessionReply } = makeDeps(ds); + const stale = makeSelectEvent('repo_switch', '/repos/alpha'); + stale.context.open_message_id = 'om_stale_picker'; + + await handleCardAction(stale, deps, APP_ID); + + expect(ds.pendingRepo).toBe(true); + expect(ds.pendingPrompt).toBe('OPENING_N'); + expect(ds.repoCardMessageId).toBe('om_current_picker'); + expect(forkWorker).not.toHaveBeenCalled(); + expect(sessionReply).not.toHaveBeenCalled(); + }); + + it('invalidates the exact picker before awaiting confirmation and keeps replays inert when confirmation fails', async () => { + const ds = makeDs({ pendingRepo: true, pendingPrompt: 'OPENING_N', worker: null }); + const { deps, sessionReply } = makeDeps(ds); + sessionReply.mockRejectedValueOnce(new Error('confirmation unavailable')); + const event = makeSelectEvent('repo_switch', '/repos/alpha'); + + await expect(handleCardAction(event, deps, APP_ID)).rejects.toThrow('confirmation unavailable'); + + expect(forkWorker).toHaveBeenCalledTimes(1); + expect(ds.pendingRepo).toBe(false); + expect(ds.repoCardMessageId).toBeUndefined(); + + await handleCardAction(event, deps, APP_ID); + expect(forkWorker).toHaveBeenCalledTimes(1); + }); + it('pendingRepo selection forks the CLI with the buffered prompt', async () => { const ds = makeDs({ pendingRepo: true, @@ -318,7 +385,7 @@ describe('repo select card — plain switch', () => { content: 'mock-prompt', codexAppInput, }); - expect(vi.mocked(forkWorker).mock.calls[0]).toHaveLength(2); + expect(vi.mocked(forkWorker).mock.calls[0]![2]).toBe(false); expect(vi.mocked(buildNewTopicCliInput).mock.calls[0]![11]).toEqual(expect.objectContaining({ substituteTrigger, })); @@ -365,269 +432,55 @@ describe('repo select card — plain switch', () => { })); }); - it('keeps the pending repo card untouched when skip_repo is clicked while a worktree is being created', async () => { + it('keeps the pending reservation and opening buffers when forkWorker throws synchronously', async () => { const ds = makeDs({ pendingRepo: true, - pendingPrompt: 'hello world', - pendingTurnId: 'om_pending_turn', - repoCardMessageId: 'om_card', - worktreeCreating: true, + initialStartPending: true, + pendingPrompt: 'first prompt', + pendingFollowUps: ['buffered follow-up'], worker: null, }); const { deps } = makeDeps(ds); + vi.mocked(forkWorker).mockImplementationOnce(() => { + expect(ds.pendingRepo).toBe(true); + expect(ds.initialStartPending).toBe(true); + expect(ds.pendingPrompt).toBe('first prompt'); + expect(ds.pendingFollowUps).toEqual(['buffered follow-up']); + throw new Error('fork preaccept failed'); + }); - const result = await handleCardAction(makeSkipEvent(), deps, APP_ID); + await expect(handleCardAction( + makeSelectEvent('repo_switch', '/repos/alpha'), + deps, + APP_ID, + )).rejects.toThrow('fork preaccept failed'); - expect(result?.toast?.content).toContain('已有一个 worktree 正在创建'); - expect(forkWorker).not.toHaveBeenCalled(); expect(ds.pendingRepo).toBe(true); - expect(ds.pendingPrompt).toBe('hello world'); - expect(ds.pendingTurnId).toBe('om_pending_turn'); - expect(ds.repoCardMessageId).toBe('om_card'); - expect(deleteMessage).not.toHaveBeenCalled(); + expect(ds.initialStartPending).toBe(true); + expect(ds.pendingPrompt).toBe('first prompt'); + expect(ds.pendingFollowUps).toEqual(['buffered follow-up']); }); - it('lets a pending plain selection win over a concurrent skip_repo during prompt preparation', async () => { - const ds = makeDs({ pendingRepo: true, pendingPrompt: 'hello world', worker: null }); - const { deps } = makeDeps(ds); - let releaseBots: (() => void) | undefined; - vi.mocked(getAvailableBots).mockImplementationOnce(() => new Promise(resolve => { - releaseBots = () => resolve([]); - })); + it('skip_repo keeps its reservation through roster lookup and cannot resurrect a closed session', async () => { + const ds = makeDs({ pendingRepo: true, pendingPrompt: 'first prompt', worker: null }); + const { deps, sessionReply } = makeDeps(ds); + const roster = deferred(); + vi.mocked(getAvailableBots).mockReturnValueOnce(roster.promise); - const first = handleCardAction(makeSelectEvent('repo_switch', '/repos/alpha'), deps, APP_ID); - await vi.waitFor(() => expect(releaseBots).toBeTruthy()); + const action = handleCardAction(makeSkipEvent(), deps, APP_ID); + await vi.waitFor(() => expect(getAvailableBots).toHaveBeenCalledTimes(1)); expect(ds.pendingRepo).toBe(true); - expect(ds.pendingRepoCommitInFlight).toBe(true); - - const late = await handleCardAction(makeSkipEvent(), deps, APP_ID); - expect(late?.toast?.content).toContain('已有一个 worktree 正在创建'); - expect(forkWorker).not.toHaveBeenCalled(); - - releaseBots!(); - await first; - - expect(forkWorker).toHaveBeenCalledTimes(1); - expect(ds.workingDir).toBe('/repos/alpha'); - expect(ds.pendingRepo).toBe(false); - expect(ds.pendingRepoCommitInFlight).toBe(false); - }); - - it('lets skip_repo win over a concurrent plain selection during prompt preparation', async () => { - const ds = makeDs({ pendingRepo: true, pendingPrompt: 'hello world', worker: null }); - const { deps } = makeDeps(ds); - let releaseBots: (() => void) | undefined; - vi.mocked(getAvailableBots).mockImplementationOnce(() => new Promise(resolve => { - releaseBots = () => resolve([]); - })); + expect(ds.pendingPrompt).toBe('first prompt'); - const first = handleCardAction(makeSkipEvent(), deps, APP_ID); - await vi.waitFor(() => expect(releaseBots).toBeTruthy()); - expect(ds.pendingRepo).toBe(true); - expect(ds.pendingRepoCommitInFlight).toBe(true); + deps.activeSessions.delete(sessionKey(ROOT_ID, APP_ID)); + ds.session.status = 'closed'; + roster.resolve([]); + await action; - const late = await handleCardAction(makeSelectEvent('repo_switch', '/repos/alpha'), deps, APP_ID); - expect(late?.toast?.content).toContain('已有一个 worktree 正在创建'); expect(forkWorker).not.toHaveBeenCalled(); - - releaseBots!(); - await first; - - expect(forkWorker).toHaveBeenCalledTimes(1); - expect(ds.pendingRepo).toBe(false); - expect(ds.pendingRepoCommitInFlight).toBe(false); - }); - - it('holds the pending claim through post-fork confirmation so a second card click cannot mid-session-switch', async () => { - // Regression: previously pendingRepoCommitInFlight was cleared right after - // forkWorker, while sessionReply was still awaiting. A second card option - // in that window saw pendingRepo=false + claim released → treated as - // mid-session switch → killWorker + empty new session. - const ds = makeDs({ - pendingRepo: true, - pendingPrompt: 'hello world', - pendingTurnId: 'om_first_turn', - repoCardMessageId: 'om_card', - worker: null, - }); - const { deps, sessionReply } = makeDeps(ds); - let releaseReply: (() => void) | undefined; - sessionReply.mockImplementation(async () => new Promise(res => { - releaseReply = () => res('om_confirm'); - })); - - const first = handleCardAction(makeSelectEvent('repo_switch', '/repos/alpha'), deps, APP_ID); - await vi.waitFor(() => expect(forkWorker).toHaveBeenCalledTimes(1)); - await vi.waitFor(() => expect(releaseReply).toBeTruthy()); - expect(ds.pendingRepo).toBe(false); - expect(ds.pendingRepoCommitInFlight).toBe(true); - expect(ds.session.sessionId).toBe('uuid-old'); - - const late = await handleCardAction(makeSelectEvent('repo_switch', '/repos/beta'), deps, APP_ID); - // After successful fork the card is marked consumed before confirm reply, - // so the second click is rejected even while claim is still held. - expect(late?.toast?.content).toMatch(/仓库已选定|worktree 正在创建|ignore the old card/i); - expect(forkWorker).toHaveBeenCalledTimes(1); - expect(killWorker).not.toHaveBeenCalled(); - expect(createSession).not.toHaveBeenCalled(); - expect(ds.session.sessionId).toBe('uuid-old'); - expect(ds.workingDir).toBe('/repos/alpha'); - expect(ds.consumedRepoCardMessageIds).toContain('om_card'); - - releaseReply!(); - await first; - - expect(forkWorker).toHaveBeenCalledTimes(1); - expect(ds.pendingRepoCommitInFlight).toBe(false); - expect(ds.session.sessionId).toBe('uuid-old'); - expect(deleteMessage).toHaveBeenCalledWith(APP_ID, 'om_card'); - expect(ds.repoCardMessageId).toBeUndefined(); - }); - - it('skip_repo does not persist the default $HOME cwd onto session.workingDir', async () => { - // Regression: skip reused commitRepoSelection which always pinned dirPath. - // getSessionWorkingDir falls back to $HOME when unset; pinning that lets a - // sibling bot inherit HOME instead of getting its own repo card. - const home = homedir(); - vi.mocked(getSessionWorkingDir).mockReturnValueOnce(home); - const ds = makeDs({ - pendingRepo: true, - pendingPrompt: 'hello world', - pendingTurnId: 'om_skip_home', - repoCardMessageId: 'om_card', - worker: null, - }); - // Explicitly unpinned — the default-dir case. - ds.workingDir = undefined; - ds.session.workingDir = undefined; - const { deps, sessionReply } = makeDeps(ds); - - await handleCardAction(makeSkipEvent(), deps, APP_ID); - - expect(forkWorker).toHaveBeenCalledTimes(1); - expect(ds.pendingRepo).toBe(false); - expect(ds.workingDir).toBeUndefined(); - expect(ds.session.workingDir).toBeUndefined(); - expect(sessionReply.mock.calls.map(c => c[1]).join()).toContain('已直接开启会话'); - expect(deleteMessage).toHaveBeenCalledWith(APP_ID, 'om_card'); - }); - - it('rejects a stale card click while deleteMessage is still pending (local consume mark)', async () => { - // Regression: deleteMessage used to be fire-and-forget after claim release. - // We now mark the card consumed before network awaits and hold the claim - // through best-effort withdraw. A second click on the still-visible card - // must never mid-session-switch (killWorker), whether claim is still held - // or Feishu withdraw hangs / returns false. - const ds = makeDs({ - pendingRepo: true, - pendingPrompt: 'hello world', - pendingTurnId: 'om_first_turn', - repoCardMessageId: 'om_card', - worker: null, - }); - // Simulate a live worker after the first fork so a mid-session path would - // call killWorker if the stale click were allowed through. - vi.mocked(forkWorker).mockImplementationOnce((session) => { - (session as any).worker = { killed: false, send: vi.fn() }; - }); - const { deps } = makeDeps(ds); - let releaseDelete: (() => void) | undefined; - vi.mocked(deleteMessage).mockImplementationOnce(() => new Promise(res => { - releaseDelete = () => res(false); // withdraw "failed" / no-op - }) as any); - - const first = handleCardAction(makeSelectEvent('repo_switch', '/repos/alpha'), deps, APP_ID); - await vi.waitFor(() => expect(forkWorker).toHaveBeenCalledTimes(1)); - await vi.waitFor(() => expect(releaseDelete).toBeTruthy()); - // Consume is local and synchronous before the hung delete. - expect(ds.consumedRepoCardMessageIds).toContain('om_card'); - expect(ds.session.sessionId).toBe('uuid-old'); - - const lateWhileDeletePending = await handleCardAction(makeSelectEvent('repo_switch', '/repos/beta'), deps, APP_ID); - expect(lateWhileDeletePending?.toast?.content).toMatch(/仓库已选定|worktree 正在创建|ignore the old card/i); - expect(killWorker).not.toHaveBeenCalled(); - expect(createSession).not.toHaveBeenCalled(); - expect(forkWorker).toHaveBeenCalledTimes(1); - - releaseDelete!(); - await first; - expect(ds.pendingRepoCommitInFlight).toBe(false); - expect(ds.session.sessionId).toBe('uuid-old'); - expect(ds.workingDir).toBe('/repos/alpha'); - - // After claim release, Feishu may still show the card (delete returned false). - // Consume mark alone must keep rejecting. - const lateAfter = await handleCardAction(makeSelectEvent('repo_switch', '/repos/beta'), deps, APP_ID); - expect(lateAfter?.toast?.content).toMatch(/仓库已选定|ignore the old card/i); - expect(killWorker).not.toHaveBeenCalled(); - expect(createSession).not.toHaveBeenCalled(); - expect(forkWorker).toHaveBeenCalledTimes(1); - expect(ds.session.sessionId).toBe('uuid-old'); - }); - - it('keeps the first-spawn session when confirm sessionReply throws (card still marked consumed)', async () => { - // Sibling window: sessionReply throws → finally releases claim, card may - // still be visible. Consume mark must still reject the second click. - const ds = makeDs({ - pendingRepo: true, - pendingPrompt: 'hello world', - pendingTurnId: 'om_first_turn', - repoCardMessageId: 'om_card', - worker: null, - }); - vi.mocked(forkWorker).mockImplementationOnce((session) => { - (session as any).worker = { killed: false, send: vi.fn() }; - }); - const { deps, sessionReply } = makeDeps(ds); - sessionReply.mockRejectedValueOnce(new Error('reply boom')); - - await handleCardAction(makeSelectEvent('repo_switch', '/repos/alpha'), deps, APP_ID); - - expect(forkWorker).toHaveBeenCalledTimes(1); - expect(ds.pendingRepo).toBe(false); - expect(ds.pendingRepoCommitInFlight).toBe(false); - expect(ds.consumedRepoCardMessageIds).toContain('om_card'); - expect(ds.workingDir).toBe('/repos/alpha'); - - const late = await handleCardAction(makeSelectEvent('repo_switch', '/repos/beta'), deps, APP_ID); - expect(late?.toast?.content).toMatch(/仓库已选定|ignore the old card/i); - expect(killWorker).not.toHaveBeenCalled(); - expect(createSession).not.toHaveBeenCalled(); - expect(forkWorker).toHaveBeenCalledTimes(1); - expect(ds.session.sessionId).toBe('uuid-old'); - }); - - it('keeps pending buffers and releases the claim when forkWorker throws, then allows retry', async () => { - const pendingFollowUps = ['buffered follow-up']; - const ds = makeDs({ - pendingRepo: true, - pendingPrompt: 'hello world', - pendingFollowUps, - pendingTurnId: 'om_buffered_turn', - worker: null, - }); - const { deps } = makeDeps(ds); - vi.mocked(forkWorker).mockImplementationOnce(() => { throw new Error('fork boom'); }); - - await expect( - handleCardAction(makeSelectEvent('repo_switch', '/repos/alpha'), deps, APP_ID), - ).rejects.toThrow('fork boom'); - expect(ds.pendingRepo).toBe(true); - expect(ds.pendingRepoCommitInFlight).toBe(false); - expect(ds.pendingPrompt).toBe('hello world'); - expect(ds.pendingFollowUps).toBe(pendingFollowUps); - expect(ds.pendingTurnId).toBe('om_buffered_turn'); - expect(deleteMessage).not.toHaveBeenCalled(); - - await handleCardAction(makeSelectEvent('repo_switch', '/repos/alpha'), deps, APP_ID); - - expect(forkWorker).toHaveBeenCalledTimes(2); - expect(ds.pendingRepo).toBe(false); - expect(ds.pendingRepoCommitInFlight).toBe(false); - expect(ds.pendingPrompt).toBeUndefined(); - expect(ds.pendingFollowUps).toBeUndefined(); - expect(ds.pendingTurnId).toBeUndefined(); + expect(ds.pendingPrompt).toBe('first prompt'); + expect(sessionReply.mock.calls.some(call => String(call[1]).includes('已直接开启'))).toBe(false); }); it('mid-session selection closes the old session and forks a fresh one', async () => { @@ -637,8 +490,8 @@ describe('repo select card — plain switch', () => { await handleCardAction(makeSelectEvent('repo_switch', '/repos/beta'), deps, APP_ID); - expect(killWorker).toHaveBeenCalledTimes(1); - expect(closeSession).toHaveBeenCalledWith('uuid-old'); + expect(killWorker).not.toHaveBeenCalled(); + expect(closeWorkerPoolSession).toHaveBeenCalledWith('uuid-old'); expect(ds.session.sessionId).toMatch(/^uuid-new-/); expect(ds.workingDir).toBe('/repos/beta'); expect(ds.session.workingDir).toBe('/repos/beta'); @@ -683,8 +536,9 @@ describe('repo select card — plain switch', () => { const sessionAfterFirst = ds.session.sessionId; const late = await handleCardAction(makeSelectEvent('repo_switch', '/repos/beta'), deps, APP_ID); - expect(late?.toast?.content).toMatch(/仓库已选定|ignore the old card/i); - expect(killWorker).toHaveBeenCalledTimes(1); + expect(late?.toast?.content).toMatch(/失效|最新卡片|仓库已选定|ignore the old card/i); + expect(killWorker).not.toHaveBeenCalled(); + expect(closeWorkerPoolSession).toHaveBeenCalledTimes(1); expect(createSession).toHaveBeenCalledTimes(1); expect(forkWorker).toHaveBeenCalledTimes(1); expect(ds.session.sessionId).toBe(sessionAfterFirst); @@ -692,7 +546,8 @@ describe('repo select card — plain switch', () => { releaseReply!(); await first; - expect(killWorker).toHaveBeenCalledTimes(1); + expect(killWorker).not.toHaveBeenCalled(); + expect(closeWorkerPoolSession).toHaveBeenCalledTimes(1); expect(forkWorker).toHaveBeenCalledTimes(1); expect(ds.session.sessionId).toBe(sessionAfterFirst); }); @@ -705,7 +560,7 @@ describe('repo select card — plain switch', () => { const { deps } = makeDeps(ds); const late = await handleCardAction(makeSelectEvent('repo_switch', '/repos/beta'), deps, APP_ID); - expect(late?.toast?.content).toMatch(/仓库已选定|ignore the old card/i); + expect(late?.toast?.content).toMatch(/失效|最新卡片|仓库已选定|ignore the old card/i); expect(killWorker).not.toHaveBeenCalled(); expect(createSession).not.toHaveBeenCalled(); expect(forkWorker).not.toHaveBeenCalled(); @@ -753,6 +608,39 @@ describe('repo select card — plain switch', () => { })); }); + it('holds the canonical anchor lock across mid-session close and replacement publication', async () => { + const ds = makeDs(); + const { deps } = makeDeps(ds); + const close = deferred(); + vi.mocked(closeWorkerPoolSession).mockReturnValueOnce(close.promise); + + const switching = handleCardAction( + makeSelectEvent('repo_switch', '/repos/beta'), + deps, + APP_ID, + ); + await vi.waitFor(() => expect(closeWorkerPoolSession).toHaveBeenCalledWith('uuid-old')); + + let contenderEntered = false; + const contender = withActiveSessionKeyLock( + deps.activeSessions, + sessionKey(ROOT_ID, APP_ID), + () => { + contenderEntered = true; + return deps.activeSessions.get(sessionKey(ROOT_ID, APP_ID)); + }, + ); + await Promise.resolve(); + expect(contenderEntered).toBe(false); + + close.resolve({ ok: true, alreadyClosed: false }); + await switching; + const ownerAfterSwitch = await contender; + expect(contenderEntered).toBe(true); + expect(ownerAfterSwitch).toBe(ds); + expect(ds.session.sessionId).toMatch(/^uuid-new-/); + }); + it('ignores a keyless dropdown (option + root_id, no repo_switch/repo_worktree key)', async () => { // Security seal: a hand-crafted card (e.g. via `botmux send --card-json`) can // supply a bare `option + value.root_id` with no recognized key. It must NOT @@ -776,6 +664,30 @@ describe('repo select card — plain switch', () => { }); describe('repo select card — worktree open', () => { + it('rejects a stale direct single-select picker before any worktree side effect', async () => { + const ds = makeDs({ + pendingRepo: true, + pendingPrompt: 'hi', + worker: null, + repoCardMessageId: 'om_current_picker', + }); + const { deps, sessionReply } = makeDeps(ds); + + const result = await handleCardAction( + makeSelectEvent('repo_worktree', '/repos/alpha'), + deps, + APP_ID, + ); + + expect(result?.toast?.type).toBe('warning'); + expect(createRepoWorktree).not.toHaveBeenCalled(); + expect(pushWorktreeBranch).not.toHaveBeenCalled(); + expect(sessionReply).not.toHaveBeenCalled(); + expect(forkWorker).not.toHaveBeenCalled(); + expect(killWorker).not.toHaveBeenCalled(); + expect(ds.worktreeCreating).not.toBe(true); + }); + it('double click starts ONE background creation and commits once', async () => { const ds = makeDs({ pendingRepo: true, pendingPrompt: 'hi', worker: null }); const { deps, sessionReply } = makeDeps(ds); @@ -1089,7 +1001,7 @@ describe('repo select card — worktree open', () => { await handleCardAction(makeWorktreeSubmitEvent('feat/mid', ['/repos/alpha', '/repos/beta']), deps, APP_ID); await vi.waitFor(() => expect(ds.worktreeCreating).toBe(false)); - expect(closeSession).toHaveBeenCalledWith('uuid-old'); + expect(closeWorkerPoolSession).toHaveBeenCalledWith('uuid-old'); expect(ds.session).not.toBe(oldSession); expect(oldSession.riffRepoDirs).toBeUndefined(); expect(ds.session.riffRepoDirs).toEqual([ @@ -1219,14 +1131,37 @@ describe('repo select card — worktree open', () => { expect(ds.workingDir).toBe('/repos/alpha-feat-one'); }); + it('rejects a stale worktree form before slug generation or git work', async () => { + const ds = makeDs({ + pendingRepo: true, + pendingPrompt: 'hi', + worker: null, + repoCardMessageId: 'om_current_picker', + }); + const { deps, sessionReply } = makeDeps(ds); + + const result = await handleCardAction( + makeWorktreeSubmitEvent('feat/stale', ['/repos/alpha']), + deps, + APP_ID, + ); + + expect(result?.toast?.type).toBe('warning'); + expect(createRepoWorktree).not.toHaveBeenCalled(); + expect(pushWorktreeBranch).not.toHaveBeenCalled(); + expect(sessionReply).not.toHaveBeenCalled(); + expect(forkWorker).not.toHaveBeenCalled(); + expect(ds.worktreeCreating).not.toBe(true); + }); + it('worktree_toggle_mode flips the persisted picker mode and re-sends a fresh repo card', async () => { - // Callback open_message_id must match the live posted card. - const ds = makeDs({ pendingRepo: true, pendingPrompt: 'hi', worker: null, repoCardMessageId: 'om_card' }); + const ds = makeDs({ pendingRepo: true, pendingPrompt: 'hi', worker: null, repoCardMessageId: 'om_old_card' }); + ds.session.pendingRepoSetup = { mode: 'picker', prompt: 'hi', repoCardMessageId: 'om_old_card' }; const { deps, sessionReply } = makeDeps(ds); const event = { operator: { open_id: OWNER }, action: { value: { action: 'worktree_toggle_mode', root_id: ROOT_ID } }, - context: { open_message_id: 'om_card' }, + context: { open_message_id: 'om_old_card' }, }; const res = await handleCardAction(event, deps, APP_ID); @@ -1235,51 +1170,69 @@ describe('repo select card — worktree open', () => { // persisted the flipped mode (config undefined → true) expect(vi.mocked(applyConfigField)).toHaveBeenCalledWith('app_test', expect.objectContaining({ configKey: 'worktreeMultiPicker' }), true); // withdrew the old card and posted a fresh interactive repo card - expect(vi.mocked(deleteMessage)).toHaveBeenCalledWith('app_test', 'om_card'); + expect(vi.mocked(deleteMessage)).toHaveBeenCalledWith('app_test', 'om_old_card'); const interactiveCall = sessionReply.mock.calls.find(c => c[2] === 'interactive'); expect(interactiveCall).toBeDefined(); + expect(ds.repoCardMessageId).toBe('om_reply'); + expect(ds.session.pendingRepoSetup.repoCardMessageId).toBe('om_reply'); expect(createRepoWorktree).not.toHaveBeenCalled(); expect(forkWorker).not.toHaveBeenCalled(); expect(ds.worktreeCreating).not.toBe(true); }); - it('worktree_toggle_mode rejects a stale/wrong card id before flipping config', async () => { - const ds = makeDs({ pendingRepo: true, pendingPrompt: 'hi', worker: null, repoCardMessageId: 'om_live_card' }); - const { deps } = makeDeps(ds); - const event = { + it('rejects a stale worktree mode toggle before config, publish, or deletion', async () => { + const ds = makeDs({ pendingRepo: true, worker: null, repoCardMessageId: 'om_current_picker' }); + const { deps, sessionReply } = makeDeps(ds); + + const result = await handleCardAction({ operator: { open_id: OWNER }, action: { value: { action: 'worktree_toggle_mode', root_id: ROOT_ID } }, - context: { open_message_id: 'om_stale_card' }, - }; + context: { open_message_id: 'om_old_picker' }, + }, deps, APP_ID); - const res = await handleCardAction(event, deps, APP_ID); + expect(result?.toast?.type).toBe('warning'); + expect(applyConfigField).not.toHaveBeenCalled(); + expect(sessionReply).not.toHaveBeenCalled(); + expect(deleteMessage).not.toHaveBeenCalled(); + expect(ds.repoCardMessageId).toBe('om_current_picker'); + }); - expect(res?.toast?.content).toMatch(/仓库已选定|ignore the old card/i); - expect(vi.mocked(applyConfigField)).not.toHaveBeenCalled(); - expect(vi.mocked(deleteMessage)).not.toHaveBeenCalled(); - expect(ds.repoCardMessageId).toBe('om_live_card'); + it('keeps the old picker authoritative when replacement publication fails', async () => { + const ds = makeDs({ pendingRepo: true, pendingPrompt: 'hi', worker: null, repoCardMessageId: 'om_old_card' }); + ds.session.pendingRepoSetup = { mode: 'picker', prompt: 'hi', repoCardMessageId: 'om_old_card' }; + const { deps, sessionReply } = makeDeps(ds); + sessionReply.mockRejectedValueOnce(new Error('publish unavailable')); + + const result = await handleCardAction({ + operator: { open_id: OWNER }, + action: { value: { action: 'worktree_toggle_mode', root_id: ROOT_ID } }, + context: { open_message_id: 'om_old_card' }, + }, deps, APP_ID); + + expect(result?.toast?.type).toBe('error'); + expect(ds.repoCardMessageId).toBe('om_old_card'); + expect(ds.session.pendingRepoSetup.repoCardMessageId).toBe('om_old_card'); + expect(deleteMessage).not.toHaveBeenCalled(); }); - it('worktree_toggle_mode rejects restart-like state with no live repoCardMessageId', async () => { - const ds = makeDs({ - pendingRepo: true, - pendingPrompt: 'hi', - worker: null, - repoCardMessageId: undefined, - consumedRepoCardMessageIds: undefined, - }); + it('keeps the old picker authoritative when the replacement id cannot persist', async () => { + const ds = makeDs({ pendingRepo: true, pendingPrompt: 'hi', worker: null, repoCardMessageId: 'om_old_card' }); + ds.session.pendingRepoSetup = { mode: 'picker', prompt: 'hi', repoCardMessageId: 'om_old_card' }; const { deps } = makeDeps(ds); - const event = { + vi.mocked(updateSession).mockImplementationOnce(() => { + throw new Error('picker id save unavailable'); + }); + + const result = await handleCardAction({ operator: { open_id: OWNER }, action: { value: { action: 'worktree_toggle_mode', root_id: ROOT_ID } }, - context: { open_message_id: 'om_card' }, - }; - - const res = await handleCardAction(event, deps, APP_ID); + context: { open_message_id: 'om_old_card' }, + }, deps, APP_ID); - expect(res?.toast?.content).toMatch(/仓库已选定|ignore the old card/i); - expect(vi.mocked(applyConfigField)).not.toHaveBeenCalled(); - expect(vi.mocked(deleteMessage)).not.toHaveBeenCalled(); + expect(result?.toast?.type).toBe('error'); + expect(ds.repoCardMessageId).toBe('om_old_card'); + expect(ds.session.pendingRepoSetup.repoCardMessageId).toBe('om_old_card'); + expect(deleteMessage).not.toHaveBeenCalled(); }); it('worktree_toggle_mode requires canOperate — a non-operator (even the pending-session owner) cannot flip bot config', async () => { @@ -1342,6 +1295,43 @@ describe('repo select card — worktree open', () => { }); }); +describe('auto-worktree detached commit admission', () => { + it('holds the delayed commit/fork behind a same-bot mutation after the caller lease ended', async () => { + const ds = makeDs({ + pendingRepo: true, + pendingPrompt: 'delayed first turn', + worker: null, + }); + const { deps } = makeDeps(ds); + const { activeSessions } = deps; + const worktreeReady = deferred<{ dir: string }>(); + vi.mocked(maybeCreateDefaultWorktree).mockReturnValueOnce(worktreeReady.promise); + + const detached = runAutoWorktreeCommit({ + ds, + anchor: ROOT_ID, + larkAppId: APP_ID, + baseDir: '/repos/alpha', + prompt: 'delayed first turn', + activeSessions, + notify: vi.fn(), + }); + await vi.waitFor(() => expect(maybeCreateDefaultWorktree).toHaveBeenCalledOnce()); + + const finishMutation = deferred(); + const mutation = withBotTurnMutation(APP_ID, () => finishMutation.promise); + worktreeReady.resolve({ dir: '/repos/alpha-wt' }); + await Promise.resolve(); + await Promise.resolve(); + expect(forkWorker).not.toHaveBeenCalled(); + + finishMutation.resolve(); + await Promise.all([mutation, detached]); + expect(forkWorker).toHaveBeenCalledOnce(); + expect(ds.workingDir).toBe('/repos/alpha-wt'); + }); +}); + describe('repo select card — manual directory entry', () => { let tmpDir: string; beforeEach(() => { tmpDir = mkdtempSync(join(tmpdir(), 'botmux-manual-repo-')); }); @@ -1370,8 +1360,8 @@ describe('repo select card — manual directory entry', () => { await handleCardAction(makeManualEvent(tmpDir), deps, APP_ID); - expect(killWorker).toHaveBeenCalledTimes(1); - expect(closeSession).toHaveBeenCalledWith('uuid-old'); + expect(killWorker).not.toHaveBeenCalled(); + expect(closeWorkerPoolSession).toHaveBeenCalledWith('uuid-old'); expect(ds.session.sessionId).toMatch(/^uuid-new-/); expect(ds.session.workingDir).toBe(tmpDir); expect(forkWorker).toHaveBeenCalledTimes(1); diff --git a/test/card-handler-voice.test.ts b/test/card-handler-voice.test.ts index b25c0759b..8aa6765f3 100644 --- a/test/card-handler-voice.test.ts +++ b/test/card-handler-voice.test.ts @@ -107,6 +107,21 @@ describe('card-handler voice_summary', () => { expect(send).toHaveBeenCalledTimes(1); }); + it('does not consume the dedupe key when live IPC rejects the voice turn', async () => { + const { types, handler } = await fresh(); + const send = vi.fn() + .mockImplementationOnce(() => { throw new Error('worker exited'); }) + .mockImplementationOnce(() => {}); + deps.activeSessions.set(types.sessionKey('om_root', 'h1'), fakeSession(send)); + + const failed = await handler.handleCardAction(voiceAction('om_retryable_voice'), deps, 'h1'); + const retried = await handler.handleCardAction(voiceAction('om_retryable_voice'), deps, 'h1'); + + expect(failed?.toast?.type).toBe('warning'); + expect(retried?.toast?.type).toBe('success'); + expect(send).toHaveBeenCalledTimes(2); + }); + it('a re-click while the voice is still generating (worker back to working) says "already", not "busy"', async () => { const { types, handler } = await fresh(); const send = vi.fn(); @@ -176,6 +191,34 @@ describe('card-handler voice_summary', () => { }); describe('card-handler retry_last_task', () => { + it('keeps retry eligibility and visible state unchanged when live IPC rejects acceptance', async () => { + const { types, handler } = await fresh(); + const send = vi.fn() + .mockImplementationOnce(() => { throw new Error('worker exited'); }) + .mockImplementationOnce(() => {}); + const ds = fakeSession(send); + ds.lastCliInput = 'retry me'; + ds.lastUserPrompt = 'retry me'; + ds.usageLimit = { retryReady: true, retryAtMs: 0, retryLabel: 'now' }; + deps.activeSessions.set(types.sessionKey('om_root', 'h1'), ds); + + await handler.handleCardAction(retryAction(), deps, 'h1'); + + expect(ds.usageLimit).toMatchObject({ retryReady: true, retryLabel: 'now' }); + expect(ds.lastScreenStatus).toBe('idle'); + expect(deps.sessionReply).toHaveBeenCalledWith( + 'om_root', + expect.stringContaining('重试状态已失效'), + undefined, + 'h1', + ); + + await handler.handleCardAction(retryAction(), deps, 'h1'); + expect(send).toHaveBeenCalledTimes(2); + expect(ds.usageLimit).toBeUndefined(); + expect(ds.lastScreenStatus).toBe('working'); + }); + it('preserves the clean Codex App sidecar when retrying a completed turn', async () => { const dir = mkdtempSync(join(tmpdir(), 'botmux-cardretry-clean-')); const cfg = join(dir, 'bots.json'); @@ -214,6 +257,12 @@ describe('card-handler retry_last_task', () => { additionalContext: ds.lastCodexAppInput.additionalContext, }, }); - expect(send.mock.calls[0][0].codexAppInput).not.toHaveProperty('clientUserMessageId'); + // A replay must never reuse the completed native turn's routing id. The + // durable Codex App dispatcher assigns a fresh synthetic identity so this + // retry has its own ledger ownership and final attribution. + expect(send.mock.calls[0][0].codexAppInput.clientUserMessageId) + .toMatch(/^codex-app-dispatch-/); + expect(send.mock.calls[0][0].codexAppInput.clientUserMessageId).not.toBe('om_original'); + expect(send.mock.calls[0][0].codexAppDispatchId).toEqual(expect.any(String)); }); }); diff --git a/test/card-integration.test.ts b/test/card-integration.test.ts index f5bf3943d..fb65b6dd0 100644 --- a/test/card-integration.test.ts +++ b/test/card-integration.test.ts @@ -162,8 +162,7 @@ vi.mock('@larksuiteoapi/node-sdk', () => ({ // ─── Imports ────────────────────────────────────────────────────────────── import { handleCardAction, type CardHandlerDeps } from '../src/im/lark/card-handler.js'; -import { scheduleCardPatch } from '../src/core/worker-pool.js'; -import { killWorker, forkWorker } from '../src/core/worker-pool.js'; +import { scheduleCardPatch, forkWorker, setActiveSessionsRegistry } from '../src/core/worker-pool.js'; import { sessionKey } from '../src/core/types.js'; import type { DaemonSession } from '../src/core/types.js'; import { buildStreamingCard } from '../src/im/lark/card-builder.js'; @@ -189,7 +188,14 @@ function makeDaemonSession(overrides?: Partial): DaemonSession { chatType: 'group', scope: 'chat', }, - worker: { killed: false, send: vi.fn() } as any, + worker: { + killed: false, + send: vi.fn(), + kill: vi.fn(), + once: vi.fn(), + exitCode: null, + signalCode: null, + } as any, workerPort: 8080, workerToken: 'tok_secret', larkAppId: APP_ID, @@ -214,6 +220,10 @@ function makeDaemonSession(overrides?: Partial): DaemonSession { function makeDeps(activeSessions: Map): CardHandlerDeps { sessionReplyCallIndex = 0; + // closeSession is intentionally real in this partial worker-pool mock; point + // its authoritative registry at the integration fixture so close-card tests + // exercise the same identity-safe eviction contract as production. + setActiveSessionsRegistry(activeSessions); return { activeSessions, sessionReply: vi.fn(async () => { @@ -471,7 +481,7 @@ describe('Card integration: full event flow', () => { await handleCardAction(makeRestartEvent(ROOT_ID), deps, APP_ID); - expect(workerSend).toHaveBeenCalledWith({ type: 'restart' }); + expect(workerSend).toHaveBeenCalledWith({ type: 'restart', reason: 'operator' }); // The confirmation is delivered ephemeral to the clicker (group chat + an // operator open_id), not as a visible group reply. expect(vi.mocked(clientMod.sendEphemeralCard)).toHaveBeenCalledWith( @@ -491,7 +501,7 @@ describe('Card integration: full event flow', () => { expect(forkWorker).toHaveBeenCalledWith(ds, '', false); }); - it('close should kill worker and remove session', async () => { + it('close should remove session and deliver the closed card', async () => { const clientMod = await import('../src/im/lark/client.js'); const ds = makeDaemonSession(); const sessions = new Map(); @@ -501,7 +511,6 @@ describe('Card integration: full event flow', () => { await handleCardAction(makeCloseEvent(ROOT_ID), deps, APP_ID); - expect(killWorker).toHaveBeenCalledWith(ds); expect(sessions.has(sKey)).toBe(false); // Closed reply is an interactive card with a Resume button, delivered // ephemeral to the clicker (group chat + operator open_id); the mocked @@ -531,7 +540,6 @@ describe('Card integration: full event flow', () => { await handleCardAction(makeCloseEvent(ROOT_ID, 'ou_owner'), deps, APP_ID); - expect(killWorker).toHaveBeenCalledWith(ds); expect(sessions.has(sKey)).toBe(false); // Closed card goes ephemeral to the owner … expect(vi.mocked(clientMod.sendEphemeralCard)).toHaveBeenCalledWith( @@ -572,7 +580,6 @@ describe('Card integration: full event flow', () => { await handleCardAction(makeCloseEvent(ROOT_ID, 'ou_owner', 'private'), deps, APP_ID); - expect(killWorker).toHaveBeenCalledWith(ds); // Closed card still goes ephemeral to the owner … expect(vi.mocked(clientMod.sendEphemeralCard)).toHaveBeenCalledWith( APP_ID, ds.chatId, 'ou_owner', expect.stringContaining('"type":"closed"'), @@ -788,7 +795,7 @@ describe('Card integration: full event flow', () => { await handleCardAction(makeRestartEvent(ROOT_ID), deps, APP_ID); - expect(workerSend).toHaveBeenCalledWith({ type: 'restart' }); + expect(workerSend).toHaveBeenCalledWith({ type: 'restart', reason: 'operator' }); expect(vi.mocked(clientMod.sendEphemeralCard)).not.toHaveBeenCalled(); expect(deps.sessionReply).toHaveBeenCalledWith( ROOT_ID, expect.stringContaining('重启'), undefined, APP_ID, @@ -805,7 +812,6 @@ describe('Card integration: full event flow', () => { await handleCardAction(makeCloseEvent(ROOT_ID), deps, APP_ID); - expect(killWorker).toHaveBeenCalledWith(ds); expect(sessions.has(sKey)).toBe(false); expect(vi.mocked(clientMod.sendEphemeralCard)).not.toHaveBeenCalled(); expect(deps.sessionReply).toHaveBeenCalledWith( @@ -1137,6 +1143,31 @@ describe('Card integration: full event flow', () => { // sessionReply was used to surface the rejection message. expect(deps.sessionReply).toHaveBeenCalled(); }); + + it('restart on a live Riff session is rejected without replacing its lineage owner', async () => { + const ds = makeDaemonSession({ + initConfig: { backendType: 'riff' } as any, + }); + ds.session.backendType = 'riff'; + ds.session.cliId = 'riff'; + const sessions = new Map(); + sessions.set(sessionKey(ROOT_ID, APP_ID), ds); + const deps = makeDeps(sessions); + + await handleCardAction(makeRestartEvent(ROOT_ID), deps, APP_ID); + await flush(); + + expect((ds.worker as any).send).not.toHaveBeenCalledWith( + expect.objectContaining({ type: 'restart' }), + ); + expect(forkWorker).not.toHaveBeenCalled(); + expect(deps.sessionReply).toHaveBeenCalledWith( + ROOT_ID, + expect.stringContaining('Riff'), + undefined, + APP_ID, + ); + }); }); describe('Scenario 8: usage-limit retry action', () => { diff --git a/test/claude-turn-terminal-contract.test.ts b/test/claude-turn-terminal-contract.test.ts index 66f5565ba..82ba8f4f8 100644 --- a/test/claude-turn-terminal-contract.test.ts +++ b/test/claude-turn-terminal-contract.test.ts @@ -212,7 +212,7 @@ describe('Claude durable turn terminal contract', () => { expect(source).toContain("emitDurableFailure(`submit_impossible:${reason}`)"); expect(source).toContain("emitDurableFailure('submit_usage_limit')"); expect(source).toContain("emitDurableFailure('submit_unconfirmed')"); - expect(source).toContain('bridgeQueue.dropPendingTurn(bridgeTurnId, turnIdentity?.dispatchAttempt)'); + expect(source).toContain('bridgeQueue.dropPendingTurn(bridgeTurnId, dispatchAttempt)'); expect(source).toContain("'terminal_bridge_unavailable'"); expect(source).toMatch(/emitTurnTerminal\([\s\S]*?'ambiguous',[\s\S]*?'cli_exit'/); expect(source).toContain('requireExplicitTerminalForDurable: true'); diff --git a/test/cli-send-hook-context.test.ts b/test/cli-send-hook-context.test.ts index cd2dfd9bf..5ca8bfc66 100644 --- a/test/cli-send-hook-context.test.ts +++ b/test/cli-send-hook-context.test.ts @@ -1,11 +1,48 @@ -import { readFileSync } from 'node:fs'; +import { spawn, spawnSync } from 'node:child_process'; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; import { fileURLToPath } from 'node:url'; +import { tmpdir } from 'node:os'; import { dirname, join } from 'node:path'; import { describe, expect, it } from 'vitest'; +import { startOutboxWatcher } from '../src/adapters/backend/sandbox.js'; +import { + managedOriginCapabilityPath, + RELAY_ORIGIN_CAPABILITY_BASENAME, + replaceManagedOriginCapabilityFile, +} from '../src/core/managed-origin-capability.js'; const __dirname = dirname(fileURLToPath(import.meta.url)); const cliSource = readFileSync(join(__dirname, '..', 'src', 'cli.ts'), 'utf8'); +function runCli(args: string[], env: NodeJS.ProcessEnv): Promise<{ + code: number | null; + stdout: string; + stderr: string; +}> { + return new Promise((resolve, reject) => { + const child = spawn(process.execPath, [ + '--import', 'tsx', join(__dirname, '..', 'src', 'cli.ts'), ...args, + ], { + cwd: join(__dirname, '..'), + env, + stdio: ['ignore', 'pipe', 'pipe'], + }); + let stdout = ''; + let stderr = ''; + child.stdout.on('data', chunk => { stdout += String(chunk); }); + child.stderr.on('data', chunk => { stderr += String(chunk); }); + const timer = setTimeout(() => { + child.kill('SIGKILL'); + reject(new Error(`CLI timed out: ${stderr}`)); + }, 10_000); + child.on('error', reject); + child.on('close', code => { + clearTimeout(timer); + resolve({ code, stdout, stderr }); + }); + }); +} + describe('cmdSend hook context wiring', () => { it('repairs scope-less chat records in both CLI session file loaders', () => { const loadSessionsStart = cliSource.indexOf('function loadSessions()'); @@ -32,6 +69,360 @@ describe('cmdSend hook context wiring', () => { expect(cliSource).toMatch(/mentions\.push\(\{ open_id: replyTargetSenderOpenId, name: '' \}\)/); }); + it('prefers the current Codex App ledger entry over mutable shared-chat reply state', () => { + expect(cliSource).toContain( + 'originSession?.codexAppDispatchLedger', + ); + expect(cliSource).toContain("turnMatches.filter(entry => entry.dispatchAttempt === originDispatchAttempt)"); + expect(cliSource).toContain('if (exactMatches.length !== 1)'); + expect(cliSource).toContain('const frozenTurnDispatch = originSessionId === sid'); + expect(cliSource).toMatch( + /const sendTarget = !sendInto && !sendTopLevel && !overrideChatId && frozenTurnReplyTarget\s*\? frozenTurnReplyTarget/, + ); + expect(cliSource).toContain('?? frozenTurnDispatch?.replyTargetSenderOpenId'); + expect(cliSource).toContain('?? frozenTurnDispatch?.quoteTargetId'); + expect(cliSource).toContain('lastCallerOpenId: frozenTurnDispatch.replyTargetSenderOpenId'); + expect(cliSource).toContain('lastCallerIsBot: frozenTurnDispatch.replyTargetSenderIsBot'); + }); + + it('fails closed when a durable turn is bound to a non-Lark delivery sink', () => { + expect(cliSource).toContain("exactOriginDispatch?.deliverySink === 'http_wait'"); + expect(cliSource).toContain("exactOriginDispatch?.deliverySink === 'http_async'"); + expect(cliSource).toContain("exactOriginDispatch?.deliverySink === 'suppressed'"); + expect(cliSource).toContain("exactOriginDispatch?.deliverySink === 'doc_comment'"); + expect(cliSource).toContain('a document-comment turn supports only its exact plain-text comment reply'); + }); + + it('retains per-turn document-comment routing for non-Codex CLI adapters only', () => { + expect(cliSource).toContain("originSession?.cliId !== 'codex-app' && !!docTarget"); + expect(cliSource).toContain('const isOriginDocCommentTurn ='); + expect(cliSource.match(/if \(isOriginDocCommentTurn\)/g)).toHaveLength(2); + }); + + it('binds delivery-sink authority to the trusted origin, not --session-id destination', () => { + const cmdSendStart = cliSource.indexOf('async function cmdSend('); + const cmdDispatchStart = cliSource.indexOf('async function cmdDispatch(', cmdSendStart); + const cmdSend = cliSource.slice(cmdSendStart, cmdDispatchStart); + expect(cmdSend).toContain('const originSessionId = trustedRelayCtx?.sessionId'); + expect(cmdSend).toContain('const authoritativeOriginTurnCtx = trustedRelayCtx'); + expect(cmdSend).toContain(': (liveMarkerCtx?.turnId'); + expect(cmdSend).toContain(': isolatedManagedOriginCtx?.turnId'); + expect(cmdSend).not.toMatch(/const originSessionId =[\s\S]{0,200}sessionIdArg/); + expect(cmdSend).toContain('const sid = sessionIdArg ?? ancestorCtx?.sessionId'); + expect(cmdSend).toContain('const exactOriginDispatch = (() => {'); + expect(cmdSend).toContain('if (sid !== originSessionId'); + expect(cmdSend).toContain('const exactDocSession = originSession!;'); + }); + + it('does not promote detached spawn-time turn env while durable output is unsettled', () => { + const dataDir = mkdtempSync(join(tmpdir(), 'botmux-send-stale-origin-')); + try { + writeFileSync(join(dataDir, 'sessions-app-a.json'), JSON.stringify({ + origin: { + sessionId: 'origin', + chatId: 'oc_origin', + rootMessageId: 'om_origin', + title: 'origin', + status: 'active', + createdAt: new Date(0).toISOString(), + larkAppId: 'app-a', + codexAppDispatchLedger: [{ + dispatchId: 'dispatch-live', + turnId: 'turn-live', + dispatchAttempt: 1, + state: 'accepted', + content: 'private result', + deliverySink: 'http_wait', + }], + }, + destination: { + sessionId: 'destination', + chatId: 'oc_destination', + rootMessageId: 'om_destination', + title: 'destination', + status: 'active', + createdAt: new Date(0).toISOString(), + larkAppId: 'app-a', + }, + })); + const result = spawnSync(process.execPath, [ + '--import', 'tsx', + join(__dirname, '..', 'src', 'cli.ts'), + 'send', 'must-not-leak', '--session-id', 'destination', '--no-mention', + ], { + cwd: join(__dirname, '..'), + encoding: 'utf8', + env: { + ...process.env, + SESSION_DATA_DIR: dataDir, + BOTMUX_SESSION_ID: 'origin', + // These are inherited spawn-time fallbacks, not a live marker or + // protected capability. They must not select (or bypass) a sink. + BOTMUX_TURN_ID: 'turn-stale', + BOTMUX_DISPATCH_ATTEMPT: '99', + BOTMUX_HOST_RELAY_AUTHORIZED: '', + BOTMUX_SEND_RELAY: '', + BOTMUX_WORKFLOW: '', + BOTMUX_LARK_APP_ID: '', + BOTMUX_LARK_APP_SECRET: '', + }, + }); + expect(result.status).toBe(2); + expect(result.stderr).toContain('unsettled durable output but no fresh authoritative dispatch identity'); + expect(result.stderr).not.toContain('must-not-leak'); + } finally { + rmSync(dataDir, { recursive: true, force: true }); + } + }); + + it('routes a read-isolated Codex App send through the capability-gated host relay', async () => { + const root = mkdtempSync(join(tmpdir(), 'botmux-send-read-iso-relay-')); + const dataDir = join(root, 'data'); + const outbox = join(root, 'outbox'); + mkdirSync(dataDir, { recursive: true }); + mkdirSync(outbox, { recursive: true }); + const capability = 'ab'.repeat(32); + replaceManagedOriginCapabilityFile(join(outbox, RELAY_ORIGIN_CAPABILITY_BASENAME), JSON.stringify({ + token: capability, + turnId: 'turn-live', + dispatchAttempt: 4, + })); + const ledger = [{ + dispatchId: 'dispatch-live', + turnId: 'turn-live', + dispatchAttempt: 4, + state: 'prepared', + content: 'prompt', + deliverySink: 'lark', + }]; + writeFileSync(join(dataDir, 'sessions-app-a.json'), JSON.stringify({ + session: { + sessionId: 'session', chatId: 'oc_chat', rootMessageId: 'om_root', + title: 'read isolated', status: 'active', createdAt: new Date(0).toISOString(), + larkAppId: 'app-a', cliId: 'codex-app', codexAppDispatchLedger: ledger, + }, + })); + const fixture = join(root, 'host-send.mjs'); + writeFileSync(fixture, ` + const argv = process.argv.slice(2); + process.stdout.write(JSON.stringify({ + command: argv[0], + sessionId: argv[argv.indexOf('--session-id') + 1], + turnId: process.env.BOTMUX_TURN_ID, + dispatchAttempt: process.env.BOTMUX_DISPATCH_ATTEMPT, + requiresLedger: process.env.BOTMUX_HOST_RELAY_REQUIRES_CODEX_APP_LEDGER, + })); + `); + const authorize = (claim: { capability?: string }) => { + const exact = ledger.filter(entry => entry.turnId === 'turn-live' && entry.dispatchAttempt === 4); + return claim.capability === capability && exact.length === 1 + ? { + ok: true as const, + origin: { + turnId: 'turn-live', dispatchAttempt: 4, requiresCodexAppLedger: true, + }, + } + : { ok: false as const, error: 'stale' }; + }; + const stop = startOutboxWatcher(outbox, { ...process.env }, 'session', { + cliPath: fixture, + authorize, + }); + try { + const result = await runCli( + ['send', 'relay body', '--session-id', 'session', '--no-mention'], + { + ...process.env, + SESSION_DATA_DIR: dataDir, + BOTMUX_SESSION_ID: 'session', + BOTMUX_SEND_RELAY: outbox, + BOTMUX_HOST_RELAY_AUTHORIZED: '', + BOTMUX_WORKFLOW: '', + }, + ); + expect(result.code, result.stderr).toBe(0); + expect(JSON.parse(result.stdout)).toEqual({ + command: 'send', + sessionId: 'session', + turnId: 'turn-live', + dispatchAttempt: '4', + requiresLedger: '1', + }); + } finally { + stop(); + rmSync(root, { recursive: true, force: true }); + } + }); + + it('fails closed when relay capability is missing even if the default protected snapshot remains', async () => { + const root = mkdtempSync(join(tmpdir(), 'botmux-send-missing-relay-cap-')); + const dataDir = join(root, 'data'); + const outbox = join(root, 'outbox'); + mkdirSync(outbox, { recursive: true }); + replaceManagedOriginCapabilityFile( + managedOriginCapabilityPath(dataDir, 'session', 'ef'.repeat(32)), + JSON.stringify({ + sessionId: 'session', capability: 'bc'.repeat(32), + channelId: 'ef'.repeat(32), + turnId: 'turn-stale', dispatchAttempt: 2, + }), + ); + try { + const result = await runCli( + ['send', 'must not send', '--session-id', 'session', '--no-mention'], + { + ...process.env, + SESSION_DATA_DIR: dataDir, + BOTMUX_SESSION_ID: 'session', + BOTMUX_SEND_RELAY: outbox, + BOTMUX_HOST_RELAY_AUTHORIZED: '', + BOTMUX_WORKFLOW: '', + }, + ); + expect(result.code).toBe(2); + expect(result.stderr).toContain('managed host relay capability is stale or missing'); + expect(result.stderr).not.toContain('must not send'); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); + + it('lets a visible live marker win over a stale relay/default capability', async () => { + const root = mkdtempSync(join(tmpdir(), 'botmux-send-live-marker-wins-')); + const dataDir = join(root, 'data'); + const outbox = join(root, 'outbox'); + mkdirSync(join(dataDir, '.botmux-cli-pids'), { recursive: true }); + mkdirSync(outbox, { recursive: true }); + writeFileSync(join(dataDir, '.botmux-cli-pids', String(process.pid)), JSON.stringify({ + sessionId: 'session', turnId: 'turn-live', + })); + writeFileSync(join(outbox, RELAY_ORIGIN_CAPABILITY_BASENAME), JSON.stringify({ + token: 'cd'.repeat(32), turnId: 'turn-stale', dispatchAttempt: 9, + })); + replaceManagedOriginCapabilityFile( + managedOriginCapabilityPath(dataDir, 'session', 'fe'.repeat(32)), + JSON.stringify({ + sessionId: 'session', capability: 'cd'.repeat(32), + channelId: 'fe'.repeat(32), + turnId: 'turn-stale', dispatchAttempt: 9, + }), + ); + writeFileSync(join(dataDir, 'sessions-app-a.json'), JSON.stringify({ + session: { + sessionId: 'session', chatId: 'oc_chat', rootMessageId: 'om_root', + title: 'host', status: 'active', createdAt: new Date(0).toISOString(), + larkAppId: 'app-a', cliId: 'codex-app', + codexAppDispatchLedger: [{ + dispatchId: 'dispatch-live', turnId: 'turn-live', + state: 'prepared', content: 'prompt', deliverySink: 'http_wait', + }], + }, + })); + try { + const result = await runCli( + ['send', 'must not relay', '--session-id', 'session', '--no-mention'], + { + ...process.env, + SESSION_DATA_DIR: dataDir, + BOTMUX_SESSION_ID: 'session', + BOTMUX_SEND_RELAY: outbox, + BOTMUX_HOST_RELAY_AUTHORIZED: '', + BOTMUX_WORKFLOW: '', + BOTMUX_LARK_APP_ID: '', BOTMUX_LARK_APP_SECRET: '', + }, + ); + expect(result.code).toBe(2); + expect(result.stderr).toContain('origin turn turn-live is bound to the http_wait host sink'); + expect(result.stderr).not.toContain('managed host relay capability'); + } finally { + rmSync(root, { recursive: true, force: true }); + } + }); + + it('rejects a trusted host re-exec when its authorized Codex App ledger was already settled', async () => { + const dataDir = mkdtempSync(join(tmpdir(), 'botmux-send-host-ledger-gone-')); + writeFileSync(join(dataDir, 'sessions-app-a.json'), JSON.stringify({ + session: { + sessionId: 'session', chatId: 'oc_chat', rootMessageId: 'om_root', + title: 'settled', status: 'active', createdAt: new Date(0).toISOString(), + larkAppId: 'app-a', cliId: 'codex-app', pid: process.pid, + }, + })); + try { + const result = await runCli( + ['send', 'must not downgrade', '--session-id', 'session', '--no-mention'], + { + ...process.env, + SESSION_DATA_DIR: dataDir, + BOTMUX_SESSION_ID: 'session', + BOTMUX_TURN_ID: 'turn-settled', + BOTMUX_DISPATCH_ATTEMPT: '4', + BOTMUX_HOST_RELAY_AUTHORIZED: '1', + BOTMUX_HOST_RELAY_REQUIRES_CODEX_APP_LEDGER: '1', + BOTMUX_SEND_RELAY: '', + BOTMUX_WORKFLOW: '', + BOTMUX_LARK_APP_ID: '', BOTMUX_LARK_APP_SECRET: '', + }, + ); + expect(result.code).toBe(2); + expect(result.stderr).toContain('authorized Codex App origin session/turn-settled is no longer unsettled'); + expect(result.stderr).not.toContain('must not downgrade'); + } finally { + rmSync(dataDir, { recursive: true, force: true }); + } + }); + + it('authorizes a host relay from its forced trusted origin before any provider side effect', () => { + const cmdSendStart = cliSource.indexOf('async function cmdSend('); + const cmdDispatchStart = cliSource.indexOf('async function cmdDispatch(', cmdSendStart); + const cmdSend = cliSource.slice(cmdSendStart, cmdDispatchStart); + expect(cmdSend).toContain("const trustedHostRelay = process.env.BOTMUX_HOST_RELAY_AUTHORIZED === '1';"); + expect(cmdSend).toContain('isTrustedVcMeetingHostRelayParent('); + expect(cmdSend).toContain("BOTMUX_HOST_RELAY_REQUIRES_CODEX_APP_LEDGER === '1'"); + expect(cmdSend.indexOf('const exactOriginDispatch = (() => {')) + .toBeLessThan(cmdSend.indexOf("const { synthesizeVoiceOpus }")); + expect(cmdSend.indexOf("exactOriginDispatch?.deliverySink === 'http_wait'")) + .toBeLessThan(cmdSend.indexOf("const { sendMessage, replyMessage, uploadImage, uploadFile")); + }); + + it('validates the exact document text path before reading content or invoking TTS/uploads', () => { + const cmdSendStart = cliSource.indexOf('async function cmdSend('); + const cmdDispatchStart = cliSource.indexOf('async function cmdDispatch(', cmdSendStart); + const cmdSend = cliSource.slice(cmdSendStart, cmdDispatchStart); + const docGuard = cmdSend.indexOf('if (isOriginDocCommentTurn)'); + expect(docGuard).toBeGreaterThan(-1); + expect(docGuard).toBeLessThan(cmdSend.indexOf('// Read content from:')); + expect(docGuard).toBeLessThan(cmdSend.indexOf('content = readFileSync(contentFile')); + expect(docGuard).toBeLessThan(cmdSend.indexOf('content = await readStdin()')); + expect(docGuard).toBeLessThan(cmdSend.indexOf("const { synthesizeVoiceOpus }")); + const guardSource = cmdSend.slice(docGuard, cmdSend.indexOf('// Read content from:')); + for (const forbidden of [ + 'asVoice', + 'images.length > 0', + 'files.length > 0', + 'videoAttachments.length > 0', + 'videoCovers.length > 0', + 'customCardRequested', + 'sendTopLevel', + 'overrideChatId', + 'sendInto', + ]) { + expect(guardSource).toContain(forbidden); + } + }); + + it('never writes the startup session snapshot after an async document reply', () => { + const cmdSendStart = cliSource.indexOf('async function cmdSend('); + const cmdDispatchStart = cliSource.indexOf('async function cmdDispatch(', cmdSendStart); + const cmdSend = cliSource.slice(cmdSendStart, cmdDispatchStart); + const docSendStart = cmdSend.indexOf('if (isOriginDocCommentTurn)', cmdSend.indexOf('// Read content from:')); + const mentionParsing = cmdSend.indexOf('// Parse mentions:', docSendStart); + const docSend = cmdSend.slice(docSendStart, mentionParsing); + expect(docSend).toContain('Daemon settlement owns exact target retirement.'); + expect(docSend).not.toContain('saveSession('); + expect(docSend).not.toMatch(/delete\s+exactDocSession\.docCommentTargets/); + }); + it('freezes VC listener replay content and indexes only the successful primary output', () => { const cmdSendStart = cliSource.indexOf('async function cmdSend('); const cmdDispatchStart = cliSource.indexOf('async function cmdDispatch(', cmdSendStart); @@ -43,8 +434,9 @@ describe('cmdSend hook context wiring', () => { expect(cmdSend).toContain('msgType: canonicalOutput.msgType'); expect(cmdSend).toContain('quoteTargetId: canonicalOutput.quoteTargetId'); expect(cmdSend).toMatch( - /const dispatch = async \([^)]*\): Promise => \{[\s\S]*?revalidateVcMeetingManagedSend\(\);/, + /const dispatchAfterOriginGate = async \([^)]*\): Promise => \{[\s\S]*?revalidateVcMeetingManagedSend\(\);/, ); + expect(cmdSend).toMatch(/const dispatch = async \([^)]*\): Promise => \{[\s\S]*?dispatchAfterOriginGate\(/); expect(cmdSend).toMatch( /const dispatchPrimary = async \([^)]*\): Promise => \{\s*\/\/[^\n]*\n\s*\/\/[^\n]*\n\s*revalidateVcMeetingManagedSend\(\);/, ); @@ -67,14 +459,14 @@ describe('cmdSend hook context wiring', () => { expect(cmdSend.indexOf('const managedRenderedPayloadError = managedVcSendPayloadError({')) .toBeGreaterThan(cmdSend.indexOf('BOTMUX_CARD_PREPARED_CONTENT_FILE')); expect(cmdSend.indexOf('const managedRenderedPayloadError = managedVcSendPayloadError({')) - .toBeLessThan(cmdSend.indexOf('const results = await Promise.all(images.map')); + .toBeLessThan(cmdSend.indexOf('imageKeys.push(await uploadImage')); expect(cmdSend).toContain('const managedQuoteError = managedVcQuoteError({'); expect(cmdSend).toContain('const managedCustomCardError = managedVcCustomCardError('); expect(cmdSend).toMatch(/sessionQuoteTargetId: vcMeetingDeliveryReplyOrigin\s*\? undefined/); expect(cmdSend).toContain('const prepared = prepareVcMeetingListenerReply(proposedOutput);'); expect(cmdSend).toMatch(/canonicalOutput\.msgType,[\s\S]*?prepared\?\.providerKey/); expect(cmdSend).toContain('...(prepared ? { suppressHook: true } : {})'); - expect(cmdSend).toContain('const managedProviderOptions = prepared'); + expect(cmdSend).toContain('const managedProviderOptions = outboundMessageOptions(!!prepared);'); expect(cmdSend).toContain('...(vcMeetingManagedSendOrigin ? { maxMessages: 1 } : {})'); }); }); diff --git a/test/codex-app-control.test.ts b/test/codex-app-control.test.ts new file mode 100644 index 000000000..c47282b63 --- /dev/null +++ b/test/codex-app-control.test.ts @@ -0,0 +1,1722 @@ +import { + chmodSync, + existsSync, + linkSync, + mkdirSync, + mkdtempSync, + readFileSync, + readdirSync, + rmSync, + statSync, + symlinkSync, + writeFileSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { basename, dirname, join } from 'node:path'; +import { createConnection, createServer, type Server } from 'node:net'; +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { + CODEX_APP_CONTROL_LINE_MAX_BYTES, + CODEX_APP_CONTROL_STARTUP_TIMEOUT_MS, + CodexAppControlFinalAssembler, + CodexAppControlLineDecoder, + CodexAppControlEndpointTracker, + CodexAppControlReplayWindow, + CodexAppControlProofDeadline, + CodexAppControlRunnerHandshake, + CodexAppControlSequenceFence, + activateCodexAppControlIdentity, + acquireCodexAppControlOwnerLease, + acquireCodexAppPosixOwnerLease, + armCodexAppControlHandshakeTimeout, + armCodexAppControlStartupTimeout, + authenticateCodexAppControlCandidate, + bindThenPublishCodexAppControlLocator, + cleanupStaleCodexAppControlBootstraps, + codexAppControlFilesystemPolicy, + codexAppControlLocatorPath, + codexAppPosixControlRoot, + codexAppPosixOwnerLeaseDirectory, + codexAppPosixProcessProbeEnv, + codexAppControlStatePathForPlatform, + codexAppWindowsOwnerPipeEndpoint, + codexAppWindowsControlRoot, + consumeCodexAppControlBootstrap, + createCodexAppControlBootstrap, + decodeWindowsAclSnapshot, + encodeCodexAppControlAccepted, + encodeCodexAppControlAck, + encodeCodexAppControlAuth, + encodeCodexAppControlChallenge, + encodeCodexAppSignedControlMarker, + generateCodexAppControlChallenge, + generateCodexAppControlEpoch, + generateCodexAppPosixSocketEndpoint, + generateCodexAppWindowsPipeEndpoint, + mergeCodexAppControlCandidate, + parseCodexAppControlWireRecord, + parseWindowsCurrentSid, + readCodexAppControlState, + readCodexAppControlLocator, + shouldColdStartCodexAppReattach, + shouldFailCodexAppControlChannel, + takeCodexAppControlLocatorEndpoint, + validateCodexAppControlLocator, + verifyCodexAppControlAuth, + verifyCodexAppSignedControlMarker, + verifyWindowsCodexAppControlDacl, + writeCodexAppControlLocator, + writeCodexAppControlState, +} from '../src/utils/codex-app-control.js'; + +const tempDirs: string[] = []; + +function tempDir(): string { + const dir = mkdtempSync(join(tmpdir(), 'botmux-codex-control-')); + tempDirs.push(dir); + return dir; +} + +function shortPosixControlRoot(): string { + const dir = mkdtempSync('/tmp/bca-'); + tempDirs.push(dir); + return dir; +} + +function writePosixLeaseActorRecord(input: { + path: string; + sessionId: string; + nonce: string; + pid: number; + processStartToken: string; + createdAtMs?: number; + intendedDirectory?: string; + targetOwnerPath?: string; + version?: 2 | 3; +}): void { + const role = basename(input.path).startsWith('reap-') ? 'reaper' : 'owner'; + const intendedDirectory = input.intendedDirectory ?? dirname(input.path); + const directoryStat = statSync(intendedDirectory, { bigint: true }); + const generationNames = readdirSync(intendedDirectory) + .filter(name => /^generation-[a-f0-9]{64}\.json$/.test(name)); + const directoryGeneration = generationNames.length === 1 + ? generationNames[0]!.slice('generation-'.length, -'.json'.length) + : undefined; + const version = input.version ?? (directoryGeneration ? 3 : 2); + let targetOwner: Record | undefined; + if (role === 'reaper') { + const targetPath = input.targetOwnerPath ?? readdirSync(dirname(input.path)) + .find(name => /^owner-[a-f0-9]{64}\.json$/.test(name)); + const resolvedTargetPath = targetPath + ? (targetPath.includes('/') ? targetPath : join(dirname(input.path), targetPath)) + : undefined; + if (resolvedTargetPath && existsSync(resolvedTargetPath)) { + const targetRecord = JSON.parse(readFileSync(resolvedTargetPath, 'utf8')) as { + nonce: string; + pid: number; + processStartToken: string; + }; + const targetStat = statSync(resolvedTargetPath, { bigint: true }); + targetOwner = { + nonce: targetRecord.nonce, + recordIdentity: { dev: targetStat.dev.toString(10), ino: targetStat.ino.toString(10) }, + pid: targetRecord.pid, + processStartToken: targetRecord.processStartToken, + }; + } else { + targetOwner = { + nonce: '0'.repeat(64), + recordIdentity: { dev: '0', ino: '0' }, + pid: null, + processStartToken: null, + }; + } + } + writeFileSync(input.path, JSON.stringify({ + version, + role, + sessionId: input.sessionId, + nonce: input.nonce, + pid: input.pid, + processStartToken: input.processStartToken, + createdAtMs: input.createdAtMs ?? Date.now(), + directoryIdentity: { + dev: directoryStat.dev.toString(10), + ino: directoryStat.ino.toString(10), + ...(version === 3 ? { generation: directoryGeneration } : {}), + }, + ...(targetOwner ? { targetOwner } : {}), + }), { mode: 0o600 }); + chmodSync(input.path, 0o600); +} + +afterEach(() => { + for (const dir of tempDirs.splice(0)) rmSync(dir, { recursive: true, force: true }); +}); + +describe('Codex App asymmetric control bootstrap and state', () => { + it('keeps a delayed runner alive beyond 30 seconds and expires at the shared 90-second hard cap', async () => { + vi.useFakeTimers(); + const onTimeout = vi.fn(); + const timer = armCodexAppControlStartupTimeout(onTimeout); + try { + expect(CODEX_APP_CONTROL_STARTUP_TIMEOUT_MS).toBe(90_000); + await vi.advanceTimersByTimeAsync(30_001); + expect(onTimeout).not.toHaveBeenCalled(); + await vi.advanceTimersByTimeAsync(59_998); + expect(onTimeout).not.toHaveBeenCalled(); + await vi.advanceTimersByTimeAsync(1); + expect(onTimeout).toHaveBeenCalledTimes(1); + } finally { + clearTimeout(timer); + vi.useRealTimers(); + } + }); + + it('returns only a public identity, consumes the 0600 private bootstrap once, and binds session', () => { + const dir = tempDir(); + const bootstrap = createCodexAppControlBootstrap(dir, 'session-one'); + const raw = readFileSync(bootstrap.path, 'utf8'); + const serialized = JSON.parse(raw); + + expect(Object.keys(bootstrap).sort()).toEqual(['identity', 'path']); + expect(bootstrap.identity.publicKey).toEqual(expect.any(String)); + expect(JSON.stringify(bootstrap)).not.toContain(serialized.privateKey); + expect(serialized).toMatchObject({ + version: 3, + sessionId: 'session-one', + generation: bootstrap.identity.generation, + privateKey: expect.any(String), + socketPath: expect.stringMatching(/\.sock$/), + }); + expect(statSync(bootstrap.path).mode & 0o777).toBe(0o600); + + const consumed = consumeCodexAppControlBootstrap(bootstrap.path, 'session-one'); + expect(consumed.generation).toBe(bootstrap.identity.generation); + expect(consumed.privateKey.type).toBe('private'); + expect(consumed.privateKey.asymmetricKeyType).toBe('ed25519'); + expect(existsSync(bootstrap.path)).toBe(false); + expect(() => consumeCodexAppControlBootstrap(bootstrap.path, 'session-one')).toThrow(); + + const wrongSession = createCodexAppControlBootstrap(dir, 'session-two'); + expect(() => consumeCodexAppControlBootstrap(wrongSession.path, 'session-other')).toThrow(/invalid/); + expect(existsSync(wrongSession.path)).toBe(false); + }); + + it('fails closed on broad mode, symlink, and hard-linked bootstraps', () => { + const dir = tempDir(); + const loose = createCodexAppControlBootstrap(dir, 'loose'); + chmodSync(loose.path, 0o644); + expect(() => consumeCodexAppControlBootstrap(loose.path, 'loose')).toThrow(/regular 0600 file/); + expect(existsSync(loose.path)).toBe(false); + + const target = join(dir, 'target'); + const symlink = join(dir, 'symlink.bootstrap'); + writeFileSync(target, '{}', { mode: 0o600 }); + symlinkSync(target, symlink); + expect(() => consumeCodexAppControlBootstrap(symlink, 'x')).toThrow(); + expect(existsSync(symlink)).toBe(false); + + const linked = createCodexAppControlBootstrap(dir, 'linked'); + const secondLink = join(dir, 'second-link.bootstrap'); + linkSync(linked.path, secondLink); + expect(() => consumeCodexAppControlBootstrap(linked.path, 'linked')).toThrow(/single-link/); + rmSync(secondLink, { force: true }); + }); + + it('pins POSIX locator bootstraps to the fixed per-UID control root', () => { + const dir = tempDir(); + const sessionId = 'session-locator-root'; + const canonical = createCodexAppControlBootstrap(dir, sessionId, { + kind: 'locator', + locatorPath: codexAppControlLocatorPath(codexAppPosixControlRoot(), sessionId), + }); + expect(consumeCodexAppControlBootstrap(canonical.path, sessionId).locatorPath) + .toBe(codexAppControlLocatorPath(codexAppPosixControlRoot(), sessionId)); + + const attackerRoot = tempDir(); + const forged = createCodexAppControlBootstrap(dir, sessionId, { + kind: 'locator', + locatorPath: codexAppControlLocatorPath(attackerRoot, sessionId), + }); + expect(() => consumeCodexAppControlBootstrap(forged.path, sessionId)).toThrow(/invalid/); + }); + + it('cleans crash-orphaned bootstrap files for only the requested session', () => { + const dir = tempDir(); + const first = createCodexAppControlBootstrap(dir, 'session-clean'); + const other = createCodexAppControlBootstrap(dir, 'session-other'); + cleanupStaleCodexAppControlBootstraps(dir, 'session-clean'); + expect(existsSync(first.path)).toBe(false); + expect(existsSync(other.path)).toBe(true); + }); + + it('persists public candidates pending, then atomically collapses the proven generation active', () => { + const dir = tempDir(); + const statePath = join(dir, 'state', 'session.json'); + const oldBootstrap = createCodexAppControlBootstrap(dir, 'session-state'); + const oldPending = mergeCodexAppControlCandidate(undefined, oldBootstrap.identity, 10); + const oldActive = activateCodexAppControlIdentity(oldPending, oldBootstrap.identity.generation, 20); + writeCodexAppControlState(statePath, oldActive); + + const fresh = createCodexAppControlBootstrap(dir, 'session-state'); + const pending = mergeCodexAppControlCandidate(oldActive, fresh.identity, 30); + writeCodexAppControlState(statePath, pending); + const persistedPending = readCodexAppControlState(statePath)!; + expect(persistedPending.status).toBe('pending'); + expect(persistedPending.identities.map(identity => identity.generation)).toEqual([ + fresh.identity.generation, + oldBootstrap.identity.generation, + ]); + expect(JSON.stringify(persistedPending)).not.toContain('privateKey'); + expect(statSync(statePath).mode & 0o777).toBe(0o600); + + const challenge = generateCodexAppControlChallenge(); + const oldKey = consumeCodexAppControlBootstrap(oldBootstrap.path, 'session-state'); + const oldAuth = parseCodexAppControlWireRecord(encodeCodexAppControlAuth( + oldKey.privateKey, + 'session-state', + oldKey.generation, + challenge, + )); + if (!oldAuth || oldAuth.type !== 'auth') throw new Error('old auth parse failed'); + expect(authenticateCodexAppControlCandidate({ + state: persistedPending, + auth: oldAuth, + sessionId: 'session-state', + challenge, + })?.generation).toBe(oldBootstrap.identity.generation); + expect(authenticateCodexAppControlCandidate({ + state: persistedPending, + auth: { ...oldAuth, challenge: generateCodexAppControlChallenge() }, + sessionId: 'session-state', + challenge, + })).toBeUndefined(); + + const reused = activateCodexAppControlIdentity( + persistedPending, + oldBootstrap.identity.generation, + 40, + ); + writeCodexAppControlState(statePath, reused); + expect(readCodexAppControlState(statePath)).toMatchObject({ + status: 'active', + identities: [{ generation: oldBootstrap.identity.generation }], + activatedAtMs: 40, + }); + }); + + it('cold-starts only persistent panes that have no valid public state', () => { + const dir = tempDir(); + const bootstrap = createCodexAppControlBootstrap(dir, 'session-cold'); + const pending = mergeCodexAppControlCandidate(undefined, bootstrap.identity); + for (const backendType of ['tmux', 'herdr', 'zellij'] as const) { + expect(shouldColdStartCodexAppReattach({ + cliId: 'codex-app', backendType, isReattach: true, + })).toBe(true); + expect(shouldColdStartCodexAppReattach({ + cliId: 'codex-app', backendType, isReattach: true, persistedState: pending, + })).toBe(false); + } + expect(shouldColdStartCodexAppReattach({ + cliId: 'codex-app', backendType: 'pty', isReattach: true, + })).toBe(false); + expect(shouldColdStartCodexAppReattach({ + cliId: 'codex', backendType: 'tmux', isReattach: true, + })).toBe(false); + }); +}); + +describe('Codex App final transaction fencing', () => { + it('allows an arbitrary first sequence after replacement but requires continuity thereafter', () => { + const fence = new CodexAppControlSequenceFence(); + expect(fence.accept(41)).toBe(true); + expect(fence.accept(42)).toBe(true); + expect(fence.accept(44)).toBe(false); + + const duplicate = new CodexAppControlSequenceFence(); + expect(duplicate.accept(9)).toBe(true); + expect(duplicate.accept(9)).toBe(false); + }); + + it('publishes a final only after a complete ordered transaction', () => { + const assembler = new CodexAppControlFinalAssembler(); + expect(assembler.accept('final-start', { + id: 'turn:1', total: 2, turnId: 'om_1', completedAtMs: 100, + })).toEqual({ status: 'accepted' }); + expect(assembler.accept('final-chunk', { + id: 'turn:1', index: 0, data: Buffer.from('hello ').toString('base64'), + })).toEqual({ status: 'accepted' }); + expect(assembler.accept('final-chunk', { + id: 'turn:1', index: 1, data: Buffer.from('world').toString('base64'), + })).toEqual({ status: 'accepted' }); + expect(assembler.accept('final-end', { id: 'turn:1', total: 2 })).toEqual({ + status: 'complete', + payload: { turnId: 'om_1', completedAtMs: 100, content: 'hello world' }, + }); + }); + + it('rejects an incomplete final-end and accepts a full replay from a fresh connection', () => { + const incomplete = new CodexAppControlFinalAssembler(); + expect(incomplete.accept('final-start', { id: 'turn:2', total: 2 })).toEqual({ status: 'accepted' }); + expect(incomplete.accept('final-chunk', { + id: 'turn:2', index: 0, data: Buffer.from('first').toString('base64'), + })).toEqual({ status: 'accepted' }); + expect(incomplete.accept('final-end', { id: 'turn:2', total: 2 })).toMatchObject({ status: 'reject' }); + + const replay = new CodexAppControlFinalAssembler(); + expect(replay.accept('final-start', { id: 'turn:2', total: 2 })).toEqual({ status: 'accepted' }); + expect(replay.accept('final-chunk', { + id: 'turn:2', index: 0, data: Buffer.from('first').toString('base64'), + })).toEqual({ status: 'accepted' }); + expect(replay.accept('final-chunk', { + id: 'turn:2', index: 1, data: Buffer.from('second').toString('base64'), + })).toEqual({ status: 'accepted' }); + expect(replay.accept('final-end', { id: 'turn:2', total: 2 })).toMatchObject({ + status: 'complete', + payload: { content: 'firstsecond' }, + }); + }); + + it.each([ + ['out-of-order chunk', [ + ['final-start', { id: 'turn:3', total: 2 }], + ['final-chunk', { id: 'turn:3', index: 1, data: Buffer.from('late').toString('base64') }], + ]], + ['duplicate chunk', [ + ['final-start', { id: 'turn:3', total: 2 }], + ['final-chunk', { id: 'turn:3', index: 0, data: Buffer.from('one').toString('base64') }], + ['final-chunk', { id: 'turn:3', index: 0, data: Buffer.from('again').toString('base64') }], + ]], + ['invalid base64', [ + ['final-start', { id: 'turn:3', total: 1 }], + ['final-chunk', { id: 'turn:3', index: 0, data: '***not-base64***' }], + ]], + ['interleaved marker', [ + ['final-start', { id: 'turn:3', total: 1 }], + ['state', { busy: false }], + ]], + ['mismatched final total', [ + ['final-start', { id: 'turn:3', total: 1 }], + ['final-chunk', { id: 'turn:3', index: 0, data: Buffer.from('one').toString('base64') }], + ['final-end', { id: 'turn:3', total: 2 }], + ]], + ] as const)('rejects %s instead of making it cumulatively ACK-eligible', (_name, records) => { + const assembler = new CodexAppControlFinalAssembler(); + let result: ReturnType = { status: 'not-final' }; + for (const [kind, payload] of records) result = assembler.accept(kind, payload); + expect(result.status).toBe('reject'); + }); +}); + +describe('Windows Codex App control root, locator, and filesystem policy', () => { + it('anchors state and locator paths under LOCALAPPDATA instead of SESSION_DATA_DIR', () => { + const options = { + platform: 'win32' as const, + localAppData: 'C:\\Users\\alice\\AppData\\Local', + homeDirectory: 'C:\\Users\\alice', + }; + const root = codexAppWindowsControlRoot(options); + expect(root).toBe('C:\\Users\\alice\\AppData\\Local\\Botmux\\codex-app-control'); + const state = codexAppControlStatePathForPlatform('Z:\\shared\\untrusted', 'session-win', options); + expect(state.startsWith(root)).toBe(true); + expect(state).not.toContain('shared'); + expect(codexAppControlLocatorPath(root, 'session-win', 'win32').startsWith(root)).toBe(true); + expect(codexAppWindowsOwnerPipeEndpoint('session-win')) + .toMatch(/^\\\\\?\\pipe\\botmux-codex-app-owner-[a-f0-9]{64}$/); + expect(() => codexAppWindowsControlRoot({ + platform: 'win32', + localAppData: '\\\\fileserver\\profiles\\alice', + homeDirectory: 'C:\\Users\\alice', + })).toThrow(/local drive-qualified/); + }); + + it('uses Windows file semantics without weakening POSIX owner-only policy', () => { + expect(codexAppControlFilesystemPolicy('win32')).toEqual({ + useNoFollow: false, + verifyUid: false, + verifyExactMode: false, + chmodAfterCreate: false, + verifyPostUnlinkLinkCount: false, + fsyncDirectory: false, + }); + expect(codexAppControlFilesystemPolicy('linux')).toEqual({ + useNoFollow: true, + verifyUid: true, + verifyExactMode: true, + chmodAfterCreate: true, + verifyPostUnlinkLinkCount: true, + fsyncDirectory: true, + }); + }); + + it('parses the current SID and accepts only a protected exact current-SID + SYSTEM DACL', () => { + const sid = 'S-1-5-21-111-222-333-1001'; + expect(parseWindowsCurrentSid(`"DOMAIN\\alice","${sid}"`)).toBe(sid); + // The username column is attacker-influenced and can itself be SID-shaped; + // whoami's second CSV column is the only SID authority. + expect(parseWindowsCurrentSid(`"S-1-5-18","${sid}"\r\n`)).toBe(sid); + expect(parseWindowsCurrentSid(`"DOMAIN\\alice, admin","${sid}"\r\n`)).toBe(sid); + expect(parseWindowsCurrentSid(`"${sid}","not-a-sid"`)).toBeUndefined(); + expect(parseWindowsCurrentSid(`"DOMAIN\\alice","${sid}","extra"`)).toBeUndefined(); + const exact = [ + 'D:\\Botmux\\codex-app-control', + `D:P(A;OICI;FA;;;${sid})(A;OICI;FA;;;SY)`, + '', + ].join('\r\n'); + expect(verifyWindowsCodexAppControlDacl(exact, sid)).toBe(true); + expect(verifyWindowsCodexAppControlDacl( + `D:\\Botmux\\control D:P(A;OICI;FA;;;${sid})(A;OICI;FA;;;SY)`, + sid, + )).toBe(true); + const utf16 = Buffer.concat([ + Buffer.from([0xff, 0xfe]), + Buffer.from(exact, 'utf16le'), + ]); + expect(verifyWindowsCodexAppControlDacl(decodeWindowsAclSnapshot(utf16), sid)).toBe(true); + expect(verifyWindowsCodexAppControlDacl( + `D:P(A;;FA;;;${sid})(A;;FA;;;SY)`, + sid, + 'file', + )).toBe(true); + expect(verifyWindowsCodexAppControlDacl(exact, sid, 'file')).toBe(false); + expect(verifyWindowsCodexAppControlDacl( + `${exact.trim()}(A;OICI;FA;;;S-1-1-0)`, + sid, + )).toBe(false); + expect(verifyWindowsCodexAppControlDacl( + `D:AI(A;OICI;FA;;;${sid})(A;OICI;FA;;;SY)`, + sid, + )).toBe(false); + expect(verifyWindowsCodexAppControlDacl( + `D:P(A;OICIID;FA;;;${sid})(A;OICI;FA;;;SY)`, + sid, + )).toBe(false); + }); + + it('uses trusted absolute whoami/icacls argv without a shell and verifies saved SDDL', () => { + const source = readFileSync(join(process.cwd(), 'src/utils/codex-app-control.ts'), 'utf8'); + const start = source.indexOf('function defaultWindowsControlCommandRunner('); + const end = source.indexOf('/**\n * Remove inherited ACLs', start); + const acl = source.slice(start, end); + expect(acl).toContain('spawnSync(command, args, {'); + expect(acl).toContain('shell: false'); + expect(source).toContain("win32.join(systemRoot, 'System32', 'whoami.exe')"); + expect(source).toContain("win32.join(systemRoot, 'System32', 'icacls.exe')"); + expect(source).toContain("['/user', '/fo', 'csv', '/nh']"); + expect(source).toContain("[path, '/inheritance:r', '/q']"); + expect(source).toContain("[path, '/save', snapshotPath, '/q']"); + expect(source).toContain('verifyWindowsCodexAppControlDacl(snapshot, sid, kind)'); + }); + + it('validates random 256-bit pipe endpoints and rejects wrong/corrupt locators', () => { + const endpoint = generateCodexAppWindowsPipeEndpoint(); + const epoch = generateCodexAppControlEpoch(); + expect(endpoint).toMatch(/^\\\\\?\\pipe\\botmux-codex-app-[a-f0-9]{64}$/); + const locator = validateCodexAppControlLocator({ + version: 1, + sessionId: 'session-win', + epoch, + endpoint, + }, 'session-win', { platform: 'win32' }); + expect(locator).toEqual({ version: 1, sessionId: 'session-win', epoch, endpoint }); + expect(validateCodexAppControlLocator(locator, 'wrong-session', { platform: 'win32' })).toBeUndefined(); + expect(validateCodexAppControlLocator( + { ...locator, endpoint: '\\\\?\\pipe\\fixed' }, + 'session-win', + { platform: 'win32' }, + )) + .toBeUndefined(); + expect(validateCodexAppControlLocator( + { ...locator, epoch: 'short' }, + 'session-win', + { platform: 'win32' }, + )).toBeUndefined(); + }); + + it('publishes only after listen succeeds and never publishes on bind failure', async () => { + const order: string[] = []; + const endpoint = generateCodexAppWindowsPipeEndpoint(); + const epoch = generateCodexAppControlEpoch(); + await bindThenPublishCodexAppControlLocator({ + sessionId: 'session-bind', + epoch, + endpoint, + platform: 'win32', + listen: async () => { order.push('listen'); }, + publish: () => { order.push('publish'); }, + }); + expect(order).toEqual(['listen', 'publish']); + + const publish = vi.fn(); + await expect(bindThenPublishCodexAppControlLocator({ + sessionId: 'session-bind', + epoch: generateCodexAppControlEpoch(), + endpoint: generateCodexAppWindowsPipeEndpoint(), + platform: 'win32', + listen: async () => { throw new Error('EADDRINUSE'); }, + publish, + })).rejects.toThrow('EADDRINUSE'); + expect(publish).not.toHaveBeenCalled(); + }); + + it('serializes owner publishers, bounds EADDRINUSE waiting, and fails other bind errors immediately', async () => { + let nowMs = 0; + let oldLeaseHeld = true; + const bind = vi.fn(async () => { + if (oldLeaseHeld) throw Object.assign(new Error('old worker owns lease'), { code: 'EADDRINUSE' }); + return 'new-lease'; + }); + await expect(acquireCodexAppControlOwnerLease({ + bind, + timeoutMs: 1_000, + retryDelayMs: 100, + now: () => nowMs, + wait: async delayMs => { + nowMs += delayMs; + if (nowMs >= 200) oldLeaseHeld = false; + }, + })).resolves.toBe('new-lease'); + expect(bind).toHaveBeenCalledTimes(3); + + nowMs = 0; + const alwaysBusy = vi.fn(async () => { + throw Object.assign(new Error('still owned'), { code: 'EADDRINUSE' }); + }); + await expect(acquireCodexAppControlOwnerLease({ + bind: alwaysBusy, + timeoutMs: 200, + retryDelayMs: 100, + now: () => nowMs, + wait: async delayMs => { nowMs += delayMs; }, + })).rejects.toThrow('still owned'); + expect(nowMs).toBe(200); + + const denied = Object.assign(new Error('access denied'), { code: 'EACCES' }); + const failFast = vi.fn(async () => { throw denied; }); + await expect(acquireCodexAppControlOwnerLease({ bind: failFast })).rejects.toBe(denied); + expect(failFast).toHaveBeenCalledTimes(1); + }); + + it('keeps B published when superseded A finishes binding late', async () => { + let releaseListen!: () => void; + const listen = new Promise(resolve => { releaseListen = resolve; }); + let currentChannelId = 1; + let stopping = false; + const publishA = vi.fn(); + const retireA = vi.fn(); + const pendingA = bindThenPublishCodexAppControlLocator({ + sessionId: 'session-retired', + epoch: generateCodexAppControlEpoch(), + endpoint: generateCodexAppWindowsPipeEndpoint(), + platform: 'win32', + listen: () => listen, + publish: publishA, + isCurrent: () => !stopping && currentChannelId === 1, + retire: retireA, + }); + // Model stop(A), then bind+publish B, before A's delayed bind resolves. + stopping = true; + currentChannelId = 2; + stopping = false; + let published = ''; + await bindThenPublishCodexAppControlLocator({ + sessionId: 'session-retired', + epoch: generateCodexAppControlEpoch(), + endpoint: generateCodexAppWindowsPipeEndpoint(), + platform: 'win32', + listen: async () => {}, + publish: () => { published = 'B'; }, + isCurrent: () => !stopping && currentChannelId === 2, + }); + releaseListen(); + await expect(pendingA).resolves.toBeUndefined(); + expect(published).toBe('B'); + expect(publishA).not.toHaveBeenCalled(); + expect(retireA).toHaveBeenCalledTimes(1); + expect(shouldFailCodexAppControlChannel({ + channelId: 1, + currentChannelId, + stopping, + })).toBe(false); + + const retireB = vi.fn(); + await expect(bindThenPublishCodexAppControlLocator({ + sessionId: 'session-retired', + epoch: generateCodexAppControlEpoch(), + endpoint: generateCodexAppWindowsPipeEndpoint(), + platform: 'win32', + listen: async () => {}, + publish: () => { throw new Error('B locator write failed'); }, + isCurrent: () => !stopping && currentChannelId === 2, + retire: retireB, + })).rejects.toThrow('B locator write failed'); + expect(retireB).toHaveBeenCalledTimes(1); + expect(shouldFailCodexAppControlChannel({ + channelId: 2, + currentChannelId, + stopping, + })).toBe(true); + }); + + it('retires a bound endpoint when locator publication fails', async () => { + const retire = vi.fn(); + await expect(bindThenPublishCodexAppControlLocator({ + sessionId: 'session-publish-failure', + epoch: generateCodexAppControlEpoch(), + endpoint: generateCodexAppWindowsPipeEndpoint(), + platform: 'win32', + listen: async () => {}, + publish: () => { throw new Error('locator rename failed'); }, + retire, + })).rejects.toThrow('locator rename failed'); + expect(retire).toHaveBeenCalledTimes(1); + }); + + it('polls missing locators, retries unaccepted A, burns accepted A, then selects B', () => { + const dir = shortPosixControlRoot(); + const sessionId = 'session-track'; + const locatorPath = codexAppControlLocatorPath(dir, sessionId); + const socketDirectory = join(dir, 'sockets'); + mkdirSync(join(dir, 'locators'), { recursive: true, mode: 0o700 }); + mkdirSync(socketDirectory, { recursive: true, mode: 0o700 }); + const tracker = new CodexAppControlEndpointTracker(); + const first = { + version: 1 as const, + sessionId, + epoch: generateCodexAppControlEpoch(), + endpoint: generateCodexAppPosixSocketEndpoint(socketDirectory), + }; + expect(takeCodexAppControlLocatorEndpoint({ + locatorPath, + sessionId: first.sessionId, + tracker, + expectedControlRoot: dir, + })).toBeUndefined(); + writeCodexAppControlLocator(locatorPath, first, process.platform, dir); + for (let attempt = 1; attempt <= 5; attempt++) { + expect(takeCodexAppControlLocatorEndpoint({ + locatorPath, + sessionId: first.sessionId, + tracker, + expectedControlRoot: dir, + })).toEqual({ endpoint: first.endpoint, epoch: first.epoch }); + expect(tracker.attemptCount(first.endpoint)).toBe(attempt); + } + tracker.noteAccepted(first.endpoint); + expect(takeCodexAppControlLocatorEndpoint({ + locatorPath, + sessionId: first.sessionId, + tracker, + expectedControlRoot: dir, + })).toBeUndefined(); + expect(tracker.wasAttempted(first.endpoint)).toBe(true); + expect(tracker.wasAccepted(first.endpoint)).toBe(true); + const rotated = { + ...first, + epoch: generateCodexAppControlEpoch(), + endpoint: generateCodexAppPosixSocketEndpoint(socketDirectory), + }; + writeCodexAppControlLocator(locatorPath, rotated, process.platform, dir); + expect(takeCodexAppControlLocatorEndpoint({ + locatorPath, + sessionId: first.sessionId, + tracker, + expectedControlRoot: dir, + })).toEqual({ endpoint: rotated.endpoint, epoch: rotated.epoch }); + }); +}); + +describe('POSIX Codex App locator replacement', () => { + it('pins non-Linux process-start probes to a stable locale and timezone', () => { + expect(codexAppPosixProcessProbeEnv({ + PATH: '/trusted/bin', + LC_ALL: 'zh_CN.UTF-8', + LANG: 'zh_CN.UTF-8', + TZ: 'Asia/Shanghai', + })).toMatchObject({ + PATH: '/trusted/bin', + LC_ALL: 'C', + LANG: 'C', + TZ: 'UTC', + }); + }); + + it('accepts only random endpoints inside the locator control root', () => { + const root = shortPosixControlRoot(); + const sessionId = 'session-posix-locator'; + const locatorPath = codexAppControlLocatorPath(root, sessionId); + const socketDirectory = join(root, 'sockets'); + mkdirSync(join(root, 'locators'), { recursive: true, mode: 0o700 }); + mkdirSync(socketDirectory, { recursive: true, mode: 0o700 }); + const locator = { + version: 1 as const, + sessionId, + epoch: generateCodexAppControlEpoch(), + endpoint: generateCodexAppPosixSocketEndpoint(socketDirectory), + }; + + expect(validateCodexAppControlLocator(locator, sessionId, { + platform: 'linux', + locatorPath, + expectedControlRoot: root, + })).toEqual(locator); + expect(validateCodexAppControlLocator( + { ...locator, endpoint: join(root, 'outside.sock') }, + sessionId, + { platform: 'linux', locatorPath, expectedControlRoot: root }, + )).toBeUndefined(); + expect(validateCodexAppControlLocator(locator, sessionId, { + platform: 'linux', + locatorPath: join(root, 'locator.json'), + expectedControlRoot: root, + })).toBeUndefined(); + expect(validateCodexAppControlLocator(locator, sessionId, { + platform: 'linux', + locatorPath, + })).toBeUndefined(); + expect(validateCodexAppControlLocator(locator, sessionId, { + platform: 'linux', + locatorPath, + expectedControlRoot: join(root, 'attacker-selected'), + })).toBeUndefined(); + }); + + it('keeps B in the POSIX locator when superseded A finishes binding late', async () => { + const root = shortPosixControlRoot(); + const sessionId = 'session-posix-delayed-publish'; + const locatorPath = codexAppControlLocatorPath(root, sessionId); + const socketDirectory = join(root, 'sockets'); + mkdirSync(join(root, 'locators'), { recursive: true, mode: 0o700 }); + mkdirSync(socketDirectory, { recursive: true, mode: 0o700 }); + let channelId = 1; + let releaseA!: () => void; + const delayedListen = new Promise(resolvePromise => { releaseA = resolvePromise; }); + const retireA = vi.fn(); + const endpointA = generateCodexAppPosixSocketEndpoint(socketDirectory); + const pendingA = bindThenPublishCodexAppControlLocator({ + sessionId, + epoch: generateCodexAppControlEpoch(), + endpoint: endpointA, + platform: process.platform, + locatorPath, + expectedControlRoot: root, + listen: () => delayedListen, + isCurrent: () => channelId === 1, + publish: locator => writeCodexAppControlLocator( + locatorPath, + locator, + process.platform, + root, + ), + retire: retireA, + }); + + channelId = 2; + const endpointB = generateCodexAppPosixSocketEndpoint(socketDirectory); + await bindThenPublishCodexAppControlLocator({ + sessionId, + epoch: generateCodexAppControlEpoch(), + endpoint: endpointB, + platform: process.platform, + locatorPath, + expectedControlRoot: root, + listen: async () => undefined, + isCurrent: () => channelId === 2, + publish: locator => writeCodexAppControlLocator( + locatorPath, + locator, + process.platform, + root, + ), + }); + releaseA(); + await expect(pendingA).resolves.toBeUndefined(); + expect(retireA).toHaveBeenCalledTimes(1); + expect(readCodexAppControlLocator(locatorPath, sessionId, process.platform, root)?.endpoint) + .toBe(endpointB); + }); + + it('holds publisher ownership across overlap and fails closed on unknown owner liveness', async () => { + const root = tempDir(); + const statuses = new Map([ + [101, 'alive'], + [202, 'alive'], + ]); + const inspectOwner = (pid: number) => statuses.get(pid) ?? 'unknown'; + const first = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId: 'session-posix-owner', + pid: 101, + processStartToken: 'start-A', + inspectOwner, + }); + let nowMs = 0; + let waits = 0; + const second = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId: 'session-posix-owner', + pid: 202, + processStartToken: 'start-B', + inspectOwner, + timeoutMs: 1_000, + retryDelayMs: 10, + now: () => nowMs, + wait: async delayMs => { + nowMs += delayMs; + waits++; + if (waits === 2) first.release(); + }, + }); + expect(waits).toBeGreaterThanOrEqual(2); + expect(first.isOwned()).toBe(false); + expect(second.isOwned()).toBe(true); + second.release(); + + const unknown = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId: 'session-posix-unknown', + pid: 101, + processStartToken: 'start-A', + inspectOwner, + }); + await expect(acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId: 'session-posix-unknown', + pid: 202, + processStartToken: 'start-B', + inspectOwner: () => 'unknown', + timeoutMs: 20, + retryDelayMs: 10, + now: () => nowMs, + wait: async delayMs => { nowMs += delayMs; }, + })).rejects.toThrow(/timed out/); + expect(unknown.isOwned()).toBe(true); + unknown.release(); + }); + + it('does not reap a fresh empty initialization window before its grace expires', async () => { + const root = tempDir(); + const sessionId = 'session-posix-initializing'; + const directory = codexAppPosixOwnerLeaseDirectory(root, sessionId); + mkdirSync(join(root, 'leases'), { recursive: true, mode: 0o700 }); + mkdirSync(directory, { mode: 0o700 }); + let nowMs = statSync(directory).mtimeMs; + let waits = 0; + const lease = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 304, + processStartToken: 'start-after-empty', + inspectOwner: () => 'alive', + initializationGraceMs: 30, + retryDelayMs: 10, + timeoutMs: 100, + now: () => nowMs, + wait: async delayMs => { waits++; nowMs += delayMs; }, + }); + expect(waits).toBeGreaterThanOrEqual(3); + expect(lease.isOwned()).toBe(true); + lease.release(); + }); + + it('does not let a v2 actor claim authority over a v3 generation directory', async () => { + const root = tempDir(); + const sessionId = 'session-posix-v2-v3-boundary'; + const directory = codexAppPosixOwnerLeaseDirectory(root, sessionId); + mkdirSync(join(root, 'leases'), { recursive: true, mode: 0o700 }); + mkdirSync(directory, { mode: 0o700 }); + const generation = '1'.repeat(64); + writeFileSync(join(directory, `generation-${generation}.json`), generation, { mode: 0o600 }); + writePosixLeaseActorRecord({ + path: join(directory, `owner-${'2'.repeat(64)}.json`), + sessionId, + nonce: '2'.repeat(64), + pid: 401, + processStartToken: 'legacy-live', + version: 2, + }); + + const lease = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 402, + processStartToken: 'v3-successor', + inspectOwner: pid => pid === 401 ? 'alive' : 'unknown', + initializationGraceMs: 0, + }); + expect(lease.isOwned()).toBe(true); + lease.release(); + }); + + it('recovers multiple valid generation markers after the crash residue is actor-free', async () => { + const root = tempDir(); + const sessionId = 'session-posix-multiple-generations'; + const directory = codexAppPosixOwnerLeaseDirectory(root, sessionId); + mkdirSync(join(root, 'leases'), { recursive: true, mode: 0o700 }); + mkdirSync(directory, { mode: 0o700 }); + for (const generation of ['3'.repeat(64), '4'.repeat(64)]) { + writeFileSync(join(directory, `generation-${generation}.json`), generation, { mode: 0o600 }); + } + + const lease = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 403, + processStartToken: 'multiple-generation-successor', + inspectOwner: () => 'unknown', + initializationGraceMs: 0, + }); + expect(lease.isOwned()).toBe(true); + lease.release(); + }); + + it('keeps an ambiguous multi-generation directory fail-closed while an actor is live', async () => { + const root = tempDir(); + const sessionId = 'session-posix-multiple-generations-live'; + const directory = codexAppPosixOwnerLeaseDirectory(root, sessionId); + mkdirSync(join(root, 'leases'), { recursive: true, mode: 0o700 }); + mkdirSync(directory, { mode: 0o700 }); + for (const generation of ['5'.repeat(64), '6'.repeat(64)]) { + writeFileSync(join(directory, `generation-${generation}.json`), generation, { mode: 0o600 }); + } + writePosixLeaseActorRecord({ + path: join(directory, `owner-${'7'.repeat(64)}.json`), + sessionId, + nonce: '7'.repeat(64), + pid: 404, + processStartToken: 'ambiguous-live', + }); + let nowMs = Date.now(); + await expect(acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 405, + processStartToken: 'blocked-successor', + inspectOwner: pid => pid === 404 ? 'alive' : 'unknown', + initializationGraceMs: 0, + timeoutMs: 20, + retryDelayMs: 10, + now: () => nowMs, + wait: async delayMs => { nowMs += delayMs; }, + })).rejects.toThrow(/timed out/); + + const lease = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 405, + processStartToken: 'recovered-successor', + inspectOwner: pid => pid === 404 ? 'dead' : 'unknown', + initializationGraceMs: 0, + }); + expect(lease.isOwned()).toBe(true); + lease.release(); + }); + + it('recovers a dead reaper crash both before and after stale-owner retirement', async () => { + const root = tempDir(); + const statuses = new Map([ + [311, 'alive'], + [312, 'dead'], + [313, 'alive'], + ]); + const inspectOwner = (pid: number) => statuses.get(pid) ?? 'unknown'; + + const beforeSession = 'session-posix-reaper-before-retire'; + const stale = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId: beforeSession, + pid: 311, + processStartToken: 'owner-before', + inspectOwner, + }); + statuses.set(311, 'dead'); + const beforeReaperNonce = 'a'.repeat(64); + writePosixLeaseActorRecord({ + path: join(stale.directory, `reap-${beforeReaperNonce}.json`), + sessionId: beforeSession, + nonce: beforeReaperNonce, + pid: 312, + processStartToken: 'reaper-before', + }); + const recoveredBefore = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId: beforeSession, + pid: 313, + processStartToken: 'successor-before', + inspectOwner, + }); + expect(stale.isOwned()).toBe(false); + expect(recoveredBefore.isOwned()).toBe(true); + recoveredBefore.release(); + + const afterSession = 'session-posix-reaper-after-retire'; + const afterDirectory = codexAppPosixOwnerLeaseDirectory(root, afterSession); + mkdirSync(afterDirectory, { recursive: true, mode: 0o700 }); + chmodSync(afterDirectory, 0o700); + const afterReaperNonce = 'b'.repeat(64); + writePosixLeaseActorRecord({ + path: join(afterDirectory, `reap-${afterReaperNonce}.json`), + sessionId: afterSession, + nonce: afterReaperNonce, + pid: 312, + processStartToken: 'reaper-after', + }); + const recoveredAfter = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId: afterSession, + pid: 313, + processStartToken: 'successor-after', + inspectOwner, + initializationGraceMs: 0, + }); + expect(recoveredAfter.isOwned()).toBe(true); + recoveredAfter.release(); + }); + + it('keeps live and unknown reaper actors fail-closed', async () => { + for (const [suffix, reaperStatus] of [ + ['live', 'alive'], + ['unknown', 'unknown'], + ] as const) { + const root = tempDir(); + const sessionId = `session-posix-reaper-${suffix}`; + const statuses = new Map([ + [321, 'dead'], + [322, reaperStatus], + [323, 'alive'], + ]); + const inspectOwner = (pid: number) => statuses.get(pid) ?? 'unknown'; + const stale = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 321, + processStartToken: `owner-${suffix}`, + inspectOwner, + }); + const reaperNonce = suffix === 'live' ? 'c'.repeat(64) : 'd'.repeat(64); + writePosixLeaseActorRecord({ + path: join(stale.directory, `reap-${reaperNonce}.json`), + sessionId, + nonce: reaperNonce, + pid: 322, + processStartToken: `reaper-${suffix}`, + }); + let nowMs = Date.now(); + await expect(acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 323, + processStartToken: `successor-${suffix}`, + inspectOwner, + timeoutMs: 30, + retryDelayMs: 10, + now: () => nowMs, + wait: async delayMs => { nowMs += delayMs; }, + })).rejects.toThrow(/timed out/); + expect(stale.isOwned()).toBe(false); // a reaper record suspends owner authority + rmSync(stale.directory, { recursive: true, force: true }); + } + }); + + it('grace-recovers secure crash-partial owner and reaper records', async () => { + for (const kind of ['owner', 'reap'] as const) { + const root = tempDir(); + const sessionId = `session-posix-partial-${kind}`; + const directory = codexAppPosixOwnerLeaseDirectory(root, sessionId); + mkdirSync(directory, { recursive: true, mode: 0o700 }); + chmodSync(directory, 0o700); + const nonce = (kind === 'owner' ? 'e' : 'f').repeat(64); + const partialPath = join(directory, `${kind}-${nonce}.json`); + writeFileSync(partialPath, '{"version":1', { mode: 0o600 }); + chmodSync(partialPath, 0o600); + let nowMs = statSync(partialPath).mtimeMs; + let waits = 0; + const lease = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 333, + processStartToken: `successor-partial-${kind}`, + inspectOwner: () => 'alive', + initializationGraceMs: 30, + retryDelayMs: 10, + timeoutMs: 200, + now: () => nowMs, + wait: async delayMs => { waits++; nowMs += delayMs; }, + }); + expect(waits).toBeGreaterThanOrEqual(3); + expect(lease.isOwned()).toBe(true); + lease.release(); + } + }); + + it('retries when graceful release wins exactly after mkdir reports EEXIST', async () => { + const root = tempDir(); + const sessionId = 'session-posix-release-observation-gap'; + const first = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 341, + processStartToken: 'first-owner', + inspectOwner: () => 'alive', + }); + let contended = 0; + const second = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 342, + processStartToken: 'second-owner', + inspectOwner: () => 'alive', + onContended: () => { + contended++; + first.release(); + }, + }); + expect(contended).toBeGreaterThanOrEqual(1); + expect(first.isOwned()).toBe(false); + expect(second.isOwned()).toBe(true); + second.release(); + }); + + it('never lets a delayed original creator publish into a successor directory inode', async () => { + const root = tempDir(); + const sessionId = 'session-posix-delayed-original-writer'; + const successorNonce = '3'.repeat(64); + let replaced = false; + let nowMs = Date.now(); + await expect(acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 346, + processStartToken: 'delayed-original', + inspectOwner: pid => pid === 347 ? 'alive' : 'dead', + timeoutMs: 30, + retryDelayMs: 10, + now: () => nowMs, + wait: async delayMs => { nowMs += delayMs; }, + onOwnerDirectoryCreated: directory => { + if (replaced) return; + replaced = true; + rmSync(directory, { recursive: true, force: true }); + mkdirSync(directory, { mode: 0o700 }); + writePosixLeaseActorRecord({ + path: join(directory, `owner-${successorNonce}.json`), + sessionId, + nonce: successorNonce, + pid: 347, + processStartToken: 'successor-owner', + }); + }, + })).rejects.toThrow(/timed out/); + expect(replaced).toBe(true); + const successorPath = join( + codexAppPosixOwnerLeaseDirectory(root, sessionId), + `owner-${successorNonce}.json`, + ); + expect(existsSync(successorPath)).toBe(true); + }); + + it('retires a live foreign owner left in the publish-to-directory-CAS crash window', async () => { + const root = tempDir(); + const sessionId = 'session-posix-foreign-owner-crash-window'; + const statuses = new Map([ + [361, 'alive'], + [362, 'alive'], + [363, 'alive'], + ]); + const inspectOwner = (pid: number) => statuses.get(pid) ?? 'unknown'; + let successor!: Awaited>; + let replaced = false; + let paused = false; + let foreignOwnerPath = ''; + let signalPublished!: () => void; + const published = new Promise(resolvePromise => { signalPublished = resolvePromise; }); + let resumePublisher!: () => void; + const publisherMayResume = new Promise(resolvePromise => { resumePublisher = resolvePromise; }); + + const delayed = acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 361, + processStartToken: 'delayed-owner', + inspectOwner, + timeoutMs: 1_000, + retryDelayMs: 5, + onOwnerDirectoryCreated: async directory => { + if (replaced) return; + replaced = true; + rmSync(directory, { recursive: true, force: true }); + successor = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 362, + processStartToken: 'successor-owner', + inspectOwner, + }); + }, + onOwnerRecordPublished: async (_directory, ownerRecordPath) => { + if (paused) return; + paused = true; + foreignOwnerPath = ownerRecordPath; + signalPublished(); + await publisherMayResume; + }, + }); + + await published; + expect(existsSync(foreignOwnerPath)).toBe(true); + expect(successor.isOwned()).toBe(true); + await expect(acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 363, + processStartToken: 'observing-contender', + inspectOwner, + timeoutMs: 30, + retryDelayMs: 5, + })).rejects.toThrow(/timed out/); + // The delayed actor is still live, but its D1-bound record has no authority + // in D2 and cannot permanently create a multiple-owner poison pill. + expect(existsSync(foreignOwnerPath)).toBe(false); + expect(successor.isOwned()).toBe(true); + + successor.release(); + resumePublisher(); + const recovered = await delayed; + expect(recovered.isOwned()).toBe(true); + recovered.release(); + }); + + it('reconciles a dead same-directory losing owner without disturbing the live winner', async () => { + const root = tempDir(); + const sessionId = 'session-posix-same-directory-owner-crash'; + const statuses = new Map([ + [365, 'alive'], + [366, 'dead'], + [367, 'alive'], + ]); + const inspectOwner = (pid: number) => statuses.get(pid) ?? 'unknown'; + const winner = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 365, + processStartToken: 'same-dir-winner', + inspectOwner, + }); + const losingNonce = '4'.repeat(64); + const losingPath = join(winner.directory, `owner-${losingNonce}.json`); + writePosixLeaseActorRecord({ + path: losingPath, + sessionId, + nonce: losingNonce, + pid: 366, + processStartToken: 'same-dir-dead-loser', + }); + expect(winner.isOwned()).toBe(true); + + await expect(acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 367, + processStartToken: 'same-dir-observer', + inspectOwner, + timeoutMs: 30, + retryDelayMs: 5, + })).rejects.toThrow(/timed out/); + expect(existsSync(losingPath)).toBe(false); + expect(winner.isOwned()).toBe(true); + + winner.release(); + const successor = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 367, + processStartToken: 'same-dir-observer', + inspectOwner, + }); + expect(successor.isOwned()).toBe(true); + successor.release(); + }); + + it('ignores and retires a live stale reaper published into a replacement directory', async () => { + const root = tempDir(); + const sessionId = 'session-posix-foreign-reaper-crash-window'; + const statuses = new Map([ + [371, 'alive'], + [372, 'alive'], + [373, 'alive'], + [374, 'alive'], + ]); + const inspectOwner = (pid: number) => statuses.get(pid) ?? 'unknown'; + const stale = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 371, + processStartToken: 'stale-owner', + inspectOwner, + }); + statuses.set(371, 'dead'); + + let successor!: Awaited>; + let replaced = false; + let paused = false; + let foreignReaperPath = ''; + let signalPublished!: () => void; + const published = new Promise(resolvePromise => { signalPublished = resolvePromise; }); + let resumePublisher!: () => void; + const publisherMayResume = new Promise(resolvePromise => { resumePublisher = resolvePromise; }); + const delayedCleaner = acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 372, + processStartToken: 'delayed-reaper', + inspectOwner, + timeoutMs: 1_000, + retryDelayMs: 5, + onBeforeReaperRecordPublished: async directory => { + if (replaced) return; + replaced = true; + rmSync(directory, { recursive: true, force: true }); + successor = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 373, + processStartToken: 'replacement-owner', + inspectOwner, + }); + }, + onReaperRecordPublished: async (_directory, reaperRecordPath) => { + if (paused) return; + paused = true; + foreignReaperPath = reaperRecordPath; + signalPublished(); + await publisherMayResume; + }, + }); + + await published; + expect(existsSync(foreignReaperPath)).toBe(true); + // A foreign live reaper must not revoke the real D2 owner while its delayed + // publisher is paused at the exact post-publication crash boundary. + expect(successor.isOwned()).toBe(true); + await expect(acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 374, + processStartToken: 'replacement-observer', + inspectOwner, + timeoutMs: 30, + retryDelayMs: 5, + })).rejects.toThrow(/timed out/); + expect(existsSync(foreignReaperPath)).toBe(false); + expect(successor.isOwned()).toBe(true); + + successor.release(); + resumePublisher(); + const recovered = await delayedCleaner; + expect(recovered.isOwned()).toBe(true); + recovered.release(); + stale.release(); + }); + + it('hard-fails wrong record mode instead of grace-reclassifying it as partial', async () => { + const root = tempDir(); + const sessionId = 'session-posix-insecure-partial'; + const directory = codexAppPosixOwnerLeaseDirectory(root, sessionId); + mkdirSync(directory, { recursive: true, mode: 0o700 }); + chmodSync(directory, 0o700); + const nonce = '1'.repeat(64); + writeFileSync(join(directory, `owner-${nonce}.json`), '{', { mode: 0o644 }); + chmodSync(join(directory, `owner-${nonce}.json`), 0o644); + await expect(acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId, + pid: 351, + processStartToken: 'insecure-successor', + inspectOwner: () => 'dead', + initializationGraceMs: 0, + })).rejects.toThrow(/0600/); + }); + + it('reclaims SIGKILL residue with multiple contenders but never grants two owners', async () => { + const root = tempDir(); + const statuses = new Map([ + [301, 'alive'], + [302, 'alive'], + [303, 'alive'], + [304, 'dead'], + ]); + const inspectOwner = (pid: number) => statuses.get(pid) ?? 'unknown'; + const stale = await acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId: 'session-posix-race', + pid: 301, + processStartToken: 'start-stale', + inspectOwner, + }); + statuses.set(301, 'dead'); // model SIGKILL: no graceful release + const crashedReaperNonce = '2'.repeat(64); + writePosixLeaseActorRecord({ + path: join(stale.directory, `reap-${crashedReaperNonce}.json`), + sessionId: 'session-posix-race', + nonce: crashedReaperNonce, + pid: 304, + processStartToken: 'start-dead-reaper', + }); + + let secondResolved = false; + const contender = (pid: number, token: string) => acquireCodexAppPosixOwnerLease({ + controlRoot: root, + sessionId: 'session-posix-race', + pid, + processStartToken: token, + inspectOwner, + timeoutMs: 2_000, + retryDelayMs: 2, + }); + const bPromise = contender(302, 'start-B'); + const cPromise = contender(303, 'start-C').then(lease => { + secondResolved = true; + return lease; + }); + const firstWinner = await Promise.race([ + bPromise.then(lease => ({ name: 'B' as const, lease })), + cPromise.then(lease => ({ name: 'C' as const, lease })), + ]); + expect(stale.isOwned()).toBe(false); + expect(firstWinner.lease.isOwned()).toBe(true); + if (firstWinner.name === 'B') expect(secondResolved).toBe(false); + firstWinner.lease.release(); + const secondWinner = firstWinner.name === 'B' ? await cPromise : await bPromise; + expect(secondWinner.isOwned()).toBe(true); + expect(firstWinner.lease.isOwned()).toBe(false); + secondWinner.release(); + }); + + it('keeps random endpoint B reachable after old endpoint A closes late', async () => { + const root = shortPosixControlRoot(); + const sessionId = 'session-posix-close'; + const socketDirectory = join(root, 'sockets'); + const locatorPath = codexAppControlLocatorPath(root, sessionId); + mkdirSync(socketDirectory, { recursive: true, mode: 0o700 }); + mkdirSync(join(root, 'locators'), { recursive: true, mode: 0o700 }); + const endpointA = generateCodexAppPosixSocketEndpoint(socketDirectory); + const endpointB = generateCodexAppPosixSocketEndpoint(socketDirectory); + const serverA = createServer(socket => socket.end()); + const serverB = createServer(socket => socket.end('B')); + const listen = (server: Server, endpoint: string) => new Promise((resolvePromise, rejectPromise) => { + server.once('error', rejectPromise); + server.listen(endpoint, () => { + server.off('error', rejectPromise); + resolvePromise(); + }); + }); + await listen(serverA, endpointA); + await listen(serverB, endpointB); + const locator = { + version: 1 as const, + sessionId, + epoch: generateCodexAppControlEpoch(), + endpoint: endpointB, + }; + writeCodexAppControlLocator(locatorPath, locator, process.platform, root); + + await new Promise(resolvePromise => serverA.close(() => resolvePromise())); + expect(readCodexAppControlLocator( + locatorPath, + sessionId, + process.platform, + root, + )?.endpoint).toBe(endpointB); + expect(existsSync(endpointB)).toBe(true); + await new Promise((resolvePromise, rejectPromise) => { + const socket = createConnection(endpointB); + socket.once('data', data => { + expect(data.toString('utf8')).toBe('B'); + socket.destroy(); + resolvePromise(); + }); + socket.once('error', rejectPromise); + }); + await new Promise(resolvePromise => serverB.close(() => resolvePromise())); + }); +}); + +describe('Codex App signed challenge protocol', () => { + it('enforces challenge → matching locator epoch → ACK ordering and rejects repeats', () => { + const generation = generateCodexAppControlEpoch(); + const epoch = generateCodexAppControlEpoch(); + const challenge = generateCodexAppControlChallenge(); + const parse = (line: string) => parseCodexAppControlWireRecord(line); + const handshake = new CodexAppControlRunnerHandshake('session-handshake', generation, epoch); + + expect(handshake.handle(parse(encodeCodexAppControlChallenge( + 'session-handshake', challenge, + )), 0)).toEqual({ type: 'authenticate', challenge }); + expect(handshake.handle(parse(encodeCodexAppControlChallenge( + 'session-handshake', generateCodexAppControlChallenge(), + )), 0)).toEqual({ type: 'reject' }); + + const wrongEpoch = new CodexAppControlRunnerHandshake('session-handshake', generation, epoch); + expect(wrongEpoch.handle(parse(encodeCodexAppControlChallenge( + 'session-handshake', challenge, + )), 0).type).toBe('authenticate'); + expect(wrongEpoch.handle(parse(encodeCodexAppControlAccepted( + 'session-handshake', generation, challenge, generateCodexAppControlEpoch(), + )), 0)).toEqual({ type: 'reject' }); + + expect(handshake.handle(parse(encodeCodexAppControlAccepted( + 'session-handshake', generation, challenge, epoch, + )), 0)).toEqual({ type: 'accepted', challenge }); + expect(handshake.active).toBe(true); + expect(handshake.handle(parse(encodeCodexAppControlAck( + 'session-handshake', generation, challenge, 2, + )), 1)).toEqual({ type: 'reject' }); + expect(handshake.handle(parse(encodeCodexAppControlAck( + 'session-handshake', generation, challenge, 1, + )), 1)).toEqual({ type: 'ack', seq: 1 }); + }); + + it('uses an absolute handshake deadline that slow transport activity cannot extend', async () => { + vi.useFakeTimers(); + const timedOut = vi.fn(); + const timer = armCodexAppControlHandshakeTimeout(timedOut, 100); + try { + await vi.advanceTimersByTimeAsync(40); + // Model two partial/slow-drip transport reads. There is deliberately no + // activity/reset API: only a matching accepted record clears the timer. + await vi.advanceTimersByTimeAsync(40); + expect(timedOut).not.toHaveBeenCalled(); + await vi.advanceTimersByTimeAsync(20); + expect(timedOut).toHaveBeenCalledTimes(1); + } finally { + clearTimeout(timer); + vi.useRealTimers(); + } + }); + + it('re-arms the shared 90-second proof deadline after an authenticated disconnect', async () => { + vi.useFakeTimers(); + const deadline = new CodexAppControlProofDeadline(); + const startupTimeout = vi.fn(); + const disconnectTimeout = vi.fn(); + try { + deadline.arm(startupTimeout); + expect(deadline.armed).toBe(true); + // Successful startup authentication clears the first deadline. + deadline.clear(); + await vi.advanceTimersByTimeAsync(CODEX_APP_CONTROL_STARTUP_TIMEOUT_MS); + expect(startupTimeout).not.toHaveBeenCalled(); + + // Losing that authenticated socket starts a fresh proof deadline. + deadline.arm(disconnectTimeout); + await vi.advanceTimersByTimeAsync(CODEX_APP_CONTROL_STARTUP_TIMEOUT_MS - 1); + expect(disconnectTimeout).not.toHaveBeenCalled(); + await vi.advanceTimersByTimeAsync(1); + expect(disconnectTimeout).toHaveBeenCalledTimes(1); + expect(deadline.armed).toBe(false); + } finally { + deadline.clear(); + vi.useRealTimers(); + } + }); + + it('authenticates possession and signs every marker without putting a reusable secret on the wire', () => { + const dir = tempDir(); + const bootstrap = createCodexAppControlBootstrap(dir, 'session-signed'); + const consumed = consumeCodexAppControlBootstrap(bootstrap.path, 'session-signed'); + const challenge = generateCodexAppControlChallenge(); + + const authLine = encodeCodexAppControlAuth( + consumed.privateKey, + 'session-signed', + consumed.generation, + challenge, + ); + const auth = parseCodexAppControlWireRecord(authLine); + expect(auth?.type).toBe('auth'); + expect(authLine).not.toContain('privateKey'); + expect(auth && auth.type === 'auth' + ? verifyCodexAppControlAuth(auth, bootstrap.identity.publicKey) + : false).toBe(true); + + const markerLine = encodeCodexAppSignedControlMarker( + consumed.privateKey, + 'session-signed', + consumed.generation, + challenge, + 1, + 'activity', + { phase: 'progress', atMs: 123 }, + ); + const marker = parseCodexAppControlWireRecord(markerLine); + expect(marker?.type).toBe('marker'); + expect(marker && marker.type === 'marker' + ? verifyCodexAppSignedControlMarker(marker, bootstrap.identity.publicKey) + : false).toBe(true); + }); + + it('rejects replay across connection challenges and mutation of every signed domain field', () => { + const dir = tempDir(); + const bootstrap = createCodexAppControlBootstrap(dir, 'session-replay'); + const consumed = consumeCodexAppControlBootstrap(bootstrap.path, 'session-replay'); + const challenge = generateCodexAppControlChallenge(); + const otherChallenge = generateCodexAppControlChallenge(); + const auth = parseCodexAppControlWireRecord(encodeCodexAppControlAuth( + consumed.privateKey, 'session-replay', consumed.generation, challenge, + )); + expect(auth?.type).toBe('auth'); + if (!auth || auth.type !== 'auth') throw new Error('auth parse failed'); + expect(verifyCodexAppControlAuth({ ...auth, challenge: otherChallenge }, bootstrap.identity.publicKey)).toBe(false); + expect(verifyCodexAppControlAuth({ ...auth, sessionId: 'other-session' }, bootstrap.identity.publicKey)).toBe(false); + + const marker = parseCodexAppControlWireRecord(encodeCodexAppSignedControlMarker( + consumed.privateKey, + 'session-replay', + consumed.generation, + challenge, + 7, + 'final', + { content: 'real' }, + )); + expect(marker?.type).toBe('marker'); + if (!marker || marker.type !== 'marker') throw new Error('marker parse failed'); + for (const mutated of [ + { ...marker, sessionId: 'other' }, + { ...marker, generation: 'A'.repeat(43) }, + { ...marker, challenge: otherChallenge }, + { ...marker, seq: 8 }, + { ...marker, kind: 'activity' }, + { ...marker, payload: { content: 'forged' } }, + ]) { + expect(verifyCodexAppSignedControlMarker(mutated, bootstrap.identity.publicKey)).toBe(false); + } + }); + + it('deduplicates a re-signed sequence retry across socket reconnects', () => { + const replay = new CodexAppControlReplayWindow(); + const firstGeneration = 'A'.repeat(43); + const nextGeneration = 'B'.repeat(43); + expect(replay.hasSeen(firstGeneration, 1)).toBe(false); + replay.commit(firstGeneration, 1); + expect(replay.hasSeen(firstGeneration, 1)).toBe(true); + expect(replay.hasSeen(firstGeneration, 0)).toBe(true); + expect(replay.hasSeen(firstGeneration, 2)).toBe(false); + replay.commit(firstGeneration, 3); + expect(replay.hasSeen(firstGeneration, 2)).toBe(true); + + replay.commit(nextGeneration, 7); + replay.retainOnly(firstGeneration); + expect(replay.highWater(firstGeneration)).toBe(3); + expect(replay.highWater(nextGeneration)).toBe(0); + }); + + it('bounds socket line buffering and resynchronizes after oversized input', () => { + const decoder = new CodexAppControlLineDecoder(); + expect(decoder.push(Buffer.from('one\npar')).lines).toEqual(['one']); + expect(decoder.push(Buffer.from('tial\n')).lines).toEqual(['partial']); + expect(decoder.push(Buffer.alloc(CODEX_APP_CONTROL_LINE_MAX_BYTES + 1, 0x78)).droppedMalformed).toBe(true); + const recovered = decoder.push(Buffer.from('\nvalid\n')); + expect(recovered).toEqual({ lines: ['valid'], droppedMalformed: true }); + }); +}); diff --git a/test/codex-app-dispatch-ledger.test.ts b/test/codex-app-dispatch-ledger.test.ts new file mode 100644 index 000000000..f0574710c --- /dev/null +++ b/test/codex-app-dispatch-ledger.test.ts @@ -0,0 +1,201 @@ +import { describe, expect, it } from 'vitest'; +import { + appendAcceptedCodexAppDispatch, + cancelCodexAppDispatch, + committedCodexAppSequence, + hasUnsettledCodexAppDispatch, + prepareCodexAppDispatch, + retryPreparedCodexAppDispatch, + retireCodexAppDispatchAfterBackingMissing, + settleCodexAppDispatch, + validateCodexAppManagedSendOrigin, +} from '../src/utils/codex-app-dispatch-ledger.js'; + +describe('Codex App durable dispatch ledger', () => { + it('preserves frozen payload and settles only the exact prepared FIFO head', () => { + let ledger = appendAcceptedCodexAppDispatch([], { + dispatchId: 'dispatch-1', + turnId: 'turn-1', + replyTurnId: 'route-1', + deliverySink: 'http_wait', + content: 'frozen content', + codexAppInput: { text: 'clean content' }, + }); + ledger = appendAcceptedCodexAppDispatch(ledger, { + dispatchId: 'dispatch-2', + turnId: 'turn-2', + content: 'next content', + }); + + const first = prepareCodexAppDispatch(ledger, { + dispatchId: 'dispatch-1', turnId: 'turn-1', + }); + expect(first).toMatchObject({ ok: true }); + if (!first.ok) return; + expect(first.ledger[0]).toMatchObject({ + state: 'prepared', + replyTurnId: 'route-1', + deliverySink: 'http_wait', + content: 'frozen content', + codexAppInput: { text: 'clean content' }, + }); + + const settled = settleCodexAppDispatch( + first.ledger, + [], + { dispatchId: 'dispatch-1', turnId: 'turn-1' }, + 'generation-1', + 7, + ); + expect(settled).toMatchObject({ ok: true }); + if (!settled.ok) return; + expect(settled.ledger.map(entry => entry.dispatchId)).toEqual(['dispatch-2']); + expect(committedCodexAppSequence(settled.commits, 'generation-1', 7)).toBe(true); + }); + + it('treats every accepted or prepared entry as unsettled lifecycle ownership', () => { + expect(hasUnsettledCodexAppDispatch(undefined)).toBe(false); + expect(hasUnsettledCodexAppDispatch([])).toBe(false); + expect(hasUnsettledCodexAppDispatch([ + { dispatchId: 'd1', turnId: 't1', state: 'accepted', content: 'one' }, + ])).toBe(true); + expect(hasUnsettledCodexAppDispatch([ + { dispatchId: 'd1', turnId: 't1', state: 'prepared', content: 'one' }, + ])).toBe(true); + }); + + it('authorizes host relay only for the exact live Lark-bound Codex App entry', () => { + const ledger = [{ + dispatchId: 'd1', turnId: 't1', dispatchAttempt: 3, + state: 'prepared' as const, content: 'one', deliverySink: 'lark' as const, + }]; + expect(validateCodexAppManagedSendOrigin( + ledger, + { turnId: 't1', dispatchAttempt: 3 }, + true, + )).toEqual({ ok: true, requiresLedger: true }); + expect(validateCodexAppManagedSendOrigin( + ledger, + { turnId: 't1', dispatchAttempt: 4 }, + true, + )).toMatchObject({ ok: false }); + expect(validateCodexAppManagedSendOrigin([ + ...ledger, + { ...ledger[0]!, dispatchId: 'd2' }, + ], { turnId: 't1', dispatchAttempt: 3 }, true)).toEqual({ + ok: false, + error: '2 Codex App ledger entries match the live relay origin', + }); + }); + + it('rejects a Codex App capability when settlement removed the ledger before relay admission', () => { + expect(validateCodexAppManagedSendOrigin( + [], + { turnId: 't1', dispatchAttempt: 3 }, + true, + )).toEqual({ + ok: false, + error: '0 Codex App ledger entries match the live relay origin', + }); + // A non-Codex managed receiver may legitimately carry a dispatch attempt + // without owning the Codex ledger; do not blanket-block that path. + expect(validateCodexAppManagedSendOrigin( + [], + { turnId: 'vc-turn', dispatchAttempt: 3 }, + false, + )).toEqual({ ok: true, requiresLedger: false }); + }); + + it('rejects Codex App host sends bound to non-Lark sinks', () => { + for (const deliverySink of ['http_wait', 'http_async', 'suppressed'] as const) { + expect(validateCodexAppManagedSendOrigin([{ + dispatchId: 'd1', turnId: 't1', state: 'prepared', content: 'one', deliverySink, + }], { turnId: 't1' }, true)).toEqual({ + ok: false, + error: `Codex App output is bound to ${deliverySink}`, + }); + } + }); + + it('does not cancel a predecessor while a prepared successor exists', () => { + const ledger = [ + { dispatchId: 'd1', turnId: 't1', state: 'prepared' as const, content: 'one' }, + { dispatchId: 'd2', turnId: 't2', state: 'prepared' as const, content: 'two' }, + ]; + expect(cancelCodexAppDispatch(ledger, { dispatchId: 'd1', turnId: 't1' })) + .toEqual({ ok: false, error: 'prepared_successor_exists' }); + }); + + it('returns an exactly untouched queued activation from prepared to accepted without losing its token', () => { + const ledger = [{ + dispatchId: 'activation-dispatch', + turnId: 'activation-turn', + dispatchAttempt: 2, + state: 'prepared' as const, + content: 'exact queued opening', + queuedActivationToken: 'activation-token', + }]; + expect(retryPreparedCodexAppDispatch(ledger, { + dispatchId: 'activation-dispatch', + turnId: 'activation-turn', + dispatchAttempt: 2, + })).toEqual({ + ok: true, + ledger: [{ ...ledger[0], state: 'accepted' }], + }); + }); + + it('refuses a prepared retry when a prepared successor could overtake it', () => { + const ledger = [ + { dispatchId: 'activation', turnId: 't1', state: 'prepared' as const, content: 'one' }, + { dispatchId: 'successor', turnId: 't2', state: 'prepared' as const, content: 'two' }, + ]; + expect(retryPreparedCodexAppDispatch(ledger, { + dispatchId: 'activation', turnId: 't1', + })).toEqual({ ok: false, error: 'prepared_successor_exists' }); + }); + + it.each(['accepted', 'prepared'] as const)( + 'retires an exact %s crashed delivery after backing-missing proof without dropping its successor', + state => { + const ledger = [ + { + dispatchId: 'old-dispatch', turnId: 'delivery-old', dispatchAttempt: 3, + state, content: 'old', + }, + { + dispatchId: 'successor', turnId: 'ordinary-successor', + state: 'prepared' as const, content: 'new', + }, + ]; + expect(retireCodexAppDispatchAfterBackingMissing(ledger, 'delivery-old', 3)) + .toEqual({ ok: true, ledger: [ledger[1]] }); + }, + ); + + it('keeps exact crash retirement idempotent and fails closed on ambiguous receipt identity', () => { + expect(retireCodexAppDispatchAfterBackingMissing([], 'delivery-old', 3)) + .toEqual({ ok: true, ledger: [] }); + const ambiguous = [ + { dispatchId: 'd1', turnId: 'delivery-old', dispatchAttempt: 3, state: 'accepted' as const, content: 'one' }, + { dispatchId: 'd2', turnId: 'delivery-old', dispatchAttempt: 3, state: 'prepared' as const, content: 'two' }, + ]; + expect(retireCodexAppDispatchAfterBackingMissing(ambiguous, 'delivery-old', 3)) + .toEqual({ ok: false, error: 'dispatch_identity_ambiguous' }); + for (const conflictingAttempt of [2, undefined]) { + const conflict = [{ + dispatchId: 'old', turnId: 'delivery-old', dispatchAttempt: conflictingAttempt, + state: 'accepted' as const, content: 'old', + }]; + expect(retireCodexAppDispatchAfterBackingMissing(conflict, 'delivery-old', 3)) + .toEqual({ ok: false, error: 'dispatch_attempt_conflict' }); + } + + const duplicateDispatchId = [ + { dispatchId: 'same', turnId: 'delivery-old', dispatchAttempt: 3, state: 'accepted' as const, content: 'old' }, + { dispatchId: 'same', turnId: 'unrelated', state: 'prepared' as const, content: 'new' }, + ]; + expect(retireCodexAppDispatchAfterBackingMissing(duplicateDispatchId, 'delivery-old', 3)) + .toEqual({ ok: true, ledger: [duplicateDispatchId[1]] }); + }); +}); diff --git a/test/codex-app-runner.integration.test.ts b/test/codex-app-runner.integration.test.ts index d961f20cd..26563b978 100644 --- a/test/codex-app-runner.integration.test.ts +++ b/test/codex-app-runner.integration.test.ts @@ -3,21 +3,46 @@ import { chmodSync, copyFileSync, existsSync, + mkdirSync, mkdtempSync, readFileSync, rmSync, + unlinkSync, writeFileSync, } from 'node:fs'; +import { createServer, type Server, type Socket } from 'node:net'; import { tmpdir } from 'node:os'; import { join, resolve } from 'node:path'; import { afterEach, describe, expect, it } from 'vitest'; import { encodeRunnerInput } from '../src/adapters/cli/runner-input.js'; +import { + CodexAppControlFinalAssembler, + CodexAppControlLineDecoder, + CodexAppControlSequenceFence, + codexAppControlLocatorPath, + codexAppPosixControlRoot, + codexAppControlSocketPath, + createCodexAppControlBootstrap, + encodeCodexAppControlAck, + encodeCodexAppControlAccepted, + encodeCodexAppControlChallenge, + ensureCodexAppControlDirectory, + generateCodexAppControlChallenge, + generateCodexAppControlEpoch, + generateCodexAppPosixSocketEndpoint, + parseCodexAppControlWireRecord, + verifyCodexAppControlAuth, + verifyCodexAppSignedControlMarker, + writeCodexAppControlLocator, + type CodexAppControlLocator, + type CodexAppSignedControlMarker, +} from '../src/utils/codex-app-control.js'; import type { CodexAppTurnInput } from '../src/types.js'; const RUNNER_PATH = resolve('src/codex-app-runner.ts'); const FAKE_SERVER_FIXTURE = resolve('test/fixtures/fake-codex-app-server.mjs'); const CONTROL_PREFIX = '::botmux-codex-app:'; -const FINAL_MARKER = /\x1b\]777;botmux:final:([A-Za-z0-9+/=]+)\x07/; +const SESSION_ID = 'session-integration'; interface Harness { child: ChildProcessWithoutNullStreams; @@ -31,9 +56,388 @@ interface RunResult { imagePath: string; missingImagePath: string; final: Record; + finals: Array>; + activities: Array>; + states: Array>; + markers: Array<{ kind: string; payload: Record }>; + wireLines: string[]; + privateKeyEncoding: string; } const liveChildren = new Set(); +const liveCollectors = new Set(); +const liveLocatorCollectors = new Set(); + +class ControlCollector { + readonly bootstrap; + readonly socketPath: string; + readonly privateKeyEncoding: string; + readonly activities: Array> = []; + readonly states: Array> = []; + readonly finals: Array> = []; + readonly markers: Array<{ kind: string; payload: Record }> = []; + readonly wireLines: string[] = []; + authCount = 0; + readonly authObserved: Promise; + private resolveAuthObserved!: () => void; + private readonly server: Server; + private readonly sockets = new Set(); + private pendingAcceptance?: { socket: Socket; challenge: string }; + private lastSeq = 0; + private disconnectedFinalChunk = false; + private omittedFinalChunk = false; + private disconnectedFinalEndAck = false; + private readonly socketDirectory: string; + + constructor( + readonly directory: string, + private readonly manualAcceptance = false, + private readonly disconnectOnFirstFinalChunk = false, + private readonly omitFirstFinalChunk = false, + private readonly disconnectAfterFirstFinalEndBeforeAck = false, + ) { + this.socketDirectory = mkdtempSync('/tmp/bca-sock-'); + this.socketPath = codexAppControlSocketPath(this.socketDirectory, SESSION_ID); + this.bootstrap = createCodexAppControlBootstrap(directory, SESSION_ID, this.socketPath); + this.privateKeyEncoding = JSON.parse(readFileSync(this.bootstrap.path, 'utf8')).privateKey; + this.authObserved = new Promise(resolvePromise => { this.resolveAuthObserved = resolvePromise; }); + this.server = createServer(socket => this.accept(socket)); + liveCollectors.add(this); + } + + listen(): Promise { + return new Promise((resolvePromise, rejectPromise) => { + this.server.once('error', rejectPromise); + this.server.listen(this.socketPath, () => { + this.server.off('error', rejectPromise); + resolvePromise(); + }); + }); + } + + releaseAcceptance(): void { + const pending = this.pendingAcceptance; + if (!pending) throw new Error('no authenticated runner is awaiting acceptance'); + this.pendingAcceptance = undefined; + pending.socket.write(`${encodeCodexAppControlAccepted( + SESSION_ID, + this.bootstrap.identity.generation, + pending.challenge, + )}\n`); + } + + async restartEndpoint(): Promise { + for (const socket of this.sockets) socket.destroy(); + this.sockets.clear(); + if (this.server.listening) { + await new Promise(resolvePromise => this.server.close(() => resolvePromise())); + } + await this.listen(); + } + + async close(): Promise { + for (const socket of this.sockets) socket.destroy(); + this.sockets.clear(); + if (this.server.listening) { + await new Promise(resolvePromise => this.server.close(() => resolvePromise())); + } + liveCollectors.delete(this); + rmSync(this.socketDirectory, { recursive: true, force: true }); + } + + private accept(socket: Socket): void { + this.sockets.add(socket); + const decoder = new CodexAppControlLineDecoder(); + const sequenceFence = new CodexAppControlSequenceFence(); + const finalAssembler = new CodexAppControlFinalAssembler(); + const challenge = generateCodexAppControlChallenge(); + let authenticated = false; + socket.on('data', chunk => { + const decoded = decoder.push(chunk); + if (decoded.droppedMalformed) socket.destroy(); + for (const line of decoded.lines) { + this.wireLines.push(line); + const record = parseCodexAppControlWireRecord(line); + if (!record || record.sessionId !== SESSION_ID) { + socket.destroy(); + continue; + } + if (!authenticated) { + if (record.type !== 'auth' + || record.challenge !== challenge + || record.generation !== this.bootstrap.identity.generation + || !verifyCodexAppControlAuth(record, this.bootstrap.identity.publicKey)) { + socket.destroy(); + continue; + } + authenticated = true; + this.authCount++; + this.resolveAuthObserved(); + if (this.manualAcceptance) this.pendingAcceptance = { socket, challenge }; + else { + socket.write(`${encodeCodexAppControlAccepted( + SESSION_ID, + this.bootstrap.identity.generation, + challenge, + )}\n`); + } + continue; + } + if (record.type !== 'marker' + || record.challenge !== challenge + || record.generation !== this.bootstrap.identity.generation + || !sequenceFence.accept(record.seq) + || !verifyCodexAppSignedControlMarker(record, this.bootstrap.identity.publicKey)) { + socket.destroy(); + continue; + } + // The worker checks connection continuity before its cumulative replay + // window. A reconnect may therefore replay an already-committed final + // transaction contiguously: ACK every duplicate without reassembling + // or publishing its final side effect again. + if (record.seq <= this.lastSeq) { + socket.write(`${encodeCodexAppControlAck( + SESSION_ID, + this.bootstrap.identity.generation, + challenge, + record.seq, + )}\n`); + continue; + } + this.markers.push({ kind: record.kind, payload: record.payload }); + if (record.kind === 'final-chunk' + && this.disconnectOnFirstFinalChunk + && !this.disconnectedFinalChunk) { + this.disconnectedFinalChunk = true; + // Model a replacement worker: the per-connection assembly and + // in-memory cumulative sequence window both disappear. + this.lastSeq = 0; + socket.destroy(); + return; + } + if (record.kind === 'final-chunk' + && this.omitFirstFinalChunk + && !this.omittedFinalChunk) { + // Simulate a missing fragment inside this connection. final-end must + // be rejected without an ACK, forcing a complete replay. + this.omittedFinalChunk = true; + continue; + } + const finalResult = finalAssembler.accept(record.kind, record.payload); + if (finalResult.status === 'reject') { + socket.destroy(); + return; + } + if (finalResult.status === 'not-final') this.collectNonFinalMarker(record); + else if (finalResult.status === 'complete') this.finals.push(finalResult.payload); + if (finalResult.status === 'accepted') continue; + this.lastSeq = record.seq; + if (record.kind === 'final-end' + && this.disconnectAfterFirstFinalEndBeforeAck + && !this.disconnectedFinalEndAck) { + this.disconnectedFinalEndAck = true; + socket.destroy(); + return; + } + socket.write(`${encodeCodexAppControlAck( + SESSION_ID, + this.bootstrap.identity.generation, + challenge, + record.seq, + )}\n`); + } + }); + socket.on('error', () => undefined); + socket.on('close', () => this.sockets.delete(socket)); + socket.write(`${encodeCodexAppControlChallenge(SESSION_ID, challenge)}\n`); + } + + private collectNonFinalMarker(marker: CodexAppSignedControlMarker): void { + if (marker.kind === 'state') { + this.states.push(marker.payload); + return; + } + if (marker.kind === 'activity') { + this.activities.push(marker.payload); + } + } +} + +type LocatorEndpointMode = 'accept' | 'wrong-epoch' | 'repeat-challenge' | 'slow-drip'; + +interface LocatorEndpointHandle { + locator: CodexAppControlLocator; + readonly connections: number; + readonly closedConnections: number; + readonly authCount: number; + authObserved: Promise; + closeObserved: Promise; + close(): Promise; +} + +/** + * Runs the real runner locator loop on POSIX with the same strict random + * AF_UNIX endpoint + protected locator shape used by the worker. + */ +class LocatorControlCollector { + readonly locatorPath: string; + readonly bootstrap; + private readonly endpoints = new Set(); + private readonly socketDirectory: string; + + constructor(readonly directory: string) { + const controlRoot = codexAppPosixControlRoot(); + this.locatorPath = codexAppControlLocatorPath(controlRoot, SESSION_ID); + this.socketDirectory = join(controlRoot, 'sockets'); + ensureCodexAppControlDirectory(controlRoot); + ensureCodexAppControlDirectory(join(controlRoot, 'locators')); + ensureCodexAppControlDirectory(this.socketDirectory); + this.bootstrap = createCodexAppControlBootstrap(directory, SESSION_ID, { + kind: 'locator', + locatorPath: this.locatorPath, + }); + liveLocatorCollectors.add(this); + } + + async publish(mode: LocatorEndpointMode): Promise { + const endpoint = generateCodexAppPosixSocketEndpoint(this.socketDirectory); + const epoch = generateCodexAppControlEpoch(); + const locator: CodexAppControlLocator = { + version: 1, + sessionId: SESSION_ID, + endpoint, + epoch, + }; + const server = createServer(); + const sockets = new Set(); + let connectionCount = 0; + let closedConnectionCount = 0; + let authCount = 0; + let resolveAuth!: () => void; + let resolveClose!: () => void; + const authObserved = new Promise(resolvePromise => { resolveAuth = resolvePromise; }); + const closeObserved = new Promise(resolvePromise => { resolveClose = resolvePromise; }); + server.on('connection', socket => { + sockets.add(socket); + connectionCount++; + const decoder = new CodexAppControlLineDecoder(); + const challenge = generateCodexAppControlChallenge(); + let accepted = false; + let lastSeq = 0; + let dripTimer: ReturnType | undefined; + socket.on('data', chunk => { + const decoded = decoder.push(chunk); + if (decoded.droppedMalformed) socket.destroy(); + for (const line of decoded.lines) { + const record = parseCodexAppControlWireRecord(line); + if (!record || record.sessionId !== SESSION_ID) { + socket.destroy(); + continue; + } + if (!accepted) { + if (record.type !== 'auth' + || record.generation !== this.bootstrap.identity.generation + || record.challenge !== challenge + || !verifyCodexAppControlAuth(record, this.bootstrap.identity.publicKey)) { + socket.destroy(); + continue; + } + authCount++; + resolveAuth(); + if (mode === 'repeat-challenge') continue; + accepted = true; + socket.write(`${encodeCodexAppControlAccepted( + SESSION_ID, + this.bootstrap.identity.generation, + challenge, + mode === 'wrong-epoch' ? generateCodexAppControlEpoch() : epoch, + )}\n`); + continue; + } + if (record.type !== 'marker' + || record.generation !== this.bootstrap.identity.generation + || record.challenge !== challenge + || record.seq <= lastSeq + || !verifyCodexAppSignedControlMarker(record, this.bootstrap.identity.publicKey)) { + socket.destroy(); + continue; + } + lastSeq = record.seq; + socket.write(`${encodeCodexAppControlAck( + SESSION_ID, + this.bootstrap.identity.generation, + challenge, + record.seq, + )}\n`); + } + }); + socket.on('error', () => undefined); + socket.on('close', () => { + if (dripTimer) clearInterval(dripTimer); + sockets.delete(socket); + closedConnectionCount++; + resolveClose(); + }); + if (mode === 'slow-drip') { + const line = `${encodeCodexAppControlChallenge(SESSION_ID, challenge)}\n`; + let offset = 0; + socket.write(line[offset++]!); + dripTimer = setInterval(() => { + if (socket.destroyed || offset >= line.length) { + if (dripTimer) clearInterval(dripTimer); + dripTimer = undefined; + return; + } + socket.write(line[offset++]!); + }, 200); + } else { + socket.write(`${encodeCodexAppControlChallenge(SESSION_ID, challenge)}\n`); + } + if (mode === 'repeat-challenge') { + socket.write(`${encodeCodexAppControlChallenge( + SESSION_ID, + generateCodexAppControlChallenge(), + )}\n`); + } + }); + await new Promise((resolvePromise, rejectPromise) => { + server.once('error', rejectPromise); + server.listen(endpoint, () => { + server.off('error', rejectPromise); + resolvePromise(); + }); + }); + writeCodexAppControlLocator(this.locatorPath, locator); + let closed = false; + const handle: LocatorEndpointHandle = { + locator, + get connections() { return connectionCount; }, + get closedConnections() { return closedConnectionCount; }, + get authCount() { return authCount; }, + authObserved, + closeObserved, + close: async () => { + if (closed) return; + closed = true; + for (const socket of sockets) socket.destroy(); + sockets.clear(); + if (server.listening) { + await new Promise(resolvePromise => server.close(() => resolvePromise())); + } + try { unlinkSync(endpoint); } catch { /* libuv may already remove it */ } + this.endpoints.delete(handle); + }, + }; + this.endpoints.add(handle); + return handle; + } + + async close(): Promise { + await Promise.all([...this.endpoints].map(endpoint => endpoint.close())); + try { unlinkSync(this.locatorPath); } catch { /* absent or already replaced */ } + liveLocatorCollectors.delete(this); + } +} function startRunner( fakeCodex: string, @@ -41,27 +445,32 @@ function startRunner( logPath: string, version: string, behavior: string, + controlBootstrapPath: string | null, + options: { threadId?: string; env?: Record } = {}, ): Harness { let stdout = ''; let stderr = ''; - const child = spawn(process.execPath, [ - '--import', - 'tsx', - RUNNER_PATH, - '--session-id', - 'session-integration', - '--codex-bin', - fakeCodex, - '--cwd', - cwd, - ], { + const env = { + ...process.env, + FAKE_CODEX_LOG: logPath, + FAKE_CODEX_VERSION: version, + FAKE_CODEX_BEHAVIOR: behavior, + NODE_ENV: 'test', + ...options.env, + }; + delete env.BOTMUX_CODEX_APP_CONTROL_NONCE; + delete env.BOTMUX_CODEX_APP_CONTROL_BOOTSTRAP; + if (controlBootstrapPath !== null) env.BOTMUX_CODEX_APP_CONTROL_BOOTSTRAP = controlBootstrapPath; + const runnerArgs = [ + '--import', 'tsx', RUNNER_PATH, + '--session-id', SESSION_ID, + '--codex-bin', fakeCodex, + '--cwd', cwd, + ...(options.threadId ? ['--thread-id', options.threadId] : []), + ]; + const child = spawn(process.execPath, runnerArgs, { cwd: resolve('.'), - env: { - ...process.env, - FAKE_CODEX_LOG: logPath, - FAKE_CODEX_VERSION: version, - FAKE_CODEX_BEHAVIOR: behavior, - }, + env, stdio: ['pipe', 'pipe', 'pipe'], }); liveChildren.add(child); @@ -75,28 +484,31 @@ function startRunner( }; } -function waitForOutput(harness: Harness, predicate: (output: string) => boolean, timeoutMs = 10_000): Promise { - if (predicate(harness.stdout)) return Promise.resolve(); +function waitFor( + harness: Harness, + predicate: () => boolean, + timeoutMs = 10_000, +): Promise { + if (predicate()) return Promise.resolve(); return new Promise((resolvePromise, rejectPromise) => { + const poll = setInterval(() => { + if (!predicate()) return; + cleanup(); + resolvePromise(); + }, 10); const timer = setTimeout(() => { cleanup(); - rejectPromise(new Error(`runner output timed out\nstdout:\n${harness.stdout}\nstderr:\n${harness.stderr}`)); + rejectPromise(new Error(`runner timed out\nstdout:\n${harness.stdout}\nstderr:\n${harness.stderr}`)); }, timeoutMs); - const onData = () => { - if (!predicate(harness.stdout)) return; - cleanup(); - resolvePromise(); - }; const onExit = (code: number | null, signal: NodeJS.Signals | null) => { cleanup(); - rejectPromise(new Error(`runner exited before expected output (code=${code}, signal=${signal})\nstdout:\n${harness.stdout}\nstderr:\n${harness.stderr}`)); + rejectPromise(new Error(`runner exited early (code=${code}, signal=${signal})\nstdout:\n${harness.stdout}\nstderr:\n${harness.stderr}`)); }; const cleanup = () => { + clearInterval(poll); clearTimeout(timer); - harness.child.stdout.off('data', onData); harness.child.off('exit', onExit); }; - harness.child.stdout.on('data', onData); harness.child.once('exit', onExit); }); } @@ -104,9 +516,7 @@ function waitForOutput(harness: Harness, predicate: (output: string) => boolean, async function stopChild(child: ChildProcessWithoutNullStreams): Promise { if (child.exitCode !== null || child.signalCode !== null) return; await new Promise(resolvePromise => { - const forceTimer = setTimeout(() => { - child.kill('SIGKILL'); - }, 1_000); + const forceTimer = setTimeout(() => child.kill('SIGKILL'), 1_000); child.once('exit', () => { clearTimeout(forceTimer); resolvePromise(); @@ -115,25 +525,17 @@ async function stopChild(child: ChildProcessWithoutNullStreams): Promise { }); } -function decodeFinalMarker(output: string): Record { - const match = output.match(FINAL_MARKER); - if (!match) throw new Error(`final marker missing from output:\n${output}`); - return JSON.parse(Buffer.from(match[1], 'base64').toString('utf8')); -} - function readRequests(logPath: string): Array> { if (!existsSync(logPath)) return []; - return readFileSync(logPath, 'utf8') - .split('\n') - .filter(Boolean) - .map(line => JSON.parse(line)); + return readFileSync(logPath, 'utf8').split('\n').filter(Boolean).map(line => JSON.parse(line)); } async function exerciseRunner(opts: { version: string; - behavior?: 'success' | 'capability-error' | 'generic-error' | 'osc-injection'; + behavior?: 'success' | 'capability-error' | 'generic-error' | 'osc-injection' | 'empty-final' | 'start-response-last'; includeMissingImage?: boolean; includeSidecar?: boolean; + turnCount?: number; }): Promise { const dir = mkdtempSync(join(tmpdir(), 'botmux-codex-runner-')); const fakeCodex = join(dir, 'fake-codex'); @@ -146,7 +548,8 @@ async function exerciseRunner(opts: { 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAusB9Y9Zg0sAAAAASUVORK5CYII=', 'base64', )); - + const control = new ControlCollector(dir); + await control.listen(); const sidecar: CodexAppTurnInput = { text: 'clean user text', additionalContext: { @@ -161,37 +564,338 @@ async function exerciseRunner(opts: { ], clientUserMessageId: 'om_integration_123', }; - const harness = startRunner(fakeCodex, dir, logPath, opts.version, opts.behavior ?? 'success'); + const harness = startRunner( + fakeCodex, dir, logPath, opts.version, opts.behavior ?? 'success', control.bootstrap.path, + ); try { - await waitForOutput(harness, output => output.includes('Codex App connected.')); - const encoded = encodeRunnerInput( - 'legacy prompt', - opts.includeSidecar === false ? undefined : sidecar, - ); - harness.child.stdin.write(`${CONTROL_PREFIX}${encoded}\r`); - await waitForOutput(harness, output => FINAL_MARKER.test(output)); - + await waitFor(harness, () => harness.stdout.includes('Codex App connected.')); + expect(existsSync(control.bootstrap.path)).toBe(false); + const turnCount = opts.turnCount ?? 1; + for (let i = 0; i < turnCount; i++) { + const legacyContent = turnCount === 1 + ? 'legacy prompt' + : `legacy prompt ${i + 1}`; + const turnSidecar = opts.includeSidecar === false + ? undefined + : turnCount === 1 + ? sidecar + : { ...sidecar, clientUserMessageId: `om_integration_${i + 1}` }; + harness.child.stdin.write(`${CONTROL_PREFIX}${encodeRunnerInput( + legacyContent, + turnSidecar, + )}\r`); + } + await waitFor(harness, () => ( + control.finals.length >= turnCount + && control.states.filter(state => state.busy === false).length >= 2 + && (harness.stdout.match(/› /g)?.length ?? 0) >= 2 + )); const output = harness.stdout; - const final = decodeFinalMarker(output); + const requests = readRequests(logPath); await stopChild(harness.child); - return { output, requests: readRequests(logPath), imagePath, missingImagePath, final }; + return { + output, + requests, + imagePath, + missingImagePath, + final: control.finals[0]!, + finals: [...control.finals], + activities: [...control.activities], + states: [...control.states], + markers: [...control.markers], + wireLines: [...control.wireLines], + privateKeyEncoding: control.privateKeyEncoding, + }; } finally { await stopChild(harness.child); + await control.close(); rmSync(dir, { recursive: true, force: true }); } } afterEach(async () => { await Promise.all([...liveChildren].map(stopChild)); + await Promise.all([...liveCollectors].map(collector => collector.close())); + await Promise.all([...liveLocatorCollectors].map(collector => collector.close())); }); describe('codex-app-runner app-server protocol integration', () => { + it('refuses to start without a worker-established control bootstrap', async () => { + const dir = mkdtempSync(join(tmpdir(), 'botmux-codex-runner-no-key-')); + const harness = startRunner('/does/not/matter', dir, join(dir, 'requests.jsonl'), '0.136.0', 'success', null); + try { + const exitCode = harness.child.exitCode ?? await new Promise(resolvePromise => { + harness.child.once('exit', code => resolvePromise(code)); + }); + expect(exitCode).toBe(2); + expect(harness.stderr).toContain('BOTMUX_CODEX_APP_CONTROL_BOOTSTRAP is required'); + } finally { + await stopChild(harness.child); + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('does not start app-server until the worker verifies proof and accepts the generation', async () => { + const dir = mkdtempSync(join(tmpdir(), 'botmux-codex-runner-auth-gate-')); + const fakeCodex = join(dir, 'fake-codex'); + const logPath = join(dir, 'requests.jsonl'); + copyFileSync(FAKE_SERVER_FIXTURE, fakeCodex); + chmodSync(fakeCodex, 0o755); + const control = new ControlCollector(dir, true); + await control.listen(); + const harness = startRunner(fakeCodex, dir, logPath, '0.136.0', 'success', control.bootstrap.path); + try { + await control.authObserved; + expect(readRequests(logPath)).toEqual([]); + expect(harness.stdout).not.toContain('Codex App connected.'); + control.releaseAcceptance(); + await waitFor(harness, () => harness.stdout.includes('Codex App connected.')); + expect(readRequests(logPath).some(request => request.method === 'initialize')).toBe(true); + } finally { + await stopChild(harness.child); + await control.close(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('keeps retrying when the worker socket begins listening after the runner starts', async () => { + const dir = mkdtempSync(join(tmpdir(), 'botmux-codex-runner-late-socket-')); + const fakeCodex = join(dir, 'fake-codex'); + const logPath = join(dir, 'requests.jsonl'); + copyFileSync(FAKE_SERVER_FIXTURE, fakeCodex); + chmodSync(fakeCodex, 0o755); + const control = new ControlCollector(dir); + const harness = startRunner(fakeCodex, dir, logPath, '0.136.0', 'success', control.bootstrap.path); + try { + await new Promise(resolvePromise => setTimeout(resolvePromise, 300)); + expect(harness.child.exitCode).toBeNull(); + expect(readRequests(logPath)).toEqual([]); + await control.listen(); + await waitFor(harness, () => harness.stdout.includes('Codex App connected.')); + expect(control.authCount).toBe(1); + } finally { + await stopChild(harness.child); + await control.close(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('polls locators, rejects repeated/wrong-epoch/slow-drip handshakes, burns A, and connects B', async () => { + const dir = mkdtempSync(join(tmpdir(), 'botmux-codex-runner-locator-')); + const fakeCodex = join(dir, 'fake-codex'); + const logPath = join(dir, 'requests.jsonl'); + copyFileSync(FAKE_SERVER_FIXTURE, fakeCodex); + chmodSync(fakeCodex, 0o755); + const control = new LocatorControlCollector(dir); + const harness = startRunner(fakeCodex, dir, logPath, '0.136.0', 'success', control.bootstrap.path); + try { + // Missing locator is a poll miss, not a fatal bootstrap/app-server start. + await new Promise(resolvePromise => setTimeout(resolvePromise, 350)); + expect(harness.child.exitCode).toBeNull(); + expect(readRequests(logPath)).toEqual([]); + + const repeated = await control.publish('repeat-challenge'); + await waitFor(harness, () => repeated.closedConnections >= 1); + expect(readRequests(logPath)).toEqual([]); + + const wrongEpoch = await control.publish('wrong-epoch'); + await waitFor(harness, () => wrongEpoch.authCount >= 1 && wrongEpoch.closedConnections >= 1); + expect(readRequests(logPath)).toEqual([]); + + const slowDripStartedAt = Date.now(); + const slowDrip = await control.publish('slow-drip'); + await waitFor(harness, () => slowDrip.closedConnections >= 1); + expect(Date.now() - slowDripStartedAt).toBeGreaterThanOrEqual(4_500); + expect(readRequests(logPath)).toEqual([]); + + const acceptedA = await control.publish('accept'); + await waitFor(harness, () => ( + acceptedA.authCount >= 1 && harness.stdout.includes('Codex App connected.') + )); + expect(readRequests(logPath).filter(request => request.method === 'initialize')).toHaveLength(1); + + await acceptedA.close(); + const acceptedAConnections = acceptedA.connections; + await new Promise(resolvePromise => setTimeout(resolvePromise, 600)); + expect(acceptedA.connections).toBe(acceptedAConnections); + + const acceptedB = await control.publish('accept'); + await waitFor(harness, () => acceptedB.authCount >= 1); + expect(readRequests(logPath).filter(request => request.method === 'initialize')).toHaveLength(1); + } finally { + await stopChild(harness.child); + await control.close(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('emits signed submitted/progress/completed boundaries without a reusable secret on the wire', async () => { + const result = await exerciseRunner({ version: '0.136.0', turnCount: 2 }); + expect(result.activities.map(activity => activity.phase)).toEqual( + expect.arrayContaining(['submitted', 'progress', 'completed']), + ); + expect(result.activities + .filter(activity => activity.phase === 'submitted' || activity.phase === 'completed') + .map(activity => activity.phase)) + .toEqual(['submitted', 'completed', 'submitted', 'completed']); + expect(result.activities.filter(activity => activity.phase === 'completed')).toMatchObject([ + { turnId: 'turn-fake-1', atMs: expect.any(Number) }, + { turnId: 'turn-fake-2', atMs: expect.any(Number) }, + ]); + expect(result.wireLines.join('\n')).not.toContain(result.privateKeyEncoding); + expect(result.requests.find(request => request.fixtureEnv)?.fixtureEnv).toEqual({ + controlNoncePresent: false, + controlBootstrapPresent: false, + argvContainsControlNonce: false, + }); + expect(result.finals).toHaveLength(2); + expect(result.finals.map(final => final.turnId)).toEqual([ + 'om_integration_1', + 'om_integration_2', + ]); + expect(result.output.match(/› /g)).toHaveLength(2); + + // One idle state belongs to initialized startup and one to the fully + // drained two-turn queue. There must be no transient idle between turns. + const idleMarkerIndexes = result.markers + .map((marker, index) => marker.kind === 'state' && marker.payload.busy === false ? index : -1) + .filter(index => index >= 0); + expect(idleMarkerIndexes).toHaveLength(2); + const lastFinalEndIndex = result.markers.findLastIndex(marker => marker.kind === 'final-end'); + const lastCompletedIndex = result.markers.findLastIndex( + marker => marker.kind === 'activity' && marker.payload.phase === 'completed', + ); + expect(idleMarkerIndexes[1]).toBeGreaterThan(lastCompletedIndex); + expect(idleMarkerIndexes[1]).toBeGreaterThan(lastFinalEndIndex); + }); + + it('emits a zero-chunk final transaction for an empty answer before the signed idle boundary', async () => { + const result = await exerciseRunner({ version: '0.136.0', behavior: 'empty-final' }); + expect(result.finals).toEqual([ + expect.objectContaining({ + turnId: 'om_integration_123', + nativeTurnId: 'turn-fake-1', + content: '', + }), + ]); + const finalStart = result.markers.find(marker => marker.kind === 'final-start'); + expect(finalStart?.payload).toMatchObject({ total: 0, turnId: 'om_integration_123' }); + expect(result.markers.some(marker => marker.kind === 'final-chunk')).toBe(false); + const finalEndIndex = result.markers.findIndex(marker => marker.kind === 'final-end'); + const drainedIdleIndex = result.markers.findLastIndex( + marker => marker.kind === 'state' && marker.payload.busy === false, + ); + expect(finalEndIndex).toBeGreaterThan(-1); + expect(drainedIdleIndex).toBeGreaterThan(finalEndIndex); + }); + + it('re-authenticates the same live runner with a fresh challenge after worker endpoint restart', async () => { + const dir = mkdtempSync(join(tmpdir(), 'botmux-codex-runner-warm-proof-')); + const fakeCodex = join(dir, 'fake-codex'); + const logPath = join(dir, 'requests.jsonl'); + copyFileSync(FAKE_SERVER_FIXTURE, fakeCodex); + chmodSync(fakeCodex, 0o755); + const control = new ControlCollector(dir); + await control.listen(); + const harness = startRunner(fakeCodex, dir, logPath, '0.136.0', 'success', control.bootstrap.path); + try { + await waitFor(harness, () => ( + harness.stdout.includes('Codex App connected.') + && control.authCount === 1 + && control.states.length === 1 + )); + const initializeCount = readRequests(logPath).filter(request => request.method === 'initialize').length; + await control.restartEndpoint(); + await waitFor(harness, () => control.authCount === 2 && control.states.length === 2); + expect(readRequests(logPath).filter(request => request.method === 'initialize')).toHaveLength(initializeCount); + expect(control.states[1]).toMatchObject({ busy: false, atMs: expect.any(Number) }); + harness.child.stdin.write(`${CONTROL_PREFIX}${encodeRunnerInput('warm follow-up')}\r`); + await waitFor(harness, () => control.finals.length === 1); + expect(control.finals[0]).toMatchObject({ content: 'fake answer 1' }); + } finally { + await stopChild(harness.child); + await control.close(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('replays a complete final transaction when the worker is replaced after its first chunk', async () => { + const dir = mkdtempSync(join(tmpdir(), 'botmux-codex-runner-final-replay-')); + const fakeCodex = join(dir, 'fake-codex'); + const logPath = join(dir, 'requests.jsonl'); + copyFileSync(FAKE_SERVER_FIXTURE, fakeCodex); + chmodSync(fakeCodex, 0o755); + const control = new ControlCollector(dir, false, true); + await control.listen(); + const harness = startRunner(fakeCodex, dir, logPath, '0.136.0', 'success', control.bootstrap.path); + try { + await waitFor(harness, () => harness.stdout.includes('Codex App connected.')); + harness.child.stdin.write(`${CONTROL_PREFIX}${encodeRunnerInput('final replay')}\r`); + await waitFor(harness, () => control.authCount >= 2 && control.finals.length === 1); + expect(control.finals).toEqual([ + expect.objectContaining({ content: 'fake answer 1' }), + ]); + expect(readRequests(logPath).filter(request => request.method === 'turn/start')).toHaveLength(1); + } finally { + await stopChild(harness.child); + await control.close(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('does not ACK an incomplete final-end and replays the complete transaction after re-authentication', async () => { + const dir = mkdtempSync(join(tmpdir(), 'botmux-codex-runner-incomplete-final-')); + const fakeCodex = join(dir, 'fake-codex'); + const logPath = join(dir, 'requests.jsonl'); + copyFileSync(FAKE_SERVER_FIXTURE, fakeCodex); + chmodSync(fakeCodex, 0o755); + const control = new ControlCollector(dir, false, false, true); + await control.listen(); + const harness = startRunner(fakeCodex, dir, logPath, '0.136.0', 'success', control.bootstrap.path); + try { + await waitFor(harness, () => harness.stdout.includes('Codex App connected.')); + harness.child.stdin.write(`${CONTROL_PREFIX}${encodeRunnerInput('incomplete final replay')}\r`); + await waitFor(harness, () => control.authCount >= 2 && control.finals.length === 1); + expect(control.finals).toEqual([ + expect.objectContaining({ content: 'fake answer 1' }), + ]); + expect(readRequests(logPath).filter(request => request.method === 'turn/start')).toHaveLength(1); + } finally { + await stopChild(harness.child); + await control.close(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('ACKs a committed final replay after ACK loss without publishing the final twice', async () => { + const dir = mkdtempSync(join(tmpdir(), 'botmux-codex-runner-final-ack-loss-')); + const fakeCodex = join(dir, 'fake-codex'); + const logPath = join(dir, 'requests.jsonl'); + copyFileSync(FAKE_SERVER_FIXTURE, fakeCodex); + chmodSync(fakeCodex, 0o755); + const control = new ControlCollector(dir, false, false, false, true); + await control.listen(); + const harness = startRunner(fakeCodex, dir, logPath, '0.136.0', 'success', control.bootstrap.path); + try { + await waitFor(harness, () => harness.stdout.includes('Codex App connected.')); + harness.child.stdin.write(`${CONTROL_PREFIX}${encodeRunnerInput('final ACK loss')}\r`); + await waitFor(harness, () => control.authCount >= 2 && control.states.length >= 2); + expect(control.finals).toEqual([ + expect.objectContaining({ content: 'fake answer 1' }), + ]); + expect(readRequests(logPath).filter(request => request.method === 'turn/start')).toHaveLength(1); + } finally { + await stopChild(harness.child); + await control.close(); + rmSync(dir, { recursive: true, force: true }); + } + }); + it('sends clean text, hidden context, localImage, and clientUserMessageId on codex >= 0.136', async () => { const result = await exerciseRunner({ version: '0.136.0', includeMissingImage: true }); const initialize = result.requests.find(request => request.method === 'initialize'); expect(initialize?.params.capabilities).toEqual({ experimentalApi: true }); - const turns = result.requests.filter(request => request.method === 'turn/start'); expect(turns).toHaveLength(1); expect(turns[0].params.input).toEqual([ @@ -212,7 +916,339 @@ describe('codex-app-runner app-server protocol integration', () => { expect(result.final.nativeTurnId).toBe('turn-fake-1'); }); - it('preserves the full legacy prompt on codex < 0.135 even if the server would ignore new fields', async () => { + it('buffers start notifications until a response-last RPC proves the authoritative native id', async () => { + const result = await exerciseRunner({ version: '0.144.1', behavior: 'start-response-last' }); + expect(result.final).toMatchObject({ + turnId: 'om_integration_123', + nativeTurnId: 'turn-fake-1', + content: 'fake answer 1', + }); + expect(result.final.content).not.toContain('unrelated autonomous output'); + expect(result.requests.filter(request => request.method === 'thread/turns/list')).toHaveLength(0); + expect(result.markers.some(marker => marker.kind === 'diagnostic')).toBe(false); + expect(result.activities.map(activity => activity.phase)).toEqual([ + 'submitted', + 'progress', + 'completed', + ]); + }); + + it('does not let response-last A overwrite a newer Goal B native lifecycle', async () => { + const dir = mkdtempSync(join(tmpdir(), 'botmux-codex-runner-response-last-goal-')); + const fakeCodex = join(dir, 'fake-codex'); + const logPath = join(dir, 'requests.jsonl'); + copyFileSync(FAKE_SERVER_FIXTURE, fakeCodex); + chmodSync(fakeCodex, 0o755); + const control = new ControlCollector(dir); + await control.listen(); + const harness = startRunner( + fakeCodex, + dir, + logPath, + '0.144.1', + 'start-response-last-goal', + control.bootstrap.path, + ); + try { + await waitFor(harness, () => control.states.some(state => state.busy === false)); + harness.child.stdin.write(`${CONTROL_PREFIX}${encodeRunnerInput('first legacy', { + text: 'first exact', clientUserMessageId: 'om_response_last_a', + })}\r`); + await waitFor(harness, () => control.finals.length === 1 + && control.states.some(state => state.busy === true && state.tracksTurn === false)); + + harness.child.stdin.write(`${CONTROL_PREFIX}${encodeRunnerInput('confirm legacy', { + text: 'confirm exact', clientUserMessageId: 'om_response_last_confirm', + })}\r`); + await waitFor(harness, () => control.finals.length === 2 + && control.states.filter(state => state.busy === false).length >= 2); + + const requests = readRequests(logPath); + expect(requests.filter(request => request.method === 'turn/start')).toHaveLength(1); + expect(requests.filter(request => request.method === 'turn/steer')).toEqual([ + expect.objectContaining({ + params: expect.objectContaining({ + expectedTurnId: 'turn-goal-auto', + clientUserMessageId: 'om_response_last_confirm', + }), + }), + ]); + expect(control.finals).toEqual([ + expect.objectContaining({ + turnId: 'om_response_last_a', + nativeTurnId: 'turn-fake-1', + content: 'fake answer 1', + }), + expect.objectContaining({ + turnId: 'om_response_last_confirm', + nativeTurnId: 'turn-goal-auto', + content: 'goal steer answer', + }), + ]); + } finally { + await stopChild(harness.child); + await control.close(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('keeps a Goal auto-continuation native-busy and steers the next exact Lark turn into it', async () => { + const dir = mkdtempSync(join(tmpdir(), 'botmux-codex-runner-goal-steer-')); + const fakeCodex = join(dir, 'fake-codex'); + const logPath = join(dir, 'requests.jsonl'); + copyFileSync(FAKE_SERVER_FIXTURE, fakeCodex); + chmodSync(fakeCodex, 0o755); + const control = new ControlCollector(dir); + await control.listen(); + const harness = startRunner(fakeCodex, dir, logPath, '0.144.1', 'goal-continuation', control.bootstrap.path); + const sidecar = (text: string, id: string): CodexAppTurnInput => ({ + text, + clientUserMessageId: id, + }); + try { + await waitFor(harness, () => control.states.some(state => state.busy === false)); + harness.child.stdin.write(`${CONTROL_PREFIX}${encodeRunnerInput('first legacy', sidecar('first', 'om_goal_a'))}\r`); + await waitFor(harness, () => ( + control.finals.length === 1 + && control.states.some(state => state.busy === true && state.tracksTurn === false) + )); + harness.child.stdin.write(`${CONTROL_PREFIX}${encodeRunnerInput('confirm legacy', sidecar('confirm', 'om_goal_confirm'))}\r`); + await waitFor(harness, () => control.finals.length === 2 + && control.states.filter(state => state.busy === false).length >= 2); + + const requests = readRequests(logPath); + expect( + requests.filter(request => request.method === 'turn/start'), + JSON.stringify(requests.filter(request => request.method?.startsWith('turn/')), null, 2), + ).toHaveLength(1); + const steer = requests.filter(request => request.method === 'turn/steer'); + expect(steer).toHaveLength(1); + expect(steer[0].params).toMatchObject({ + threadId: 'thread-fake', + expectedTurnId: 'turn-goal-auto', + clientUserMessageId: 'om_goal_confirm', + input: [{ type: 'text', text: 'confirm', text_elements: [] }], + }); + expect(steer[0].params).not.toHaveProperty('cwd'); + expect(control.finals).toEqual([ + expect.objectContaining({ turnId: 'om_goal_a', content: 'fake answer 1' }), + expect.objectContaining({ + turnId: 'om_goal_confirm', + nativeTurnId: 'turn-goal-auto', + content: 'goal steer answer', + }), + ]); + const firstEnd = control.markers.findIndex(marker => + marker.kind === 'final-end' && marker.payload.id?.startsWith('om_goal_a:')); + const secondStart = control.markers.findIndex(marker => + marker.kind === 'final-start' && marker.payload.turnId === 'om_goal_confirm'); + expect(control.markers.slice(firstEnd + 1, secondStart)).not.toContainEqual( + expect.objectContaining({ kind: 'state', payload: expect.objectContaining({ busy: false }) }), + ); + } finally { + await stopChild(harness.child); + await control.close(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('falls back to turn/start only after an explicit stale expected-turn rejection', async () => { + const dir = mkdtempSync(join(tmpdir(), 'botmux-codex-runner-steer-race-')); + const fakeCodex = join(dir, 'fake-codex'); + const logPath = join(dir, 'requests.jsonl'); + copyFileSync(FAKE_SERVER_FIXTURE, fakeCodex); + chmodSync(fakeCodex, 0o755); + const control = new ControlCollector(dir); + await control.listen(); + const harness = startRunner(fakeCodex, dir, logPath, '0.144.1', 'goal-steer-race', control.bootstrap.path); + try { + await waitFor(harness, () => control.states.some(state => state.busy === false)); + harness.child.stdin.write(`${CONTROL_PREFIX}${encodeRunnerInput('first', { + text: 'first', clientUserMessageId: 'om_race_a', + })}\r`); + await waitFor(harness, () => control.finals.length === 1 + && control.states.some(state => state.busy === true && state.tracksTurn === false)); + harness.child.stdin.write(`${CONTROL_PREFIX}${encodeRunnerInput('confirm legacy', { + text: 'confirm exact', clientUserMessageId: 'om_race_confirm', + })}\r`); + await waitFor(harness, () => control.finals.length === 2 + && control.states.filter(state => state.busy === false).length >= 2); + + const requests = readRequests(logPath); + const starts = requests.filter(request => request.method === 'turn/start'); + const steers = requests.filter(request => request.method === 'turn/steer'); + expect(steers).toHaveLength(1); + expect(steers[0].params).toMatchObject({ + expectedTurnId: 'turn-goal-auto', + clientUserMessageId: 'om_race_confirm', + }); + expect(starts).toHaveLength(2); + expect(starts[1].params).toMatchObject({ + clientUserMessageId: 'om_race_confirm', + input: [{ type: 'text', text: 'confirm exact', text_elements: [] }], + }); + expect(control.finals[1]).toMatchObject({ + turnId: 'om_race_confirm', + nativeTurnId: 'turn-fake-2', + content: 'fake answer 2', + }); + expect(control.finals[1].content).not.toContain('autonomous goal text before Lark input'); + } finally { + await stopChild(harness.child); + await control.close(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('reconciles a mismatched completion only through one exact full-history client id match', async () => { + const dir = mkdtempSync(join(tmpdir(), 'botmux-codex-runner-history-reconcile-')); + const fakeCodex = join(dir, 'fake-codex'); + const logPath = join(dir, 'requests.jsonl'); + copyFileSync(FAKE_SERVER_FIXTURE, fakeCodex); + chmodSync(fakeCodex, 0o755); + const control = new ControlCollector(dir); + await control.listen(); + const harness = startRunner(fakeCodex, dir, logPath, '0.144.1', 'history-reconcile', control.bootstrap.path); + try { + await waitFor(harness, () => harness.stdout.includes('Codex App connected.')); + harness.child.stdin.write(`${CONTROL_PREFIX}${encodeRunnerInput('legacy', { + text: 'exact input', + clientUserMessageId: 'om_exact_reconcile', + })}\r`); + await waitFor(harness, () => control.finals.length === 1 + && control.states.filter(state => state.busy === false).length >= 2); + expect(readRequests(logPath).filter(request => request.method === 'thread/turns/list')) + .toEqual([expect.objectContaining({ + params: expect.objectContaining({ + threadId: 'thread-fake', + limit: 50, + sortDirection: 'desc', + itemsView: 'full', + }), + })]); + expect(control.finals[0]).toMatchObject({ + turnId: 'om_exact_reconcile', + nativeTurnId: 'turn-fake-1', + content: 'reconciled answer 1', + }); + expect(control.finals[0].content).not.toContain('autonomous text before exact input'); + expect(control.markers.some(marker => marker.kind === 'diagnostic')).toBe(false); + } finally { + await stopChild(harness.child); + await control.close(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + for (const [behavior, expectedReason] of [ + ['history-no-match', 'found no match'], + ['history-multi-match', 'found multiple matches'], + ] as const) { + it(`fails closed with an explicit settled error when bounded history ${expectedReason}`, async () => { + const dir = mkdtempSync(join(tmpdir(), `botmux-codex-runner-${behavior}-`)); + const fakeCodex = join(dir, 'fake-codex'); + const logPath = join(dir, 'requests.jsonl'); + copyFileSync(FAKE_SERVER_FIXTURE, fakeCodex); + chmodSync(fakeCodex, 0o755); + const control = new ControlCollector(dir); + await control.listen(); + const harness = startRunner(fakeCodex, dir, logPath, '0.144.1', behavior, control.bootstrap.path); + try { + await waitFor(harness, () => harness.stdout.includes('Codex App connected.')); + harness.child.stdin.write(`${CONTROL_PREFIX}${encodeRunnerInput('legacy', { + text: 'must match exactly', + clientUserMessageId: 'om_conflict', + })}\r`); + await waitFor(harness, () => control.markers.some(marker => marker.kind === 'diagnostic') + && control.finals.length === 1 + && control.states.filter(state => state.busy === false).length >= 2); + const diagnostic = control.markers.find(marker => marker.kind === 'diagnostic'); + expect(diagnostic?.payload).toMatchObject({ + code: 'native_turn_identity_conflict', + turnId: 'om_conflict', + message: expect.stringContaining(expectedReason), + }); + expect(control.finals[0]).toMatchObject({ + turnId: 'om_conflict', + content: expect.stringContaining('Codex App native turn identity conflict'), + }); + expect(control.states.filter(state => state.busy === false).length).toBeGreaterThanOrEqual(2); + expect(harness.child.exitCode).toBeNull(); + } finally { + await stopChild(harness.child); + await control.close(); + rmSync(dir, { recursive: true, force: true }); + } + }); + } + + it('keeps the startup deadline armed through initialize and never exposes a pre-ready prompt', async () => { + const dir = mkdtempSync(join(tmpdir(), 'botmux-codex-runner-startup-deadline-')); + const fakeCodex = join(dir, 'fake-codex'); + const logPath = join(dir, 'requests.jsonl'); + copyFileSync(FAKE_SERVER_FIXTURE, fakeCodex); + chmodSync(fakeCodex, 0o755); + const control = new ControlCollector(dir); + await control.listen(); + const harness = startRunner( + fakeCodex, + dir, + logPath, + '0.144.1', + 'hang-initialize', + control.bootstrap.path, + { env: { BOTMUX_TEST_CODEX_APP_STARTUP_TIMEOUT_MS: '1000' } }, + ); + try { + const exitCode = await new Promise(resolvePromise => harness.child.once('exit', resolvePromise)); + expect(exitCode).toBe(2); + expect(readRequests(logPath).filter(request => request.method === 'initialize')).toHaveLength(1); + expect(harness.stdout).not.toContain('Codex App connected.'); + expect(harness.stdout).not.toContain('› '); + expect(control.states).toEqual([]); + expect(harness.stderr).toContain('startup timed out before the first signed runner state'); + } finally { + await stopChild(harness.child); + await control.close(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('does not turn an ambiguous resume timeout into a fresh thread', async () => { + const dir = mkdtempSync(join(tmpdir(), 'botmux-codex-runner-resume-timeout-')); + const fakeCodex = join(dir, 'fake-codex'); + const logPath = join(dir, 'requests.jsonl'); + copyFileSync(FAKE_SERVER_FIXTURE, fakeCodex); + chmodSync(fakeCodex, 0o755); + const control = new ControlCollector(dir); + await control.listen(); + const harness = startRunner( + fakeCodex, + dir, + logPath, + '0.144.1', + 'hang-resume', + control.bootstrap.path, + { + threadId: 'thread-existing', + env: { BOTMUX_TEST_CODEX_APP_STARTUP_TIMEOUT_MS: '1000' }, + }, + ); + try { + await new Promise(resolvePromise => harness.child.once('exit', () => resolvePromise())); + const requests = readRequests(logPath); + expect(requests.filter(request => request.method === 'thread/resume')).toHaveLength(1); + expect(requests.filter(request => request.method === 'thread/start')).toHaveLength(0); + expect(harness.stdout).not.toContain('Codex App connected.'); + expect(control.states).toEqual([]); + } finally { + await stopChild(harness.child); + await control.close(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('preserves the full legacy prompt on codex < 0.135 even if the server ignores new fields', async () => { const result = await exerciseRunner({ version: '0.134.9' }); const turns = result.requests.filter(request => request.method === 'turn/start'); expect(turns).toHaveLength(1); @@ -269,11 +1305,11 @@ describe('codex-app-runner app-server protocol integration', () => { expect(result.final.nativeTurnId).toBe('turn-fake-1'); }); - it('escapes split agent/command OSC injections and emits only the trusted final marker', async () => { + it('escapes split agent/command OSC injections and emits the trusted final only out of band', async () => { const result = await exerciseRunner({ version: '0.136.0', behavior: 'osc-injection' }); expect(result.output).toContain('␛]777;botmux:final:'); - expect(result.output.match(/\x1b\]777;botmux:final:/g)).toHaveLength(1); + expect(result.output.match(/\x1b\]777;botmux:final:/g)).toBeNull(); expect(result.final).toMatchObject({ turnId: 'om_integration_123', nativeTurnId: 'turn-fake-1', diff --git a/test/codex-app-turn-dispatch.test.ts b/test/codex-app-turn-dispatch.test.ts new file mode 100644 index 000000000..846d37e1f --- /dev/null +++ b/test/codex-app-turn-dispatch.test.ts @@ -0,0 +1,116 @@ +import { describe, expect, it } from 'vitest'; +import { CodexAppTurnDispatchQueue } from '../src/utils/codex-app-turn-dispatch.js'; + +describe('CodexAppTurnDispatchQueue', () => { + it('attributes queued finals to immutable FIFO heads after later writes change worker globals', () => { + const queue = new CodexAppTurnDispatchQueue(); + queue.reserve('turn-1', 4); + queue.reserve('turn-2', 9); + + expect(queue.settleFinal({ + turnId: 'turn-1', + nativeTurnId: 'native-1', + })).toMatchObject({ + ok: true, + turnId: 'turn-1', + dispatchAttempt: 4, + nativeTurnId: 'native-1', + remaining: 1, + }); + expect(queue.settleFinal({ turnId: 'turn-2' })).toMatchObject({ + ok: true, + turnId: 'turn-2', + dispatchAttempt: 9, + remaining: 0, + }); + }); + + it('uses a complete empty final as the exact FIFO boundary', () => { + const queue = new CodexAppTurnDispatchQueue(); + queue.reserve('empty-turn'); + queue.reserve('next-turn'); + + expect(queue.settleFinal({ turnId: 'empty-turn' })).toMatchObject({ + ok: true, + turnId: 'empty-turn', + remaining: 1, + }); + expect(queue.settleFinal({ turnId: 'next-turn' })).toMatchObject({ + ok: true, + turnId: 'next-turn', + remaining: 0, + }); + }); + + it('rejects mismatched turn and attempt assertions without advancing the head', () => { + const queue = new CodexAppTurnDispatchQueue(); + queue.reserve('turn-1', 7); + queue.reserve('turn-2', 8); + + expect(queue.settleFinal({ turnId: 'turn-2' })).toEqual({ + ok: false, + reason: 'turn_mismatch', + markerTurnId: 'turn-2', + expectedTurnId: 'turn-1', + }); + expect(queue.settleFinal({ turnId: 'turn-1', dispatchAttempt: 8 })).toEqual({ + ok: false, + reason: 'dispatch_attempt_mismatch', + markerDispatchAttempt: 8, + expectedDispatchAttempt: 7, + }); + expect(queue.size()).toBe(2); + expect(queue.settleFinal({ turnId: 'turn-1', dispatchAttempt: 7 })).toMatchObject({ + ok: true, + turnId: 'turn-1', + dispatchAttempt: 7, + remaining: 1, + }); + }); + + it('cancels only the exact failed write and preserves peers in FIFO order', () => { + const queue = new CodexAppTurnDispatchQueue(); + const first = queue.reserve('turn-1', 1); + const second = queue.reserve('turn-2', 2); + const third = queue.reserve('turn-3', 3); + + expect(queue.cancelExact(second.handle)).toBe(true); + expect(queue.cancelExact(second.handle)).toBe(false); + expect(queue.settleFinal({ turnId: 'turn-1' })).toMatchObject({ + ok: true, + turnId: first.turnId, + remaining: 1, + }); + expect(queue.settleFinal({ turnId: 'turn-3' })).toMatchObject({ + ok: true, + turnId: third.turnId, + remaining: 0, + }); + // A late adapter rejection after an already-applied final must not emit a + // second ambiguous/failed terminal for the settled turn. + expect(queue.cancelExact(first.handle)).toBe(false); + }); + + it('recovers at most one daemon-frozen warm-reattach identity and clears on reset', () => { + const queue = new CodexAppTurnDispatchQueue(); + expect(queue.recoverWarmReattach(undefined, 3)).toBeUndefined(); + expect(queue.recoverWarmReattach('reattached', 3)).toMatchObject({ + turnId: 'reattached', + dispatchAttempt: 3, + }); + expect(queue.recoverWarmReattach('must-not-overwrite', 4)).toBeUndefined(); + expect(queue.settleFinal({ turnId: 'reattached' })).toMatchObject({ + ok: true, + turnId: 'reattached', + dispatchAttempt: 3, + }); + + queue.reserve('stale'); + queue.clear(); + expect(queue.size()).toBe(0); + expect(queue.settleFinal({ turnId: 'stale' })).toEqual({ + ok: false, + reason: 'no_pending_turn', + }); + }); +}); diff --git a/test/codex-app-turn-liveness.test.ts b/test/codex-app-turn-liveness.test.ts new file mode 100644 index 000000000..29e4cc3a6 --- /dev/null +++ b/test/codex-app-turn-liveness.test.ts @@ -0,0 +1,513 @@ +import { describe, expect, it, vi } from 'vitest'; +import { + applyTrustedCodexAppActivityMarker, + applyTrustedCodexAppStateMarker, + CodexAppFlushPromptReplay, + CodexAppReadyAuthority, + CodexAppTurnLiveness, + shouldBeginCodexAppReattachObservation, +} from '../src/utils/codex-app-turn-liveness.js'; +import { + CodexAppControlRecordApplicationGate, + projectCodexAppControlReadinessStatus, +} from '../src/utils/codex-app-control.js'; + +describe('CodexAppReadyAuthority', () => { + it('does not allocate a Botmux liveness slot for a native Goal continuation', () => { + const authority = new CodexAppReadyAuthority(); + const tracker = new CodexAppTurnLiveness(1_000); + const state = applyTrustedCodexAppStateMarker(tracker, authority, { + busy: true, + tracksTurn: false, + acceptingInput: true, + atMs: 10_000, + }, 10_000); + expect(state).toMatchObject({ accepted: true, busy: true, tracksTurn: false }); + expect(tracker.hasActiveTurn()).toBe(false); + expect(authority.canPublishPromptReady()).toBe(false); + + tracker.beginReattachObservation(10_100); + expect(tracker.hasActiveTurn()).toBe(true); + applyTrustedCodexAppStateMarker(tracker, authority, { + busy: true, + tracksTurn: false, + acceptingInput: true, + atMs: 10_110, + }, 10_110); + expect(tracker.hasActiveTurn()).toBe(false); + }); + + it('suppresses a terminal prompt and publishes ready only after signed idle, preserving final-before-ready', () => { + const authority = new CodexAppReadyAuthority(); + const tracker = new CodexAppTurnLiveness(1_000); + const events: string[] = []; + + tracker.begin('turn-ready', 10_000); + authority.beginWork(); + applyTrustedCodexAppActivityMarker(tracker, { + phase: 'completed', + atMs: 10_100, + }, 10_100); + // Model a prompt byte that wins the terminal/backend race and arrives + // before the signed final transaction. It is not an idle authority. + if (tracker.notePrompt(10_101) && authority.canPublishPromptReady()) events.push('ready-from-terminal'); + else events.push('terminal-prompt-suppressed'); + + events.push('final-output'); + const state = applyTrustedCodexAppStateMarker(tracker, authority, { + busy: false, + atMs: 10_102, + }, 10_102); + expect(state).toMatchObject({ accepted: true, busy: false, shouldPublishReady: true }); + if (state.shouldPublishReady && authority.canPublishPromptReady()) events.push('prompt-ready'); + + expect(events).toEqual([ + 'terminal-prompt-suppressed', + 'final-output', + 'prompt-ready', + ]); + }); + + it('does not retain an old idle grant across a new queued turn', () => { + const authority = new CodexAppReadyAuthority(); + authority.noteSignedState(false); + expect(authority.canPublishPromptReady()).toBe(true); + + authority.beginWork(); + expect(authority.canPublishPromptReady()).toBe(false); + authority.noteSignedState(true); + expect(authority.canPublishPromptReady()).toBe(false); + }); + + it('grants exactly one late terminal prompt after an exact submit cancellation', () => { + const authority = new CodexAppReadyAuthority(); + const tracker = new CodexAppTurnLiveness(1_000); + const replay = new CodexAppFlushPromptReplay(); + const handle = tracker.begin('cancelled-turn', 10_000); + authority.beginWork(); + + // No prompt was observed during writeInput; it arrives only after the + // exact failed slot has been removed and the flush has returned. + replay.cancelSubmission(tracker, authority, handle, 10_100); + expect(replay.consumeAfterFlush(tracker)).toBe(false); + expect(tracker.notePrompt(10_110)).toBe(true); + expect(authority.canPublishPromptReady()).toBe(false); + expect(authority.consumeLatePromptRecovery(!tracker.hasActiveTurn())).toBe(true); + expect(authority.canPublishPromptReady()).toBe(false); + expect(authority.consumeLatePromptRecovery(true)).toBe(false); + }); + + it('does not arm recovery for an absent or already-cancelled submit handle', () => { + const authority = new CodexAppReadyAuthority(); + const tracker = new CodexAppTurnLiveness(1_000); + const replay = new CodexAppFlushPromptReplay(); + + replay.cancelSubmission(tracker, authority, undefined); + replay.cancelSubmission(tracker, authority, 999); + expect(authority.consumeLatePromptRecovery(true)).toBe(false); + + const exact = tracker.begin('exact-turn'); + replay.cancelSubmission(tracker, authority, exact); + expect(authority.consumeLatePromptRecovery(true)).toBe(true); + replay.cancelSubmission(tracker, authority, exact); + expect(authority.consumeLatePromptRecovery(true)).toBe(false); + }); + + it('does not let a cancelled turn prompt release newer work', () => { + const authority = new CodexAppReadyAuthority(); + const tracker = new CodexAppTurnLiveness(1_000); + const replay = new CodexAppFlushPromptReplay(); + const cancelled = tracker.begin('cancelled-turn', 10_000); + authority.beginWork(); + replay.cancelSubmission(tracker, authority, cancelled, 10_100); + + tracker.begin('new-turn', 10_110); + authority.beginWork(); + expect(tracker.notePrompt(10_120)).toBe(false); + expect(authority.consumeLatePromptRecovery(!tracker.hasActiveTurn())).toBe(false); + expect(authority.canPublishPromptReady()).toBe(false); + }); + + it('clears late-prompt recovery on trusted work, signed state, and reset', () => { + const authority = new CodexAppReadyAuthority(); + const tracker = new CodexAppTurnLiveness(1_000); + const replay = new CodexAppFlushPromptReplay(); + const arm = () => { + const handle = tracker.begin('cancelled-turn'); + authority.beginWork(); + replay.cancelSubmission(tracker, authority, handle); + }; + + arm(); + authority.beginWork(); // trusted submitted/progress handler + expect(authority.consumeLatePromptRecovery(true)).toBe(false); + + arm(); + authority.noteSignedState(false); + expect(authority.consumeLatePromptRecovery(true)).toBe(false); + expect(authority.canPublishPromptReady()).toBe(true); + + arm(); + authority.reset(); + expect(authority.consumeLatePromptRecovery(true)).toBe(false); + expect(authority.canPublishPromptReady()).toBe(false); + }); +}); + +describe('shouldBeginCodexAppReattachObservation', () => { + it('covers a Zellij reattach even though Zellij is not pipe mode', () => { + for (const backendType of ['tmux', 'herdr', 'zellij'] as const) { + expect(shouldBeginCodexAppReattachObservation({ + cliId: 'codex-app', + backendType, + isReattach: true, + })).toBe(true); + } + }); + + it('does not observe a fresh spawn, non-persistent backend, or another CLI', () => { + expect(shouldBeginCodexAppReattachObservation({ + cliId: 'codex-app', + backendType: 'zellij', + isReattach: false, + })).toBe(false); + expect(shouldBeginCodexAppReattachObservation({ + cliId: 'codex-app', + backendType: 'pty', + isReattach: true, + })).toBe(false); + expect(shouldBeginCodexAppReattachObservation({ + cliId: 'codex', + backendType: 'tmux', + isReattach: true, + })).toBe(false); + }); +}); + +describe('CodexAppTurnLiveness', () => { + it('coalesces a replayed final while persistence is pending and keeps type-ahead working', async () => { + const tracker = new CodexAppTurnLiveness(1_000); + const gate = new CodexAppControlRecordApplicationGate(); + tracker.begin('turn-n', 10_000); + tracker.begin('turn-n-plus-1', 10_010); + + let completionAwaitingFinal = false; + const completed = applyTrustedCodexAppActivityMarker(tracker, { + phase: 'completed', + atMs: 10_100, + }, 10_100); + expect(completed).toMatchObject({ accepted: true, phase: 'completed' }); + completionAwaitingFinal = true; + + let releasePersistence!: () => void; + const persistence = new Promise(resolve => { + releasePersistence = resolve; + }); + let applicationCount = 0; + const applyFinal = async (): Promise => { + applicationCount++; + if (!completionAwaitingFinal) tracker.completeCurrent(10_110); + completionAwaitingFinal = false; + await persistence; + return true; + }; + + const first = gate.run('generation-a', 42, applyFinal); + await Promise.resolve(); + const replayEffect = vi.fn(async () => true); + const replay = gate.run('generation-a', 42, replayEffect); + + expect(applicationCount).toBe(1); + expect(replayEffect).not.toHaveBeenCalled(); + expect(tracker.poll(10_110)).toMatchObject({ + active: true, + turnId: 'turn-n-plus-1', + }); + expect(projectCodexAppControlReadinessStatus('idle', { + controlProven: true, + signedStateObserved: false, + inputReady: false, + })).toBe('working'); + + releasePersistence(); + await expect(first).resolves.toBe(true); + await expect(replay).resolves.toBe(true); + gate.release('generation-a', 42, first); + + expect(applicationCount).toBe(1); + expect(tracker.poll(10_120)).toMatchObject({ + active: true, + turnId: 'turn-n-plus-1', + }); + }); + + it('projects idle only after authenticated signed input readiness', () => { + expect(projectCodexAppControlReadinessStatus('idle', { + controlProven: false, + signedStateObserved: false, + inputReady: false, + })).toBe('working'); + expect(projectCodexAppControlReadinessStatus('idle', { + controlProven: true, + signedStateObserved: false, + inputReady: false, + })).toBe('working'); + expect(projectCodexAppControlReadinessStatus('idle', { + controlProven: true, + signedStateObserved: true, + inputReady: false, + })).toBe('working'); + expect(projectCodexAppControlReadinessStatus('idle', { + controlProven: true, + signedStateObserved: true, + inputReady: true, + })).toBe('idle'); + expect(projectCodexAppControlReadinessStatus('stalled', { + controlProven: false, + signedStateObserved: false, + inputReady: false, + })).toBe('stalled'); + }); + + it('stays working before the no-progress threshold, then stalls and notifies once', () => { + const tracker = new CodexAppTurnLiveness(1_000); + tracker.begin('turn-1', 10_000); + + expect(tracker.poll(10_999)).toEqual({ + active: true, + stalled: false, + newlyStalled: false, + shouldNotify: false, + turnId: 'turn-1', + }); + expect(tracker.poll(11_000)).toEqual({ + active: true, + stalled: true, + newlyStalled: true, + shouldNotify: true, + turnId: 'turn-1', + }); + expect(tracker.poll(12_000)).toEqual({ + active: true, + stalled: true, + newlyStalled: false, + shouldNotify: false, + turnId: 'turn-1', + }); + }); + + it('uses runner activity as the clock and clears a stalled projection on recovery', () => { + const tracker = new CodexAppTurnLiveness(1_000); + tracker.begin('turn-1', 10_000); + tracker.noteActivity(10_900); + expect(tracker.poll(11_500).stalled).toBe(false); + + expect(tracker.poll(11_900)).toMatchObject({ stalled: true, shouldNotify: true }); + tracker.noteActivity(12_000); + expect(tracker.poll(12_000)).toMatchObject({ stalled: false, newlyStalled: false }); + + // A recovered turn may become quiet again, but it must not spam the same + // warning a second time. + expect(tracker.poll(13_000)).toMatchObject({ + stalled: true, + newlyStalled: true, + shouldNotify: false, + }); + }); + + it('does not move the activity clock backwards for a delayed OSC marker', () => { + const tracker = new CodexAppTurnLiveness(1_000); + tracker.begin('turn-1', 10_000); + tracker.noteActivity(11_000); + tracker.noteActivity(10_500); + expect(tracker.poll(11_999).stalled).toBe(false); + expect(tracker.poll(12_000).stalled).toBe(true); + }); + + it('keeps flushPending inputs FIFO and starts a queued turn clock only after its predecessor completes', () => { + const tracker = new CodexAppTurnLiveness(1_000); + // flushPending() may write both Botmux inputs before the serial runner + // finishes the first one. Completion must advance one queue slot, not + // clear or overwrite the later turn. + tracker.begin('turn-1', 10_000); + tracker.begin('turn-2', 10_010); + tracker.noteActivity(10_900); + tracker.completeCurrent(11_000); + + expect(tracker.poll(11_999)).toEqual({ + active: true, + stalled: false, + newlyStalled: false, + shouldNotify: false, + turnId: 'turn-2', + }); + expect(tracker.poll(12_000)).toEqual({ + active: true, + stalled: true, + newlyStalled: true, + shouldNotify: true, + turnId: 'turn-2', + }); + }); + + it('cancels only the exact failed submission and preserves queued peers', () => { + const tracker = new CodexAppTurnLiveness(1_000); + const first = tracker.begin('turn-1', 10_000); + const second = tracker.begin('turn-2', 10_010); + + tracker.cancel(second, 10_100); + expect(tracker.poll(11_000)).toMatchObject({ + stalled: true, + shouldNotify: true, + turnId: 'turn-1', + }); + + tracker.cancel(first, 11_100); + expect(tracker.poll(20_000)).toEqual({ + active: false, + stalled: false, + newlyStalled: false, + shouldNotify: false, + }); + }); + + it('replays a deferred inter-turn prompt when the queued control-line write is cancelled', () => { + const tracker = new CodexAppTurnLiveness(1_000); + tracker.begin('turn-1', 10_000); + const second = tracker.begin('turn-2', 10_010); + + expect(tracker.completeCurrent(10_100)).toBe(false); + // The runner briefly became empty while turn-2 was still arriving in + // chunks, so its real prompt is held behind turn-2's liveness slot. + expect(tracker.notePrompt(10_110)).toBe(false); + expect(tracker.cancel(second, 10_120)).toBe(true); + expect(tracker.hasActiveTurn()).toBe(false); + }); + + it('replays a cancelled flush prompt exactly once after every peer liveness slot drains', () => { + const tracker = new CodexAppTurnLiveness(1_000); + const authority = new CodexAppReadyAuthority(); + const replay = new CodexAppFlushPromptReplay(); + tracker.begin('turn-1', 10_000); + const second = tracker.begin('turn-2', 10_010); + const third = tracker.begin('turn-3', 10_020); + + tracker.completeCurrent(10_100); + expect(tracker.notePrompt(10_110)).toBe(false); + replay.cancelSubmission(tracker, authority, second, 10_120); + expect(replay.consumeAfterFlush(tracker)).toBe(false); + + // A later flush can observe the same real prompt only after the remaining + // peer is cancelled; the one-shot gate cannot replay it twice. + expect(tracker.notePrompt(10_130)).toBe(false); + replay.cancelSubmission(tracker, authority, third, 10_140); + expect(replay.consumeAfterFlush(tracker)).toBe(true); + expect(replay.consumeAfterFlush(tracker)).toBe(false); + }); + + it('does not replay cancellation state after an authenticated next submit supersedes the prompt', () => { + const tracker = new CodexAppTurnLiveness(1_000); + const authority = new CodexAppReadyAuthority(); + const replay = new CodexAppFlushPromptReplay(); + tracker.begin('turn-1', 10_000); + const second = tracker.begin('turn-2', 10_010); + tracker.completeCurrent(10_100); + expect(tracker.notePrompt(10_110)).toBe(false); + tracker.noteSubmitted(10_120); + authority.beginWork(); + replay.cancelSubmission(tracker, authority, second, 10_130); + expect(replay.consumeAfterFlush(tracker)).toBe(false); + }); + + it('discards a deferred inter-turn prompt once the next turn really starts', () => { + const tracker = new CodexAppTurnLiveness(1_000); + tracker.begin('turn-1', 10_000); + tracker.begin('turn-2', 10_010); + tracker.completeCurrent(10_100); + expect(tracker.notePrompt(10_110)).toBe(false); + + tracker.noteSubmitted(10_120); + expect(tracker.completeCurrent(10_200)).toBe(false); + }); + + it('applies only valid activity from the authenticated socket and bounds future timestamps', () => { + const tracker = new CodexAppTurnLiveness(1_000); + tracker.begin('turn-1', 10_000); + tracker.begin('turn-2', 10_010); + + expect(applyTrustedCodexAppActivityMarker(tracker, { + phase: 'forged', + atMs: 999_999, + }, 10_100)).toEqual({ accepted: false }); + expect(tracker.poll(10_100)).toMatchObject({ turnId: 'turn-1', stalled: false }); + expect(tracker.poll(11_000)).toMatchObject({ turnId: 'turn-1', stalled: true }); + + expect(applyTrustedCodexAppActivityMarker(tracker, { + phase: 'completed', + atMs: 11_100, + }, 11_100)).toMatchObject({ accepted: true, phase: 'completed' }); + expect(tracker.poll(11_100)).toMatchObject({ turnId: 'turn-2', stalled: false }); + + // Even an authenticated runner timestamp is bounded by receipt time. + expect(applyTrustedCodexAppActivityMarker(tracker, { + phase: 'progress', + atMs: 999_999, + }, 11_200)).toMatchObject({ accepted: true, phase: 'progress' }); + expect(tracker.poll(12_200)).toMatchObject({ turnId: 'turn-2', stalled: true }); + }); + + it('uses a prompt only to close a synthetic authenticated reattach observation', () => { + const tracker = new CodexAppTurnLiveness(1_000); + tracker.beginReattachObservation(10_000); + expect(tracker.notePrompt(10_100)).toBe(true); + expect(tracker.poll(20_000).stalled).toBe(false); + + tracker.begin('turn-1', 20_000); + tracker.begin('turn-2', 20_010); + expect(tracker.notePrompt(20_100)).toBe(false); + expect(tracker.poll(21_000)).toMatchObject({ + stalled: true, + turnId: 'turn-1', + }); + }); + + it('does not install duplicate reattach observations over a tracked turn', () => { + const tracker = new CodexAppTurnLiveness(1_000); + tracker.begin('turn-1', 10_000); + expect(tracker.beginReattachObservation(10_100)).toBeUndefined(); + expect(tracker.poll(11_000)).toMatchObject({ + stalled: true, + turnId: 'turn-1', + }); + }); + + it('recovers an explicit active slot from a submitted marker after worker reattach', () => { + const tracker = new CodexAppTurnLiveness(1_000); + tracker.noteSubmitted(10_000); + + expect(tracker.notePrompt(10_100)).toBe(false); + expect(tracker.poll(11_000)).toMatchObject({ + active: true, + stalled: true, + shouldNotify: true, + }); + }); + + it('clears all state on CLI exit, kill, or worker reinitialization', () => { + const tracker = new CodexAppTurnLiveness(1_000); + tracker.begin('turn-1', 10_000); + tracker.begin('turn-2', 10_010); + tracker.clear(); + expect(tracker.poll(20_000)).toEqual({ + active: false, + stalled: false, + newlyStalled: false, + shouldNotify: false, + }); + }); + + it('rejects invalid timeout configuration', () => { + expect(() => new CodexAppTurnLiveness(0)).toThrow(/positive finite/); + expect(() => new CodexAppTurnLiveness(Number.NaN)).toThrow(/positive finite/); + }); +}); diff --git a/test/codex-bridge-queue.test.ts b/test/codex-bridge-queue.test.ts index 4222b95c0..fe23823dc 100644 --- a/test/codex-bridge-queue.test.ts +++ b/test/codex-bridge-queue.test.ts @@ -1,5 +1,10 @@ import { describe, it, expect } from 'vitest'; -import { CodexBridgeQueue } from '../src/services/codex-bridge-queue.js'; +import { + CodexBridgeQueue, + STRUCTURED_SUBMIT_START_GRACE_MS, + STRUCTURED_UNCONFIRMED_ATTRIBUTION_GRACE_MS, + STRUCTURED_SUBMIT_VERIFICATION_GRACE_MS, +} from '../src/services/codex-bridge-queue.js'; import { buildBridgeSendMarkerContent, shouldSuppressBridgeEmit, type BridgeSendMarker } from '../src/services/bridge-fallback-gate.js'; import type { CodexBridgeEvent } from '../src/services/codex-transcript.js'; @@ -10,6 +15,9 @@ function userEv(text: string, uuid?: string, ts = 0): CodexBridgeEvent { function asstEv(text: string, uuid?: string, ts = 0): CodexBridgeEvent { return { uuid: uuid ?? `a${++nextUuid}`, timestampMs: ts, kind: 'assistant_final', text }; } +function abortEv(reason = 'interrupted', uuid?: string, ts = 0, sourceSessionId?: string): CodexBridgeEvent { + return { uuid: uuid ?? `x${++nextUuid}`, timestampMs: ts, kind: 'turn_aborted', text: reason, sourceSessionId }; +} function markerForContent(sentAtMs: number, content: string): BridgeSendMarker { return { sentAtMs, ...buildBridgeSendMarkerContent(content) }; } @@ -45,6 +53,465 @@ function emitDecisions( } describe('CodexBridgeQueue', () => { + it('reports lifecycle busy only after transcript start until assistant_final closes the turn', () => { + const q = new CodexBridgeQueue(); + q.mark('t1', 'long-running prompt', 100); + expect(q.hasBlockingTurn()).toBe(false); + + q.ingest([userEv('long-running prompt', 'u-lifecycle', 200)]); + expect(q.hasBlockingTurn()).toBe(true); + + q.ingest([asstEv('done', 'a-lifecycle', 300)]); + expect(q.hasBlockingTurn()).toBe(false); + }); + + it('closes an interrupted durable turn without output but preserves its exact-attempt terminal', () => { + const q = new CodexBridgeQueue(); + q.mark('interrupted', 'stop this turn', 100, 7); + q.ingest([userEv('stop this turn', 'u-interrupted', 200)]); + expect(q.hasBlockingTurn()).toBe(true); + + q.ingest([abortEv('interrupted', 'x-interrupted', 300)]); + expect(q.hasBlockingTurn()).toBe(false); + expect(q.drainEmittable()).toEqual([ + expect.objectContaining({ + turnId: 'interrupted', + dispatchAttempt: 7, + finalText: '', + terminalStatus: 'ambiguous', + terminalErrorCode: 'structured_turn_aborted', + }), + ]); + expect(q.peek()).toEqual([]); + }); + + it('does not let a terminal event from another source session abort the collecting turn', () => { + const q = new CodexBridgeQueue(); + q.mark('source-a-turn', 'bound prompt', 100); + q.ingest([{ + ...userEv('bound prompt', 'u-source-a', 200), + sourceSessionId: 'source-a', + }]); + + q.ingest([abortEv('interrupted', 'x-source-b', 300, 'source-b')]); + expect(q.hasBlockingTurn()).toBe(true); + expect(q.peek()).toEqual([ + expect.objectContaining({ turnId: 'source-a-turn', started: true }), + ]); + + q.ingest([abortEv('interrupted', 'x-source-a', 301, 'source-a')]); + expect(q.hasBlockingTurn()).toBe(false); + expect(q.drainEmittable()).toEqual([ + expect.objectContaining({ + turnId: 'source-a-turn', + finalText: '', + terminalStatus: 'ambiguous', + }), + ]); + }); + + it('refreshes a queued submit after an interrupted predecessor and attributes its real final', () => { + let now = 1_000; + const q = new CodexBridgeQueue(() => now); + q.mark('first', 'interrupt first', 100); + q.mark('second', 'run second', 101); + q.confirmPendingTurn('second', 200); + q.ingest([userEv('interrupt first', 'u-first-abort', 300)]); + + now = 50_000; + q.ingest([abortEv('interrupted', 'x-first-abort', 400)]); + expect(q.peek().find(turn => turn.turnId === 'second')?.submitConfirmedAtMs).toBe(now); + expect(q.hasBlockingTurn(now + 1)).toBe(true); + + q.ingest([ + userEv('run second', 'u-after-abort', 50_001), + asstEv('second completed', 'a-after-abort', 50_002), + ]); + expect(q.hasBlockingTurn()).toBe(false); + expect(q.drainEmittable()).toEqual([ + expect.objectContaining({ turnId: 'first', finalText: '', terminalStatus: 'ambiguous' }), + expect.objectContaining({ turnId: 'second', finalText: 'second completed' }), + ]); + }); + + it('replays a buffered successor abort after a stale head is pruned', () => { + let now = 19_000; + const q = new CodexBridgeQueue(() => now); + q.mark('stale', 'never reached transcript', 100); + q.confirmPendingTurn('stale', 0); + q.mark('interrupted-successor', 'successor starts then aborts', now); + q.confirmPendingTurn('interrupted-successor', now); + + q.ingest([ + userEv('successor starts then aborts', 'u-buffered-abort', 19_001), + abortEv('interrupted', 'x-buffered-abort', 19_002), + ]); + expect(q.peek().find(turn => turn.turnId === 'interrupted-successor')?.started).toBe(false); + + now = STRUCTURED_SUBMIT_START_GRACE_MS + 1; + expect(q.pruneExpiredPreStartHeads().map(turn => turn.turnId)).toEqual(['stale']); + expect(q.hasBlockingTurn()).toBe(false); + expect(q.drainEmittable()).toEqual([ + expect.objectContaining({ + turnId: 'interrupted-successor', + finalText: '', + terminalStatus: 'ambiguous', + }), + ]); + expect(q.peek()).toEqual([]); + }); + + it('bounds an explicitly confirmed pre-start turn with a self-healing lease', () => { + const q = new CodexBridgeQueue(); + q.mark('queued', 'park this prompt', 100); + expect(q.hasBlockingTurn(1_000)).toBe(false); + + expect(q.confirmPendingTurn('queued', 1_000)).toBe(true); + expect(q.hasBlockingTurn(1_000 + STRUCTURED_SUBMIT_START_GRACE_MS - 1)).toBe(true); + expect(q.preStartLeaseRemainingMs(1_000 + STRUCTURED_SUBMIT_START_GRACE_MS - 1)).toBe(1); + expect(q.hasBlockingTurn(1_000 + STRUCTURED_SUBMIT_START_GRACE_MS + 1)).toBe(false); + expect(q.preStartLeaseRemainingMs(1_000 + STRUCTURED_SUBMIT_START_GRACE_MS + 1)).toBeUndefined(); + }); + + it('blocks the ready race while submit verification is still in flight, then self-heals', () => { + const q = new CodexBridgeQueue(); + q.mark('verifying', 'history polling prompt', 100); + expect(q.beginSubmitVerification('verifying', 1_000)).toBe(true); + expect(q.hasBlockingTurn(1_000 + STRUCTURED_SUBMIT_VERIFICATION_GRACE_MS - 1)).toBe(true); + expect(q.preStartLeaseRemainingMs(1_000 + STRUCTURED_SUBMIT_VERIFICATION_GRACE_MS - 1)).toBe(1); + expect(q.hasBlockingTurn(1_000 + STRUCTURED_SUBMIT_VERIFICATION_GRACE_MS + 1)).toBe(false); + + expect(q.finishSubmitVerification('verifying')).toBe(true); + expect(q.hasBlockingTurn(1_001)).toBe(false); + }); + + it('refreshes a finished unconfirmed write into a bounded attribution-only lease', () => { + let now = 100; + const q = new CodexBridgeQueue(() => now); + q.mark('unconfirmed', 'adapter returned undefined', now); + expect(q.peek()[0]?.unconfirmedAttributionStartedAtMs).toBe(100); + + q.beginSubmitVerification('unconfirmed', 200); + now = 500; + expect(q.finishSubmitVerification('unconfirmed')).toBe(true); + expect(q.peek()[0]).toMatchObject({ unconfirmedAttributionStartedAtMs: 500 }); + expect(q.peek()[0]?.submitVerificationStartedAtMs).toBeUndefined(); + + // Attribution-only retention must never turn an uncertain keypress into + // lifecycle busy or reject a real ready signal. + expect(q.hasBlockingTurn(now)).toBe(false); + expect(q.preStartLeaseRemainingMs(now)).toBeUndefined(); + expect(q.pruneExpiredPreStartHeads(now + STRUCTURED_UNCONFIRMED_ATTRIBUTION_GRACE_MS)).toEqual([]); + expect(q.pruneExpiredPreStartHeads(now + STRUCTURED_UNCONFIRMED_ATTRIBUTION_GRACE_MS + 1) + .map(turn => turn.turnId)).toEqual(['unconfirmed']); + }); + + it('protects an unconfirmed attribution head while bounded verification is active', () => { + const q = new CodexBridgeQueue(); + q.mark('verifying', 'slow authoritative check', 0); + q.beginSubmitVerification('verifying', 0); + + expect(q.pruneExpiredPreStartHeads(STRUCTURED_UNCONFIRMED_ATTRIBUTION_GRACE_MS + 1)).toEqual([]); + expect(q.peek().map(turn => turn.turnId)).toEqual(['verifying']); + expect(q.pruneExpiredPreStartHeads(STRUCTURED_SUBMIT_VERIFICATION_GRACE_MS + 1) + .map(turn => turn.turnId)).toEqual(['verifying']); + }); + + it('atomically transitions in-flight verification to a confirmed start lease', () => { + const q = new CodexBridgeQueue(); + q.mark('verified', 'confirmed after polling', 100); + q.beginSubmitVerification('verified', 1_000); + expect(q.confirmPendingTurn('verified', 2_000)).toBe(true); + expect(q.peek()[0]?.submitVerificationStartedAtMs).toBeUndefined(); + expect(q.peek()[0]?.submitConfirmedAtMs).toBe(2_000); + expect(q.hasBlockingTurn(2_000 + STRUCTURED_SUBMIT_START_GRACE_MS - 1)).toBe(true); + }); + + it('reports the longest active pre-start lease when several submits were confirmed', () => { + const q = new CodexBridgeQueue(); + q.mark('older', 'older confirmed prompt', 100); + q.mark('newer', 'newer confirmed prompt', 101); + q.confirmPendingTurn('older', 1_000); + q.confirmPendingTurn('newer', 2_000); + + expect(q.preStartLeaseRemainingMs(1_000 + STRUCTURED_SUBMIT_START_GRACE_MS + 1)) + .toBe(999); + }); + + it('refreshes the next confirmed type-ahead lease from local observation time, not transcript clock', () => { + let observedNow = 120; + const q = new CodexBridgeQueue(() => observedNow); + q.mark('t1', 'first prompt', 100); + q.mark('t2', 'queued second prompt', 101); + q.confirmPendingTurn('t1', 110); + q.confirmPendingTurn('t2', 111); + q.ingest([userEv('first prompt', 'u-first-lease', 200)]); + + // t2's enqueue-time lease can expire while t1 legitimately runs. + observedNow = 50_000; + const futureTranscriptTime = 9_999_999_999; + q.ingest([asstEv('first done', 'a-first-lease', futureTranscriptTime)]); + expect(q.hasBlockingTurn(observedNow + 1)).toBe(true); + expect(q.peek().find(turn => turn.turnId === 't2')?.submitConfirmedAtMs).toBe(observedNow); + + q.ingest([userEv('queued second prompt', 'u-second-lease', futureTranscriptTime + 2)]); + expect(q.hasBlockingTurn(observedNow + STRUCTURED_SUBMIT_START_GRACE_MS + 100)).toBe(true); + observedNow += 10; + q.ingest([asstEv('second done', 'a-second-lease', futureTranscriptTime + 3)]); + expect(q.hasBlockingTurn(observedNow + 1)).toBe(false); + }); + + it('drops only an unstarted failed-submit mark so status can converge to idle', () => { + const q = new CodexBridgeQueue(); + q.mark('failed', 'never submitted', 100); + expect(q.dropPendingTurn('failed')?.turnId).toBe('failed'); + expect(q.hasBlockingTurn()).toBe(false); + + q.mark('started', 'accepted prompt', 200); + q.ingest([userEv('accepted prompt', 'u-started', 300)]); + expect(q.dropPendingTurn('started')).toBeNull(); + expect(q.hasBlockingTurn()).toBe(true); + }); + + it('lets an authoritative terminal retire one exact started attempt and refreshes its successor lease', () => { + let now = 100; + const q = new CodexBridgeQueue(() => now); + q.mark('active-turn', 'active delivery', 100, 1); + q.confirmPendingTurn('active-turn', 100, 1); + q.ingest([userEv('active delivery', 'u-active-delivery', 101)]); + q.mark('successor-turn', 'replayed delivery', 102, 2); + q.confirmPendingTurn('successor-turn', 102, 2); + + now = 50_000; + expect(q.dropPendingTurn('active-turn', 1, true)).toEqual( + expect.objectContaining({ turnId: 'active-turn', dispatchAttempt: 1, started: true }), + ); + expect(q.peek()).toEqual([ + expect.objectContaining({ turnId: 'successor-turn', dispatchAttempt: 2, submitConfirmedAtMs: now }), + ]); + expect(q.hasBlockingTurn(now + 1)).toBe(true); + }); + + it('replays a later turn buffered behind a failed head mark', () => { + const q = new CodexBridgeQueue(); + q.mark('failed', 'first submit was dropped', 100); + q.mark('next', 'second submit reached the cli', 200); + + // The queue initially compares this event with the failed head and buffers + // it as unmatched. Removing that head must immediately replay it against + // the second mark. + q.ingest([userEv('second submit reached the cli', 'u-next', 300)]); + expect(q.peek().find(turn => turn.turnId === 'next')?.started).toBe(false); + expect(q.dropPendingTurn('failed')?.turnId).toBe('failed'); + expect(q.peek().find(turn => turn.turnId === 'next')?.started).toBe(true); + expect(q.hasBlockingTurn()).toBe(true); + + q.ingest([asstEv('second turn done', 'a-next', 400)]); + expect(q.hasBlockingTurn()).toBe(false); + expect(q.drainEmittable()).toEqual([ + expect.objectContaining({ turnId: 'next', finalText: 'second turn done' }), + ]); + }); + + it('preserves a later successful event across two failed head drops', () => { + const q = new CodexBridgeQueue(); + q.mark('failed-1', 'first submit failed', 100); + q.mark('failed-2', 'second submit failed', 200); + q.mark('success-3', 'third submit reached cli', 300); + q.ingest([ + userEv('third submit reached cli', 'u-third-behind-two', 400), + asstEv('third response', 'a-third-behind-two', 500), + ]); + + expect(q.dropPendingTurn('failed-1')?.turnId).toBe('failed-1'); + expect(q.peek().find(turn => turn.turnId === 'success-3')?.started).toBe(false); + expect(q.dropPendingTurn('failed-2')?.turnId).toBe('failed-2'); + expect(q.drainEmittable()).toEqual([ + expect.objectContaining({ turnId: 'success-3', finalText: 'third response' }), + ]); + }); + + it('expires a confirmed stale head and replays a buffered successor user/final lifecycle', () => { + let now = 19_000; + const q = new CodexBridgeQueue(() => now); + q.mark('stale', 'accepted but never started', 100); + q.confirmPendingTurn('stale', 0); + q.mark('live-second', 'second prompt really started', 19_000); + q.confirmPendingTurn('live-second', 19_000); + + // While the stale head lease is still active, the real second user event + // cannot fingerprint-match it and is retained in the unmatched buffer. + q.ingest([userEv('second prompt really started', 'u-live-second', 19_001)]); + expect(q.peek().find(turn => turn.turnId === 'live-second')?.started).toBe(false); + + // Expiry removes only the stale confirmed head and immediately replays the + // buffered event against its live successor. + now = STRUCTURED_SUBMIT_START_GRACE_MS + 1; + expect(q.hasBlockingTurn()).toBe(true); + expect(q.pruneExpiredPreStartHeads().map(turn => turn.turnId)).toEqual(['stale']); + expect(q.peek().some(turn => turn.turnId === 'stale')).toBe(false); + expect(q.peek().find(turn => turn.turnId === 'live-second')?.started).toBe(true); + + q.ingest([asstEv('second response', 'a-live-second', 20_002)]); + expect(q.hasBlockingTurn()).toBe(false); + expect(q.drainEmittable()).toEqual([ + expect.objectContaining({ turnId: 'live-second', finalText: 'second response' }), + ]); + }); + + it('expires a silent unconfirmed write and drains a buffered real successor lifecycle', () => { + let now = 0; + const q = new CodexBridgeQueue(() => now); + + // Hermes/Pi/MTR writeInput returns undefined. The worker therefore ends + // verification without positive submit evidence; if Enter was silently + // dropped, this attribution-only head must not live forever. + q.mark('silent-write', 'keypress disappeared', now); + q.beginSubmitVerification('silent-write', now); + q.finishSubmitVerification('silent-write', now); + + now = 19_000; + q.mark('real-successor', 'second prompt reached transcript', now); + q.ingest([ + userEv('second prompt reached transcript', 'u-real-after-silent', 19_001), + asstEv('second prompt done', 'a-real-after-silent', 19_002), + ]); + expect(q.peek().find(turn => turn.turnId === 'real-successor')?.started).toBe(false); + expect(q.hasBlockingTurn()).toBe(false); + + now = STRUCTURED_UNCONFIRMED_ATTRIBUTION_GRACE_MS + 1; + expect(q.pruneExpiredPreStartHeads().map(turn => turn.turnId)).toEqual(['silent-write']); + expect(q.drainEmittable()).toEqual([ + expect.objectContaining({ turnId: 'real-successor', finalText: 'second prompt done' }), + ]); + }); + + it('expires consecutive confirmed stale heads until a buffered live turn can start', () => { + let now = 19_000; + const q = new CodexBridgeQueue(() => now); + q.mark('stale-1', 'first stale confirmed prompt', 100); + q.confirmPendingTurn('stale-1', 0); + q.mark('stale-2', 'second stale confirmed prompt', 200); + q.confirmPendingTurn('stale-2', 500); + q.mark('live-3', 'third prompt reached transcript', 19_000); + q.confirmPendingTurn('live-3', 19_000); + q.ingest([userEv('third prompt reached transcript', 'u-live-third', 19_001)]); + + now = STRUCTURED_SUBMIT_START_GRACE_MS + 501; + expect(q.pruneExpiredPreStartHeads().map(turn => turn.turnId)).toEqual(['stale-1', 'stale-2']); + expect(q.peek().find(turn => turn.turnId === 'live-3')?.started).toBe(true); + + q.ingest([asstEv('third response', 'a-live-third', 20_502)]); + expect(q.drainEmittable()).toEqual([ + expect.objectContaining({ turnId: 'live-3', finalText: 'third response' }), + ]); + }); + + it('never prunes a started turn or queued confirmations behind its active lifecycle', () => { + let now = 1_000; + const q = new CodexBridgeQueue(() => now); + q.mark('running', 'long running first prompt', 100); + q.confirmPendingTurn('running', 100); + q.ingest([userEv('long running first prompt', 'u-running-prune', 200)]); + q.mark('queued', 'valid queued second prompt', 300); + q.confirmPendingTurn('queued', 300); + + now = 100_000; + expect(q.pruneExpiredPreStartHeads()).toEqual([]); + expect(q.peek().map(turn => turn.turnId)).toEqual(['running', 'queued']); + expect(q.hasBlockingTurn()).toBe(true); + + q.ingest([asstEv('first finished', 'a-running-prune', 100_001)]); + expect(q.peek().find(turn => turn.turnId === 'queued')?.submitConfirmedAtMs).toBe(now); + expect(q.pruneExpiredPreStartHeads()).toEqual([]); + }); + + it('refreshes a queued unconfirmed attribution lease when its started predecessor finishes', () => { + let now = 100; + const q = new CodexBridgeQueue(() => now); + q.mark('running', 'long running first prompt', now); + q.ingest([userEv('long running first prompt', 'u-running-before-bare', 101)]); + q.mark('queued-bare', 'unconfirmed typeahead', 200); + q.beginSubmitVerification('queued-bare', 200); + q.finishSubmitVerification('queued-bare', 201); + + now = 100_000; + expect(q.pruneExpiredPreStartHeads()).toEqual([]); + q.ingest([asstEv('first finished', 'a-running-before-bare', now)]); + expect(q.peek().find(turn => turn.turnId === 'queued-bare')?.unconfirmedAttributionStartedAtMs) + .toBe(now); + expect(q.pruneExpiredPreStartHeads()).toEqual([]); + + now += STRUCTURED_UNCONFIRMED_ATTRIBUTION_GRACE_MS + 1; + expect(q.pruneExpiredPreStartHeads().map(turn => turn.turnId)).toEqual(['queued-bare']); + }); + + it('keeps status queries pure when pruning would replay a complete buffered successor', () => { + let now = 19_000; + const q = new CodexBridgeQueue(() => now); + q.mark('stale', 'stale head before complete successor', 100); + q.confirmPendingTurn('stale', 0); + q.mark('complete-successor', 'successor user and final buffered', 19_000); + q.confirmPendingTurn('complete-successor', 19_000); + q.ingest([ + userEv('successor user and final buffered', 'u-complete-successor', 19_001), + asstEv('successor complete', 'a-complete-successor', 19_002), + ]); + + now = STRUCTURED_SUBMIT_START_GRACE_MS + 1; + // Projection/timer queries may observe lease expiry but must not mutate the + // queue and silently create an undrained completion. + expect(q.hasBlockingTurn()).toBe(true); + expect(q.preStartLeaseRemainingMs()).toBe(STRUCTURED_SUBMIT_START_GRACE_MS - 1_001); + expect(q.peek().find(turn => turn.turnId === 'complete-successor')?.started).toBe(false); + + // The explicit worker mutation boundary prunes, replays and can drain in + // the same call stack. + expect(q.pruneExpiredPreStartHeads().map(turn => turn.turnId)).toEqual(['stale']); + expect(q.drainEmittable()).toEqual([ + expect.objectContaining({ turnId: 'complete-successor', finalText: 'successor complete' }), + ]); + }); + + it('accepts a matching late user event at the lease boundary before pruning its head', () => { + let now = STRUCTURED_SUBMIT_START_GRACE_MS + 1; + const q = new CodexBridgeQueue(() => now); + q.mark('late-but-real', 'late transcript start', 100); + q.confirmPendingTurn('late-but-real', 0); + + q.ingest([userEv('late transcript start', 'u-late-real', now)]); + expect(q.peek().find(turn => turn.turnId === 'late-but-real')?.started).toBe(true); + expect(q.hasBlockingTurn()).toBe(true); + + now++; + q.ingest([asstEv('late turn done', 'a-late-real', now)]); + expect(q.drainEmittable()).toEqual([ + expect.objectContaining({ turnId: 'late-but-real', finalText: 'late turn done' }), + ]); + }); + + it('accepts a matching late user event for an expired attribution-only head before pruning', () => { + let now = STRUCTURED_UNCONFIRMED_ATTRIBUTION_GRACE_MS + 1; + const q = new CodexBridgeQueue(() => now); + q.mark('late-unconfirmed', 'late bare transcript start', 0); + + // Worker ingest runs before the explicit prune boundary. A real matching + // event therefore owns the mark even when the attribution clock elapsed. + q.ingest([userEv('late bare transcript start', 'u-late-unconfirmed', now)]); + expect(q.peek()[0]).toMatchObject({ + turnId: 'late-unconfirmed', + started: true, + unconfirmedAttributionStartedAtMs: undefined, + }); + expect(q.pruneExpiredPreStartHeads()).toEqual([]); + + now++; + q.ingest([asstEv('late bare turn done', 'a-late-unconfirmed', now)]); + expect(q.drainEmittable()).toEqual([ + expect.objectContaining({ turnId: 'late-unconfirmed', finalText: 'late bare turn done' }), + ]); + }); + it('marked turn whose user fingerprint matches becomes started; assistant_final closes it; drainEmittable yields finalText', () => { const q = new CodexBridgeQueue(); q.mark('t1', 'hello model please', 100); @@ -99,6 +566,35 @@ describe('CodexBridgeQueue', () => { ]); }); + it('does not let stale submit lifecycle callbacks mutate a replay attempt with the same turnId', () => { + const q = new CodexBridgeQueue(); + q.mark('same-turn', 'durable prompt', 100, 1); + expect(q.beginSubmitVerification('same-turn', 110, 1)).toBe(true); + expect(q.dropPendingTurn('same-turn', 1)).toEqual( + expect.objectContaining({ dispatchAttempt: 1 }), + ); + + q.mark('same-turn', 'durable prompt', 200, 2); + expect(q.beginSubmitVerification('same-turn', 210, 2)).toBe(true); + + // Old attempt N's delayed callbacks must not fall through to N+1. + expect(q.confirmPendingTurn('same-turn', 300, 1)).toBe(false); + expect(q.finishSubmitVerification('same-turn', 301, 1)).toBe(false); + // Omitting attempt retains ordinary-message compatibility but matches only + // an ordinary no-attempt mark, never a durable delivery generation. + expect(q.confirmPendingTurn('same-turn', 302)).toBe(false); + expect(q.finishSubmitVerification('same-turn', 303)).toBe(false); + + expect(q.peek()).toEqual([ + expect.objectContaining({ + turnId: 'same-turn', + dispatchAttempt: 2, + submitVerificationStartedAtMs: 210, + }), + ]); + expect(q.peek()[0]?.submitConfirmedAtMs).toBeUndefined(); + }); + it('user event with no fingerprint match is ignored (history / local input)', () => { const q = new CodexBridgeQueue(); q.mark('t1', 'lark message', 100); @@ -462,6 +958,143 @@ describe('CodexBridgeQueue', () => { expect(ready.sourceSessionId).toBe('h1'); }); + + // ── RPC mode: server-side execution has no local transcript ────────────── + // An RPC turn is confirmed by the app-server ack but never reaches started + // because no local transcript is ingested. Without rpcActive, the bounded + // 20s confirmation lease would expire mid-turn (long-running or approval- + // pending turns) → pruneExpiredPreStartHeads drops it → false idle. + + it('rpcActive keeps the lifecycle gate asserted past the confirmation lease', () => { + let now = 1_000; + const q = new CodexBridgeQueue(() => now); + q.mark('rpc-turn', 'run on server', 100); + q.confirmPendingTurn('rpc-turn', 200); + q.markRpcActive('rpc-turn'); + expect(q.hasBlockingTurn(now)).toBe(true); + + // 25s later — well past the 20s confirmation lease — still blocking. + now = 26_000; + expect(q.hasBlockingTurn(now)).toBe(true); + expect(q.preStartLeaseRemainingMs(now)).toBeUndefined(); + }); + + it('rpcActive turn is never pruned by expiry while active', () => { + let now = 1_000; + const q = new CodexBridgeQueue(() => now); + q.mark('rpc-prune', 'server turn', 100); + q.markRpcActive('rpc-prune'); + + now = 60_000; // far past any lease + const dropped = q.pruneExpiredPreStartHeads(now); + expect(dropped).toEqual([]); + expect(q.hasBlockingTurn(now)).toBe(true); + expect(q.peek().length).toBe(1); + }); + + it('stopRpcActive releases the lifecycle gate without a terminal edge', () => { + let now = 1_000; + const q = new CodexBridgeQueue(() => now); + q.mark('rpc-stop', 'server turn', 100); + q.markRpcActive('rpc-stop'); + expect(q.hasBlockingTurn(now)).toBe(true); + + expect(q.stopRpcActive('rpc-stop')).toBe(true); + expect(q.hasBlockingTurn(now)).toBe(false); + // Stopping again is a no-op (returns false). + expect(q.stopRpcActive('rpc-stop')).toBe(false); + }); + + it('stopRpcActive releases only the exact dispatch attempt', () => { + const q = new CodexBridgeQueue(); + q.mark('same-rpc-turn', 'attempt one', 100, 1); + q.mark('same-rpc-turn', 'attempt two', 200, 2); + expect(q.markRpcActive('same-rpc-turn', 1)).toBe(true); + expect(q.markRpcActive('same-rpc-turn', 2)).toBe(true); + + expect(q.stopRpcActive('same-rpc-turn', 1)).toBe(true); + expect(q.peek().find(turn => turn.dispatchAttempt === 1)?.rpcActive).toBeUndefined(); + expect(q.peek().find(turn => turn.dispatchAttempt === 2)?.rpcActive).toBe(true); + expect(q.stopRpcActive('same-rpc-turn', 1)).toBe(false); + expect(q.stopRpcActive('same-rpc-turn', 2)).toBe(true); + }); + + it('rpcActive does not block a normal started turn from draining', () => { + let now = 1_000; + const q = new CodexBridgeQueue(() => now); + q.mark('normal', 'local turn', 90); + q.mark('rpc', 'server turn', 100); + q.markRpcActive('rpc'); + + // Normal turn starts and finishes normally. + q.ingest([userEv('local turn', 'u-normal', 200)]); + expect(q.hasBlockingTurn(now)).toBe(true); + q.ingest([asstEv('done', 'a-normal', 300)]); + + // Normal turn is emittable; rpc turn still blocks. + const ready = q.drainEmittable(); + expect(ready.map(t => t.turnId)).toEqual(['normal']); + expect(q.hasBlockingTurn(now)).toBe(true); + + // After the RPC turn is stopped, it no longer blocks the lifecycle gate. + // It remains in the queue as a normal pending turn until its attribution + // lease expires (separate from the rpcActive gate). + q.stopRpcActive('rpc'); + expect(q.hasBlockingTurn(now)).toBe(false); + }); + + it('replays successor events drained during predecessor hydration when its ACK is delayed beyond 5s', () => { + let now = 100; + const q = new CodexBridgeQueue(() => now); + q.mark('turn-n', 'first prompt', 100, 1); + q.markRpcActive('turn-n', 1); + q.ingest([userEv('first prompt', 'u-n', 200)]); + + q.stopRpcActive('turn-n', 1); + q.ingest([ + asstEv('first answer', 'a-n', 300), + userEv('second prompt', 'u-n-plus-1', 400), + asstEv('second answer', 'a-n-plus-1', 500), + ]); + + expect(q.drainEmittable()).toEqual([ + expect.objectContaining({ + turnId: 'turn-n', + dispatchAttempt: 1, + finalText: 'first answer', + }), + ]); + + // turn/start for N+1 was sent at t=350, but its ACK continuation does not + // install the mark until t=8,000. The worker preserves t=350 for this exact + // awaiting owner, so the pair buffered at t=400/500 is still replayable. + // Using the ACK time here would put both events outside the 5s window. + now = 8_000; + q.mark('turn-n-plus-1', 'second prompt', 350, 2); + expect(q.hasTerminalTurn('turn-n-plus-1', 2)).toBe(true); + expect(q.markRpcActive('turn-n-plus-1', 2)).toBe(false); + expect(q.drainEmittable()).toEqual([ + expect.objectContaining({ + turnId: 'turn-n-plus-1', + dispatchAttempt: 2, + finalText: 'second answer', + }), + ]); + }); + + it('markRpcActive clears any pending verification/confirmation lease', () => { + let now = 1_000; + const q = new CodexBridgeQueue(() => now); + q.mark('rpc-verify', 'server turn', 100); + q.beginSubmitVerification('rpc-verify', 200); + // Move past the verification lease so it would normally be prunable. + now = 40_000; + q.markRpcActive('rpc-verify'); + // rpcActive takes over from the expired verification lease. + expect(q.hasBlockingTurn(now)).toBe(true); + expect(q.pruneExpiredPreStartHeads(now)).toEqual([]); + }); + it('clearPending wipes queue state', () => { const q = new CodexBridgeQueue(); q.mark('t1', 'one', 100); diff --git a/test/codex-rpc-bridge-mark.test.ts b/test/codex-rpc-bridge-mark.test.ts index 39d825548..3b05afae2 100644 --- a/test/codex-rpc-bridge-mark.test.ts +++ b/test/codex-rpc-bridge-mark.test.ts @@ -4,10 +4,10 @@ import { shouldPreMarkFirstTurn, shouldQueueInitialPrompt, type EngageOutcome } import type { CodexBridgeEvent } from '../src/services/codex-transcript.js'; // Consecutive-turn regressions against the REAL CodexBridgeQueue, locking the -// fresh-first-turn mark discipline (Codex P1): the pre-mark must fire ONLY for a -// confirmed-accepted turn. A stale/duplicate unstarted head wedges drainEmittable -// forever, so not-sent (paste re-marks) and ambiguous (never starts) must NOT -// leave a phantom head. +// fresh-first-turn mark discipline (Codex P1): the active pre-mark fires only +// for a confirmed-accepted turn. Ambiguous delivery now has a separate +// fail-closed attribution mark that is explicitly retired at terminal/teardown; +// not-sent still leaves no mark before the safe paste fallback. const T0 = 'first prompt alpha'; const T1 = 'second prompt beta'; const user = (uuid: string, text: string, ts: number): CodexBridgeEvent => ({ uuid, timestampMs: ts, kind: 'user', text }); @@ -47,10 +47,12 @@ describe('CodexBridgeQueue — fresh first-turn mark discipline (Codex P1 stale- expect(q.drainEmittable().map(t => t.turnId)).toEqual(['t1']); }); - it('ambiguous → NO pre-mark → the next explicit prompt starts/emits (not blocked by a phantom head)', () => { + it('ambiguous fail-closed attribution mark is explicitly retired before a successor can run', () => { const q = new CodexBridgeQueue(); - // ambiguous fresh turn leaves the queue untouched. - expect(q.size()).toBe(0); + q.mark('t0', T0, 1000); + // Engine teardown/terminal owns this explicit retirement. A successor is + // not admitted while the worker's separate fail-closed gate is asserted. + expect(q.dropPendingTurn('t0')).toBeDefined(); q.mark('t1', T1, 2000); q.ingest([user('u1', T1, 2001), asst('a1', 'reply1', 2002)]); expect(q.drainEmittable().map(t => t.turnId)).toEqual(['t1']); @@ -68,6 +70,10 @@ describe('CodexBridgeQueue — fresh first-turn mark discipline (Codex P1 stale- const preMark = preMarkAlways ? true : shouldPreMarkFirstTurn(outcome); const flushMark = shouldQueueInitialPrompt({ hasPrompt: true, rpcEngineActive: engineActive, queuePrompt: false, passesInitialPromptViaArgs: false, deferInitialPrompt: false }); if (preMark) q.mark('t0', T0, 1000); + if (outcome === 'ambiguous') { + q.mark('t0', T0, 1000); // attribution-only while fail-closed + q.dropPendingTurn('t0'); // native terminal / engine teardown + } if (flushMark) q.mark('t0', T0, 1000); // paste flush re-marks the SAME turnId q.ingest([user('u0', T0, 1001), asst('a0', 'reply0', 1002)]); q.drainEmittable(); @@ -77,7 +83,8 @@ describe('CodexBridgeQueue — fresh first-turn mark discipline (Codex P1 stale- } it('FIXED worker: accepted / not-sent(→not-engaged) / ambiguous all let the NEXT turn emit', () => { - // accepted: pre-mark(1)+flush(0)=1; not-engaged: pre-mark(0)+flush(1)=1; ambiguous: 0. + // accepted: active pre-mark(1); not-engaged: paste mark(1); + // ambiguous: fail-closed attribution mark retired before successor. expect(nextTurnAfterFirst('accepted', false)).toEqual(['t1']); expect(nextTurnAfterFirst('not-engaged', false)).toEqual(['t1']); expect(nextTurnAfterFirst('ambiguous', false)).toEqual(['t1']); diff --git a/test/codex-rpc-engine.test.ts b/test/codex-rpc-engine.test.ts index 15d10830b..1fe0751e0 100644 --- a/test/codex-rpc-engine.test.ts +++ b/test/codex-rpc-engine.test.ts @@ -19,6 +19,10 @@ function makeEngine(over: Partial[0 ...over, }); } +const owner = (turnId: string, dispatchAttempt?: number) => ({ + turnId, + ...(dispatchAttempt !== undefined ? { dispatchAttempt } : {}), +}); describe('CodexRpcEngine — happy-path lifecycle against a fake app-server', () => { it('start (spawn → /readyz → connect → initialize) then startThread → sendTurn → stop', async () => { @@ -28,7 +32,8 @@ describe('CodexRpcEngine — happy-path lifecycle against a fake app-server', () expect(tid).toBe('thread-fake-1'); expect(engine.activeThreadId).toBe('thread-fake-1'); expect(engine.wsUrl).toMatch(/^ws:\/\/127\.0\.0\.1:\d+$/); - await engine.sendTurn('hello world'); // resolves on the ack, no throw + await expect(engine.sendTurn('hello world', owner('turn-1', 1))) + .resolves.toEqual({ nativeTurnId: 'turn-fake-1' }); engine.stop(); }, 20_000); @@ -39,6 +44,96 @@ describe('CodexRpcEngine — happy-path lifecycle against a fake app-server', () expect(tid).toBe('thread-persisted-42'); engine.stop(); }, 20_000); + + it('maps an ultra-fast turn/completed to the exact Botmux attempt', async () => { + const terminals: any[] = []; + const engine = makeEngine({ + sessionId: 'terminal-map', + onTurnTerminal: terminal => terminals.push(terminal), + }); + await engine.start(); + await engine.startThread(); + await engine.sendTurn('fast', owner('botmux-fast', 7)); + await new Promise(resolve => setTimeout(resolve, 50)); + expect(terminals).toEqual([expect.objectContaining({ + identity: { turnId: 'botmux-fast', dispatchAttempt: 7 }, + nativeTurnId: 'turn-fake-1', + status: 'completed', + })]); + engine.stop(); + }, 20_000); + + it('maps turn/completed that arrives before turn/start response without resurrecting the turn', async () => { + const terminals: any[] = []; + const engine = makeEngine({ + sessionId: 'terminal-before-response', + env: { ...process.env, FAKE_TERMINAL_BEFORE_RESPONSE: '1' }, + onTurnTerminal: terminal => terminals.push(terminal), + }); + await engine.start(); + await engine.startThread(); + await expect(engine.sendTurn('fastest', owner('before-response', 8))) + .resolves.toEqual({ nativeTurnId: 'turn-fake-1' }); + expect(terminals).toEqual([expect.objectContaining({ + identity: { turnId: 'before-response', dispatchAttempt: 8 }, + nativeTurnId: 'turn-fake-1', + status: 'completed', + })]); + + // If response binding resurrected the already-terminal native id, stop() + // would attempt another terminal callback. Native-terminal de-dupe masks + // that callback, so assert the internal active ownership is actually empty. + expect((engine as any).turnOwners.size).toBe(0); + engine.stop(); + }, 20_000); + + it('deduplicates duplicate native terminal notifications', async () => { + const terminals: any[] = []; + const engine = makeEngine({ + sessionId: 'duplicate-terminal', + env: { ...process.env, FAKE_DUPLICATE_TERMINAL: '1' }, + onTurnTerminal: terminal => terminals.push(terminal), + }); + await engine.start(); + await engine.startThread(); + await engine.sendTurn('once', owner('dedupe', 9)); + await new Promise(resolve => setTimeout(resolve, 50)); + expect(terminals).toHaveLength(1); + expect(terminals[0]).toEqual(expect.objectContaining({ + identity: { turnId: 'dedupe', dispatchAttempt: 9 }, + status: 'completed', + })); + engine.stop(); + }, 20_000); + + it('keeps the first terminal buffered for an unowned native turn', () => { + const engine = makeEngine({ sessionId: 'unowned-first-terminal-wins' }); + (engine as any).emitTurnTerminal('unowned-native', 'failed', 'first_failure'); + (engine as any).emitTurnTerminal('unowned-native', 'completed'); + expect((engine as any).deferredUnownedTerminals.get('unowned-native')).toEqual({ + status: 'failed', + errorCode: 'first_failure', + }); + engine.stop(); + }); + + it('keeps sequential resume/typeahead attempts isolated by native turn id', async () => { + const terminals: any[] = []; + const engine = makeEngine({ + sessionId: 'terminal-sequential', + onTurnTerminal: terminal => terminals.push(terminal), + }); + await engine.start(); + await engine.resumeThread('thread-persisted'); + await engine.sendTurn('one', owner('same-logical', 1)); + await engine.sendTurn('two', owner('same-logical', 2)); + await new Promise(resolve => setTimeout(resolve, 50)); + expect(terminals.map(t => [t.nativeTurnId, t.identity.dispatchAttempt])).toEqual([ + ['turn-fake-1', 1], + ['turn-fake-2', 2], + ]); + engine.stop(); + }, 20_000); }); describe('CodexRpcEngine — failure/recovery paths', () => { @@ -52,11 +147,52 @@ describe('CodexRpcEngine — failure/recovery paths', () => { }); await engine.start(); await engine.startThread(); - await expect(engine.sendTurn('never answered')).rejects.toThrow(/timed out/); + await expect(engine.sendTurn('never answered', owner('turn-hang', 1))).rejects.toThrow(/timed out/); expect(deadCount).toBe(1); // failAll → onDead exactly once engine.stop(); }, 20_000); + it('does not guess a lost follow-up response from an unowned native terminal', async () => { + let deadCount = 0; + const terminals: any[] = []; + const engine = makeEngine({ + sessionId: 'followup-lost-ack-terminal', + env: { + ...process.env, + FAKE_HANG_TURN: '1', + FAKE_HANG_TURN_NOTIFY: '1', + }, + requestTimeoutMs: 400, + onDead: () => { deadCount++; }, + onTurnTerminal: terminal => terminals.push(terminal), + }); + await engine.start(); + await engine.startThread(); + await expect(engine.sendTurn('completed despite lost ack', owner('followup-proof', 4))) + .rejects.toThrow(/timed out/); + expect(deadCount).toBe(1); + expect(terminals).toEqual([]); + engine.stop(); + }, 20_000); + + it('does not bind an unrelated pre-response turn/started when the request response rejects', async () => { + const terminals: any[] = []; + const engine = makeEngine({ + sessionId: 'followup-response-error-after-started', + env: { ...process.env, FAKE_ERROR_AFTER_STARTED: '1' }, + onTurnTerminal: terminal => terminals.push(terminal), + }); + await engine.start(); + await engine.startThread(); + await expect(engine.sendTurn('started before response error', owner('started-proof', 5))) + .rejects.toThrow(/fake response failure/); + expect((engine as any).turnOwners.size).toBe(0); + await new Promise(resolve => setTimeout(resolve, 200)); + expect(terminals).toEqual([]); + expect((engine as any).turnOwners.size).toBe(0); + engine.stop(); + }, 20_000); + it('app-server crash → onDead fires so the worker can restart the pane', async () => { let dead = false; const engine = makeEngine({ @@ -96,8 +232,8 @@ describe('CodexRpcEngine — failure/recovery paths', () => { const engine = makeEngine({ sessionId: 'first-ok' }); await engine.start(); await engine.startThread(); - const outcome = await engine.sendFirstTurn('hello', 'turn-1', async () => { probed = true; return false; }); - expect(outcome).toBe('accepted'); + const outcome = await engine.sendFirstTurn('hello', owner('turn-1', 1), async () => { probed = true; return false; }); + expect(outcome).toEqual({ outcome: 'accepted', nativeTurnId: 'turn-fake-1' }); expect(probed).toBe(false); // ack answered → no need to consult the rollout engine.stop(); }, 20_000); @@ -107,8 +243,8 @@ describe('CodexRpcEngine — failure/recovery paths', () => { await engine.start(); await engine.startThread(); (engine as any).ws = undefined; // simulate ws not open → send() throws before the frame leaves - const outcome = await engine.sendFirstTurn('hello', 'turn-1', async () => true); - expect(outcome).toBe('not-sent'); + const outcome = await engine.sendFirstTurn('hello', owner('turn-1', 1), async () => true); + expect(outcome).toEqual({ outcome: 'not-sent' }); engine.stop(); }, 20_000); @@ -117,8 +253,78 @@ describe('CodexRpcEngine — failure/recovery paths', () => { await engine.start(); await engine.startThread(); // frame dispatched, no ack within 400ms, but the rollout shows the user turn. - const outcome = await engine.sendFirstTurn('hello', 'turn-1', async () => true); - expect(outcome).toBe('accepted'); // positive evidence → never resend + const outcome = await engine.sendFirstTurn('hello', owner('turn-1', 1), async () => true); + expect(outcome).toEqual({ outcome: 'accepted' }); // positive evidence → never resend + engine.stop(); + }, 20_000); + + it('uses rollout evidence, not a sole pending request, when first-turn response is lost', async () => { + const terminals: any[] = []; + const engine = makeEngine({ + sessionId: 'first-lost-ack-native', + env: { + ...process.env, + FAKE_HANG_TURN: '1', + FAKE_HANG_TURN_NOTIFY: '1', + }, + requestTimeoutMs: 400, + onTurnTerminal: terminal => terminals.push(terminal), + }); + await engine.start(); + await engine.startThread(); + const outcome = await engine.sendFirstTurn( + 'hello', + owner('first-native', 3), + async () => true, + ); + expect(outcome).toEqual({ outcome: 'accepted', nativeTurnId: undefined }); + expect(terminals).toEqual([]); + engine.stop(); + }, 20_000); + + it('does not treat an uncorrelated turn/started as first-turn delivery evidence', async () => { + let probed = false; + const engine = makeEngine({ + sessionId: 'first-lost-ack-started', + env: { + ...process.env, + FAKE_HANG_TURN: '1', + FAKE_HANG_TURN_NOTIFY: '1', + FAKE_NO_TURN_TERMINAL: '1', + }, + requestTimeoutMs: 400, + }); + await engine.start(); + await engine.startThread(); + const outcome = await engine.sendFirstTurn( + 'hello', + owner('first-started', 5), + async () => { probed = true; return false; }, + ); + expect(outcome).toEqual({ outcome: 'ambiguous' }); + expect(probed).toBe(true); + engine.stop(); + }, 20_000); + + it.each([ + ['aborted', 'aborted', undefined], + ['failed', 'failed', 'fake_failed'], + ] as const)('maps native %s completion to worker terminal %s', async (nativeStatus, expectedStatus, errorCode) => { + const terminals: any[] = []; + const engine = makeEngine({ + sessionId: `terminal-${nativeStatus}`, + env: { ...process.env, FAKE_TURN_STATUS: nativeStatus }, + onTurnTerminal: terminal => terminals.push(terminal), + }); + await engine.start(); + await engine.startThread(); + await engine.sendTurn(nativeStatus, owner(`turn-${nativeStatus}`, 6)); + await new Promise(resolve => setTimeout(resolve, 50)); + expect(terminals).toEqual([expect.objectContaining({ + identity: { turnId: `turn-${nativeStatus}`, dispatchAttempt: 6 }, + status: expectedStatus, + ...(errorCode ? { errorCode } : {}), + })]); engine.stop(); }, 20_000); @@ -126,8 +332,25 @@ describe('CodexRpcEngine — failure/recovery paths', () => { const engine = makeEngine({ sessionId: 'first-amb', env: { ...process.env, FAKE_HANG_TURN: '1' }, requestTimeoutMs: 400 }); await engine.start(); await engine.startThread(); - const outcome = await engine.sendFirstTurn('hello', 'turn-1', async () => false); - expect(outcome).toBe('ambiguous'); // absence of evidence stays ambiguous → 0 auto-paste + const outcome = await engine.sendFirstTurn('hello', owner('turn-1', 1), async () => false); + expect(outcome).toEqual({ outcome: 'ambiguous' }); // absence of evidence stays ambiguous → 0 auto-paste + engine.stop(); + }, 20_000); + + it('P1-1 sendFirstTurn: rollout probe failure stays ambiguous instead of falling back to paste', async () => { + const engine = makeEngine({ + sessionId: 'first-probe-error', + env: { ...process.env, FAKE_HANG_TURN: '1' }, + requestTimeoutMs: 400, + }); + await engine.start(); + await engine.startThread(); + const outcome = await engine.sendFirstTurn( + 'hello', + owner('turn-probe-error', 2), + async () => { throw new Error('rollout unavailable'); }, + ); + expect(outcome).toEqual({ outcome: 'ambiguous' }); engine.stop(); }, 20_000); @@ -157,4 +380,42 @@ describe('CodexRpcEngine — failure/recovery paths', () => { await new Promise((r) => setTimeout(r, 300)); expect(dead).toBe(false); }, 20_000); + + it('stop releases every still-active native turn with its exact owner', async () => { + const terminals: any[] = []; + const engine = makeEngine({ + sessionId: 'stop-active', + env: { ...process.env, FAKE_NO_TURN_TERMINAL: '1' }, + onTurnTerminal: terminal => terminals.push(terminal), + }); + await engine.start(); + await engine.startThread(); + await engine.sendTurn('keep running', owner('active-stop', 11)); + engine.stop(); + expect(terminals).toEqual([expect.objectContaining({ + identity: { turnId: 'active-stop', dispatchAttempt: 11 }, + status: 'stopped', + errorCode: 'rpc_engine_stopped', + })]); + }, 20_000); + + it('engine death releases every still-active native turn before onDead recovery', async () => { + const order: string[] = []; + const engine = makeEngine({ + sessionId: 'death-active', + env: { + ...process.env, + FAKE_NO_TURN_TERMINAL: '1', + FAKE_DIE_AFTER_MS: '600', + }, + onTurnTerminal: terminal => order.push(`terminal:${terminal.identity.turnId}:${terminal.status}`), + onDead: () => order.push('dead'), + }); + await engine.start(); + await engine.startThread(); + await engine.sendTurn('active on crash', owner('active-dead', 12)); + await new Promise(resolve => setTimeout(resolve, 1500)); + expect(order).toEqual(['terminal:active-dead:engine-dead', 'dead']); + engine.stop(); + }, 20_000); }); diff --git a/test/codex-rpc-lifecycle.test.ts b/test/codex-rpc-lifecycle.test.ts index d8ef18882..1cde018d0 100644 --- a/test/codex-rpc-lifecycle.test.ts +++ b/test/codex-rpc-lifecycle.test.ts @@ -1,5 +1,5 @@ import { describe, it, expect } from 'vitest'; -import { codexRpcEligible, paneRunsRemoteTui, orchestrateCodexRpcInit, shouldQueueInitialPrompt, rolloutUserTurnMatches, decideStartupDialogAction, killAndVerifyPersistentPane, RPC_CAPABLE_CLIS, type PaneProbes, type RpcInitEffects } from '../src/codex-rpc-lifecycle.js'; +import { CODEX_RPC_TERMINAL_HYDRATION_DELAYS_MS, RpcEngagementFence, codexRpcEligible, paneRunsRemoteTui, orchestrateCodexRpcInit, shouldQueueInitialPrompt, rolloutUserTurnMatches, decideStartupDialogAction, killAndVerifyPersistentPane, rpcTranscriptIngestBlockedByAwaitingActivation, RPC_CAPABLE_CLIS, type PaneProbes, type RpcInitEffects } from '../src/codex-rpc-lifecycle.js'; import type { DaemonToWorker } from '../src/types.js'; type InitCfg = Extract; @@ -30,6 +30,68 @@ describe('codexRpcEligible — the eligible base case', () => { }); }); +describe('RPC terminal rollout hydration window', () => { + it('keeps polling through delayed 9s visibility and remains bounded near the 12s first-turn probe', () => { + const totalMs = CODEX_RPC_TERMINAL_HYDRATION_DELAYS_MS + .reduce((sum, delayMs) => sum + delayMs, 0); + expect(totalMs).toBeGreaterThanOrEqual(10_000); + expect(totalMs).toBeLessThanOrEqual(12_000); + + // Model a rollout that becomes visible nine seconds after the native + // terminal. The retry sequence must still have a scheduled observation at + // or after visibility, rather than retiring the bridge mark early. + let elapsedMs = 0; + let observed = false; + for (const delayMs of CODEX_RPC_TERMINAL_HYDRATION_DELAYS_MS) { + elapsedMs += delayMs; + if (elapsedMs >= 9_000) { + observed = true; + break; + } + } + expect(observed).toBe(true); + }); + + it('lets an older terminal hydrate while a different successor awaits activation', () => { + expect(rpcTranscriptIngestBlockedByAwaitingActivation( + ['turn-n-plus-1'], + 'turn-n', + )).toBe(false); + expect(rpcTranscriptIngestBlockedByAwaitingActivation( + ['turn-n-plus-1'], + )).toBe(true); + expect(rpcTranscriptIngestBlockedByAwaitingActivation( + ['turn-n', 'turn-n-plus-1'], + 'turn-n', + )).toBe(true); + }); +}); + +describe('RpcEngagementFence', () => { + it('rejects a delayed first-turn result after restart invalidates its lease', async () => { + const fence = new RpcEngagementFence(); + const lease = fence.begin(); + let resolveProbe!: (value: 'accepted') => void; + const probe = new Promise<'accepted'>(resolve => { resolveProbe = resolve; }); + let published = false; + let requeued = false; + const engage = (async () => { + const outcome = await probe; + if (!fence.isCurrent(lease)) return 'superseded'; + published = true; + if (outcome !== 'accepted') requeued = true; + return outcome; + })(); + + fence.invalidate(); + resolveProbe('accepted'); + + await expect(engage).resolves.toBe('superseded'); + expect(published).toBe(false); + expect(requeued).toBe(false); + }); +}); + describe('codexRpcEligible — every fail-closed gate degrades to paste', () => { const cases: Array<[string, Partial]> = [ ['codexRpcInput not set', { codexRpcInput: false }], diff --git a/test/codex-transcript.test.ts b/test/codex-transcript.test.ts index be1448558..6b83f81d7 100644 --- a/test/codex-transcript.test.ts +++ b/test/codex-transcript.test.ts @@ -267,7 +267,7 @@ describe('drainCodexRollout', () => { expect(r.events[0].text).toBe('done'); }); - it('skips reasoning / function_call / function_call_output / event_msg', () => { + it('skips reasoning / function_call / function_call_output / non-terminal event_msg', () => { writeFileSync(path, ev({ type: 'response_item', payload: { type: 'reasoning' } }) + ev({ type: 'response_item', payload: { type: 'function_call', name: 'shell' } }) + @@ -280,6 +280,32 @@ describe('drainCodexRollout', () => { expect(r.events[0].text).toBe('actual prompt'); }); + it('extracts turn_aborted as a no-output terminal edge', () => { + writeFileSync(path, + ev(userResponseItem('interrupt me')) + + ev({ + timestamp: '2026-04-29T07:00:02.000Z', + type: 'event_msg', + payload: { type: 'turn_aborted', reason: 'interrupted' }, + })); + const r = drainCodexRollout(path, 0); + expect(r.events.map(event => ({ kind: event.kind, text: event.text }))).toEqual([ + { kind: 'user', text: 'interrupt me' }, + { kind: 'turn_aborted', text: 'interrupted' }, + ]); + }); + + it('keeps an empty final_answer as a normal completed terminal edge', () => { + writeFileSync(path, + ev(userResponseItem('finish without visible text')) + + ev(assistantFinalResponseItem(''))); + const r = drainCodexRollout(path, 0); + expect(r.events.map(event => ({ kind: event.kind, text: event.text }))).toEqual([ + { kind: 'user', text: 'finish without visible text' }, + { kind: 'assistant_final', text: '' }, + ]); + }); + it('skips messages with no input_text/output_text content', () => { writeFileSync(path, ev({ diff --git a/test/command-handler.test.ts b/test/command-handler.test.ts index 6b18d55bf..ce3d61c29 100644 --- a/test/command-handler.test.ts +++ b/test/command-handler.test.ts @@ -284,6 +284,7 @@ vi.mock('../src/core/worker-pool.js', () => ({ // resolves as idempotent close so unrelated tests don't need to think // about it. closeSession: vi.fn(async () => ({ ok: true, alreadyClosed: false })), + withActiveSessionKeyLock: vi.fn(async (_map: Map, _key: string, action: () => any) => action()), // `isRelayableRealSession(ds)` — true when ds.worker is set OR persisted // CLI markers exist (session.cliId / session.lastCliInput). The default // makeSession fixture sets cliId='claude-code' so most tests pass the @@ -414,6 +415,8 @@ vi.mock('../src/services/relay-picker.js', () => ({ if (c.chatId === currentChatId) continue; if (c.session.ownerOpenId !== operatorOpenId) continue; if (c.session.adoptedFrom) continue; + if ((c.initConfig?.backendType ?? c.session.backendType) === 'riff' + || c.session.cliId === 'riff') continue; // Real-session filter (same predicate as production picker). if (!c.worker && !c.session?.cliId && !c.session?.lastCliInput) continue; out.push({ @@ -463,7 +466,7 @@ import { sessionKey } from '../src/core/types.js'; import { setTerminalProxyPort } from '../src/core/terminal-url.js'; import type { DaemonSession } from '../src/core/types.js'; import type { LarkMessage, Session } from '../src/types.js'; -import { killWorker, suspendWorker, forkWorker, getCurrentCliVersion, deliverEphemeralOrReply, deliverWritableTerminalCardTo } from '../src/core/worker-pool.js'; +import { killWorker, forkWorker, suspendWorker, getCurrentCliVersion, deliverEphemeralOrReply, deliverWritableTerminalCardTo, closeSession as closeWorkerPoolSession, withActiveSessionKeyLock } from '../src/core/worker-pool.js'; import { getOwnerOpenId } from '../src/bot-registry.js'; import { canOperate } from '../src/im/lark/event-dispatcher.js'; import { getSessionWorkingDir, buildNewTopicPrompt, buildNewTopicCliInput, ensureSessionWhiteboard, getAvailableBots } from '../src/core/session-manager.js'; @@ -1343,9 +1346,7 @@ describe('handleCommand', () => { await handleCommand('/close', ROOT_ID, makeLarkMessage('/close'), deps, LARK_APP_ID); - expect(killWorker).toHaveBeenCalledWith(ds); - expect(sessionStore.closeSession).toHaveBeenCalledWith('sess-001'); - expect(deps.activeSessions.has(sessionKey(ROOT_ID, LARK_APP_ID))).toBe(false); + expect(closeWorkerPoolSession).toHaveBeenCalledWith('sess-001'); // The「会话已关闭」card is delivered「仅自己可见」-first: it routes through // deliverEphemeralOrReply targeting the user who ran /close (message.senderId), // so plain groups get an ephemeral (visible-to-you) card and topic groups @@ -1428,7 +1429,7 @@ describe('handleCommand', () => { await handleCommand('/restart', ROOT_ID, makeLarkMessage('/restart'), deps, LARK_APP_ID); - expect(workerSend).toHaveBeenCalledWith({ type: 'restart' }); + expect(workerSend).toHaveBeenCalledWith({ type: 'restart', reason: 'operator' }); expect(deps.sessionReply).toHaveBeenCalledWith( ROOT_ID, expect.stringContaining('正在重启'), @@ -1438,6 +1439,49 @@ describe('handleCommand', () => { ); }); + it('refuses live Riff restart without retiring the worker or task lineage', async () => { + const workerSend = vi.fn(); + const worker = { killed: false, send: workerSend } as any; + const session = makeSession({ + cliId: 'riff', + backendType: 'riff', + riffParentTaskId: 'task-riff-restart', + }); + const ds = makeDaemonSession({ + session, + worker, + initConfig: { backendType: 'riff' } as any, + }); + const deps = makeDeps(ds); + + await handleCommand('/restart', ROOT_ID, makeLarkMessage('/restart'), deps, LARK_APP_ID); + + expect(workerSend).not.toHaveBeenCalled(); + expect(killWorker).not.toHaveBeenCalled(); + expect(ds.worker).toBe(worker); + expect(ds.session.riffParentTaskId).toBe('task-riff-restart'); + expect((deps.sessionReply as ReturnType).mock.calls[0][1]).toContain('Riff'); + }); + + it('refuses workerless Riff restart without falsely reporting termination or clearing lineage', async () => { + const session = makeSession({ + cliId: 'riff', + backendType: 'riff', + riffParentTaskId: 'task-riff-workerless-restart', + }); + const ds = makeDaemonSession({ session, worker: null }); + const deps = makeDeps(ds); + + await handleCommand('/restart', ROOT_ID, makeLarkMessage('/restart'), deps, LARK_APP_ID); + + expect(killWorker).not.toHaveBeenCalled(); + expect(ds.worker).toBeNull(); + expect(ds.session.riffParentTaskId).toBe('task-riff-workerless-restart'); + const reply = (deps.sessionReply as ReturnType).mock.calls[0][1] as string; + expect(reply).toContain('Riff'); + expect(reply).not.toContain('进程已终止'); + }); + it('should kill dead worker and reply recovery message when worker is already killed', async () => { const ds = makeDaemonSession({ worker: { killed: true, send: vi.fn() } as any, @@ -1657,6 +1701,58 @@ describe('handleCommand', () => { expect(replyContent).toContain('已自动创建并切换'); }); + it('refuses Riff /cd before path creation and leaves route, cwd, worker, and lineage untouched', async () => { + vi.mocked(existsSync).mockReturnValue(false); + const worker = { killed: false, send: vi.fn() } as any; + const session = makeSession({ + cliId: 'riff', + backendType: 'riff', + riffParentTaskId: 'task-riff-cd', + workingDir: '/source/repo', + riffRepoDirs: ['/source/repo'], + }); + const ds = makeDaemonSession({ + session, + worker, + workingDir: '/source/repo', + initConfig: { backendType: 'riff' } as any, + }); + const deps = makeDeps(ds); + + await handleCommand('/cd', ROOT_ID, makeLarkMessage('/cd /brand-new/riff'), deps, LARK_APP_ID); + + expect(existsSync).not.toHaveBeenCalled(); + expect(mkdirSync).not.toHaveBeenCalled(); + expect(killWorker).not.toHaveBeenCalled(); + expect(sessionStore.updateSession).not.toHaveBeenCalled(); + expect(ds.worker).toBe(worker); + expect(ds.workingDir).toBe('/source/repo'); + expect(ds.session).toMatchObject({ + workingDir: '/source/repo', + riffRepoDirs: ['/source/repo'], + riffParentTaskId: 'task-riff-cd', + }); + expect((deps.sessionReply as ReturnType).mock.calls[0][1]).toContain('Riff'); + }); + + it('should reject before path creation while Codex App dispatch ownership is non-empty', async () => { + vi.mocked(existsSync).mockReturnValue(false); + const ds = makeDaemonSession(); + ds.session.codexAppDispatchLedger = [ + { dispatchId: 'd-1', turnId: 't-1', state: 'accepted', content: 'owned' }, + ]; + const originalWorkingDir = ds.workingDir; + const deps = makeDeps(ds); + + await handleCommand('/cd', ROOT_ID, makeLarkMessage('/cd /brand-new/owned'), deps, LARK_APP_ID); + + expect(mkdirSync).not.toHaveBeenCalled(); + expect(killWorker).not.toHaveBeenCalled(); + expect(ds.workingDir).toBe(originalWorkingDir); + const replyContent = (deps.sessionReply as ReturnType).mock.calls[0][1] as string; + expect(replyContent).toContain('未结算'); + }); + it('should reject /cd when auto-create fails', async () => { vi.mocked(existsSync).mockReturnValue(false); vi.mocked(mkdirSync).mockImplementationOnce(() => { throw new Error('EACCES: permission denied'); }); @@ -1866,14 +1962,18 @@ describe('handleCommand', () => { expect(ds.workingDir).toBe('/home/testuser/project-b'); // ds.session is replaced by createSession result (pendingRepo is false → else branch) - expect(killWorker).toHaveBeenCalledWith(ds); - expect(sessionStore.closeSession).toHaveBeenCalled(); + expect(closeWorkerPoolSession).toHaveBeenCalledWith('sess-001'); expect(sessionStore.createSession).toHaveBeenCalledWith( CHAT_ID, ROOT_ID, 'project-b (dev)', 'group', undefined, ); expect(ds.session.sessionId).toBe('new-session-123'); expect(ds.hasHistory).toBe(false); expect(forkWorker).toHaveBeenCalledWith(ds, '', false); + expect(withActiveSessionKeyLock).toHaveBeenCalledWith( + deps.activeSessions, + sessionKey(ROOT_ID, LARK_APP_ID), + expect.any(Function), + ); }); it('mid-session chat switch preserves scope and the original message root', async () => { @@ -2215,25 +2315,29 @@ describe('handleCommand', () => { // ─── bare /repo while pending (replaces the old /skip command) ──────────── describe('/repo (bare) while pending', () => { - it('does not consume the pending launch while a card commit owns the shared claim', async () => { + it('retains the opening reservation and buffers when forkWorker fails before accept', async () => { const ds = makeDaemonSession({ pendingRepo: true, - pendingRepoCommitInFlight: true, - pendingPrompt: 'hello world', - pendingTurnId: 'om_pending_turn', + initialStartPending: true, + pendingPrompt: 'first prompt', + pendingFollowUps: ['buffered follow-up'], }); const deps = makeDeps(ds); + vi.mocked(forkWorker).mockImplementationOnce(() => { + expect(ds.pendingRepo).toBe(true); + expect(ds.initialStartPending).toBe(true); + expect(ds.pendingPrompt).toBe('first prompt'); + expect(ds.pendingFollowUps).toEqual(['buffered follow-up']); + throw new Error('fork preaccept failed'); + }); await handleCommand('/repo', ROOT_ID, makeLarkMessage('/repo'), deps, LARK_APP_ID); - expect(forkWorker).not.toHaveBeenCalled(); - expect(getAvailableBots).not.toHaveBeenCalled(); expect(ds.pendingRepo).toBe(true); - expect(ds.pendingRepoCommitInFlight).toBe(true); - expect(ds.pendingPrompt).toBe('hello world'); - expect(ds.pendingTurnId).toBe('om_pending_turn'); - const replies = vi.mocked(deps.sessionReply).mock.calls.map(c => c[1]).join(); - expect(replies).toContain('已有一个 worktree 正在创建'); + expect(ds.initialStartPending).toBe(true); + expect(ds.pendingPrompt).toBe('first prompt'); + expect(ds.pendingFollowUps).toEqual(['buffered follow-up']); + expect(deleteMessage).not.toHaveBeenCalled(); }); it('should boot the CLI idle (no prompt submitted) when launched via /repo itself', async () => { @@ -2249,7 +2353,7 @@ describe('handleCommand', () => { // No buffered message → spawn idle with an empty prompt so the user's NEXT // message becomes the first prompt (not an empty/boilerplate user_message). - expect(forkWorker).toHaveBeenCalledWith(ds, '', false); + expect(forkWorker).toHaveBeenCalledWith(ds, '', { turnId: 'om_repo_command_only' }); expect(buildNewTopicPrompt).not.toHaveBeenCalled(); expect(killWorker).not.toHaveBeenCalled(); expect(sessionStore.createSession).not.toHaveBeenCalled(); @@ -2388,7 +2492,7 @@ describe('handleCommand', () => { expect(forkWorker).toHaveBeenCalledWith(ds, { content: 'WRAPPED:clean', codexAppInput, - }); + }, false); expect(ds.pendingSubstituteTrigger).toBeUndefined(); }); @@ -3446,6 +3550,32 @@ describe('handleCommand', () => { vi.mocked(dd.findOnlineDaemon).mockReturnValue(null); }); + it('refuses Riff --create before resolving members or creating a destination group', async () => { + const worker = { killed: false, send: vi.fn() } as any; + const ds = makeDaemonSession({ + worker, + initConfig: { backendType: 'riff' } as any, + session: makeSession({ + ownerOpenId: 'ou_sender', + cliId: 'riff', + backendType: 'riff', + riffParentTaskId: 'task-riff-create-relay', + }), + }); + const deps = makeDeps(ds); + + await handleCommand('/relay', ROOT_ID, makeLarkMessage('/relay --create G @Codex', { + mentions: [{ key: '@_user_1', name: 'Codex', openId: 'ou_codex' }], + }), deps, LARK_APP_ID); + + expect(mockedListBots).not.toHaveBeenCalled(); + expect(mockedCreate).not.toHaveBeenCalled(); + expect(mockedSend).not.toHaveBeenCalled(); + expect(worker.send).not.toHaveBeenCalled(); + expect(ds.session.riffParentTaskId).toBe('task-riff-create-relay'); + expect((deps.sessionReply as ReturnType).mock.calls[0][1]).toContain('Riff'); + }); + it('renders the relay picker card when invoked without --create', async () => { // Existing session in the current chat (ds) — picker should NOT list it // (self-targeting is rejected by the cant_relay_same_chat filter). diff --git a/test/crash-loop-diagnostic.test.ts b/test/crash-loop-diagnostic.test.ts index d79c97787..5e3fb210f 100644 --- a/test/crash-loop-diagnostic.test.ts +++ b/test/crash-loop-diagnostic.test.ts @@ -180,7 +180,7 @@ describe("crash-loop diagnostic terminal (daemon 'claude_exit' handler)", () => // First 3 auto-restart in place; the 4th asks the worker to park a // diagnostic shell (deferred park) and keeps it alive (no close). - expect(worker.send).toHaveBeenCalledWith({ type: 'restart' }); + expect(worker.send).toHaveBeenCalledWith({ type: 'restart', reason: 'cli_crash' }); expect(worker.send).toHaveBeenCalledWith({ type: 'park_diagnostic' }); expect(worker.send).not.toHaveBeenCalledWith({ type: 'close' }); // Survives daemon restart: lazy cold-resume + idle, restart counter reset. @@ -216,4 +216,26 @@ describe("crash-loop diagnostic terminal (daemon 'claude_exit' handler)", () => expect(worker.send).toHaveBeenCalledWith({ type: 'close' }); expect(ds.session.suspendedColdResume).toBeFalsy(); }); + + it('fails closed on an unexpected Riff exit without restarting or retiring its generation', async () => { + const worker = makeFakeWorker(); + const ds = makeDs('sid-riff-unexpected-exit', worker); + ds.session.cliId = 'riff'; + ds.session.backendType = 'riff'; + ds.session.riffParentTaskId = 'task-riff-unexpected-exit'; + ds.initConfig = { backendType: 'riff' } as any; + __testOnly_setupWorkerHandlers(ds, worker); + + await crashTimes(worker, 1); + + expect(worker.send).not.toHaveBeenCalledWith({ type: 'restart', reason: 'cli_crash' }); + expect(worker.send).not.toHaveBeenCalledWith({ type: 'close' }); + expect(worker.send).not.toHaveBeenCalledWith({ type: 'suspend' }); + expect(worker.send).not.toHaveBeenCalledWith({ type: 'park_diagnostic' }); + expect(ds.worker).toBe(worker); + expect(ds.session.riffParentTaskId).toBe('task-riff-unexpected-exit'); + expect(ds.session.status).toBe('active'); + expect(ds.lastScreenStatus).toBe('idle'); + expect(restartCounts.has('sid-riff-unexpected-exit')).toBe(false); + }); }); diff --git a/test/daemon-codex-app-workflow-wiring.test.ts b/test/daemon-codex-app-workflow-wiring.test.ts index 7d374fc3a..0824d0cb7 100644 --- a/test/daemon-codex-app-workflow-wiring.test.ts +++ b/test/daemon-codex-app-workflow-wiring.test.ts @@ -24,7 +24,8 @@ describe('daemon Codex App workflow prompt lanes', () => { expect(block).toContain("const codexAppMessageContext = codexAppQuoteContext + (workflowGrillPrompt ?? '');"); expect(block).toContain('const promptContent = codexAppQuoteContext + codexAppApplicationContext + content;'); expect(block).toContain('pendingCodexAppText: codexAppVisibleText'); - expect(block.match(/codexAppText: codexAppVisibleText/g)).toHaveLength(2); + expect(source).toContain('codexAppText: ds.pendingCodexAppText'); + expect(block.match(/forkReservedInitialSession\(ds, availableBots\)/g)).toHaveLength(2); }); it('retains VC lifecycle context in rewritten legacy prompts without demoting it to untrusted', () => { @@ -46,4 +47,13 @@ describe('daemon Codex App workflow prompt lanes', () => { 'const codexAppApplicationContext = initialCodexAppApplicationContext;', ); }); + + it('does not buffer ordinary turns solely because repo commit UI cleanup is still in flight', () => { + const block = region( + 'async function handleThreadReply', + 'async function autoCreateDocSession', + ); + expect(block).toContain('if (ds?.pendingRepo || initialStartPending) {'); + expect(block).not.toContain('if (ds?.pendingRepo || ds?.pendingRepoCommitInFlight || initialStartPending)'); + }); }); diff --git a/test/daemon-heartbeat.test.ts b/test/daemon-heartbeat.test.ts index 158b9f7c4..b7dd4feb7 100644 --- a/test/daemon-heartbeat.test.ts +++ b/test/daemon-heartbeat.test.ts @@ -7,6 +7,7 @@ import { anyDaemonBusyTo, heartbeatDirIn, HEARTBEAT_FRESH_MS, + isMaintenanceBusyStatus, } from '../src/core/daemon-heartbeat.js'; const T0 = Date.parse('2026-06-07T04:00:00.000Z'); @@ -58,4 +59,11 @@ describe('daemon heartbeat / anyDaemonBusy', () => { writeHeartbeatTo(dir, 'cli_app_a', 1, iso(T0)); expect(readdirSync(heartbeatDirIn(dir)).filter(f => f.endsWith('.tmp'))).toEqual([]); }); + + it('keeps stalled turns inside the fleet-wide maintenance restart gate', () => { + expect(isMaintenanceBusyStatus('working')).toBe(true); + expect(isMaintenanceBusyStatus('stalled')).toBe(true); + expect(isMaintenanceBusyStatus('idle')).toBe(false); + expect(isMaintenanceBusyStatus(undefined)).toBe(false); + }); }); diff --git a/test/daemon-refork-substitute-wiring.test.ts b/test/daemon-refork-substitute-wiring.test.ts index f02424866..d5ef63e66 100644 --- a/test/daemon-refork-substitute-wiring.test.ts +++ b/test/daemon-refork-substitute-wiring.test.ts @@ -13,13 +13,15 @@ describe('daemon stopped-worker substitute refork wiring', () => { expect(forkStart).toBeGreaterThan(start); expect(end).toBeGreaterThan(start); const reforkBlock = source.slice(start, end); - expect(reforkBlock).toMatch(/\n\s+substituteTrigger,\n/); + expect(reforkBlock).toContain( + 'substituteTrigger: queuedHasDurableTail ? undefined : substituteTrigger', + ); expect(reforkBlock).toContain('substituteTrigger ? SUBSTITUTE_RECEIVED_REACTION_EMOJI_TYPE : undefined'); expect(reforkBlock).toContain('applyQueuedCodexAppLegacyFallback(builtReforkInput'); expect(reforkBlock).toContain('Legacy queued dashboard task has no clean-input text'); expect(reforkBlock).toContain('wrappedInput !== builtReforkInput && dsBotCfgForFork.codexAppCleanInput === true'); expect(reforkBlock.indexOf('applyQueuedCodexAppLegacyFallback(builtReforkInput')) - .toBeLessThan(reforkBlock.indexOf('rememberLastCliInput(ds, promptContent, wrappedInput)')); + .toBeLessThan(reforkBlock.indexOf('rememberLastCliInput(ds, queuedHasDurableTail')); }); it('clears stale multi-Riff repo state when doc-watch replaces a stopped session cwd', () => { diff --git a/test/daemon-rename-route.test.ts b/test/daemon-rename-route.test.ts index d7169e3c2..304ea5bd7 100644 --- a/test/daemon-rename-route.test.ts +++ b/test/daemon-rename-route.test.ts @@ -33,6 +33,7 @@ const mocks = vi.hoisted(() => { delete process.env.BOTMUX_SESSION_ID; delete process.env.BOTMUX_LARK_APP_ID; let seq = 0; + const sessions = new Map(); return { replyMessage: vi.fn(async () => 'om_reply'), sendMessage: vi.fn(async () => 'om_top'), @@ -42,17 +43,32 @@ const mocks = vi.hoisted(() => { ? { openId, type: senderType === 'app' || senderType === 'bot' ? 'bot' as const : 'user' as const } : undefined )), - forkWorker: vi.fn(), - createSession: vi.fn((chatId: string, rootMessageId: string, title: string, chatType?: 'group' | 'p2p') => ({ - sessionId: `sess-fake-${++seq}`, - chatId, - rootMessageId, - title, - status: 'active' as const, - createdAt: new Date().toISOString(), - chatType, - })), - updateSession: vi.fn(), + sessions, + createSession: vi.fn((chatId: string, rootMessageId: string, title: string, chatType?: 'group' | 'p2p') => { + const session = { + sessionId: `sess-fake-${++seq}`, + chatId, + rootMessageId, + title, + status: 'active' as const, + createdAt: new Date().toISOString(), + chatType, + }; + sessions.set(session.sessionId, session); + return session; + }), + updateSession: vi.fn((session: any) => { sessions.set(session.sessionId, session); }), + getSession: vi.fn((sessionId: string) => sessions.get(sessionId)), + closeSession: vi.fn((sessionId: string) => { + const session = sessions.get(sessionId); + if (session) session.status = 'closed'; + }), + forkWorker: vi.fn((ds: any) => { + ds.worker = { killed: false, send: vi.fn() }; + }), + scanMultipleProjects: vi.fn(() => [] as any[]), + getAvailableBots: vi.fn(async () => [] as any[]), + downloadResources: vi.fn(async () => ({ attachments: [], needLogin: false })), }; }); @@ -68,7 +84,32 @@ vi.mock('../src/im/lark/client.js', async () => { vi.mock('../src/services/session-store.js', async () => { const actual = await vi.importActual('../src/services/session-store.js'); - return { ...actual, createSession: mocks.createSession, updateSession: mocks.updateSession }; + return { + ...actual, + createSession: mocks.createSession, + updateSession: mocks.updateSession, + getSession: mocks.getSession, + closeSession: mocks.closeSession, + }; +}); + +vi.mock('../src/core/worker-pool.js', async () => { + const actual = await vi.importActual('../src/core/worker-pool.js'); + return { ...actual, forkWorker: mocks.forkWorker }; +}); + +vi.mock('../src/core/session-manager.js', async () => { + const actual = await vi.importActual('../src/core/session-manager.js'); + return { + ...actual, + getAvailableBots: mocks.getAvailableBots, + downloadResources: mocks.downloadResources, + }; +}); + +vi.mock('../src/services/project-scanner.js', async () => { + const actual = await vi.importActual('../src/services/project-scanner.js'); + return { ...actual, scanMultipleProjects: mocks.scanMultipleProjects }; }); vi.mock('../src/im/lark/identity-cache.js', async () => { @@ -76,19 +117,26 @@ vi.mock('../src/im/lark/identity-cache.js', async () => { return { ...actual, resolveSender: (...args: any[]) => mocks.resolveSender(...args) }; }); -vi.mock('../src/core/worker-pool.js', async () => { - const actual = await vi.importActual('../src/core/worker-pool.js'); - return { ...actual, forkWorker: (...args: any[]) => mocks.forkWorker(...args) }; -}); - import { registerBot } from '../src/bot-registry.js'; -import { sessionKey } from '../src/core/types.js'; +import { sessionAnchorId, sessionKey } from '../src/core/types.js'; import { __testOnly_activeSessions as activeSessions, + __testOnly_claimNewDaemonSession as claimNewDaemonSession, + __testOnly_handleChatModeConverted as handleChatModeConverted, + __testOnly_handleDocComment as handleDocComment, __testOnly_handleNewTopic as handleNewTopic, __testOnly_handleThreadReply as handleThreadReply, + __testOnly_onQueuedActivationSubmitted as onQueuedActivationSubmitted, + __testOnly_prewarmDocCommentSession as prewarmDocCommentSession, + __testOnly_releaseQueuedActivationReservation as releaseQueuedActivationReservation, + __testOnly_reserveAsyncQueuedActivationTailAdmission as reserveAsyncQueuedActivationTailAdmission, + __testOnly_resetDocCommentClaims as resetDocCommentClaims, + __testOnly_settleAsyncQueuedActivationTailAdmission as settleAsyncQueuedActivationTailAdmission, } from '../src/daemon.js'; +import { admitQueuedActivationTail } from '../src/core/worker-pool.js'; import type { DaemonSession } from '../src/core/types.js'; +import { getDocSubscription, putDocSubscription, removeDocSubscription } from '../src/services/doc-subs-store.js'; +import { config } from '../src/config.js'; const APP = 'rename_route_app'; const CHAT = 'oc_rename_route_chat'; @@ -148,56 +196,47 @@ function seedThreadSession(anchor: string, title: string): DaemonSession { return ds; } -function seedLiveChatSession(send = vi.fn()): DaemonSession { - const ds = { - scope: 'chat', - chatId: CHAT, - chatType: 'group', +function makeChatSession(sessionId: string, chatId: string, options?: { + pendingLedger?: boolean; + pendingRepo?: boolean; + queued?: boolean; +}): DaemonSession { + const session = { + sessionId, + chatId, + rootMessageId: `om_${sessionId}`, + title: sessionId, + status: 'active' as const, + createdAt: NOW, larkAppId: APP, - worker: { killed: false, send }, + scope: 'chat' as const, + chatType: 'group' as const, + queued: options?.queued, + codexAppDispatchLedger: options?.pendingLedger ? [{ + dispatchId: `dispatch-${sessionId}`, + turnId: `turn-${sessionId}`, + dispatchAttempt: 1, + state: 'prepared' as const, + content: 'durable work', + deliverySink: 'lark' as const, + }] : undefined, + }; + mocks.sessions.set(sessionId, session); + return { + session, + worker: null, workerPort: null, workerToken: null, + larkAppId: APP, + chatId, + chatType: 'group', + scope: 'chat', spawnedAt: Date.now(), cliVersion: '1.0.0', lastMessageAt: Date.now(), hasHistory: false, - ownerOpenId: OWNER, - currentReplyTarget: { - rootMessageId: 'om_stale_root', - turnId: 'om_stale_turn', - updatedAt: NOW, - }, - session: { - sessionId: 'sess-live-chat-' + Math.random().toString(36).slice(2), - chatId: CHAT, - rootMessageId: 'om_original_root', - title: 'live chat', - status: 'active', - createdAt: NOW, - larkAppId: APP, - scope: 'chat', - quoteTargetId: 'om_stale_quote', - quoteTargetSenderOpenId: 'ou_stale_caller', - lastCallerOpenId: 'ou_stale_caller', - currentReplyTarget: { - rootMessageId: 'om_stale_root', - turnId: 'om_stale_turn', - updatedAt: NOW, - }, - }, - } as unknown as DaemonSession; - activeSessions.set(sessionKey(CHAT, APP), ds); - return ds; -} - -function seedPendingRawSession(anchor: string): DaemonSession { - const ds = seedThreadSession(anchor, 'pending raw'); - ds.pendingRepo = true; - ds.pendingPrompt = ''; - ds.pendingRawInput = '/goal start'; - ds.pendingRawTurnId = 'om_initial_raw'; - ds.pendingSender = { openId: OWNER, type: 'user' }; - return ds; + pendingRepo: options?.pendingRepo, + } as DaemonSession; } /** All text replied through the mocked Lark client in this test, joined. */ @@ -213,14 +252,16 @@ describe('/rename production routing — must not pre-create a session (review P mocks.replyMessage.mockResolvedValue('om_reply'); mocks.sendMessage.mockResolvedValue('om_top'); mocks.getChatMode.mockResolvedValue('group'); - activeSessions.clear(); - const bot = registerBot({ - larkAppId: APP, - larkAppSecret: 's', - cliId: 'claude-code', - allowedUsers: [OWNER], - oncallChats: [{ chatId: CHAT, workingDir: '/tmp' }], + mocks.sessions.clear(); + mocks.forkWorker.mockImplementation((ds: any) => { + ds.worker = { killed: false, send: vi.fn() }; }); + mocks.scanMultipleProjects.mockReturnValue([]); + mocks.getAvailableBots.mockResolvedValue([]); + mocks.downloadResources.mockResolvedValue({ attachments: [], needLogin: false }); + activeSessions.clear(); + resetDocCommentClaims(); + const bot = registerBot({ larkAppId: APP, larkAppSecret: 's', cliId: 'claude-code', allowedUsers: [OWNER] }); bot.resolvedAllowedUsers = [OWNER]; }); @@ -297,95 +338,1053 @@ describe('/rename production routing — must not pre-create a session (review P expect(activeSessions.has(sessionKey('om_new_2', APP))).toBe(true); }); - it('new topic: passes the accepted Lark message id into the first worker', async () => { + it('routes a colliding daemon command to the canonical pending owner and closes only the loser', async () => { + const anchor = 'om_pending_owner'; + const incumbent = seedThreadSession(anchor, 'durable owner'); + incumbent.session.codexAppDispatchLedger = [{ + dispatchId: 'dispatch-incumbent', + turnId: 'turn-incumbent', + dispatchAttempt: 1, + state: 'prepared', + content: 'accepted input', + deliverySink: 'lark', + }]; + + await handleNewTopic(makeEventData(anchor, '/status'), makeCtx(anchor, anchor)); + + expect(activeSessions.get(sessionKey(anchor, APP))).toBe(incumbent); + expect(mocks.createSession).toHaveBeenCalledTimes(1); + const incomingId = mocks.createSession.mock.results[0]!.value.sessionId; + expect(mocks.closeSession).toHaveBeenCalledWith(incomingId); + expect(mocks.closeSession).not.toHaveBeenCalledWith(incumbent.session.sessionId); + expect(repliedText()).toContain(`Session: ${incumbent.session.sessionId}`); + }); + + it('delivers an ordinary loser turn to the canonical owner exactly once', async () => { + const anchor = 'om_collision_delivery'; + const incumbent = seedThreadSession(anchor, 'canonical owner'); + const send = vi.fn(); + incumbent.worker = { killed: false, send } as any; + await handleNewTopic( - makeEventData('om_workflow_new', '/workflow new 修复首轮授权'), - makeCtx('om_workflow_new', 'om_workflow_new'), + makeEventData(anchor, 'deliver this once'), + makeCtx(anchor, anchor), ); - expect(mocks.forkWorker).toHaveBeenCalledTimes(1); - expect(mocks.forkWorker.mock.calls[0]?.[2]).toEqual({ turnId: 'om_workflow_new' }); + expect(activeSessions.get(sessionKey(anchor, APP))).toBe(incumbent); + expect(mocks.createSession).toHaveBeenCalledTimes(1); + expect(mocks.closeSession).toHaveBeenCalledWith( + mocks.createSession.mock.results[0]!.value.sessionId, + ); + const inputCalls = send.mock.calls.filter(call => call[0]?.type === 'message'); + expect(inputCalls).toHaveLength(1); + expect(JSON.stringify(inputCalls[0]![0])).toContain('deliver this once'); }); - it('thread safety-net: passes the accepted reply id into the first worker', async () => { + it('buffers a later turn while the winning initial start is paused, then forks once in input order', async () => { + registerBot({ + larkAppId: APP, + larkAppSecret: 's', + cliId: 'claude-code', + allowedUsers: [OWNER], + defaultWorkingDir: '/tmp', + }).resolvedAllowedUsers = [OWNER]; + + let announcePreparation!: () => void; + const preparationStarted = new Promise(resolve => { announcePreparation = resolve; }); + let releasePreparation!: (bots: any[]) => void; + const preparationGate = new Promise(resolve => { releasePreparation = resolve; }); + mocks.getAvailableBots.mockImplementationOnce(async () => { + announcePreparation(); + return preparationGate; + }); + + const anchor = 'om_initial_order_root'; + const first = handleNewTopic( + makeEventData(anchor, 'first task'), + makeCtx(anchor, anchor), + ); + await preparationStarted; + + const owner = activeSessions.get(sessionKey(anchor, APP))!; + expect(owner.initialStartPending).toBe(true); + expect(owner.worker).toBeNull(); + await handleThreadReply( - makeEventData('om_workflow_reply', '/workflow new 修复首轮授权', 'om_fresh_root'), - makeCtx('om_fresh_root', 'om_workflow_reply'), + makeEventData('om_initial_order_second', 'second task', anchor), + makeCtx(anchor, 'om_initial_order_second'), ); + expect(mocks.forkWorker).not.toHaveBeenCalled(); + expect(owner.pendingFollowUps).toBeUndefined(); + expect(owner.session.queuedActivationTail).toEqual([ + expect.objectContaining({ + turnId: 'om_initial_order_second', + cliInput: expect.objectContaining({ + content: expect.stringContaining('second task'), + }), + }), + ]); + + releasePreparation([]); + await first; + expect(mocks.forkWorker).toHaveBeenCalledTimes(1); - expect(mocks.forkWorker.mock.calls[0]?.[2]).toEqual({ turnId: 'om_workflow_reply' }); + const openingInput = mocks.forkWorker.mock.calls[0]![1]; + expect(openingInput.content.indexOf('first task')).toBeGreaterThanOrEqual(0); + expect(openingInput.content).not.toContain('second task'); + expect(owner.worker!.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'message', + turnId: 'om_initial_order_second', + content: expect.stringContaining('second task'), + queuedActivationToken: expect.any(String), + })); + expect(owner.initialStartPending).toBe(true); + expect(owner.pendingFollowUps).toBeUndefined(); }); - it('live passthrough binds raw input and reply metadata to the accepted message', async () => { + it('keeps the no-project text fallback gated until its queued activation ACK', async () => { + const anchor = 'om_no_project_text_cutpoint'; + const openingToken = 'token-no-project-text'; const send = vi.fn(); - const ds = seedLiveChatSession(send); - const messageId = 'om_model_turn'; - const replyRootId = 'om_model_reply_root'; + mocks.forkWorker.mockImplementationOnce((owner: any, input: any) => { + expect(owner.session.queued).toBe(true); + expect(input.content).toContain('OPENING_TEXT_N'); + Object.assign(owner.session, { + queued: false, + queuedActivationPending: true, + queuedActivationToken: openingToken, + queuedActivationInput: input, + queuedActivationTurnId: anchor, + queuedActivationResume: false, + }); + owner.worker = { killed: false, send }; + }); + + await handleNewTopic( + makeEventData(anchor, 'OPENING_TEXT_N'), + makeCtx(anchor, anchor), + ); + + const owner = activeSessions.get(sessionKey(anchor, APP))!; + expect(mocks.scanMultipleProjects).toHaveBeenCalled(); + expect(owner.pendingRepo).toBe(false); + expect(owner.initialStartPending).toBe(true); + expect(owner.session.queuedActivationToken).toBe(openingToken); await handleThreadReply( - makeEventData(messageId, '/model opus', replyRootId), - { - chatId: CHAT, - messageId, - chatType: 'group' as const, - scope: 'chat' as const, - anchor: CHAT, - replyRootId, - larkAppId: APP, + makeEventData('om_no_project_text_n1', 'FOLLOWER_TEXT_N_PLUS_1', anchor), + makeCtx(anchor, 'om_no_project_text_n1'), + ); + + expect(send).not.toHaveBeenCalled(); + expect(owner.session.queuedActivationTail).toEqual([ + expect.objectContaining({ + turnId: 'om_no_project_text_n1', + cliInput: expect.objectContaining({ + content: expect.stringContaining('FOLLOWER_TEXT_N_PLUS_1'), + }), + }), + ]); + + Object.assign(owner.session, { + queuedActivationPending: undefined, + queuedActivationToken: undefined, + queuedActivationInput: undefined, + queuedActivationTurnId: undefined, + queuedActivationResume: undefined, + pendingRepoSetup: undefined, + }); + expect(onQueuedActivationSubmitted(owner, openingToken)).toBe(true); + expect(send).toHaveBeenCalledTimes(1); + expect(send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'message', + turnId: 'om_no_project_text_n1', + content: expect.stringContaining('FOLLOWER_TEXT_N_PLUS_1'), + queuedActivationToken: expect.any(String), + })); + }); + + it('keeps the no-project raw fallback gated through raw text-to-Enter ACK', async () => { + const anchor = 'om_no_project_raw_cutpoint'; + const openingToken = 'token-no-project-raw'; + const rawOpening = '/goal OPENING_RAW_N'; + const send = vi.fn(); + mocks.forkWorker.mockImplementationOnce((owner: any, input: any) => { + expect(owner.session.queued).toBe(true); + expect(input).toBe(''); + expect(owner.pendingRawInput).toBe(rawOpening); + Object.assign(owner.session, { + queued: false, + queuedActivationPending: true, + queuedActivationToken: openingToken, + queuedActivationInput: { content: '' }, + queuedActivationTurnId: anchor, + queuedActivationResume: false, + }); + owner.worker = { killed: false, send }; + }); + + await handleNewTopic( + makeEventData(anchor, rawOpening), + makeCtx(anchor, anchor), + ); + + const owner = activeSessions.get(sessionKey(anchor, APP))!; + expect(mocks.scanMultipleProjects).toHaveBeenCalled(); + expect(owner.pendingRepo).toBe(false); + expect(owner.initialStartPending).toBe(true); + expect(owner.session.queuedActivationToken).toBe(openingToken); + + await handleThreadReply( + makeEventData('om_no_project_raw_n1', 'FOLLOWER_AFTER_RAW_N_PLUS_1', anchor), + makeCtx(anchor, 'om_no_project_raw_n1'), + ); + + // The follower must stay durable until the adapter confirms that both the + // raw command text and its Enter beat were submitted. + expect(send).not.toHaveBeenCalled(); + expect(owner.session.queuedActivationTail).toEqual([ + expect.objectContaining({ + turnId: 'om_no_project_raw_n1', + cliInput: expect.objectContaining({ + content: expect.stringContaining('FOLLOWER_AFTER_RAW_N_PLUS_1'), + }), + }), + ]); + + Object.assign(owner.session, { + queuedActivationPending: undefined, + queuedActivationToken: undefined, + queuedActivationInput: undefined, + queuedActivationTurnId: undefined, + queuedActivationResume: undefined, + pendingRepoSetup: undefined, + }); + expect(onQueuedActivationSubmitted(owner, openingToken)).toBe(true); + expect(send).toHaveBeenCalledTimes(1); + expect(send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'message', + turnId: 'om_no_project_raw_n1', + content: expect.stringContaining('FOLLOWER_AFTER_RAW_N_PLUS_1'), + queuedActivationToken: expect.any(String), + })); + }); + + it('refuses an existing raw CLI passthrough while activation admission owns the route', async () => { + const anchor = 'om_passthrough_activation'; + const owner = seedThreadSession(anchor, 'activation owner'); + const send = vi.fn(); + owner.worker = { killed: false, send } as any; + Object.assign(owner.session, { + queuedActivationPending: true, + queuedActivationToken: 'passthrough-token', + queuedActivationInput: { content: 'OPENING_N' }, + queuedActivationTurnId: 'turn-opening', + }); + + await handleThreadReply( + makeEventData('om_passthrough_n1', '/model opus', anchor), + makeCtx(anchor, 'om_passthrough_n1'), + ); + + expect(send).not.toHaveBeenCalled(); + expect(owner.currentTurnTitle).toBeUndefined(); + expect(owner.session.queuedActivationToken).toBe('passthrough-token'); + expect(repliedText()).toContain('仍在提交中'); + }); + + it.each([ + ['close preparing', { riffCloseState: { phase: 'preparing', requestId: 'close-preparing' } }], + ['close prepared', { riffCloseState: { phase: 'prepared', requestId: 'close-prepared' } }], + ['shutdown prepared', { + riffShutdownState: { + phase: 'prepared', requestId: 'shutdown-prepared', taskId: 'task-fenced', }, + }], + ] as const)('refuses raw CLI passthrough during Riff retirement (%s)', async (_label, state) => { + const anchor = `om_passthrough_${_label.replace(/ /g, '_')}`; + const owner = seedThreadSession(anchor, 'retirement owner'); + const send = vi.fn(); + owner.worker = { killed: false, send } as any; + Object.assign(owner, state); + const priorActivity = owner.lastMessageAt; + + await handleThreadReply( + makeEventData(`om_passthrough_turn_${_label}`, '/model opus', anchor), + makeCtx(anchor, `om_passthrough_turn_${_label}`), ); - expect(send).toHaveBeenCalledWith({ - type: 'raw_input', - content: '/model opus', - turnId: messageId, + expect(send).not.toHaveBeenCalled(); + expect(owner.currentTurnTitle).toBeUndefined(); + expect(owner.lastMessageAt).toBe(priorActivity); + expect(repliedText()).toContain('Riff'); + }); + + it('preserves an ordinary inbound turn before acceptance after the fenced worker exited', async () => { + const anchor = 'om_workerless_shutdown_fence'; + const owner = seedThreadSession(anchor, 'workerless fenced owner'); + owner.worker = null; + owner.session.backendType = 'riff'; + owner.riffShutdownState = { + phase: 'prepared', + requestId: 'shutdown-worker-exited', + taskId: 'task-unverified', + }; + + await handleThreadReply( + makeEventData('om_workerless_fenced_turn', 'MUST_RETRY_EXACTLY', anchor), + makeCtx(anchor, 'om_workerless_fenced_turn'), + ); + + expect(mocks.forkWorker).not.toHaveBeenCalled(); + expect(owner.riffShutdownState).toMatchObject({ requestId: 'shutdown-worker-exited' }); + expect(repliedText()).toContain('Riff'); + }); + + it('replays a retained queued activation exactly, then releases the later inbound with its own turn id', async () => { + const anchor = 'om_reparked_activation_root'; + const ds = seedThreadSession(anchor, 're-parked activation'); + const exactOpening = { + content: 'BACKLOG_N\n\nPRIOR_TRIGGER_REPLY', + }; + Object.assign(ds.session, { + cliId: 'claude-code', + workingDir: '/tmp', + queued: true, + queuedPrompt: 'STALE_REBUILD_SOURCE_MUST_NOT_BE_USED', + queuedActivationInput: exactOpening, + queuedActivationTurnId: 'om_prior_trigger', + queuedActivationDispatchAttempt: 4, }); - expect(ds.session.quoteTargetId).toBe(messageId); - expect(ds.session.quoteTargetSenderOpenId).toBe(OWNER); - expect(ds.session.lastCallerOpenId).toBe(OWNER); - expect(ds.currentReplyTarget).toMatchObject({ rootMessageId: replyRootId, turnId: messageId }); - expect(ds.session.currentReplyTarget).toMatchObject({ rootMessageId: replyRootId, turnId: messageId }); - expect(mocks.updateSession).toHaveBeenCalledWith(ds.session); + ds.workingDir = '/tmp'; + + await handleThreadReply( + makeEventData('om_later_n_plus_1', 'LATER_REPLY_N_PLUS_1', anchor), + makeCtx(anchor, 'om_later_n_plus_1'), + ); + + expect(mocks.forkWorker).toHaveBeenCalledTimes(1); + expect(mocks.forkWorker).toHaveBeenCalledWith(ds, exactOpening, { + resume: false, + turnId: 'om_prior_trigger', + dispatchAttempt: 4, + }); + expect(mocks.forkWorker.mock.calls[0]![1]).toBe(exactOpening); + expect(JSON.stringify(mocks.forkWorker.mock.calls[0]![1])) + .not.toContain('STALE_REBUILD_SOURCE_MUST_NOT_BE_USED'); + expect(ds.session.queuedActivationTail).toEqual([ + expect.objectContaining({ + turnId: 'om_later_n_plus_1', + userPrompt: expect.stringContaining('LATER_REPLY_N_PLUS_1'), + cliInput: expect.objectContaining({ + content: expect.stringContaining('LATER_REPLY_N_PLUS_1'), + }), + }), + ]); + + const send = vi.mocked(ds.worker!.send); + expect(releaseQueuedActivationReservation(ds)).toBe(true); + expect(send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'message', + turnId: 'om_later_n_plus_1', + content: expect.stringContaining('LATER_REPLY_N_PLUS_1'), + queuedActivationToken: expect.any(String), + })); + expect(send).toHaveBeenCalledTimes(1); + expect(mocks.forkWorker.mock.invocationCallOrder[0]) + .toBeLessThan(send.mock.invocationCallOrder[0]!); + expect(ds.session.queuedActivationTail).toBeUndefined(); + expect(ds.session.queuedActivationPending).toBe(true); + expect(ds.session.queuedActivationInput?.content).toContain('LATER_REPLY_N_PLUS_1'); + const successorToken = ds.session.queuedActivationToken!; + // The worker-pool clears the tokened journal durably before invoking the + // daemon callback. Model that adapter ACK boundary explicitly here. + Object.assign(ds.session, { + queuedActivationPending: undefined, + queuedActivationToken: undefined, + queuedActivationInput: undefined, + queuedActivationTurnId: undefined, + queuedActivationDispatchAttempt: undefined, + queuedActivationResume: undefined, + }); + expect(onQueuedActivationSubmitted(ds, successorToken)).toBe(true); + expect(ds.initialStartPending).toBe(false); }); - it('pending raw same-caller follow-up rotates both staged turns to the latest message', async () => { - const anchor = 'om_pending_raw_root'; - const ds = seedPendingRawSession(anchor); - const messageId = 'om_pending_raw_followup'; + it('atomically claims a fresh queued refork so a concurrent reply buffers behind its owner', async () => { + const anchor = 'om_fresh_queued_claim_root'; + const ds = seedThreadSession(anchor, 'fresh queued claim'); + Object.assign(ds.session, { + cliId: 'claude-code', + workingDir: '/tmp', + queued: true, + queuedPrompt: 'BACKLOG_N', + }); + ds.workingDir = '/tmp'; + + let announceFirstDownload!: () => void; + const firstDownloadStarted = new Promise(resolve => { announceFirstDownload = resolve; }); + let releaseFirstDownload!: () => void; + const firstDownloadGate = new Promise(resolve => { releaseFirstDownload = resolve; }); + mocks.downloadResources.mockImplementationOnce(async () => { + announceFirstDownload(); + await firstDownloadGate; + return { attachments: [], needLogin: false }; + }); + + const first = handleThreadReply( + makeEventData('om_fresh_owner_n', 'OWNER_REPLY_N', anchor), + makeCtx(anchor, 'om_fresh_owner_n'), + ); + await firstDownloadStarted; + + expect(ds.initialStartPending).toBe(true); + expect(ds.initialStartClaimToken).toEqual(expect.any(String)); + const ownerToken = ds.initialStartClaimToken; + + const follower = handleThreadReply( + makeEventData('om_fresh_follower_n1', 'FOLLOWER_REPLY_N_PLUS_1', anchor), + makeCtx(anchor, 'om_fresh_follower_n1'), + ); + expect(mocks.forkWorker).not.toHaveBeenCalled(); + expect(ds.initialStartClaimToken).toBe(ownerToken); + + releaseFirstDownload(); + await first; + await follower; + + expect(ds.pendingFollowUps).toBeUndefined(); + expect(ds.session.queuedActivationTail).toEqual([ + expect.objectContaining({ + turnId: 'om_fresh_follower_n1', + cliInput: expect.objectContaining({ + content: expect.stringContaining('FOLLOWER_REPLY_N_PLUS_1'), + }), + }), + ]); + + expect(mocks.forkWorker).toHaveBeenCalledTimes(1); + const opening = mocks.forkWorker.mock.calls[0]![1]; + expect(opening.content.indexOf('BACKLOG_N')).toBeGreaterThanOrEqual(0); + expect(opening.content.indexOf('OWNER_REPLY_N')).toBeGreaterThan(opening.content.indexOf('BACKLOG_N')); + expect(opening.content).not.toContain('FOLLOWER_REPLY_N_PLUS_1'); + expect(ds.initialStartPending).toBe(true); + expect(ds.initialStartClaimToken).toBe(ownerToken); + + const send = vi.mocked(ds.worker!.send); + expect(onQueuedActivationSubmitted(ds)).toBe(true); + expect(send).toHaveBeenCalledTimes(1); + expect(send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'message', + turnId: 'om_fresh_follower_n1', + content: expect.stringContaining('FOLLOWER_REPLY_N_PLUS_1'), + queuedActivationToken: expect.any(String), + })); + expect(ds.session.queuedActivationPending).toBe(true); + expect(ds.session.queuedActivationInput?.content).toContain('FOLLOWER_REPLY_N_PLUS_1'); + const successorToken = ds.session.queuedActivationToken!; + Object.assign(ds.session, { + queuedActivationPending: undefined, + queuedActivationToken: undefined, + queuedActivationInput: undefined, + queuedActivationTurnId: undefined, + queuedActivationDispatchAttempt: undefined, + queuedActivationResume: undefined, + }); + expect(onQueuedActivationSubmitted(ds, successorToken)).toBe(true); + expect(ds.initialStartPending).toBe(false); + expect(ds.initialStartClaimToken).toBeUndefined(); + }); + + it.each([ + { arrivalGate: true, laterGate: false, expectsSidecar: true, label: 'ON→OFF' }, + { arrivalGate: false, laterGate: true, expectsSidecar: false, label: 'OFF→ON' }, + ])( + 'freezes a queued follower clean-input decision at reservation time ($label)', + ({ arrivalGate, laterGate, expectsSidecar }) => { + const bot = registerBot({ + larkAppId: APP, + larkAppSecret: 's', + cliId: 'codex-app', + codexAppCleanInput: arrivalGate, + allowedUsers: [OWNER], + }); + bot.resolvedAllowedUsers = [OWNER]; + const ds = seedThreadSession(`om_gate_${arrivalGate}`, 'clean-input reservation'); + const send = vi.fn(); + ds.worker = { killed: false, send } as any; + ds.initialStartPending = true; + ds.hasHistory = true; + ds.session.cliId = 'codex-app'; + + const reservation = reserveAsyncQueuedActivationTailAdmission(ds); + expect(ds.queuedActivationTailAdmissionsOutstanding).toBe(1); + bot.config.codexAppCleanInput = laterGate; + + // Model N's ACK landing while N+1 is still awaiting prompt materialization. + expect(releaseQueuedActivationReservation(ds, 'opening-token')).toBe(false); + const sidecar = { + text: 'FOLLOWER_CLEAN_N1', + additionalContext: { + hidden: { kind: 'application' as const, value: 'arrival' }, + }, + }; + admitQueuedActivationTail(ds, { + userPrompt: 'FOLLOWER_CLEAN_N1', + cliInput: { + content: 'FOLLOWER_LEGACY_N1', + codexAppInput: sidecar, + }, + turnId: 'turn-clean-follower', + dispatchAttempt: 2, + }, reservation); + settleAsyncQueuedActivationTailAdmission(ds); + + const expectedSidecar = expectsSidecar + ? { ...sidecar, clientUserMessageId: 'turn-clean-follower' } + : undefined; + expect(ds.queuedActivationTailAdmissionsOutstanding).toBeUndefined(); + expect(ds.queuedActivationTailReleasePending).toBeUndefined(); + expect(ds.session.queuedActivationTail).toBeUndefined(); + expect(ds.session.queuedActivationInput?.codexAppInput).toEqual(expectedSidecar); + expect(ds.session.codexAppDispatchLedger?.at(-1)?.codexAppInput) + .toEqual(expectedSidecar); + expect(ds.session.lastCodexAppInput).toEqual(expectedSidecar); + expect(send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'message', + turnId: 'turn-clean-follower', + ...(expectedSidecar ? { codexAppInput: expectedSidecar } : {}), + })); + if (!expectedSidecar) { + expect(send.mock.calls[0]![0]).not.toHaveProperty('codexAppInput'); + } + }, + ); + + it('parks a late follower after ACK→worker-exit and reforks it before the next inbound', async () => { + const anchor = 'om_ack_exit_late_admission'; + const ds = seedThreadSession(anchor, 'ACK exit late admission'); + ds.session.cliId = 'claude-code'; + ds.workingDir = '/tmp'; + ds.session.workingDir = '/tmp'; + ds.hasHistory = true; + ds.initialStartPending = true; + ds.worker = { killed: false, send: vi.fn() } as any; + + const reservation = reserveAsyncQueuedActivationTailAdmission(ds); + expect(releaseQueuedActivationReservation(ds, 'opening-token')).toBe(false); + // N's worker exits after its ACK but before the reserved N+1 finishes. + ds.worker = null; + admitQueuedActivationTail(ds, { + userPrompt: 'LATE_N_PLUS_1', + cliInput: { content: 'LATE_N_PLUS_1' }, + turnId: 'turn-late-n1', + }, reservation); + settleAsyncQueuedActivationTailAdmission(ds); + + expect(ds.session).toMatchObject({ + queuedActivationPending: true, + queuedActivationInput: { content: 'LATE_N_PLUS_1' }, + queuedActivationTurnId: 'turn-late-n1', + }); + expect(ds.session.queuedActivationTail).toBeUndefined(); + expect(ds.initialStartPending).toBe(false); + expect(ds.initialStartClaimToken).toBeUndefined(); + expect(ds.queuedActivationTailReleaseRetryTimer).toBeUndefined(); + + mocks.forkWorker.mockClear(); await handleThreadReply( - makeEventData(messageId, '补充同一个人的要求', anchor), - makeCtx(anchor, messageId), + makeEventData('turn-after-late-n2', 'AFTER_LATE_N_PLUS_2', anchor), + makeCtx(anchor, 'turn-after-late-n2'), ); - expect(ds.pendingRawTurnId).toBe(messageId); - expect(ds.pendingFollowUpTurnId).toBe(messageId); - expect(ds.pendingFollowUps).toHaveLength(1); - expect(ds.session.quoteTargetId).toBe(messageId); + expect(mocks.forkWorker).toHaveBeenCalledTimes(1); + expect(mocks.forkWorker.mock.calls[0]![1]).toBe(ds.session.queuedActivationInput); + expect(mocks.forkWorker.mock.calls[0]![1].content).toBe('LATE_N_PLUS_1'); + expect(ds.session.queuedActivationTail).toEqual([ + expect.objectContaining({ + turnId: 'turn-after-late-n2', + cliInput: expect.objectContaining({ + content: expect.stringContaining('AFTER_LATE_N_PLUS_2'), + }), + }), + ]); }); - it('pending raw mixed-caller follow-up clears both staged turns', async () => { - const anchor = 'om_pending_raw_mixed_root'; - const ds = seedPendingRawSession(anchor); - const ownerMessageId = 'om_pending_owner_followup'; + it('releases the route when a post-ACK async follower admission fails', () => { + const ds = seedThreadSession('om_failed_late_admission', 'failed late admission'); + ds.session.cliId = 'claude-code'; + ds.initialStartPending = true; + ds.worker = { killed: false, send: vi.fn() } as any; + const reservation = reserveAsyncQueuedActivationTailAdmission(ds); + expect(releaseQueuedActivationReservation(ds, 'opening-token')).toBe(false); + mocks.updateSession.mockImplementationOnce(() => { + throw new Error('tail persistence unavailable'); + }); + expect(() => { + try { + admitQueuedActivationTail(ds, { + userPrompt: 'FAILED_N_PLUS_1', + cliInput: { content: 'FAILED_N_PLUS_1' }, + turnId: 'turn-failed-n1', + }, reservation); + } finally { + settleAsyncQueuedActivationTailAdmission(ds); + } + }).toThrow('tail persistence unavailable'); + + expect(ds.session.queuedActivationTail).toBeUndefined(); + expect(ds.queuedActivationTailAdmissionsOutstanding).toBeUndefined(); + expect(ds.queuedActivationTailReleasePending).toBeUndefined(); + expect(ds.initialStartPending).toBe(false); + expect(ds.initialStartClaimToken).toBeUndefined(); + expect(ds.queuedActivationTailReleaseRetryTimer).toBeUndefined(); + }); + + it('releases a failed queued-refork claim so a later inbound can become the owner', async () => { + const anchor = 'om_failed_queued_claim_root'; + const ds = seedThreadSession(anchor, 'failed queued claim'); + Object.assign(ds.session, { + cliId: 'claude-code', + workingDir: '/tmp', + queued: true, + queuedPrompt: 'BACKLOG_RETRY', + }); + ds.workingDir = '/tmp'; + mocks.forkWorker.mockImplementationOnce(() => { + throw new Error('pre-fork acceptance failed'); + }); + + await expect(handleThreadReply( + makeEventData('om_failed_owner', 'FAILED_OWNER_REPLY', anchor), + makeCtx(anchor, 'om_failed_owner'), + )).rejects.toThrow('pre-fork acceptance failed'); + + expect(ds.worker).toBeNull(); + expect(ds.initialStartPending).toBe(false); + expect(ds.initialStartClaimToken).toBeUndefined(); + + mocks.forkWorker.mockImplementation((owner: any) => { + owner.worker = { killed: false, send: vi.fn() }; + }); await handleThreadReply( - makeEventData(ownerMessageId, '先由原调用者补充', anchor), - makeCtx(anchor, ownerMessageId), + makeEventData('om_retry_owner', 'RETRY_OWNER_REPLY', anchor), + makeCtx(anchor, 'om_retry_owner'), ); - expect(ds.pendingRawTurnId).toBe(ownerMessageId); - expect(ds.pendingFollowUpTurnId).toBe(ownerMessageId); - - const strangerMessageId = 'om_pending_stranger_followup'; - const strangerData = makeEventData(strangerMessageId, '再由另一个人补充', anchor); - strangerData.sender = { sender_id: { open_id: 'ou_stranger' }, sender_type: 'user' }; - await handleThreadReply(strangerData, makeCtx(anchor, strangerMessageId)); - - expect(ds.pendingRawTurnId).toBeUndefined(); - expect(ds.pendingFollowUpTurnId).toBeUndefined(); - expect(ds.pendingFollowUps).toHaveLength(2); - expect(ds.session.quoteTargetId).toBe(strangerMessageId); - expect(ds.session.lastCallerOpenId).toBe('ou_stranger'); + + expect(mocks.forkWorker).toHaveBeenCalledTimes(2); + expect(mocks.forkWorker.mock.calls[1]![1].content).toContain('RETRY_OWNER_REPLY'); + expect(ds.initialStartPending).toBe(true); + expect(ds.initialStartClaimToken).toEqual(expect.any(String)); + }); + + it('retains a tokened generic ACK successor after IPC failure and recovers it before the next inbound', async () => { + const anchor = 'om_ack_tail_handoff_root'; + const ds = seedThreadSession(anchor, 'ACK tail handoff'); + Object.assign(ds.session, { + cliId: 'claude-code', + workingDir: '/tmp', + queued: false, + }); + ds.workingDir = '/tmp'; + ds.hasHistory = true; + ds.initialStartPending = true; + ds.pendingFollowUps = ['GENERIC_TAIL_N_PLUS_1']; + ds.pendingFollowUpTurnIds = ['om_generic_tail_n_plus_1']; + const failedSend = vi.fn(() => { throw new Error('worker exited before accepting tail'); }); + const kill = vi.fn(); + ds.worker = { killed: false, send: failedSend, kill } as any; + + // Promotion is already a durable acceptance boundary: an IPC throw fences + // this child but keeps one tokened journal owner for exact recovery. + expect(onQueuedActivationSubmitted(ds)).toBe(true); + expect(failedSend).toHaveBeenCalledTimes(1); + expect(kill).toHaveBeenCalledTimes(1); + expect(ds.pendingFollowUps).toBeUndefined(); + expect(ds.session.queuedActivationTail).toBeUndefined(); + expect(ds.session).toMatchObject({ + queued: false, + queuedActivationPending: true, + queuedActivationToken: expect.any(String), + queuedActivationTurnId: 'om_generic_tail_n_plus_1', + queuedActivationInput: expect.objectContaining({ + content: expect.stringContaining('GENERIC_TAIL_N_PLUS_1'), + }), + }); + const retainedToken = ds.session.queuedActivationToken; + // Model the worker error/exit fence (the route test uses a lightweight + // child stub without worker-pool event handlers). + ds.worker = null; + ds.initialStartPending = false; + ds.initialStartClaimToken = undefined; + expect(ds.initialStartPending).toBe(false); + + mocks.forkWorker.mockClear(); + await handleThreadReply( + makeEventData('om_after_tail_n_plus_2', 'AFTER_TAIL_N_PLUS_2', anchor), + makeCtx(anchor, 'om_after_tail_n_plus_2'), + ); + + expect(mocks.forkWorker).toHaveBeenCalledTimes(1); + const reforkedHead = mocks.forkWorker.mock.calls[0]![1]; + expect(reforkedHead).toBe(ds.session.queuedActivationInput); + expect(reforkedHead.content).toContain('GENERIC_TAIL_N_PLUS_1'); + expect(reforkedHead.content).not.toContain('AFTER_TAIL_N_PLUS_2'); + expect(ds.session.queuedActivationTail).toEqual([ + expect.objectContaining({ turnId: 'om_after_tail_n_plus_2' }), + ]); + + const resumedSend = vi.mocked(ds.worker!.send); + Object.assign(ds.session, { + queuedActivationPending: undefined, + queuedActivationToken: undefined, + queuedActivationInput: undefined, + queuedActivationTurnId: undefined, + queuedActivationDispatchAttempt: undefined, + queuedActivationResume: undefined, + }); + expect(onQueuedActivationSubmitted(ds, retainedToken)).toBe(true); + expect(resumedSend).toHaveBeenCalledTimes(1); + expect(resumedSend).toHaveBeenCalledWith(expect.objectContaining({ + type: 'message', + turnId: 'om_after_tail_n_plus_2', + content: expect.stringContaining('AFTER_TAIL_N_PLUS_2'), + queuedActivationToken: expect.any(String), + })); + }); +}); + +describe('daemon live-session registration claims', () => { + beforeEach(() => { + vi.clearAllMocks(); + mocks.sessions.clear(); + mocks.getAvailableBots.mockResolvedValue([]); + }); + + it('is first-owner-wins under concurrent creation and closes the empty loser', async () => { + const registry = new Map(); + const first = makeChatSession('claim-first', 'oc_claim'); + const second = makeChatSession('claim-second', 'oc_claim'); + + const [firstResult, secondResult] = await Promise.all([ + claimNewDaemonSession(registry, first), + claimNewDaemonSession(registry, second), + ]); + + expect(firstResult.accepted).toBe(true); + expect(secondResult).toMatchObject({ + accepted: false, + reason: 'existing_owner', + owner: first, + closedIncomingSessionId: second.session.sessionId, + }); + expect(registry.get(sessionKey('oc_claim', APP))).toBe(first); + expect(mocks.closeSession).toHaveBeenCalledTimes(1); + expect(mocks.closeSession).toHaveBeenCalledWith(second.session.sessionId); + }); + + it('fails closed and preserves both persistence rows when both owners are pending', async () => { + const registry = new Map(); + const incumbent = makeChatSession('claim-pending-a', 'oc_pending', { pendingLedger: true }); + const incoming = makeChatSession('claim-pending-b', 'oc_pending', { pendingLedger: true }); + registry.set(sessionKey('oc_pending', APP), incumbent); + + const result = await claimNewDaemonSession(registry, incoming); + + expect(result).toMatchObject({ + accepted: false, + reason: 'both_pending', + owner: incumbent, + preservedIncomingSessionId: incoming.session.sessionId, + }); + expect(registry.get(sessionKey('oc_pending', APP))).toBe(incumbent); + expect(mocks.closeSession).not.toHaveBeenCalled(); + expect(mocks.sessions.get(incoming.session.sessionId)?.status).toBe('active'); + }); + + it('retires a losing group-join-style pending runtime before any durable setup is staged', async () => { + const registry = new Map(); + const incumbent = makeChatSession('join-canonical', 'oc_join_claim'); + const duplicate = makeChatSession('join-duplicate', 'oc_join_claim'); + duplicate.pendingRepo = true; + duplicate.pendingPrompt = 'SYNTHETIC_JOIN_OPENING'; + duplicate.initialStartPending = true; + registry.set(sessionKey('oc_join_claim', APP), incumbent); + + const result = await claimNewDaemonSession(registry, duplicate); + + expect(result).toMatchObject({ + accepted: false, + reason: 'existing_owner', + owner: incumbent, + closedIncomingSessionId: duplicate.session.sessionId, + }); + expect(duplicate.session.pendingRepoSetup).toBeUndefined(); + expect(mocks.closeSession).toHaveBeenCalledWith(duplicate.session.sessionId); + expect(registry.get(sessionKey('oc_join_claim', APP))).toBe(incumbent); + }); +}); + +describe('chat-mode conversion ownership', () => { + beforeEach(() => { + vi.clearAllMocks(); + mocks.sessions.clear(); + mocks.getAvailableBots.mockResolvedValue([]); + activeSessions.clear(); + }); + + it.each([ + ['durable ledger', { pendingLedger: true }], + ['repo selection', { pendingRepo: true }], + ['queued dashboard task', { queued: true }], + ])('preserves a pending chat owner (%s)', (_label, options) => { + const owner = makeChatSession(`converted-${_label}`, CHAT, options); + activeSessions.set(sessionKey(CHAT, APP), owner); + + expect(handleChatModeConverted(CHAT, APP)).toBe(false); + expect(activeSessions.get(sessionKey(CHAT, APP))).toBe(owner); + }); + + it('still evicts a ledger-empty idle owner', () => { + const owner = makeChatSession('converted-idle', CHAT); + activeSessions.set(sessionKey(CHAT, APP), owner); + + expect(handleChatModeConverted(CHAT, APP)).toBe(true); + expect(activeSessions.has(sessionKey(CHAT, APP))).toBe(false); + }); + + it.each([ + 'live frozen backend', + 'session backend', + 'session cli id', + 'frozen cli id', + 'workerless durable lineage', + ])( + 'preserves an idle Riff owner across conversion (%s)', + variant => { + const owner = makeChatSession(`converted-riff-${variant}`, CHAT); + if (variant === 'live frozen backend') { + owner.initConfig = { backendType: 'riff' } as any; + owner.worker = { killed: false, send: vi.fn() } as any; + } else if (variant === 'session backend') { + owner.session.backendType = 'riff'; + } else if (variant === 'session cli id') { + owner.session.cliId = 'riff'; + } else if (variant === 'frozen cli id') { + owner.initConfig = { cliId: 'riff' } as any; + } else { + owner.session.backendType = 'riff'; + owner.session.riffParentTaskId = 'task-owned'; + } + activeSessions.set(sessionKey(CHAT, APP), owner); + + expect(handleChatModeConverted(CHAT, APP)).toBe(false); + expect(activeSessions.get(sessionKey(CHAT, APP))).toBe(owner); + }, + ); +}); + +describe('document comment canonical ownership and single-flight delivery', () => { + beforeEach(() => { + vi.clearAllMocks(); + mocks.sessions.clear(); + mocks.forkWorker.mockImplementation((ds: any) => { + ds.worker = { killed: false, send: vi.fn() }; + }); + mocks.getAvailableBots.mockResolvedValue([]); + activeSessions.clear(); + resetDocCommentClaims(); + const bot = registerBot({ larkAppId: APP, larkAppSecret: 's', cliId: 'claude-code', allowedUsers: [OWNER] }); + bot.resolvedAllowedUsers = [OWNER]; + }); + + function docSub(fileToken: string): any { + const sub = { + fileToken, + fileType: 'docx', + sessionAnchor: `om_legacy_${fileToken}`, + scope: 'thread' as const, + chatId: CHAT, + commentTriggerMode: 'all' as const, + managedBy: 'watch-comment' as const, + createdAt: Date.now(), + }; + putDocSubscription(config.session.dataDir, APP, sub); + return sub; + } + + function docCtx(sub: any, suffix: string): any { + return { + larkAppId: APP, + sub, + commentId: `comment-${suffix}`, + replyId: `reply-${suffix}`, + text: `question ${suffix}`, + }; + } + + function bindSubToSession(sub: any, ds: DaemonSession): void { + Object.assign(sub, { + sessionAnchor: sessionAnchorId(ds), + sessionId: ds.session.sessionId, + scope: ds.scope, + chatId: ds.chatId, + }); + putDocSubscription(config.session.dataDir, APP, sub); + } + + it('leaves the provider cursor retryable when a worker-null setup owner exists', async () => { + const sub = docSub(`doc-protected-${Date.now()}`); + const ds = seedThreadSession(sub.sessionAnchor, 'protected doc owner'); + ds.pendingRepo = true; + ds.session.queued = true; + ds.session.queuedPrompt = 'OPENING_N'; + ds.session.pendingRepoSetup = { mode: 'picker', prompt: 'OPENING_N' }; + bindSubToSession(sub, ds); + mocks.forkWorker.mockClear(); + + await expect(handleDocComment(docCtx(sub, 'protected'))).resolves.toBe(false); + + expect(mocks.forkWorker).not.toHaveBeenCalled(); + expect(ds.worker).toBeNull(); + expect(ds.session.docCommentTargets).toBeUndefined(); + expect(ds.session.pendingRepoSetup).toMatchObject({ mode: 'picker', prompt: 'OPENING_N' }); + removeDocSubscription(config.session.dataDir, APP, sub.fileToken); + }); + + it('durably queues a live document comment behind the activation head without worker IPC', async () => { + const sub = docSub(`doc-live-gate-${Date.now()}`); + const ds = seedThreadSession(sub.sessionAnchor, 'live gated doc owner'); + const send = vi.fn(); + ds.worker = { killed: false, send } as any; + Object.assign(ds.session, { + queuedActivationPending: true, + queuedActivationToken: 'doc-opening-token', + queuedActivationInput: { content: 'OPENING_N' }, + queuedActivationTurnId: 'opening-turn', + }); + bindSubToSession(sub, ds); + + await expect(handleDocComment(docCtx(sub, 'live'))).resolves.toBe(true); + + expect(send).not.toHaveBeenCalled(); + expect(ds.session.queuedActivationTail).toEqual([ + expect.objectContaining({ + turnId: 'reply-live', + cliInput: expect.objectContaining({ content: expect.stringContaining('question live') }), + }), + ]); + removeDocSubscription(config.session.dataDir, APP, sub.fileToken); + }); + + it('refuses doc-watch prewarm before turn mutation when a worker-null setup owner exists', async () => { + const sub = docSub(`doc-prewarm-protected-${Date.now()}`); + const ds = seedThreadSession(sub.sessionAnchor, 'prewarm protected'); + ds.session.pendingRepoSetup = { mode: 'picker', prompt: 'OPENING_N' }; + ds.pendingRepo = true; + const priorLastMessageAt = ds.lastMessageAt; + mocks.forkWorker.mockClear(); + + await expect(prewarmDocCommentSession(ds, sub)).rejects.toThrow('durable opening ownership'); + + expect(ds.lastMessageAt).toBe(priorLastMessageAt); + expect(ds.currentTurnTitle).toBeUndefined(); + expect(mocks.forkWorker).not.toHaveBeenCalled(); + removeDocSubscription(config.session.dataDir, APP, sub.fileToken); + }); + + it('durably queues live doc-watch prewarm behind activation without worker IPC', async () => { + const sub = docSub(`doc-prewarm-live-${Date.now()}`); + const ds = seedThreadSession(sub.sessionAnchor, 'prewarm live'); + const send = vi.fn(); + ds.worker = { killed: false, send } as any; + Object.assign(ds.session, { + queuedActivationPending: true, + queuedActivationToken: 'prewarm-opening-token', + queuedActivationInput: { content: 'OPENING_N' }, + queuedActivationTurnId: 'opening-turn', + }); + + await expect(prewarmDocCommentSession(ds, sub)).resolves.toBeUndefined(); + + expect(send).not.toHaveBeenCalled(); + expect(ds.session.queuedActivationTail).toEqual([ + expect.objectContaining({ + turnId: expect.stringMatching(/^doc-watch-/), + cliInput: expect.objectContaining({ content: expect.stringContaining(sub.fileToken) }), + }), + ]); + removeDocSubscription(config.session.dataDir, APP, sub.fileToken); + }); + + it('serializes concurrent get-or-create, merges targets, and reuses canonical state after restart', async () => { + const fileToken = `doc-concurrent-${Date.now()}`; + const sub = docSub(fileToken); + + await expect(Promise.all([ + handleDocComment(docCtx(sub, 'one')), + handleDocComment(docCtx(sub, 'two')), + ])).resolves.toEqual([true, true]); + + const key = sessionKey(`doc:${fileToken}`, APP); + const owner = activeSessions.get(key)!; + expect(owner).toBeDefined(); + expect(mocks.createSession).toHaveBeenCalledTimes(1); + expect(mocks.forkWorker).toHaveBeenCalledTimes(1); + expect(Object.keys(owner.session.docCommentTargets ?? {}).sort()).toEqual(['reply-one', 'reply-two']); + + const persisted = getDocSubscription(config.session.dataDir, APP, fileToken)!; + expect(persisted).toMatchObject({ + sessionAnchor: `doc:${fileToken}`, + sessionId: owner.session.sessionId, + scope: 'chat', + chatId: `doc:${fileToken}`, + }); + // This is the exact anchor closeSession uses to find subscriptions. + expect(sessionAnchorId(owner)).toBe(persisted.sessionAnchor); + + // Simulate a daemon memory restart restoring the same persisted session at + // activeSessionKey(ds), then deliver another comment from a stale snapshot. + activeSessions.clear(); + owner.worker = null; + activeSessions.set(key, owner); + const staleSnapshot = { ...sub }; + await expect(handleDocComment(docCtx(staleSnapshot, 'three'))).resolves.toBe(true); + expect(mocks.createSession).toHaveBeenCalledTimes(1); + expect(activeSessions.get(key)).toBe(owner); + expect(Object.keys(owner.session.docCommentTargets ?? {}).sort()).toEqual([ + 'reply-one', + 'reply-three', + 'reply-two', + ]); + + removeDocSubscription(config.session.dataDir, APP, fileToken); + }); + + it('makes duplicate WS/poll deliveries share failure so neither advances its cursor', async () => { + const fileToken = `doc-failure-${Date.now()}`; + const sub = docSub(fileToken); + const ctx = docCtx(sub, 'same'); + mocks.forkWorker.mockImplementationOnce(() => { throw new Error('simulated fork failure'); }); + + const results = await Promise.all([ + handleDocComment(ctx), + handleDocComment({ ...ctx, sub: { ...sub } }), + ]); + + expect(results).toEqual([false, false]); + expect(mocks.forkWorker).toHaveBeenCalledTimes(1); + + // Failure was not recorded as completed: a later poll retry can deliver. + mocks.forkWorker.mockImplementation((ds: any) => { + ds.worker = { killed: false, send: vi.fn() }; + }); + await expect(handleDocComment(ctx)).resolves.toBe(true); + expect(mocks.forkWorker).toHaveBeenCalledTimes(2); + + removeDocSubscription(config.session.dataDir, APP, fileToken); }); }); diff --git a/test/dashboard-attention-signals.test.ts b/test/dashboard-attention-signals.test.ts index 956179f31..71306a12e 100644 --- a/test/dashboard-attention-signals.test.ts +++ b/test/dashboard-attention-signals.test.ts @@ -218,10 +218,11 @@ describe('attention signals', () => { const src = readFileSync(new URL('../src/daemon.ts', import.meta.url), 'utf-8'); const start = src.indexOf('async function handleThreadReply('); expect(start).toBeGreaterThanOrEqual(0); - // 20000：窗口需罩住函数头到最后一个拦截点 (findPendingAskByAnchor) 的全部 - // 源码——passthrough 冷启动等合法插入会把后续 marker 往后推，窗口太紧会误报。 - // 语义断言不变：clear 在所有拦截点之前。 - const region = src.slice(start, start + 20000); + const end = src.indexOf('async function autoCreateDocSession', start); + expect(end).toBeGreaterThan(start); + // Bound the assertion by the function's next top-level sibling rather than + // a byte count: admission and recovery guards legitimately grow over time. + const region = src.slice(start, end); const clearIdx = region.indexOf('clearAgentAttentionForHumanInbound();'); expect(clearIdx).toBeGreaterThanOrEqual(0); for (const marker of [ diff --git a/test/dashboard-create-session.test.ts b/test/dashboard-create-session.test.ts index 94cca8e97..1cc4fb001 100644 --- a/test/dashboard-create-session.test.ts +++ b/test/dashboard-create-session.test.ts @@ -35,27 +35,49 @@ vi.mock('../src/services/message-queue.js', () => ({ ensureQueue: vi.fn() })); const sendMessageMock = vi.fn(async () => 'om_banner_123'); const uploadImageMock = vi.fn(async () => 'img_dashboard_123'); +const replyMessageMock = vi.fn(async () => 'om_reply_123'); +const deleteMessageMock = vi.fn(async () => {}); vi.mock('../src/im/lark/client.js', () => ({ sendMessage: (...a: any[]) => sendMessageMock(...a), uploadImage: (...a: any[]) => uploadImageMock(...a), downloadMessageResource: vi.fn(), listChatBotMembers: vi.fn(async () => []), - getChatMode: vi.fn(), - replyMessage: vi.fn(), + getChatMode: vi.fn(async () => 'topic'), + replyMessage: (...a: any[]) => replyMessageMock(...a), + deleteMessage: (...a: any[]) => deleteMessageMock(...a), UserTokenMissingError: class extends Error {}, })); +const scanMultipleProjectsMock = vi.fn(() => [] as Array>); +vi.mock('../src/services/project-scanner.js', () => ({ + scanMultipleProjects: (...a: any[]) => scanMultipleProjectsMock(...a), +})); + const forkWorkerMock = vi.fn(); +const sendWorkerInputMock = vi.fn(); +const closeWorkerSessionMock = vi.fn(async () => ({ ok: true, alreadyClosed: false })); +const runAutoWorktreeCommitMock = vi.fn(async () => {}); +let activeRegistryMock: Map | null = null; vi.mock('../src/core/worker-pool.js', () => ({ forkWorker: (...a: any[]) => forkWorkerMock(...a), + sendWorkerInput: (...a: any[]) => sendWorkerInputMock(...a), forkAdoptWorker: vi.fn(), killStalePids: vi.fn(), getCurrentCliVersion: vi.fn(() => 'test-cli-v1'), restoreUsageLimitRuntimeState: vi.fn(), - setActiveSessionSafe: vi.fn(async (map: Map, k: string, ds: any) => { map.set(k, ds); }), - getActiveSessionsRegistry: vi.fn(() => null), - isRelayableRealSession: vi.fn(() => false), - closeSession: vi.fn(), + setActiveSessionSafe: vi.fn(async (map: Map, k: string, ds: any) => { + map.set(k, ds); + return { accepted: true }; + }), + getActiveSessionsRegistry: vi.fn(() => activeRegistryMock), + isRelayableRealSession: vi.fn((ds: any) => !!ds?.worker || !!ds?.session?.cliId || !!ds?.session?.lastCliInput), + closeSession: (...a: any[]) => closeWorkerSessionMock(...a), + promoteQueuedActivationTail: vi.fn(() => false), + withActiveSessionKeyLock: vi.fn(async (_map: Map, _key: string, action: () => any) => action()), +})); + +vi.mock('../src/im/lark/card-handler.js', () => ({ + runAutoWorktreeCommit: (...args: any[]) => runAutoWorktreeCommitMock(...args), })); vi.mock('../src/bot-registry.js', () => ({ @@ -93,6 +115,7 @@ vi.mock('../src/services/whiteboard-store.js', () => ({ import { activateQueuedSession, buildReforkCliInput, + executeScheduledTask, rememberLastCliInput, restoreActiveSessions, spawnDashboardSession, @@ -100,10 +123,14 @@ import { import { sessionKey } from '../src/core/types.js'; import { dashboardEventBus } from '../src/core/dashboard-events.js'; import { getBot } from '../src/bot-registry.js'; +import { getAllBots } from '../src/bot-registry.js'; +import type { ScheduledTask } from '../src/types.js'; +import { setActiveSessionSafe } from '../src/core/worker-pool.js'; import { applyQueuedCodexAppLegacyFallback, mergeQueuedCodexAppTurn, } from '../src/core/session-create.js'; +import * as sessionStore from '../src/services/session-store.js'; const APP = 'cli_app_test'; const CHAT = 'oc_newgroup'; @@ -114,6 +141,21 @@ beforeEach(() => { forkWorkerMock.mockClear(); sendMessageMock.mockClear(); uploadImageMock.mockClear(); + replyMessageMock.mockClear(); + deleteMessageMock.mockClear(); + scanMultipleProjectsMock.mockReset(); + scanMultipleProjectsMock.mockReturnValue([]); + sendWorkerInputMock.mockClear(); + closeWorkerSessionMock.mockClear(); + runAutoWorktreeCommitMock.mockClear(); + vi.mocked(sessionStore.closeSession).mockClear(); + activeRegistryMock = null; + vi.mocked(sessionStore.updateSession).mockImplementation((s: Session) => { store.set(s.sessionId, s); }); + vi.mocked(setActiveSessionSafe).mockImplementation(async (map: Map, key: string, ds: any) => { + map.set(key, ds); + return { accepted: true } as any; + }); + vi.mocked(getAllBots).mockReturnValue([]); (dashboardEventBus.publish as any).mockClear(); vi.mocked(getBot).mockReturnValue({ config: { cliId: 'claude-code', cliPathOverride: undefined, defaultWorkingDir: '/tmp' }, @@ -248,8 +290,91 @@ describe('spawnDashboardSession — backlog (待办池) parks without starting t .toEqual(expect.arrayContaining([ expect.objectContaining({ kind: 'untrusted', value: expect.stringContaining('') }), ])); - expect(restored.session.queuedCodexAppText).toBeUndefined(); - expect(restored.session.queuedCodexAppMessageContext).toBeUndefined(); + // Activation ownership is accepted, but the exact queued journal remains + // until the real worker reports adapter-level submission. + expect(restored.session.queuedCodexAppText).toBe('重启后仍保持纯净'); + expect(restored.session.queuedCodexAppMessageContext).toContain(''); + }); + + it('retains and eagerly replays a non-Codex pre-init journal after a daemon crash', async () => { + const beforeRestart = new Map(); + await spawnDashboardSession(beforeRestart, undefined, { + larkAppId: APP, + chatId: CHAT, + content: 'recover at least once', + column: 'backlog', + role: 'solo', + }); + const interrupted = beforeRestart.get(sessionKey(CHAT, APP))!; + interrupted.session.queued = false; + interrupted.session.queuedActivationPending = true; + interrupted.session.queuedActivationToken = 'activation-before-crash'; + interrupted.session.queuedActivationInput = { content: interrupted.session.queuedPrompt! }; + interrupted.session.queuedActivationTurnId = 'turn-before-crash'; + interrupted.session.queuedActivationResume = false; + // The journal deliberately retains all queued payload fields until init + // IPC has returned and the success cleanup is durably committed. + store.set(interrupted.session.sessionId, interrupted.session); + + const afterRestart = new Map(); + await restoreActiveSessions(afterRestart); + const restored = afterRestart.get(sessionKey(CHAT, APP))!; + + expect(restored.session.queued).toBe(false); + expect(restored.session.queuedActivationPending).toBe(true); + expect(restored.session.queuedActivationToken).toBe('activation-before-crash'); + expect(restored.session.queuedActivationInput).toEqual({ + content: expect.stringContaining('recover at least once'), + }); + expect(restored.session.queuedPrompt).toContain('recover at least once'); + expect(restored.pendingPrompt).toBeUndefined(); + expect(restored.hasHistory).toBe(false); + expect(forkWorkerMock).toHaveBeenCalledWith( + restored, + restored.session.queuedActivationInput, + { + resume: false, + turnId: 'turn-before-crash', + dispatchAttempt: undefined, + }, + ); + }); + + it('restores a Codex pre-init journal through its accepted ledger without re-parking', async () => { + vi.mocked(getBot).mockReturnValue({ + config: { cliId: 'codex-app', cliPathOverride: undefined, defaultWorkingDir: '/tmp' }, + botName: 'TestBot', + botOpenId: 'ou_bot', + } as any); + const beforeRestart = new Map(); + await spawnDashboardSession(beforeRestart, undefined, { + larkAppId: APP, + chatId: CHAT, + content: 'recover from exact FIFO', + column: 'backlog', + role: 'solo', + }); + const interrupted = beforeRestart.get(sessionKey(CHAT, APP))!; + interrupted.session.cliId = 'codex-app'; + interrupted.session.queued = false; + interrupted.session.queuedActivationPending = true; + interrupted.session.codexAppDispatchLedger = [{ + dispatchId: 'dispatch-before-crash', + turnId: 'turn-before-crash', + state: 'accepted', + content: 'recover from exact FIFO', + }]; + store.set(interrupted.session.sessionId, interrupted.session); + forkWorkerMock.mockClear(); + + const afterRestart = new Map(); + await restoreActiveSessions(afterRestart); + const restored = afterRestart.get(sessionKey(CHAT, APP))!; + + expect(restored.session.queued).toBe(false); + expect(restored.hasHistory).toBe(false); + expect(restored.session.codexAppDispatchLedger).toHaveLength(1); + expect(forkWorkerMock).toHaveBeenCalledWith(restored, '', true); }); it('starts a restored pre-clean-input backlog task safely from the Dashboard button', async () => { @@ -339,6 +464,265 @@ describe('spawnDashboardSession — backlog (待办池) parks without starting t expect(restored.lastCodexAppInput).toBeUndefined(); expect(restored.session.lastCodexAppInput).toBeUndefined(); }); + + it('restores a durable picker by publishing and persisting a fresh authoritative card id', async () => { + const pending: Session = { + sessionId: 'pending-picker', + chatId: CHAT, + rootMessageId: CHAT, + scope: 'chat', + larkAppId: APP, + title: 'pick a repo', + status: 'active', + createdAt: new Date('2026-01-01T00:00:00Z').toISOString(), + queued: true, + queuedPrompt: 'OPENING_N', + pendingRepoSetup: { + mode: 'picker', + prompt: 'OPENING_N', + repoCardMessageId: 'om_stale_picker', + }, + }; + store.set(pending.sessionId, pending); + scanMultipleProjectsMock.mockReturnValue([{ + name: 'repo', path: '/tmp', type: 'repo', branch: 'main', + }]); + sendMessageMock.mockResolvedValueOnce('om_fresh_picker'); + + const active = new Map(); + await restoreActiveSessions(active); + + const restored = active.get(sessionKey(CHAT, APP))!; + expect(restored.pendingRepo).toBe(true); + expect(restored.pendingPrompt).toBe('OPENING_N'); + expect(restored.repoCardMessageId).toBe('om_fresh_picker'); + expect(restored.session.pendingRepoSetup?.repoCardMessageId).toBe('om_fresh_picker'); + expect(sendMessageMock).toHaveBeenCalledWith(APP, CHAT, expect.any(String), 'interactive'); + expect(deleteMessageMock).toHaveBeenCalledWith(APP, 'om_stale_picker'); + expect(forkWorkerMock).not.toHaveBeenCalled(); + }); + + it('isolates a picker publish failure and keeps restoring later sessions with the exact setup journal', async () => { + const pending: Session = { + sessionId: 'pending-picker-failure', + chatId: CHAT, + rootMessageId: CHAT, + scope: 'chat', + larkAppId: APP, + title: 'pick a repo', + status: 'active', + createdAt: new Date('2026-01-01T00:00:00Z').toISOString(), + queued: true, + queuedPrompt: 'OPENING_N', + pendingRepoSetup: { + mode: 'picker', + prompt: 'OPENING_N', + repoCardMessageId: 'om_old_picker', + }, + }; + const sibling: Session = { + sessionId: 'later-backlog', + chatId: 'oc_later', + rootMessageId: 'oc_later', + scope: 'chat', + larkAppId: APP, + title: 'later', + status: 'active', + createdAt: new Date('2026-01-01T00:00:01Z').toISOString(), + queued: true, + queuedPrompt: 'LATER_TASK', + }; + store.set(pending.sessionId, pending); + store.set(sibling.sessionId, sibling); + scanMultipleProjectsMock.mockReturnValue([{ + name: 'repo', path: '/tmp', type: 'repo', branch: 'main', + }]); + sendMessageMock.mockRejectedValueOnce(new Error('picker publish unavailable')); + + const active = new Map(); + await expect(restoreActiveSessions(active)).resolves.toBeUndefined(); + + const restored = active.get(sessionKey(CHAT, APP))!; + expect(restored.pendingRepo).toBe(true); + expect(restored.pendingPrompt).toBe('OPENING_N'); + expect(restored.repoCardMessageId).toBe('om_old_picker'); + expect(restored.session.pendingRepoSetup).toMatchObject({ + mode: 'picker', prompt: 'OPENING_N', repoCardMessageId: 'om_old_picker', + }); + expect(active.get(sessionKey('oc_later', APP))?.session.queuedPrompt).toBe('LATER_TASK'); + }); + + it('contains detached auto-worktree recovery rejection and leaves the setup retryable', async () => { + const pending: Session = { + sessionId: 'pending-auto-worktree', + chatId: CHAT, + rootMessageId: CHAT, + scope: 'chat', + larkAppId: APP, + title: 'make worktree', + status: 'active', + createdAt: new Date('2026-01-01T00:00:00Z').toISOString(), + queued: true, + queuedPrompt: 'OPENING_N', + pendingRepoSetup: { + mode: 'auto_worktree', prompt: 'OPENING_N', baseDir: '/tmp', + }, + }; + store.set(pending.sessionId, pending); + runAutoWorktreeCommitMock.mockRejectedValueOnce(new Error('worktree publish unavailable')); + + const active = new Map(); + await expect(restoreActiveSessions(active)).resolves.toBeUndefined(); + await Promise.resolve(); + + const restored = active.get(sessionKey(CHAT, APP))!; + expect(restored.pendingRepo).toBe(true); + expect(restored.pendingPrompt).toBe('OPENING_N'); + expect(restored.session.pendingRepoSetup).toMatchObject({ + mode: 'auto_worktree', prompt: 'OPENING_N', baseDir: '/tmp', + }); + }); + + it('isolates a historical same-anchor protected loser so one collision cannot abort daemon restart', async () => { + const incumbent: Session = { + sessionId: 'canonical-pending-owner', + chatId: CHAT, + rootMessageId: CHAT, + scope: 'chat', + larkAppId: APP, + title: 'canonical', + status: 'active', + createdAt: new Date('2026-01-01T00:00:00Z').toISOString(), + cliId: 'claude-code', + queuedActivationPending: true, + queuedActivationToken: 'canonical-token', + queuedActivationInput: { content: 'CANONICAL_N' }, + queuedActivationTurnId: 'turn-canonical', + queuedActivationResume: false, + }; + const stagedLoser: Session = { + sessionId: 'historical-staged-loser', + chatId: CHAT, + rootMessageId: CHAT, + scope: 'chat', + larkAppId: APP, + title: 'loser', + status: 'active', + createdAt: new Date('2026-01-01T00:00:01Z').toISOString(), + queued: true, + queuedPrompt: 'LOSER_OPENING_N', + pendingRepoSetup: { mode: 'picker', prompt: 'LOSER_OPENING_N' }, + }; + store.set(incumbent.sessionId, incumbent); + store.set(stagedLoser.sessionId, stagedLoser); + vi.mocked(setActiveSessionSafe).mockImplementation(async (map, key, ds) => { + const current = map.get(key); + if (!current) { + map.set(key, ds); + return { accepted: true } as any; + } + return { + accepted: false, + reason: 'both_pending', + keptSessionId: current.session.sessionId, + preservedIncomingSessionId: ds.session.sessionId, + } as any; + }); + + const active = new Map(); + await expect(restoreActiveSessions(active)).resolves.toBeUndefined(); + + expect(active.get(sessionKey(CHAT, APP))?.session.sessionId).toBe(incumbent.sessionId); + expect(store.get(stagedLoser.sessionId)).toMatchObject({ + status: 'active', + queued: true, + queuedPrompt: 'LOSER_OPENING_N', + pendingRepoSetup: { mode: 'picker', prompt: 'LOSER_OPENING_N' }, + }); + }); + + it('isolates a malformed queued+unsettled row and restores the later healthy row', async () => { + const malformed: Session = { + sessionId: 'malformed-queued', chatId: CHAT, rootMessageId: CHAT, scope: 'chat', larkAppId: APP, + title: 'bad', status: 'active', createdAt: new Date('2026-01-01T00:00:00Z').toISOString(), + queued: true, queuedPrompt: 'BAD_N', + codexAppDispatchLedger: [{ dispatchId: 'bad-dispatch', turnId: 'bad-turn', state: 'prepared', content: 'BAD_N' }], + }; + const healthy: Session = { + sessionId: 'healthy-after-malformed', chatId: 'oc_healthy_1', rootMessageId: 'oc_healthy_1', scope: 'chat', larkAppId: APP, + title: 'good', status: 'active', createdAt: new Date('2026-01-01T00:00:01Z').toISOString(), + queued: true, queuedPrompt: 'GOOD_N', + }; + store.set(malformed.sessionId, malformed); + store.set(healthy.sessionId, healthy); + + const active = new Map(); + await expect(restoreActiveSessions(active)).resolves.toBeUndefined(); + + expect(active.has(sessionKey(CHAT, APP))).toBe(false); + expect(store.get(malformed.sessionId)).toMatchObject({ status: 'active', queued: true, queuedPrompt: 'BAD_N' }); + expect(active.get(sessionKey('oc_healthy_1', APP))?.session.sessionId).toBe(healthy.sessionId); + }); + + it('isolates a tail-promotion write failure, rolls the row back, and restores the later row', async () => { + const failed: Session = { + sessionId: 'failed-tail-promotion', chatId: CHAT, rootMessageId: CHAT, scope: 'chat', larkAppId: APP, + title: 'bad tail', status: 'active', createdAt: new Date('2026-01-01T00:00:00Z').toISOString(), + cliId: 'claude-code', + queuedActivationTail: [{ id: 'tail-1', order: 1, userPrompt: 'TAIL_N', cliInput: { content: 'TAIL_N' }, turnId: 'tail-turn' }], + queuedActivationTailNextOrder: 1, + }; + const healthy: Session = { + sessionId: 'healthy-after-tail', chatId: 'oc_healthy_2', rootMessageId: 'oc_healthy_2', scope: 'chat', larkAppId: APP, + title: 'good', status: 'active', createdAt: new Date('2026-01-01T00:00:01Z').toISOString(), + queued: true, queuedPrompt: 'GOOD_AFTER_TAIL', + }; + store.set(failed.sessionId, failed); + store.set(healthy.sessionId, healthy); + vi.mocked(sessionStore.updateSession).mockImplementation((s: Session) => { + if (s.sessionId === failed.sessionId) throw new Error('tail promotion save unavailable'); + store.set(s.sessionId, s); + }); + + const active = new Map(); + await expect(restoreActiveSessions(active)).resolves.toBeUndefined(); + + expect(active.has(sessionKey(CHAT, APP))).toBe(false); + expect(failed.queuedActivationPending).toBeUndefined(); + expect(failed.queuedActivationTail?.[0]?.cliInput.content).toBe('TAIL_N'); + expect(active.get(sessionKey('oc_healthy_2', APP))?.session.sessionId).toBe(healthy.sessionId); + }); + + it('rolls back a failed terminal-empty cleanup and continues restoring later rows', async () => { + const failed: Session = { + sessionId: 'failed-terminal-cleanup', chatId: CHAT, rootMessageId: CHAT, scope: 'chat', larkAppId: APP, + title: 'terminal', status: 'active', createdAt: new Date('2026-01-01T00:00:00Z').toISOString(), + cliId: 'codex-app', queuedActivationPending: true, queuedActivationToken: 'terminal-token', + queuedActivationInput: { content: 'SETTLED_N' }, queuedActivationTurnId: 'terminal-turn', + }; + const healthy: Session = { + sessionId: 'healthy-after-terminal', chatId: 'oc_healthy_3', rootMessageId: 'oc_healthy_3', scope: 'chat', larkAppId: APP, + title: 'good', status: 'active', createdAt: new Date('2026-01-01T00:00:01Z').toISOString(), + queued: true, queuedPrompt: 'GOOD_AFTER_TERMINAL', + }; + store.set(failed.sessionId, failed); + store.set(healthy.sessionId, healthy); + vi.mocked(sessionStore.updateSession).mockImplementation((s: Session) => { + if (s.sessionId === failed.sessionId) throw new Error('terminal cleanup save unavailable'); + store.set(s.sessionId, s); + }); + + const active = new Map(); + await expect(restoreActiveSessions(active)).resolves.toBeUndefined(); + + expect(active.has(sessionKey(CHAT, APP))).toBe(false); + expect(failed).toMatchObject({ + queuedActivationPending: true, + queuedActivationToken: 'terminal-token', + queuedActivationInput: { content: 'SETTLED_N' }, + }); + expect(active.get(sessionKey('oc_healthy_3', APP))?.session.sessionId).toBe(healthy.sessionId); + }); }); describe('spawnDashboardSession — in_progress starts immediately', () => { @@ -366,6 +750,108 @@ describe('spawnDashboardSession — in_progress starts immediately', () => { }); expect(prompt.content).toContain('Sub1'); }); + + it('unpublishes and closes only the new row when fork pre-accept throws', async () => { + const active = new Map(); + forkWorkerMock.mockImplementationOnce(() => { throw new Error('spawn preaccept failed'); }); + + await expect(spawnDashboardSession(active, undefined, { + larkAppId: APP, chatId: CHAT, content: 'opening task', column: 'in_progress', role: 'solo', + })).resolves.toEqual({ ok: false, error: 'spawn preaccept failed' }); + + expect(active.has(sessionKey(CHAT, APP))).toBe(false); + expect(sessionStore.closeSession).toHaveBeenCalledWith('sess-1'); + }); + + it('unpublishes and closes only the new row when auto-worktree staging fails', async () => { + vi.mocked(getBot).mockReturnValue({ + config: { + cliId: 'claude-code', defaultWorkingDir: '/tmp', defaultWorkingDirAutoWorktree: true, + }, + botName: 'TestBot', botOpenId: 'ou_bot', + } as any); + vi.mocked(sessionStore.updateSession).mockImplementation((s: Session) => { + if (s.pendingRepoSetup?.mode === 'auto_worktree') throw new Error('stage setup unavailable'); + store.set(s.sessionId, s); + }); + const active = new Map(); + activeRegistryMock = active; + + await expect(spawnDashboardSession(active, undefined, { + larkAppId: APP, chatId: CHAT, content: 'opening worktree task', column: 'in_progress', role: 'solo', + })).resolves.toEqual({ ok: false, error: 'stage setup unavailable' }); + + expect(active.has(sessionKey(CHAT, APP))).toBe(false); + expect(sessionStore.closeSession).toHaveBeenCalledWith('sess-1'); + expect(runAutoWorktreeCommitMock).not.toHaveBeenCalled(); + expect(forkWorkerMock).not.toHaveBeenCalled(); + }); + + it('keeps a visible picker fail-closed when persisting its authoritative id fails', async () => { + vi.mocked(getBot).mockReturnValue({ + config: { cliId: 'claude-code', workingDir: '/tmp', defaultWorkingDir: undefined }, + botName: 'TestBot', botOpenId: 'ou_bot', + } as any); + scanMultipleProjectsMock.mockReturnValue([{ name: 'repo', path: '/tmp', type: 'repo', branch: 'main' }]); + sendMessageMock.mockResolvedValueOnce('om_visible_picker'); + vi.mocked(sessionStore.updateSession).mockImplementation((s: Session) => { + if (s.pendingRepoSetup?.repoCardMessageId === 'om_visible_picker') { + throw new Error('picker id save unavailable'); + } + store.set(s.sessionId, s); + }); + const active = new Map(); + + const result = await spawnDashboardSession(active, undefined, { + larkAppId: APP, chatId: CHAT, content: 'pick before start', column: 'in_progress', role: 'solo', + }); + + expect(result).toEqual({ ok: true, sessionId: 'sess-1' }); + const ds = active.get(sessionKey(CHAT, APP))!; + expect(ds.pendingRepo).toBe(true); + expect(ds.repoCardMessageId).toBe('om_visible_picker'); + expect(ds.session.pendingRepoSetup).toMatchObject({ mode: 'picker', prompt: expect.stringContaining('pick before start') }); + expect(ds.session.pendingRepoSetup?.repoCardMessageId).toBeUndefined(); + expect(forkWorkerMock).not.toHaveBeenCalled(); + expect(sessionStore.closeSession).not.toHaveBeenCalled(); + expect(dashboardEventBus.publish).toHaveBeenCalledWith(expect.objectContaining({ + type: 'session.spawned', + })); + }); + + it('retains a durable picker owner when publish fallback and fork both fail', async () => { + vi.mocked(getBot).mockReturnValue({ + config: { cliId: 'claude-code', workingDir: '/tmp', defaultWorkingDir: undefined }, + botName: 'TestBot', botOpenId: 'ou_bot', + } as any); + scanMultipleProjectsMock.mockReturnValue([{ name: 'repo', path: '/tmp', type: 'repo', branch: 'main' }]); + sendMessageMock.mockRejectedValueOnce(new Error('picker publish unavailable')); + forkWorkerMock.mockImplementationOnce(() => { throw new Error('fallback fork unavailable'); }); + const active = new Map(); + + const result = await spawnDashboardSession(active, undefined, { + larkAppId: APP, chatId: CHAT, content: 'retain exact opening', column: 'in_progress', role: 'solo', + }); + + expect(result).toEqual({ ok: false, error: 'fallback fork unavailable' }); + const ds = active.get(sessionKey(CHAT, APP))!; + expect(ds).toBeDefined(); + expect(ds.pendingRepo).toBe(true); + expect(ds.pendingPrompt).toContain('retain exact opening'); + expect(ds.session).toMatchObject({ + status: 'active', + queued: true, + queuedPrompt: expect.stringContaining('retain exact opening'), + pendingRepoSetup: { + mode: 'picker', + prompt: expect.stringContaining('retain exact opening'), + }, + }); + expect(sessionStore.closeSession).not.toHaveBeenCalled(); + expect(dashboardEventBus.publish).toHaveBeenCalledWith(expect.objectContaining({ + type: 'session.spawned', + })); + }); }); describe('spawnDashboardSession — guards', () => { @@ -375,6 +861,64 @@ describe('spawnDashboardSession — guards', () => { const r2 = await spawnDashboardSession(active, undefined, { larkAppId: APP, chatId: CHAT, content: 'b', column: 'in_progress', role: 'solo' }); expect(r2).toMatchObject({ ok: false, error: 'session_exists' }); }); + + it.each([ + ['pendingRepo', { pendingRepo: true }], + ['initialStartPending', { initialStartPending: true }], + ['worktreeCreating', { worktreeCreating: true }], + ])('does not replace an existing %s opening reservation', async (_name, flags) => { + const reserved = { + session: { sessionId: 'reserved', status: 'active', queued: false }, + worker: null, + larkAppId: APP, + chatId: CHAT, + scope: 'chat', + ...flags, + } as unknown as DaemonSession; + const active = new Map([[sessionKey(CHAT, APP), reserved]]); + + const result = await spawnDashboardSession(active, undefined, { + larkAppId: APP, chatId: CHAT, content: 'must not replace', column: 'in_progress', role: 'solo', + }); + + expect(result).toEqual({ ok: false, error: 'session_exists' }); + expect(active.get(sessionKey(CHAT, APP))).toBe(reserved); + expect(closeWorkerSessionMock).not.toHaveBeenCalled(); + expect(forkWorkerMock).not.toHaveBeenCalled(); + }); + + it('rechecks first-owner after banner preparation and preserves a concurrent reservation', async () => { + const active = new Map(); + let bannerStarted!: () => void; + let releaseBanner!: () => void; + const started = new Promise(resolve => { bannerStarted = resolve; }); + const paused = new Promise(resolve => { releaseBanner = resolve; }); + sendMessageMock.mockImplementationOnce(async () => { + bannerStarted(); + await paused; + return 'om_delayed_banner'; + }); + + const spawning = spawnDashboardSession(active, undefined, { + larkAppId: APP, chatId: CHAT, content: 'late contender', column: 'in_progress', role: 'solo', postBanner: true, + }); + await started; + const incumbent = { + session: { sessionId: 'concurrent-reservation', status: 'active', queued: false }, + worker: null, + initialStartPending: true, + larkAppId: APP, + chatId: CHAT, + scope: 'chat', + } as unknown as DaemonSession; + active.set(sessionKey(CHAT, APP), incumbent); + releaseBanner(); + + await expect(spawning).resolves.toEqual({ ok: false, error: 'session_exists' }); + expect(active.get(sessionKey(CHAT, APP))).toBe(incumbent); + expect(closeWorkerSessionMock).not.toHaveBeenCalled(); + expect(forkWorkerMock).not.toHaveBeenCalled(); + }); }); describe('activateQueuedSession', () => { @@ -394,7 +938,9 @@ describe('activateQueuedSession', () => { expect(prompt).toMatchObject({ content: expect.stringContaining('排队的任务') }); expect(prompt.content).toContain(''); // preamble survived park→activate expect(ds.session.queued).toBe(false); - expect(ds.session.queuedPrompt).toBeUndefined(); + // The worker-pool ACK handler, not activation acceptance, clears this + // exact replay source after adapter submission. + expect(ds.session.queuedPrompt).toContain('排队的任务'); expect(ds.session.kanbanColumn).toBe('in_progress'); }); @@ -402,4 +948,226 @@ describe('activateQueuedSession', () => { const ds = { worker: null, session: { queued: false } } as unknown as DaemonSession; expect(await activateQueuedSession(ds)).toMatchObject({ ok: false, error: 'not_queued' }); }); + + it('keeps the backlog payload retryable when fork pre-accept throws', async () => { + const active = new Map(); + await spawnDashboardSession(active, undefined, { + larkAppId: APP, chatId: CHAT, content: 'do not lose this task', column: 'backlog', role: 'solo', + }); + activeRegistryMock = active; + const ds = active.get(sessionKey(CHAT, APP))!; + forkWorkerMock.mockImplementationOnce(() => { throw new Error('fork preaccept failed'); }); + + const result = await activateQueuedSession(ds); + + expect(result).toEqual({ ok: false, error: 'fork preaccept failed' }); + expect(ds.session.queued).toBe(true); + expect(ds.session.queuedPrompt).toContain('do not lose this task'); + expect(ds.pendingPrompt).toContain('do not lose this task'); + expect(ds.initialStartPending).toBe(false); + expect(ds.pendingRepo).toBe(false); + }); + + it('returns success after worker ownership even when kanban metadata persistence fails', async () => { + const active = new Map(); + await spawnDashboardSession(active, undefined, { + larkAppId: APP, + chatId: CHAT, + content: 'owned before metadata write', + column: 'backlog', + role: 'solo', + }); + const ds = active.get(sessionKey(CHAT, APP))!; + forkWorkerMock.mockImplementationOnce((owned: DaemonSession) => { + owned.worker = { killed: false, send: vi.fn() } as any; + owned.session.queuedActivationPending = true; + owned.session.queuedActivationToken = 'activation-token'; + }); + vi.mocked(sessionStore.updateSession) + .mockImplementationOnce((s: Session) => { store.set(s.sessionId, s); }) + .mockImplementationOnce(() => { + throw new Error('kanban projection unavailable'); + }); + + await expect(activateQueuedSession(ds)).resolves.toEqual({ ok: true }); + expect(forkWorkerMock).toHaveBeenCalledOnce(); + expect(ds.worker).toBeTruthy(); + expect(ds.session.queued).toBe(false); + expect(ds.session.kanbanColumn).toBe('in_progress'); + expect(ds.session.queuedPrompt).toContain('owned before metadata write'); + }); + + it('keeps the durable queued payload while an auto-worktree owns the pending fork', async () => { + vi.mocked(getBot).mockReturnValue({ + config: { + cliId: 'codex-app', + cliPathOverride: undefined, + defaultWorkingDir: '/tmp', + defaultWorkingDirAutoWorktree: true, + codexAppCleanInput: true, + }, + botName: 'TestBot', + botOpenId: 'ou_bot', + } as any); + const active = new Map(); + await spawnDashboardSession(active, undefined, { + larkAppId: APP, + chatId: CHAT, + content: 'survive pending worktree restart', + column: 'backlog', + role: 'solo', + }); + activeRegistryMock = active; + const ds = active.get(sessionKey(CHAT, APP))!; + + expect(await activateQueuedSession(ds)).toEqual({ ok: true }); + + expect(forkWorkerMock).not.toHaveBeenCalled(); + expect(runAutoWorktreeCommitMock).toHaveBeenCalledTimes(1); + expect(ds.pendingRepo).toBe(true); + expect(ds.session).toMatchObject({ + queued: true, + queuedPrompt: expect.stringContaining('survive pending worktree restart'), + queuedCodexAppText: 'survive pending worktree restart', + kanbanColumn: 'in_progress', + }); + + // A repeated dashboard start is idempotent while the delayed commit owns + // the attempt; it must not schedule a second worktree or picker. + expect(await activateQueuedSession(ds)).toEqual({ ok: true }); + expect(runAutoWorktreeCommitMock).toHaveBeenCalledTimes(1); + }); +}); + +describe('executeScheduledTask — workerless owner semantics', () => { + const ROOT = 'om_scheduler_owner'; + + function task(): ScheduledTask { + return { + id: 'schedule-owner-test', + name: 'owner test', + schedule: 'once', + parsed: { kind: 'once', at: '2026-01-01T00:00:00.000Z' }, + prompt: 'scheduled prompt', + workingDir: '/tmp', + chatId: CHAT, + rootMessageId: ROOT, + scope: 'thread', + larkAppId: APP, + enabled: true, + createdAt: '2026-01-01T00:00:00.000Z', + } as ScheduledTask; + } + + function owner(overrides: Partial & { session?: Partial } = {}): DaemonSession { + const { session: sessionPatch, ...daemonPatch } = overrides; + return { + session: { + sessionId: 'existing-owner', chatId: CHAT, rootMessageId: ROOT, + status: 'active', scope: 'thread', createdAt: '2026-01-01T00:00:00.000Z', + ...sessionPatch, + } as Session, + worker: null, + workerPort: null, + workerToken: null, + larkAppId: APP, + chatId: CHAT, + chatType: 'group', + scope: 'thread', + spawnedAt: 1, + cliVersion: 'test', + lastMessageAt: 1, + hasHistory: false, + ...daemonPatch, + } as DaemonSession; + } + + beforeEach(() => { + vi.mocked(getAllBots).mockReturnValue([{ + config: { larkAppId: APP, cliId: 'claude-code' }, + botName: 'TestBot', botOpenId: 'ou_bot', resolvedAllowedUsers: [], + }] as any); + }); + + it.each([ + ['pending_repo', { pendingRepo: true }], + ['initial_start_pending', { initialStartPending: true }], + ['worktree_creating', { worktreeCreating: true }], + ['queued_backlog', { session: { queued: true } }], + ])('preserves a %s reservation instead of forking the scheduled prompt', async (state, patch) => { + const ds = owner(patch as any); + const active = new Map([[sessionKey(ROOT, APP), ds]]); + + await expect(executeScheduledTask(task(), active, vi.fn())).rejects.toThrow(state); + + expect(active.get(sessionKey(ROOT, APP))).toBe(ds); + expect(forkWorkerMock).not.toHaveBeenCalled(); + expect(sendWorkerInputMock).not.toHaveBeenCalled(); + expect(closeWorkerSessionMock).not.toHaveBeenCalled(); + }); + + it('reforks a cold real owner and keeps its history', async () => { + const ds = owner({ session: { cliId: 'claude-code', lastCliInput: 'prior turn' }, hasHistory: true }); + const active = new Map([[sessionKey(ROOT, APP), ds]]); + + await executeScheduledTask(task(), active, vi.fn()); + + expect(active.get(sessionKey(ROOT, APP))).toBe(ds); + expect(forkWorkerMock).toHaveBeenCalledWith(ds, expect.anything(), expect.objectContaining({ + resume: true, + turnId: expect.stringMatching(/^schedule:schedule-owner-test:/), + })); + expect(closeWorkerSessionMock).not.toHaveBeenCalled(); + }); + + it('treats a persisted Riff task id as a cold real owner', async () => { + const ds = owner({ session: { riffParentTaskId: 'riff-task-123' }, hasHistory: true }); + const active = new Map([[sessionKey(ROOT, APP), ds]]); + + await executeScheduledTask(task(), active, vi.fn()); + + expect(active.get(sessionKey(ROOT, APP))).toBe(ds); + expect(forkWorkerMock).toHaveBeenCalledWith(ds, expect.anything(), expect.objectContaining({ + resume: true, + turnId: expect.stringMatching(/^schedule:schedule-owner-test:/), + })); + expect(closeWorkerSessionMock).not.toHaveBeenCalled(); + }); + + it('retires a scratch owner and creates a fresh scheduled session', async () => { + const scratch = owner(); + const active = new Map([[sessionKey(ROOT, APP), scratch]]); + + await executeScheduledTask(task(), active, vi.fn()); + + expect(closeWorkerSessionMock).toHaveBeenCalledWith('existing-owner'); + expect(active.get(sessionKey(ROOT, APP))).not.toBe(scratch); + expect(forkWorkerMock).toHaveBeenCalledTimes(1); + }); + + it('does not delete an owner when explicit close reports retryable cleanup failure', async () => { + const scratch = owner(); + const active = new Map([[sessionKey(ROOT, APP), scratch]]); + closeWorkerSessionMock.mockResolvedValueOnce({ + ok: false, alreadyClosed: false, error: 'riff_cancel_failed', retryable: true, taskId: 'task-1', + }); + + await expect(executeScheduledTask(task(), active, vi.fn())) + .rejects.toThrow('riff_cancel_failed'); + + expect(active.get(sessionKey(ROOT, APP))).toBe(scratch); + expect(forkWorkerMock).not.toHaveBeenCalled(); + }); + + it('retains a fresh-session reservation and opening prompt when fork pre-accept throws', async () => { + const active = new Map(); + forkWorkerMock.mockImplementationOnce(() => { throw new Error('scheduler fork failed'); }); + + await expect(executeScheduledTask(task(), active, vi.fn())).rejects.toThrow('scheduler fork failed'); + + const ds = active.get(sessionKey(ROOT, APP)); + expect(ds?.initialStartPending).toBe(true); + expect(ds?.pendingPrompt).toBe('scheduled prompt'); + expect(ds?.worker).toBeNull(); + }); }); diff --git a/test/dashboard-ipc.test.ts b/test/dashboard-ipc.test.ts index bad81e336..1ee069919 100644 --- a/test/dashboard-ipc.test.ts +++ b/test/dashboard-ipc.test.ts @@ -1,7 +1,8 @@ // test/dashboard-ipc.test.ts import { describe, it, expect, afterEach, vi } from 'vitest'; import { createHmac, randomBytes } from 'node:crypto'; -import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { existsSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from 'node:fs'; +import { request as httpRequest } from 'node:http'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { ipcRoute, startIpcServer, setLarkAppId, setIpcAuthSecret, setBotRenamer, setBotAvatarChanger, type IpcServerHandle } from '../src/core/dashboard-ipc-server.js'; @@ -11,12 +12,16 @@ import * as groupsStore from '../src/services/groups-store.js'; import * as larkClient from '../src/im/lark/client.js'; import * as oncallStore from '../src/services/oncall-store.js'; import * as sessionStore from '../src/services/session-store.js'; +import * as sandboxStore from '../src/services/sandbox-store.js'; import * as workerPool from '../src/core/worker-pool.js'; import * as scheduler from '../src/core/scheduler.js'; -import { __testOnly_resetBotRegistry, loadBotConfigs, registerBot } from '../src/bot-registry.js'; +import * as persistentBackend from '../src/core/persistent-backend.js'; +import { __testOnly_resetBotRegistry, getBot, loadBotConfigs, registerBot } from '../src/bot-registry.js'; import { config } from '../src/config.js'; import { sessionKey } from '../src/core/types.js'; import { writeRoleFile, writeTeamRoleFile } from '../src/core/role-resolver.js'; +import { managedOriginAttestationProofPath } from '../src/core/managed-origin-capability.js'; +import { MANAGED_ORIGIN_PROOF_DOMAIN } from '../src/core/managed-origin-attestation.js'; // Loopback-HMAC the write-link routes require. Inject a known secret per test // (setIpcAuthSecret) and sign with it, so the suite doesn't depend on a real @@ -117,6 +122,40 @@ describe('dashboard IPC server', () => { expect(res.status).toBe(404); }); + it('binds and serves health early but holds authenticated state routes behind readiness', async () => { + setIpcAuthSecret(TEST_IPC_SECRET); + let releaseReady!: () => void; + const ready = new Promise(resolve => { releaseReady = resolve; }); + let mutations = 0; + const path = '/api/test-startup-readiness-mutation'; + ipcRoute('POST', path, (_req, res) => { + mutations += 1; + res.writeHead(200, { 'content-type': 'application/json' }); + res.end(JSON.stringify({ ok: true })); + }); + handle = await startIpcServer({ + port: 0, + host: '127.0.0.1', + authRequired: true, + ready, + }); + const base = `http://127.0.0.1:${handle.port}`; + + const health = await fetch(`${base}/__health`); + expect(health.status).toBe(200); + const pending = fetch(`${base}${path}`, { + method: 'POST', + headers: trustedHostHeaders('POST', path, handle.port), + }); + await new Promise(resolve => setTimeout(resolve, 20)); + expect(mutations).toBe(0); + + releaseReady(); + const response = await pending; + expect(response.status).toBe(200); + expect(mutations).toBe(1); + }); + it('denies sandbox-like loopback reads and mutations but accepts route-bound trusted-host calls', async () => { setIpcAuthSecret(TEST_IPC_SECRET); let mutations = 0; @@ -169,6 +208,349 @@ describe('dashboard IPC server', () => { }); }); +describe('POST /api/session-origin/attest', () => { + const CHANNEL = '77'.repeat(32); + const CAPABILITY = 'ab'.repeat(32); + const TURN_ID = 'turn-managed-origin'; + const DISPATCH_ATTEMPT = 3; + + function installManagedOriginFixture(options: { + worker?: Record | null; + origin?: Record | null; + ledger?: unknown[]; + } = {}) { + const dataDir = mkdtempSync(join(tmpdir(), 'dashboard-ipc-origin-attest-')); + const previousDataDir = config.session.dataDir; + const previousRegistry = workerPool.getActiveSessionsRegistry(); + const sessionId = `origin-attest-${randomBytes(8).toString('hex')}`; + const defaultWorker = { + pid: process.pid, + connected: true, + killed: false, + exitCode: null, + signalCode: null, + send: vi.fn(), + }; + const defaultOrigin = { + capability: CAPABILITY, + originChannelId: CHANNEL, + turnId: TURN_ID, + dispatchAttempt: DISPATCH_ATTEMPT, + }; + const worker = options.worker === null + ? null + : { ...defaultWorker, ...(options.worker ?? {}) }; + const managedTurnOrigin = options.origin === null + ? undefined + : { ...defaultOrigin, ...(options.origin ?? {}) }; + const session = { + sessionId, + cliId: 'codex-app', + codexAppDispatchLedger: options.ledger ?? [{ + dispatchId: 'dispatch-managed-origin', + turnId: TURN_ID, + dispatchAttempt: DISPATCH_ATTEMPT, + state: 'prepared', + content: 'prompt', + deliverySink: 'lark', + }], + }; + const active = { + session, + worker, + managedTurnOrigin, + initConfig: { cliId: 'codex-app' }, + larkAppId: 'app-managed-origin', + } as any; + config.session.dataDir = dataDir; + workerPool.setActiveSessionsRegistry(new Map([[sessionId, active]])); + + return { + active, + dataDir, + sessionId, + proofPath: (nonce: string, channelId = CHANNEL) => + managedOriginAttestationProofPath(dataDir, sessionId, channelId, nonce), + cleanup: () => { + workerPool.setActiveSessionsRegistry(previousRegistry ?? new Map()); + config.session.dataDir = previousDataDir; + rmSync(dataDir, { recursive: true, force: true }); + }, + }; + } + + async function postAttestation( + port: number, + body: Record, + ): Promise { + return fetch(`http://127.0.0.1:${port}/api/session-origin/attest`, { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify(body), + }); + } + + it('writes an exact nonce/channel/turn/ledger proof only for the live worker capability', async () => { + const fixture = installManagedOriginFixture(); + const nonce = 'cd'.repeat(32); + const issuedAfter = Date.now(); + try { + setIpcAuthSecret(TEST_IPC_SECRET); + handle = await startIpcServer({ port: 0, host: '127.0.0.1', authRequired: true }); + const res = await postAttestation(handle.port, { + sessionId: fixture.sessionId, + channelId: CHANNEL, + originCapability: CAPABILITY, + nonce, + }); + + expect(res.status).toBe(200); + expect(await res.json()).toEqual({ ok: true }); + const path = fixture.proofPath(nonce); + expect(statSync(path).mode & 0o777).toBe(0o600); + const proof = JSON.parse(readFileSync(path, 'utf8')); + expect(proof).toMatchObject({ + domain: MANAGED_ORIGIN_PROOF_DOMAIN, + version: 1, + nonce, + channelId: CHANNEL, + sessionId: fixture.sessionId, + turnId: TURN_ID, + dispatchAttempt: DISPATCH_ATTEMPT, + requiresCodexAppLedger: true, + }); + expect(proof.issuedAtMs).toBeGreaterThanOrEqual(issuedAfter); + expect(proof.issuedAtMs).toBeLessThanOrEqual(Date.now()); + } finally { + fixture.cleanup(); + } + }); + + it('rejects missing, disconnected, or dead exact workers without writing a proof', async () => { + setIpcAuthSecret(TEST_IPC_SECRET); + handle = await startIpcServer({ port: 0, host: '127.0.0.1', authRequired: true }); + const cases = [ + { name: 'missing', worker: null }, + { name: 'disconnected', worker: { connected: false } }, + { name: 'dead', worker: { pid: undefined } }, + ] as const; + for (const candidate of cases) { + const fixture = installManagedOriginFixture({ worker: candidate.worker }); + const nonce = randomBytes(32).toString('hex'); + try { + const res = await postAttestation(handle.port, { + sessionId: fixture.sessionId, + channelId: CHANNEL, + originCapability: CAPABILITY, + nonce, + }); + expect(res.status, candidate.name).toBe(403); + expect(await res.json(), candidate.name).toEqual({ ok: false, error: 'origin_unproven' }); + expect(existsSync(fixture.proofPath(nonce)), candidate.name).toBe(false); + } finally { + fixture.cleanup(); + } + } + }); + + it('rejects a wrong rotating capability without writing a proof', async () => { + const fixture = installManagedOriginFixture(); + const nonce = 'de'.repeat(32); + try { + setIpcAuthSecret(TEST_IPC_SECRET); + handle = await startIpcServer({ port: 0, host: '127.0.0.1', authRequired: true }); + const res = await postAttestation(handle.port, { + sessionId: fixture.sessionId, + channelId: CHANNEL, + originCapability: 'ef'.repeat(32), + nonce, + }); + + expect(res.status).toBe(403); + expect(await res.json()).toEqual({ ok: false, error: 'origin_unproven' }); + expect(existsSync(fixture.proofPath(nonce))).toBe(false); + } finally { + fixture.cleanup(); + } + }); + + it('rejects a missing or malformed live authority channel without writing a proof', async () => { + setIpcAuthSecret(TEST_IPC_SECRET); + handle = await startIpcServer({ port: 0, host: '127.0.0.1', authRequired: true }); + const cases = [ + { name: 'missing', originChannelId: undefined }, + { name: 'malformed', originChannelId: 'not-a-channel' }, + ] as const; + for (const candidate of cases) { + const fixture = installManagedOriginFixture({ + origin: { originChannelId: candidate.originChannelId }, + }); + const nonce = randomBytes(32).toString('hex'); + try { + const res = await postAttestation(handle.port, { + sessionId: fixture.sessionId, + channelId: CHANNEL, + originCapability: CAPABILITY, + nonce, + }); + expect(res.status, candidate.name).toBe(403); + expect(await res.json(), candidate.name).toEqual({ + ok: false, + error: 'origin_channel_unproven', + }); + expect(existsSync(fixture.proofPath(nonce)), candidate.name).toBe(false); + } finally { + fixture.cleanup(); + } + } + }); + + it('rejects a missing, malformed, or non-matching claimed channel without writing a proof', async () => { + setIpcAuthSecret(TEST_IPC_SECRET); + handle = await startIpcServer({ port: 0, host: '127.0.0.1', authRequired: true }); + const cases = [ + { name: 'missing', channelId: undefined, status: 400, error: 'bad_attestation_request' }, + { name: 'malformed', channelId: 'not-a-channel', status: 400, error: 'bad_attestation_request' }, + { name: 'non-matching', channelId: '88'.repeat(32), status: 403, error: 'origin_channel_unproven' }, + ] as const; + for (const candidate of cases) { + const fixture = installManagedOriginFixture(); + const nonce = randomBytes(32).toString('hex'); + try { + const res = await postAttestation(handle.port, { + sessionId: fixture.sessionId, + ...(candidate.channelId === undefined ? {} : { channelId: candidate.channelId }), + originCapability: CAPABILITY, + nonce, + }); + expect(res.status, candidate.name).toBe(candidate.status); + expect(await res.json(), candidate.name).toEqual({ + ok: false, + error: candidate.error, + }); + expect(existsSync(fixture.proofPath(nonce)), candidate.name).toBe(false); + } finally { + fixture.cleanup(); + } + } + }); + + it('rejects missing or non-exact Codex App ledger ownership without writing a proof', async () => { + setIpcAuthSecret(TEST_IPC_SECRET); + handle = await startIpcServer({ port: 0, host: '127.0.0.1', authRequired: true }); + const cases = [ + { name: 'missing', ledger: [] }, + { + name: 'wrong-turn', + ledger: [{ + dispatchId: 'dispatch-wrong-turn', + turnId: 'turn-other', + dispatchAttempt: DISPATCH_ATTEMPT, + state: 'prepared', + content: 'prompt', + deliverySink: 'lark', + }], + }, + { + name: 'wrong-attempt', + ledger: [{ + dispatchId: 'dispatch-wrong-attempt', + turnId: TURN_ID, + dispatchAttempt: DISPATCH_ATTEMPT + 1, + state: 'prepared', + content: 'prompt', + deliverySink: 'lark', + }], + }, + ]; + for (const candidate of cases) { + const fixture = installManagedOriginFixture({ ledger: candidate.ledger }); + const nonce = randomBytes(32).toString('hex'); + try { + const res = await postAttestation(handle.port, { + sessionId: fixture.sessionId, + channelId: CHANNEL, + originCapability: CAPABILITY, + nonce, + }); + expect(res.status, candidate.name).toBe(409); + expect(await res.json(), candidate.name).toEqual({ + ok: false, + error: 'origin_not_sendable', + }); + expect(existsSync(fixture.proofPath(nonce)), candidate.name).toBe(false); + } finally { + fixture.cleanup(); + } + } + }); + + it('rejects an oversized unauthenticated body before capability lookup and writes no proof', async () => { + const fixture = installManagedOriginFixture(); + const nonce = 'f0'.repeat(32); + try { + setIpcAuthSecret(TEST_IPC_SECRET); + handle = await startIpcServer({ port: 0, host: '127.0.0.1', authRequired: true }); + const res = await postAttestation(handle.port, { + sessionId: fixture.sessionId, + channelId: CHANNEL, + originCapability: CAPABILITY, + nonce, + padding: 'x'.repeat(3_000), + }); + + expect(res.status).toBe(413); + expect(res.headers.get('connection')).toBe('close'); + expect(await res.json()).toEqual({ ok: false, error: 'body_too_large' }); + expect(existsSync(fixture.proofPath(nonce))).toBe(false); + } finally { + fixture.cleanup(); + } + }); + + it('times out a slow partial unauthenticated body and writes no proof', async () => { + const fixture = installManagedOriginFixture(); + const nonce = 'f1'.repeat(32); + try { + setIpcAuthSecret(TEST_IPC_SECRET); + handle = await startIpcServer({ port: 0, host: '127.0.0.1', authRequired: true }); + const result = await new Promise<{ status: number; headers: Record; body: string }>((resolve, reject) => { + const req = httpRequest({ + host: '127.0.0.1', + port: handle!.port, + path: '/api/session-origin/attest', + method: 'POST', + headers: { 'content-type': 'application/json' }, + }, res => { + const chunks: Buffer[] = []; + res.on('data', chunk => chunks.push(Buffer.from(chunk))); + res.on('end', () => resolve({ + status: res.statusCode ?? 0, + headers: res.headers, + body: Buffer.concat(chunks).toString('utf8'), + })); + }); + req.once('error', reject); + // Send a complete JSON value but deliberately omit the terminating + // chunk, exercising the pre-auth slow-body deadline. + req.write(JSON.stringify({ + sessionId: fixture.sessionId, + channelId: CHANNEL, + originCapability: CAPABILITY, + nonce, + })); + }); + + expect(result.status).toBe(408); + expect(result.headers.connection).toBe('close'); + expect(JSON.parse(result.body)).toEqual({ ok: false, error: 'body_timeout' }); + expect(existsSync(fixture.proofPath(nonce))).toBe(false); + } finally { + fixture.cleanup(); + } + }, 5_000); +}); + describe('PUT /api/bot-card-prefs — Codex App clean history', () => { it('is default-off and persists explicit on/off changes immediately', async () => { const dir = mkdtempSync(join(tmpdir(), 'dashboard-ipc-codex-clean-')); @@ -361,6 +743,97 @@ describe('POST /api/sessions/:sessionId/lock', () => { }); }); +describe('POST /api/sessions/:sessionId/board queued activation', () => { + it('returns the activation failure without publishing a false in-progress success', async () => { + const dataDir = mkdtempSync(join(tmpdir(), 'dashboard-ipc-board-activation-')); + const previousDataDir = config.session.dataDir; + const previousRegistry = workerPool.getActiveSessionsRegistry(); + const appId = 'test-board-activation-app'; + const events: any[] = []; + const off = dashboardEventBus.subscribe(event => events.push(event)); + try { + config.session.dataDir = dataDir; + registerBot({ + larkAppId: appId, + larkAppSecret: 'secret', + cliId: 'codex', + defaultWorkingDir: '/tmp', + workingDir: '/tmp', + workingDirs: ['/tmp'], + } as any); + setLarkAppId(appId); + sessionStore.init(appId); + const session = sessionStore.createSession('oc_board', 'om_board', 'queued board task', 'group'); + Object.assign(session, { + larkAppId: appId, + scope: 'thread', + workingDir: '/tmp', + queued: true, + queuedPrompt: 'queued board payload', + kanbanColumn: 'backlog', + }); + sessionStore.updateSession(session); + const ds = { + session, + worker: null, + workerPort: null, + workerToken: null, + larkAppId: appId, + chatId: session.chatId, + chatType: 'group', + scope: 'thread', + spawnedAt: Date.now(), + cliVersion: 'test', + lastMessageAt: Date.now(), + hasHistory: false, + workingDir: '/tmp', + pendingPrompt: session.queuedPrompt, + } as any; + workerPool.setActiveSessionsRegistry(new Map([[sessionKey(session.rootMessageId, appId), ds]])); + workerPool.initWorkerPool({ + sessionReply: vi.fn(async () => 'om_reply'), + getSessionWorkingDir: () => { throw new Error('forced pre-init failure'); }, + getActiveCount: () => 1, + closeSession: vi.fn(), + }); + handle = await startIpcServer({ port: 0, host: '127.0.0.1' }); + + const res = await fetch( + `http://127.0.0.1:${handle.port}/api/sessions/${session.sessionId}/board`, + { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ column: 'in_progress', position: 7 }), + }, + ); + + expect(res.status).toBe(500); + expect(await res.json()).toEqual({ ok: false, error: 'forced pre-init failure' }); + expect(sessionStore.getSession(session.sessionId)).toMatchObject({ + queued: true, + queuedPrompt: 'queued board payload', + kanbanColumn: 'backlog', + }); + expect(events).not.toContainEqual(expect.objectContaining({ + type: 'session.update', + body: expect.objectContaining({ sessionId: session.sessionId }), + })); + } finally { + off(); + workerPool.setActiveSessionsRegistry(previousRegistry ?? new Map()); + workerPool.initWorkerPool({ + sessionReply: vi.fn(async () => 'om_reply'), + getSessionWorkingDir: () => '/tmp', + getActiveCount: () => 0, + closeSession: vi.fn(), + }); + sessionStore.init(); + config.session.dataDir = previousDataDir; + rmSync(dataDir, { recursive: true, force: true }); + } + }); +}); + describe('POST /api/sessions/:sessionId/restart', () => { it('sends a restart IPC message to the live worker', async () => { const send = vi.fn(); @@ -375,7 +848,7 @@ describe('POST /api/sessions/:sessionId/restart', () => { expect(res.status).toBe(200); expect(await res.json()).toMatchObject({ ok: true, sessionId: 's-restart', cliId: 'codex' }); - expect(send).toHaveBeenCalledWith({ type: 'restart' }); + expect(send).toHaveBeenCalledWith({ type: 'restart', reason: 'operator' }); findSpy.mockRestore(); }); @@ -410,6 +883,31 @@ describe('POST /api/sessions/:sessionId/restart', () => { forkSpy.mockRestore(); }); + it.each([ + ['live', { send: vi.fn(), killed: false }], + ['worker-less', null], + ])('rejects %s Riff sessions without restart or refork', async (_label, worker) => { + const forkSpy = vi.spyOn(workerPool, 'forkWorker').mockImplementation(() => {}); + const findSpy = vi.spyOn(workerPool, 'findActiveBySessionId').mockReturnValue({ + session: { sessionId: 's-riff', cliId: 'riff', backendType: 'riff' }, + initConfig: { backendType: 'riff' }, + worker, + adoptedFrom: undefined, + hasHistory: true, + larkAppId: 'app', + } as any); + + handle = await startIpcServer({ port: 0, host: '127.0.0.1' }); + const res = await fetch(`http://127.0.0.1:${handle.port}/api/sessions/s-riff/restart`, { method: 'POST' }); + + expect(res.status).toBe(409); + expect(await res.json()).toEqual({ ok: false, error: 'riff_restart_unsupported' }); + if (worker) expect(worker.send).not.toHaveBeenCalled(); + expect(forkSpy).not.toHaveBeenCalled(); + findSpy.mockRestore(); + forkSpy.mockRestore(); + }); + it('revives a worker-less but active session by re-forking (matches the Feishu card path)', async () => { const forkSpy = vi.spyOn(workerPool, 'forkWorker').mockImplementation(() => {}); const findSpy = vi.spyOn(workerPool, 'findActiveBySessionId').mockReturnValue({ @@ -480,6 +978,31 @@ describe('POST /api/sessions/:sessionId/suspend', () => { findSpy.mockRestore(); }); + it('409s before suspension while durable Codex App dispatch ownership is non-empty', async () => { + const ds = { + session: { + sessionId: 's-owned', + cliId: 'codex-app', + codexAppDispatchLedger: [ + { dispatchId: 'd-1', turnId: 't-1', state: 'prepared', content: 'owned' }, + ], + }, + worker: { send: vi.fn(), killed: false }, + adoptedFrom: undefined, + } as any; + const findSpy = vi.spyOn(workerPool, 'findActiveBySessionId').mockReturnValue(ds); + const suspendSpy = vi.spyOn(workerPool, 'suspendWorker').mockReturnValue(true); + + handle = await startIpcServer({ port: 0, host: '127.0.0.1' }); + const res = await fetch(`http://127.0.0.1:${handle.port}/api/sessions/s-owned/suspend`, { method: 'POST' }); + + expect(res.status).toBe(409); + expect(await res.json()).toMatchObject({ ok: false, error: 'codex_app_dispatch_pending' }); + expect(suspendSpy).not.toHaveBeenCalled(); + findSpy.mockRestore(); + suspendSpy.mockRestore(); + }); + it('rejects adopt/observed sessions (suspending would kill the user pane)', async () => { const suspendSpy = vi.spyOn(workerPool, 'suspendWorker').mockReturnValue(true); const findSpy = vi.spyOn(workerPool, 'findActiveBySessionId').mockReturnValue({ @@ -534,6 +1057,347 @@ describe('POST /api/sessions/:sessionId/suspend', () => { }); }); +describe('PUT /api/bot-read-isolation', () => { + for (const enabled of [false, true]) { + it(`treats ${enabled}→${enabled} as a no-op even with active and persisted pending owners`, async () => { + const appId = `test-read-isolation-noop-${enabled}`; + registerBot({ + larkAppId: appId, + larkAppSecret: 'secret', + cliId: 'codex-app', + workingDir: process.cwd(), + workingDirs: [process.cwd()], + readIsolation: enabled, + } as any); + setLarkAppId(appId); + const pendingLedger = [ + { dispatchId: 'd-noop', turnId: 't-noop', state: 'accepted', content: 'owned' }, + ]; + const previousRegistry = workerPool.getActiveSessionsRegistry(); + workerPool.setActiveSessionsRegistry(new Map([['active-noop', { + larkAppId: appId, + session: { sessionId: 's-active-noop', codexAppDispatchLedger: pendingLedger }, + worker: { send: vi.fn(), killed: false }, + } as any]])); + const listSpy = vi.spyOn(sessionStore, 'listSessions').mockReturnValue([{ + sessionId: 's-persisted-noop', + chatId: 'oc_noop', + rootMessageId: 'om_noop', + title: 'persisted pending no-op', + status: 'active', + createdAt: new Date().toISOString(), + larkAppId: appId, + backendType: 'tmux', + codexAppDispatchLedger: pendingLedger, + } as any]); + const updateSpy = vi.spyOn(sandboxStore, 'updateBotReadIsolation'); + const probeSpy = vi.spyOn(persistentBackend, 'probePersistentSession'); + const suspendSpy = vi.spyOn(workerPool, 'suspendWorker'); + try { + handle = await startIpcServer({ port: 0, host: '127.0.0.1' }); + const res = await fetch(`http://127.0.0.1:${handle.port}/api/bot-read-isolation`, { + method: 'PUT', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ enabled }), + }); + + expect(res.status).toBe(200); + expect(await res.json()).toEqual({ + ok: true, + readIsolation: enabled, + suspendedSessions: 0, + changed: false, + }); + expect(listSpy).not.toHaveBeenCalled(); + expect(updateSpy).not.toHaveBeenCalled(); + expect(probeSpy).not.toHaveBeenCalled(); + expect(suspendSpy).not.toHaveBeenCalled(); + } finally { + suspendSpy.mockRestore(); + probeSpy.mockRestore(); + updateSpy.mockRestore(); + listSpy.mockRestore(); + workerPool.setActiveSessionsRegistry(previousRegistry ?? new Map()); + } + }); + } + + it('rejects before persisting or suspending when any bot session owns a Codex App dispatch', async () => { + const appId = 'test-read-isolation-owned'; + registerBot({ + larkAppId: appId, + larkAppSecret: 'secret', + cliId: 'codex-app', + workingDir: process.cwd(), + workingDirs: [process.cwd()], + readIsolation: true, + } as any); + const owned = { + larkAppId: appId, + session: { + sessionId: 's-read-isolation-owned', + codexAppDispatchLedger: [ + { dispatchId: 'd-1', turnId: 't-1', state: 'prepared', content: 'owned' }, + ], + }, + worker: { send: vi.fn(), killed: false }, + } as any; + const previousRegistry = workerPool.getActiveSessionsRegistry(); + workerPool.setActiveSessionsRegistry(new Map([['owned', owned]])); + setLarkAppId(appId); + const suspendSpy = vi.spyOn(workerPool, 'suspendWorker'); + try { + handle = await startIpcServer({ port: 0, host: '127.0.0.1' }); + const res = await fetch(`http://127.0.0.1:${handle.port}/api/bot-read-isolation`, { + method: 'PUT', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ enabled: false }), + }); + + expect(res.status).toBe(409); + expect(await res.json()).toMatchObject({ ok: false, error: 'codex_app_dispatch_pending' }); + expect(suspendSpy).not.toHaveBeenCalled(); + } finally { + suspendSpy.mockRestore(); + workerPool.setActiveSessionsRegistry(previousRegistry ?? new Map()); + } + }); + + it('refuses read-isolation disable before persistence while an old-policy active session can resume', async () => { + const appId = 'test-read-isolation-active-disable'; + registerBot({ + larkAppId: appId, + larkAppSecret: 'secret', + cliId: 'codex-app', + workingDir: process.cwd(), + workingDirs: [process.cwd()], + readIsolation: true, + } as any); + const workerless = { + larkAppId: appId, + session: { sessionId: 's-read-isolation-active-disable', backendType: 'tmux' }, + initConfig: { backendType: 'tmux' }, + // A quiet restart/crash can leave worker=null while its old read-isolated + // pane survives and remains attachable. + worker: null, + } as any; + const previousRegistry = workerPool.getActiveSessionsRegistry(); + workerPool.setActiveSessionsRegistry(new Map([['workerless', workerless]])); + setLarkAppId(appId); + const updateSpy = vi.spyOn(sandboxStore, 'persistBotReadIsolation'); + try { + handle = await startIpcServer({ port: 0, host: '127.0.0.1' }); + const res = await fetch(`http://127.0.0.1:${handle.port}/api/bot-read-isolation`, { + method: 'PUT', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ enabled: false }), + }); + + expect(res.status).toBe(409); + expect(await res.json()).toMatchObject({ ok: false, error: 'read_isolation_active_sessions' }); + expect(updateSpy).not.toHaveBeenCalled(); + expect(sandboxStore.getBotReadIsolation(appId)).toBe(true); + } finally { + updateSpy.mockRestore(); + workerPool.setActiveSessionsRegistry(previousRegistry ?? new Map()); + } + }); + + it.runIf(process.platform === 'darwin')('refuses read-isolation enable before persistence while a write-only pane can survive restart', async () => { + const appId = 'test-read-isolation-active-enable'; + registerBot({ + larkAppId: appId, + larkAppSecret: 'secret', + cliId: 'codex-app', + workingDir: process.cwd(), + workingDirs: [process.cwd()], + sandbox: true, + } as any); + const workerless = { + larkAppId: appId, + session: { sessionId: 's-write-only-active', backendType: 'tmux', sandbox: true }, + initConfig: { backendType: 'tmux', sandbox: true, readIsolation: false }, + worker: null, + } as any; + const previousRegistry = workerPool.getActiveSessionsRegistry(); + workerPool.setActiveSessionsRegistry(new Map([['workerless-write-only', workerless]])); + setLarkAppId(appId); + const updateSpy = vi.spyOn(sandboxStore, 'persistBotReadIsolation'); + try { + handle = await startIpcServer({ port: 0, host: '127.0.0.1' }); + const res = await fetch(`http://127.0.0.1:${handle.port}/api/bot-read-isolation`, { + method: 'PUT', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ enabled: true }), + }); + + expect(res.status).toBe(409); + expect(await res.json()).toMatchObject({ ok: false, error: 'read_isolation_active_sessions' }); + expect(updateSpy).not.toHaveBeenCalled(); + expect(sandboxStore.getBotReadIsolation(appId)).toBe(false); + } finally { + updateSpy.mockRestore(); + workerPool.setActiveSessionsRegistry(previousRegistry ?? new Map()); + } + }); + + it('refuses before persistence for a durable active row omitted from the runtime registry', async () => { + const appId = 'test-read-isolation-persisted-active'; + registerBot({ + larkAppId: appId, + larkAppSecret: 'secret', + cliId: 'codex-app', + workingDir: process.cwd(), + workingDirs: [process.cwd()], + readIsolation: true, + } as any); + const previousRegistry = workerPool.getActiveSessionsRegistry(); + workerPool.setActiveSessionsRegistry(new Map()); + setLarkAppId(appId); + const listSpy = vi.spyOn(sessionStore, 'listSessions').mockReturnValue([{ + sessionId: 's-persisted-not-restored', + chatId: 'oc_persisted', + rootMessageId: 'om_persisted', + title: 'persisted active', + status: 'active', + createdAt: new Date().toISOString(), + larkAppId: appId, + backendType: 'tmux', + // Deliberately points at this live Vitest process. A closed row must not + // treat a reused pid as teardown authority; the stamped pane probe is. + pid: process.pid, + } as any]); + const updateSpy = vi.spyOn(sandboxStore, 'updateBotReadIsolation'); + try { + handle = await startIpcServer({ port: 0, host: '127.0.0.1' }); + const res = await fetch(`http://127.0.0.1:${handle.port}/api/bot-read-isolation`, { + method: 'PUT', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ enabled: false }), + }); + + expect(res.status).toBe(409); + expect(await res.json()).toMatchObject({ ok: false, error: 'read_isolation_active_sessions' }); + expect(updateSpy).not.toHaveBeenCalled(); + } finally { + updateSpy.mockRestore(); + listSpy.mockRestore(); + workerPool.setActiveSessionsRegistry(previousRegistry ?? new Map()); + } + }); + + it('waits for a just-closed persistent backing to disappear before changing policy', async () => { + const appId = 'test-read-isolation-close-teardown'; + registerBot({ + larkAppId: appId, + larkAppSecret: 'secret', + cliId: 'codex-app', + workingDir: process.cwd(), + workingDirs: [process.cwd()], + readIsolation: true, + } as any); + const previousRegistry = workerPool.getActiveSessionsRegistry(); + workerPool.setActiveSessionsRegistry(new Map()); + setLarkAppId(appId); + const listSpy = vi.spyOn(sessionStore, 'listSessions').mockReturnValue([{ + sessionId: 's-just-closed', + chatId: 'oc_closed', + rootMessageId: 'om_closed', + title: 'just closed', + status: 'closed', + createdAt: new Date().toISOString(), + larkAppId: appId, + backendType: 'tmux', + } as any]); + const probeSpy = vi.spyOn(persistentBackend, 'probePersistentSession') + .mockReturnValueOnce('exists') + .mockReturnValue('missing'); + const updateSpy = vi.spyOn(sandboxStore, 'updateBotReadIsolation') + .mockResolvedValue({ ok: true, readIsolation: false }); + try { + handle = await startIpcServer({ port: 0, host: '127.0.0.1' }); + const endpoint = `http://127.0.0.1:${handle.port}/api/bot-read-isolation`; + const request = () => fetch(endpoint, { + method: 'PUT', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ enabled: false }), + }); + + const first = await request(); + expect(first.status).toBe(409); + expect(await first.json()).toMatchObject({ + ok: false, + error: 'read_isolation_teardown_unverified', + }); + expect(updateSpy).not.toHaveBeenCalled(); + + const second = await request(); + expect(second.status).toBe(200); + expect(await second.json()).toMatchObject({ + ok: true, + readIsolation: false, + suspendedSessions: 0, + }); + expect(updateSpy).toHaveBeenCalledOnce(); + expect(probeSpy).toHaveBeenCalledWith('tmux', 'bmx-s-just-c'); + } finally { + updateSpy.mockRestore(); + probeSpy.mockRestore(); + listSpy.mockRestore(); + workerPool.setActiveSessionsRegistry(previousRegistry ?? new Map()); + } + }); + + it('does not synchronously fan out legacy closed rows across every persistent backend', async () => { + const appId = 'test-read-isolation-legacy-backing'; + registerBot({ + larkAppId: appId, + larkAppSecret: 'secret', + cliId: 'codex-app', + workingDir: process.cwd(), + workingDirs: [process.cwd()], + readIsolation: true, + } as any); + const previousRegistry = workerPool.getActiveSessionsRegistry(); + workerPool.setActiveSessionsRegistry(new Map()); + setLarkAppId(appId); + const listSpy = vi.spyOn(sessionStore, 'listSessions').mockReturnValue([{ + sessionId: 's-legacy-no-backend', + chatId: 'oc_legacy', + rootMessageId: 'om_legacy', + title: 'legacy closed', + status: 'closed', + createdAt: new Date().toISOString(), + larkAppId: appId, + // Deliberately no backendType stamp. + } as any]); + const probeSpy = vi.spyOn(persistentBackend, 'probePersistentSession'); + const updateSpy = vi.spyOn(sandboxStore, 'updateBotReadIsolation') + .mockResolvedValue({ ok: true, readIsolation: false }); + try { + handle = await startIpcServer({ port: 0, host: '127.0.0.1' }); + const res = await fetch(`http://127.0.0.1:${handle.port}/api/bot-read-isolation`, { + method: 'PUT', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ enabled: false }), + }); + + expect(res.status).toBe(200); + expect(await res.json()).toMatchObject({ + ok: true, + readIsolation: false, + }); + expect(updateSpy).toHaveBeenCalledOnce(); + expect(probeSpy).not.toHaveBeenCalled(); + } finally { + updateSpy.mockRestore(); + probeSpy.mockRestore(); + listSpy.mockRestore(); + workerPool.setActiveSessionsRegistry(previousRegistry ?? new Map()); + } + }); +}); + describe('POST /api/sessions/:sessionId/resume', () => { it('rejects a managed VC receiver without reactivating or waking it', async () => { const dataDir = mkdtempSync(join(tmpdir(), 'dashboard-ipc-resume-')); @@ -1116,6 +1980,196 @@ describe('PUT /api/bot-agent', () => { rmSync(dir, { recursive: true, force: true }); } }); + + it('rejects an unsettled Codex App session before config/readIsolation mutation or close', async () => { + const dir = mkdtempSync(join(tmpdir(), 'botmux-agent-pending-ipc-')); + const dataDir = join(dir, 'data'); + const configPath = join(dir, 'bots.json'); + const appId = 'test-agent-pending-app'; + const prevBotsConfig = process.env.BOTS_CONFIG; + const prevDataDir = config.session.dataDir; + try { + process.env.BOTS_CONFIG = configPath; + config.session.dataDir = dataDir; + writeFileSync(configPath, JSON.stringify([{ + larkAppId: appId, + larkAppSecret: 'secret', + cliId: 'codex-app', + model: 'old-model', + readIsolation: true, + }], null, 2)); + loadBotConfigs().forEach((c: any) => registerBot(c)); + sessionStore.init(appId); + const session = sessionStore.createSession('oc_pending', 'om_pending', 'Pending', 'group'); + session.larkAppId = appId; + session.cliId = 'codex-app'; + session.codexAppDispatchLedger = [{ + dispatchId: 'dispatch-pending', turnId: 'turn-pending', + state: 'prepared', content: 'prompt', deliverySink: 'lark', + }]; + sessionStore.updateSession(session); + const send = vi.fn(); + const registry = new Map([[sessionKey(session.rootMessageId, appId), { + session, + worker: { killed: false, send }, + workerPort: 1, + workerToken: 'token', + larkAppId: appId, + chatId: session.chatId, + chatType: 'group', + scope: 'thread', + spawnedAt: Date.now(), + cliVersion: 'test', + lastMessageAt: Date.now(), + hasHistory: true, + } as any]]); + workerPool.setActiveSessionsRegistry(registry); + setLarkAppId(appId); + handle = await startIpcServer({ port: 0, host: '127.0.0.1' }); + + const beforeFile = readFileSync(configPath, 'utf8'); + const beforeLive = structuredClone(getBot(appId).config); + const res = await fetch(`http://127.0.0.1:${handle.port}/api/bot-agent`, { + method: 'PUT', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ cliId: 'ttadk-x-codex', model: 'new-model' }), + }); + + expect(res.status).toBe(409); + expect(await res.json()).toEqual({ + ok: false, + error: 'codex_app_dispatch_pending', + blockingSessions: [{ + sessionId: session.sessionId, + cliId: 'codex-app', + reasons: ['codex_app_dispatch'], + }], + }); + expect(readFileSync(configPath, 'utf8')).toBe(beforeFile); + expect(getBot(appId).config).toEqual(beforeLive); + expect(sessionStore.getSession(session.sessionId)).toMatchObject({ + status: 'active', + codexAppDispatchLedger: [{ dispatchId: 'dispatch-pending' }], + }); + expect(registry.has(sessionKey(session.rootMessageId, appId))).toBe(true); + expect(send).not.toHaveBeenCalled(); + } finally { + workerPool.setActiveSessionsRegistry(new Map()); + sessionStore.init(); + config.session.dataDir = prevDataDir; + if (prevBotsConfig === undefined) delete process.env.BOTS_CONFIG; + else process.env.BOTS_CONFIG = prevBotsConfig; + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('reports non-Codex pending work with a backend-neutral error and actionable sessions', async () => { + const dir = mkdtempSync(join(tmpdir(), 'botmux-agent-generic-pending-ipc-')); + const dataDir = join(dir, 'data'); + const configPath = join(dir, 'bots.json'); + const appId = 'test-agent-generic-pending-app'; + const prevBotsConfig = process.env.BOTS_CONFIG; + const prevDataDir = config.session.dataDir; + try { + process.env.BOTS_CONFIG = configPath; + config.session.dataDir = dataDir; + writeFileSync(configPath, JSON.stringify([{ + larkAppId: appId, + larkAppSecret: 'secret', + cliId: 'traex', + }], null, 2)); + loadBotConfigs().forEach((c: any) => registerBot(c)); + sessionStore.init(appId); + const session = sessionStore.createSession( + 'oc_generic_pending', + 'om_generic_pending', + 'Generic pending', + 'group', + ); + session.larkAppId = appId; + session.cliId = 'traex'; + session.queued = true; + session.pendingRepoSetup = { mode: 'picker', prompt: 'OPENING_N' }; + sessionStore.updateSession(session); + setLarkAppId(appId); + handle = await startIpcServer({ port: 0, host: '127.0.0.1' }); + + const res = await fetch(`http://127.0.0.1:${handle.port}/api/bot-agent`, { + method: 'PUT', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ cliId: 'codex', model: '' }), + }); + + expect(res.status).toBe(409); + expect(await res.json()).toEqual({ + ok: false, + error: 'session_mutation_pending', + blockingSessions: [{ + sessionId: session.sessionId, + cliId: 'traex', + reasons: ['queued_todo', 'repository_setup'], + }], + }); + expect(JSON.parse(readFileSync(configPath, 'utf8'))[0].cliId).toBe('traex'); + } finally { + workerPool.setActiveSessionsRegistry(new Map()); + sessionStore.init(); + config.session.dataDir = prevDataDir; + if (prevBotsConfig === undefined) delete process.env.BOTS_CONFIG; + else process.env.BOTS_CONFIG = prevBotsConfig; + rmSync(dir, { recursive: true, force: true }); + } + }); + + it('keeps the hot-switch mismatch close after a settled Codex App ledger', async () => { + const dir = mkdtempSync(join(tmpdir(), 'botmux-agent-settled-ipc-')); + const dataDir = join(dir, 'data'); + const configPath = join(dir, 'bots.json'); + const appId = 'test-agent-settled-app'; + const prevBotsConfig = process.env.BOTS_CONFIG; + const prevDataDir = config.session.dataDir; + try { + process.env.BOTS_CONFIG = configPath; + config.session.dataDir = dataDir; + writeFileSync(configPath, JSON.stringify([{ + larkAppId: appId, larkAppSecret: 'secret', cliId: 'codex-app', + }], null, 2)); + loadBotConfigs().forEach((c: any) => registerBot(c)); + sessionStore.init(appId); + const session = sessionStore.createSession('oc_settled', 'om_settled', 'Settled', 'group'); + session.larkAppId = appId; + session.cliId = 'codex-app'; + session.codexAppDispatchLedger = []; + sessionStore.updateSession(session); + const registry = new Map([[sessionKey(session.rootMessageId, appId), { + session, worker: null, workerPort: null, workerToken: null, + larkAppId: appId, chatId: session.chatId, chatType: 'group', scope: 'thread', + spawnedAt: Date.now(), cliVersion: 'test', lastMessageAt: Date.now(), + hasHistory: true, + } as any]]); + workerPool.setActiveSessionsRegistry(registry); + setLarkAppId(appId); + handle = await startIpcServer({ port: 0, host: '127.0.0.1' }); + + const res = await fetch(`http://127.0.0.1:${handle.port}/api/bot-agent`, { + method: 'PUT', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ cliId: 'codex', model: '' }), + }); + + expect(res.status).toBe(200); + expect(await res.json()).toMatchObject({ ok: true, closedMismatchedSessions: 1 }); + expect(sessionStore.getSession(session.sessionId)?.status).toBe('closed'); + expect(registry.has(sessionKey(session.rootMessageId, appId))).toBe(false); + } finally { + workerPool.setActiveSessionsRegistry(new Map()); + sessionStore.init(); + config.session.dataDir = prevDataDir; + if (prevBotsConfig === undefined) delete process.env.BOTS_CONFIG; + else process.env.BOTS_CONFIG = prevBotsConfig; + rmSync(dir, { recursive: true, force: true }); + } + }); }); describe('PUT /api/bot-riff config safety (finding H)', () => { diff --git a/test/dashboard-sessions-ui.test.ts b/test/dashboard-sessions-ui.test.ts index 1cb3974b3..79c45a661 100644 --- a/test/dashboard-sessions-ui.test.ts +++ b/test/dashboard-sessions-ui.test.ts @@ -6,6 +6,8 @@ import { SessionsKanbanView, type SessionsKanbanCallbacks, type SessionsKanbanSt import { canRestartSession, CLI_FILTER_OPTIONS, + SESSION_STATUS_OPTIONS, + deriveSessionBoardColumn, groupSessionsByTopic, isUnknownChatSession, restartConfirmMessage, @@ -271,6 +273,11 @@ describe('dashboard sessions filters', () => { expect((html.match(/
{ // ─── Imports (must be after mocks) ────────────────────────────────────────── import { __resetAnchorQueues } from '../src/utils/anchor-serializer.js'; -import { __resetEventClaimsForTest, canOperate, canTalk, decideRouting, ensureBotOpenId, isBotMentioned, mentionsAnotherMember, markForwardFollowupsSessionsReady, startLarkEventDispatcher, writeBotInfoFile, type EventHandlers } from '../src/im/lark/event-dispatcher.js'; +import { __resetEventClaimsForTest, canOperate, canTalk, decideRouting, ensureBotOpenId, isBotMentioned, mentionsAnotherMember, markForwardFollowupsSessionsReady, rawMessageIngressAnchor, startLarkEventDispatcher, writeBotInfoFile, type EventHandlers } from '../src/im/lark/event-dispatcher.js'; import { VC_BOT_MEETING_ACTIVITY_EVENT, VC_BOT_MEETING_ENDED_EVENT, @@ -5870,6 +5870,213 @@ describe('im.message.receive_v1 — ack-safe duplicate delivery', () => { await flushEventWork(); }); + it('reserves same-chat arrival order before the first async routing lookup', async () => { + let releaseFirstMode!: (mode: 'group') => void; + const firstMode = new Promise<'group'>(resolve => { releaseFirstMode = resolve; }); + mockGetChatMode.mockReset() + .mockImplementationOnce(async () => firstMode) + .mockResolvedValue('group'); + + const handled: string[] = []; + handlers.handleNewTopic.mockImplementation(async (data: any) => { + handled.push(data.message.message_id); + }); + const event = (messageId: string) => makeUserMessageEvent({ + senderOpenId: USER_OPEN_ID, + content: JSON.stringify({ text: `@BotA ${messageId}` }), + messageId, + chatId: 'chat-ingress-order', + chatType: 'group', + mentions: [{ key: '@_bot_a', name: 'BotA', id: { open_id: MY_OPEN_ID } }], + }); + + capturedHandlers['im.message.receive_v1'](event('msg-ingress-n')); + capturedHandlers['im.message.receive_v1'](event('msg-ingress-n-plus-1')); + await new Promise(resolve => setImmediate(resolve)); + await Promise.resolve(); + + // N is blocked inside decideRouting. N+1 must not even enter that lookup; + // ordering at the later canonical anchor is already too late. + expect(mockGetChatMode).toHaveBeenCalledTimes(1); + expect(handled).toEqual([]); + + releaseFirstMode('group'); + await flushEventWork(); + await flushEventWork(); + expect(handled).toEqual(['msg-ingress-n', 'msg-ingress-n-plus-1']); + }); + + it('orders a topic seed before its thread-shaped reply across raw topology', async () => { + let releaseSeedMode!: (mode: 'topic') => void; + const seedMode = new Promise<'topic'>(resolve => { releaseSeedMode = resolve; }); + mockGetChatMode.mockReset() + .mockImplementationOnce(async () => seedMode) + .mockResolvedValue('topic'); + + const handled: string[] = []; + handlers.handleNewTopic.mockImplementation(async (data: any) => { + handled.push(data.message.message_id); + }); + handlers.handleThreadReply.mockImplementation(async (data: any) => { + handled.push(data.message.message_id); + }); + const mentions = [{ key: '@_bot_a', name: 'BotA', id: { open_id: MY_OPEN_ID } }]; + const seed = makeUserMessageEvent({ + senderOpenId: USER_OPEN_ID, + content: JSON.stringify({ text: '@BotA seed' }), + messageId: 'msg-topic-seed', + chatId: 'chat-topic-ingress', + chatType: 'group', + mentions, + }); + const reply = makeUserMessageEvent({ + senderOpenId: USER_OPEN_ID, + content: JSON.stringify({ text: '@BotA reply' }), + messageId: 'msg-topic-reply', + rootId: 'msg-topic-seed', + threadId: 'omt-topic-thread', + chatId: 'chat-topic-ingress', + chatType: 'group', + mentions, + }); + + capturedHandlers['im.message.receive_v1'](seed); + capturedHandlers['im.message.receive_v1'](reply); + await new Promise(resolve => setImmediate(resolve)); + await Promise.resolve(); + + // A thread-specific raw lane lets the reply bypass the seed here even + // though both later canonicalize to anchor=msg-topic-seed. + expect(handled).toEqual([]); + + releaseSeedMode('topic'); + await flushEventWork(); + await flushEventWork(); + expect(handled).toEqual(['msg-topic-seed', 'msg-topic-reply']); + }); + + it('bounds a reply wait when the earlier seed routing lookup is wedged', async () => { + vi.useFakeTimers(); + try { + capturedHandlers = {}; + __resetAnchorQueues(); + __resetEventClaimsForTest(); + config.daemon.forwardFollowupWaitMs = 25; + let releaseSeedMode!: (mode: 'topic') => void; + const seedMode = new Promise<'topic'>(resolve => { releaseSeedMode = resolve; }); + mockGetChatMode.mockReset() + .mockImplementationOnce(async () => seedMode) + .mockResolvedValue('topic'); + startLarkEventDispatcher(MY_APP_ID, 'secret', handlers); + + const handled: string[] = []; + handlers.handleNewTopic.mockImplementation(async data => { + handled.push(data.message.message_id); + }); + handlers.handleThreadReply.mockImplementation(async data => { + handled.push(data.message.message_id); + }); + const mentions = [{ key: '@_bot_a', name: 'BotA', id: { open_id: MY_OPEN_ID } }]; + const seed = makeUserMessageEvent({ + senderOpenId: USER_OPEN_ID, + content: JSON.stringify({ text: '@BotA seed' }), + messageId: 'msg-wedged-seed', + chatId: 'chat-wedged-seed', + chatType: 'group', + mentions, + }); + const reply = makeUserMessageEvent({ + senderOpenId: USER_OPEN_ID, + content: JSON.stringify({ text: '@BotA reply' }), + messageId: 'msg-wedged-reply', + rootId: 'msg-wedged-seed', + threadId: 'omt-wedged', + chatId: 'chat-wedged-seed', + chatType: 'group', + mentions, + }); + + capturedHandlers['im.message.receive_v1'](seed); + capturedHandlers['im.message.receive_v1'](reply); + await vi.advanceTimersByTimeAsync(5_030); + expect(handled).toContain('msg-wedged-reply'); + + releaseSeedMode('topic'); + await vi.advanceTimersByTimeAsync(0); + await Promise.resolve(); + expect(handled).toContain('msg-wedged-seed'); + } finally { + vi.useRealTimers(); + } + }); + + it('releases the chat routing barrier after enqueue so distinct topics still execute concurrently', async () => { + mockGetChatMode.mockReset().mockResolvedValue('topic'); + let releaseFirst!: () => void; + const firstPending = new Promise(resolve => { releaseFirst = resolve; }); + const started: string[] = []; + handlers.handleNewTopic.mockImplementation(async (data: any) => { + started.push(data.message.message_id); + if (data.message.message_id === 'msg-independent-topic-1') await firstPending; + }); + const event = (messageId: string) => makeUserMessageEvent({ + senderOpenId: USER_OPEN_ID, + content: JSON.stringify({ text: `@BotA ${messageId}` }), + messageId, + chatId: 'chat-independent-topics', + chatType: 'group', + mentions: [{ key: '@_bot_a', name: 'BotA', id: { open_id: MY_OPEN_ID } }], + }); + + capturedHandlers['im.message.receive_v1'](event('msg-independent-topic-1')); + capturedHandlers['im.message.receive_v1'](event('msg-independent-topic-2')); + await flushEventWork(); + + expect(started).toEqual(['msg-independent-topic-1', 'msg-independent-topic-2']); + releaseFirst(); + await flushEventWork(); + }); + + it('keeps the canonical session FIFO strict past five seconds', async () => { + vi.useFakeTimers(); + try { + mockGetChatMode.mockReset().mockResolvedValue('group'); + let releaseFirst!: () => void; + const firstPending = new Promise(resolve => { releaseFirst = resolve; }); + const started: string[] = []; + handlers.handleNewTopic.mockImplementation(async (data: any) => { + started.push(data.message.message_id); + if (data.message.message_id === 'msg-canonical-strict-1') await firstPending; + }); + const event = (messageId: string) => makeUserMessageEvent({ + senderOpenId: USER_OPEN_ID, + content: JSON.stringify({ text: `@BotA ${messageId}` }), + messageId, + chatId: 'chat-canonical-strict', + chatType: 'group', + mentions: [{ key: '@_bot_a', name: 'BotA', id: { open_id: MY_OPEN_ID } }], + }); + + capturedHandlers['im.message.receive_v1'](event('msg-canonical-strict-1')); + capturedHandlers['im.message.receive_v1'](event('msg-canonical-strict-2')); + await vi.advanceTimersByTimeAsync(0); + await Promise.resolve(); + await Promise.resolve(); + expect(started).toEqual(['msg-canonical-strict-1']); + + await vi.advanceTimersByTimeAsync(5_100); + expect(started).toEqual(['msg-canonical-strict-1']); + + releaseFirst(); + await Promise.resolve(); + await Promise.resolve(); + await vi.advanceTimersByTimeAsync(0); + expect(started).toEqual(['msg-canonical-strict-1', 'msg-canonical-strict-2']); + } finally { + vi.useRealTimers(); + } + }); + it('dedupes timeout redelivery of the same message_id', async () => { const event = makeUserMessageEvent({ senderOpenId: USER_OPEN_ID, @@ -5909,6 +6116,49 @@ describe('im.message.receive_v1 — ack-safe duplicate delivery', () => { }); }); +describe('rawMessageIngressAnchor', () => { + it('keeps top-level and quote-bubble messages on the same chat lane', () => { + const topLevel = rawMessageIngressAnchor(MY_APP_ID, { chat_id: 'oc_same' }); + const quoteBubble = rawMessageIngressAnchor(MY_APP_ID, { + chat_id: 'oc_same', + root_id: 'om_quoted', + }); + expect(quoteBubble).toBe(topLevel); + }); + + it('uses one routing barrier per chat while isolating bot apps', () => { + const first = rawMessageIngressAnchor(MY_APP_ID, { + chat_id: 'oc_same', + thread_id: 'omt_first', + }); + const second = rawMessageIngressAnchor(MY_APP_ID, { + chat_id: 'oc_same', + thread_id: 'omt_second', + }); + const otherBot = rawMessageIngressAnchor(OTHER_BOT_APP_ID, { + chat_id: 'oc_same', + thread_id: 'omt_first', + }); + expect(second).toBe(first); + expect(otherBot).not.toBe(first); + }); + + it('folds thread-shaped p2p replies together when DM routing is chat-scope', () => { + setupBotState({ p2pMode: 'chat' }); + const first = rawMessageIngressAnchor(MY_APP_ID, { + chat_id: 'oc_dm', + chat_type: 'p2p', + thread_id: 'omt_first', + }); + const second = rawMessageIngressAnchor(MY_APP_ID, { + chat_id: 'oc_dm', + chat_type: 'p2p', + thread_id: 'omt_second', + }); + expect(second).toBe(first); + }); +}); + describe('writeBotInfoFile — multi-daemon merge', () => { beforeEach(() => { mockExistsSync.mockReturnValue(true); diff --git a/test/explicit-session-backing-cleanup.test.ts b/test/explicit-session-backing-cleanup.test.ts new file mode 100644 index 000000000..eb15f436a --- /dev/null +++ b/test/explicit-session-backing-cleanup.test.ts @@ -0,0 +1,113 @@ +import { describe, expect, it, vi } from 'vitest'; +import { cleanupExplicitSessionBacking } from '../src/core/explicit-session-backing-cleanup.js'; + +const SID = 'abcd1234-1111-2222-3333-444444444444'; + +describe('offline explicit session backing cleanup', () => { + for (const backendType of ['tmux', 'herdr', 'zellij'] as const) { + it(`confirms ${backendType} is absent before reporting cleanup success`, async () => { + const killPersistent = vi.fn(); + const probePersistent = vi.fn(() => 'missing' as const); + const result = await cleanupExplicitSessionBacking( + { sessionId: SID, backendType }, + { killPersistent, probePersistent }, + ); + + expect(result).toEqual({ + ok: true, + kind: 'destroyed_persistent', + backendType, + name: 'bmx-abcd1234', + }); + expect(killPersistent).toHaveBeenCalledWith(backendType, 'bmx-abcd1234'); + expect(probePersistent).toHaveBeenCalledWith(backendType, 'bmx-abcd1234'); + }); + } + + it('does not report success when a persistent backend still exists after kill', async () => { + const result = await cleanupExplicitSessionBacking( + { sessionId: SID, backendType: 'tmux' }, + { + killPersistent: vi.fn(), + probePersistent: vi.fn(() => 'exists'), + }, + ); + expect(result).toMatchObject({ + ok: false, + kind: 'persistent_destroy_failed', + error: 'backing_session_still_exists', + }); + }); + + it('cancels Riff using the supplied authoritative bot config and returns the exact task id', async () => { + const cancelRiffTask = vi.fn(async () => true); + const result = await cleanupExplicitSessionBacking( + { + sessionId: SID, + backendType: 'riff', + riffParentTaskId: 'riff-task-1', + riffConfig: { baseUrl: 'https://riff.example', jwt: 'token' }, + }, + { cancelRiffTask }, + ); + expect(cancelRiffTask).toHaveBeenCalledWith( + { baseUrl: 'https://riff.example', jwt: 'token' }, + 'riff-task-1', + ); + expect(result).toEqual({ ok: true, kind: 'cancelled_riff', taskId: 'riff-task-1' }); + }); + + it('fails closed on Riff cancel failure so the caller can preserve its task id', async () => { + const record = { + sessionId: SID, + backendType: 'riff' as const, + riffParentTaskId: 'riff-task-retry', + riffConfig: { baseUrl: 'https://riff.example' }, + }; + const result = await cleanupExplicitSessionBacking(record, { + cancelRiffTask: vi.fn(async () => false), + }); + expect(result).toEqual({ + ok: false, + kind: 'riff_cancel_failed', + backendType: 'riff', + taskId: 'riff-task-retry', + }); + expect(record.riffParentTaskId).toBe('riff-task-retry'); + }); + + it('fails closed when current Riff config is unavailable', async () => { + expect(await cleanupExplicitSessionBacking({ + sessionId: SID, + backendType: 'riff', + riffParentTaskId: 'riff-task-no-config', + })).toEqual({ + ok: false, + kind: 'riff_config_missing', + backendType: 'riff', + taskId: 'riff-task-no-config', + }); + }); + + it('never destroys an adopted user-owned pane', async () => { + const killPersistent = vi.fn(); + const probePersistent = vi.fn(); + expect(await cleanupExplicitSessionBacking( + { sessionId: SID, backendType: 'tmux', adopted: true }, + { killPersistent, probePersistent: probePersistent as any }, + )).toEqual({ ok: true, kind: 'skipped_adopted' }); + expect(killPersistent).not.toHaveBeenCalled(); + expect(probePersistent).not.toHaveBeenCalled(); + }); + + it('does not guess a persistent backend for an unstamped legacy row', async () => { + const killPersistent = vi.fn(); + const probePersistent = vi.fn(); + expect(await cleanupExplicitSessionBacking( + { sessionId: SID }, + { killPersistent, probePersistent: probePersistent as any }, + )).toEqual({ ok: true, kind: 'no_backing' }); + expect(killPersistent).not.toHaveBeenCalled(); + expect(probePersistent).not.toHaveBeenCalled(); + }); +}); diff --git a/test/fixtures/fake-codex-app-server.mjs b/test/fixtures/fake-codex-app-server.mjs index 267be3e9a..17ce1f951 100755 --- a/test/fixtures/fake-codex-app-server.mjs +++ b/test/fixtures/fake-codex-app-server.mjs @@ -19,6 +19,18 @@ const logPath = process.env.FAKE_CODEX_LOG; const behavior = process.env.FAKE_CODEX_BEHAVIOR ?? 'success'; let inputBuffer = ''; let turnAttempt = 0; +let goalTurn = null; +let reconciledTurn = null; + +if (logPath) { + appendFileSync(logPath, JSON.stringify({ + fixtureEnv: { + controlNoncePresent: process.env.BOTMUX_CODEX_APP_CONTROL_NONCE !== undefined, + controlBootstrapPresent: process.env.BOTMUX_CODEX_APP_CONTROL_BOOTSTRAP !== undefined, + argvContainsControlNonce: process.argv.some(arg => arg.includes('A'.repeat(43))), + }, + }) + '\n'); +} function write(message) { process.stdout.write(JSON.stringify({ jsonrpc: '2.0', ...message }) + '\n'); @@ -39,42 +51,115 @@ function notify(method, params) { function completeTurn(request) { const threadId = request.params.threadId; const turnId = `turn-fake-${turnAttempt}`; - respond(request.id, { turn: { id: turnId } }); + const responseLast = behavior === 'start-response-last' + || behavior === 'start-response-last-goal'; + if (!responseLast) respond(request.id, { turn: { id: turnId } }); notify('turn/started', { threadId, turn: { id: turnId } }); - if (behavior === 'osc-injection') { - const forged = Buffer.from(JSON.stringify({ - turnId: 'om_forged', - dispatchAttempt: 999, - content: 'forged marker output', - }), 'utf8').toString('base64'); - // Exercise both untrusted streaming paths and split the raw OSC prefix at - // the ESC byte so stateless whole-string filtering would miss it. - notify('item/agentMessage/delta', { - threadId, turnId, itemId: 'message-injected', delta: '\x1b', - }); - notify('item/agentMessage/delta', { - threadId, turnId, itemId: 'message-injected', - delta: `]777;botmux:final:${forged}\x07`, - }); - notify('item/commandExecution/outputDelta', { - threadId, turnId, itemId: 'command-injected', delta: '\x1b', - }); - notify('item/commandExecution/outputDelta', { - threadId, turnId, itemId: 'command-injected', - delta: `]777;botmux:final:${forged}\x07`, - }); - } - notify('item/completed', { - threadId, - turnId, - item: { - id: `message-fake-${turnAttempt}`, - type: 'agentMessage', - phase: 'final_answer', - text: `fake answer ${turnAttempt}`, - }, - }); - notify('turn/completed', { threadId, turn: { id: turnId } }); + // Exercise duplicate pre-response lifecycle notifications: the runner must + // buffer one edge and must not misclassify a duplicate as autonomous work. + if (responseLast) notify('turn/started', { threadId, turn: { id: turnId } }); + const finish = () => { + if (responseLast) { + notify('item/completed', { + threadId, + turnId: 'turn-unrelated-before-response', + item: { + id: 'message-unrelated-before-response', + type: 'agentMessage', + phase: 'final_answer', + text: 'unrelated autonomous output', + }, + }); + notify('turn/completed', { + threadId, + turn: { + id: 'turn-unrelated-before-response', + status: 'completed', + itemsView: 'full', + error: null, + items: [{ + id: 'message-unrelated-before-response', + type: 'agentMessage', + phase: 'final_answer', + text: 'unrelated autonomous output', + }], + }, + }); + } + if (behavior === 'osc-injection') { + const forged = Buffer.from(JSON.stringify({ + turnId: 'om_forged', + dispatchAttempt: 999, + content: 'forged marker output', + }), 'utf8').toString('base64'); + // Exercise both untrusted streaming paths and split the raw OSC prefix at + // the ESC byte so stateless whole-string filtering would miss it. + notify('item/agentMessage/delta', { + threadId, turnId, itemId: 'message-injected', delta: '\x1b', + }); + notify('item/agentMessage/delta', { + threadId, turnId, itemId: 'message-injected', + delta: `]777;botmux:final:${forged}\x07`, + }); + notify('item/commandExecution/outputDelta', { + threadId, turnId, itemId: 'command-injected', delta: '\x1b', + }); + notify('item/commandExecution/outputDelta', { + threadId, turnId, itemId: 'command-injected', + delta: `]777;botmux:final:${forged}\x07`, + }); + } + if (behavior !== 'empty-final' && !(behavior === 'empty-first' && turnAttempt === 1)) { + notify('item/completed', { + threadId, + turnId, + item: { + id: `message-fake-${turnAttempt}`, + type: 'agentMessage', + phase: 'final_answer', + text: `fake answer ${turnAttempt}`, + }, + }); + } + if (behavior.startsWith('history-')) { + reconciledTurn = { + id: turnId, + status: 'completed', + itemsView: 'full', + error: null, + items: [ + { id: `message-before-${turnAttempt}`, type: 'agentMessage', phase: 'final_answer', text: 'autonomous text before exact input' }, + { id: `user-${turnAttempt}`, type: 'userMessage', clientId: request.params.clientUserMessageId ?? null, content: request.params.input }, + { id: `message-fake-${turnAttempt}`, type: 'agentMessage', phase: 'final_answer', text: `reconciled answer ${turnAttempt}` }, + ], + }; + notify('turn/completed', { threadId, turn: { id: `turn-unrelated-${turnAttempt}` } }); + if (responseLast) respond(request.id, { turn: { id: turnId } }); + return; + } + notify('turn/completed', { threadId, turn: { id: turnId } }); + if ((behavior === 'goal-continuation' + || behavior === 'goal-steer-race' + || behavior === 'start-response-last-goal') && turnAttempt === 1) { + goalTurn = { + id: 'turn-goal-auto', + threadId, + items: [{ + id: 'message-goal-before-input', + type: 'agentMessage', + phase: 'final_answer', + text: 'autonomous goal text before Lark input', + }], + }; + notify('turn/started', { + threadId, + turn: { id: goalTurn.id, status: 'inProgress', itemsView: 'full', items: goalTurn.items }, + }); + } + if (responseLast) respond(request.id, { turn: { id: turnId } }); + }; + if (behavior === 'delayed-first' && turnAttempt === 1) setTimeout(finish, 300); + else finish(); } function handle(request) { @@ -82,6 +167,7 @@ function handle(request) { if (typeof request.id !== 'number') return; if (request.method === 'initialize') { + if (behavior === 'hang-initialize') return; respond(request.id, { userAgent: 'fake-codex-app-server' }); return; } @@ -90,6 +176,11 @@ function handle(request) { return; } if (request.method === 'thread/resume') { + if (behavior === 'hang-resume') return; + if (behavior === 'resume-not-found') { + reject(request.id, -32001, `thread ${request.params.threadId} not found`); + return; + } respond(request.id, { thread: { id: request.params.threadId } }); return; } @@ -97,6 +188,67 @@ function handle(request) { respond(request.id, {}); return; } + if (request.method === 'thread/turns/list') { + const data = behavior === 'history-no-match' + ? [] + : behavior === 'history-multi-match' && reconciledTurn + ? [reconciledTurn, { ...reconciledTurn, id: `${reconciledTurn.id}-duplicate` }] + : reconciledTurn ? [reconciledTurn] : []; + respond(request.id, { + data, + nextCursor: null, + backwardsCursor: null, + }); + return; + } + if (request.method === 'turn/steer') { + if (!goalTurn || request.params.expectedTurnId !== goalTurn.id) { + reject(request.id, -32000, 'expected turn is not active'); + return; + } + if (behavior === 'goal-steer-race') { + const completedGoal = goalTurn; + goalTurn = null; + notify('turn/completed', { + threadId: completedGoal.threadId, + turn: { + id: completedGoal.id, + status: 'completed', + itemsView: 'full', + error: null, + items: completedGoal.items, + }, + }); + reject(request.id, -32000, 'expected turn is not active'); + return; + } + respond(request.id, { turnId: goalTurn.id }); + const user = { + id: 'user-goal-steer', + type: 'userMessage', + clientId: request.params.clientUserMessageId ?? null, + content: request.params.input, + }; + const answer = { + id: 'message-goal-steer', + type: 'agentMessage', + phase: 'final_answer', + text: 'goal steer answer', + }; + notify('item/completed', { threadId: goalTurn.threadId, turnId: goalTurn.id, item: answer }); + notify('turn/completed', { + threadId: goalTurn.threadId, + turn: { + id: goalTurn.id, + status: 'completed', + itemsView: 'full', + error: null, + items: [...goalTurn.items, user, answer], + }, + }); + goalTurn = null; + return; + } if (request.method !== 'turn/start') { respond(request.id, {}); return; diff --git a/test/fixtures/fake-codex-rpc-server.mjs b/test/fixtures/fake-codex-rpc-server.mjs index a9b8c8a4b..bff00718d 100755 --- a/test/fixtures/fake-codex-rpc-server.mjs +++ b/test/fixtures/fake-codex-rpc-server.mjs @@ -4,6 +4,10 @@ // SAME port (as the real app-server does), answering the handshake + thread/turn // requests. Env knobs drive the failure-path tests: // FAKE_HANG_TURN=1 → never answer turn/start (wedged app-server) +// FAKE_HANG_TURN_NOTIFY=1 → emit started/completed but lose the ack +// FAKE_TERMINAL_BEFORE_RESPONSE=1 → broadcast terminal before turn/start ack +// FAKE_ERROR_AFTER_STARTED=1 → emit turn/started, reject the response, then complete +// FAKE_DUPLICATE_TERMINAL=1 → broadcast turn/completed twice // FAKE_DIE_AFTER_MS=N → exit(1) after N ms (crash → engine onDead) import { createServer } from 'node:http'; import { WebSocketServer } from 'ws'; @@ -12,7 +16,14 @@ const listenArg = process.argv[process.argv.indexOf('--listen') + 1] || ''; const m = listenArg.match(/ws:\/\/127\.0\.0\.1:(\d+)/); const port = m ? Number(m[1]) : 0; const HANG_TURN = process.env.FAKE_HANG_TURN === '1'; +const HANG_TURN_NOTIFY = process.env.FAKE_HANG_TURN_NOTIFY === '1'; +const TERMINAL_BEFORE_RESPONSE = process.env.FAKE_TERMINAL_BEFORE_RESPONSE === '1'; +const ERROR_AFTER_STARTED = process.env.FAKE_ERROR_AFTER_STARTED === '1'; +const DUPLICATE_TERMINAL = process.env.FAKE_DUPLICATE_TERMINAL === '1'; +const NO_TURN_TERMINAL = process.env.FAKE_NO_TURN_TERMINAL === '1'; +const TURN_STATUS = process.env.FAKE_TURN_STATUS ?? ''; const DIE_AFTER = process.env.FAKE_DIE_AFTER_MS ? Number(process.env.FAKE_DIE_AFTER_MS) : 0; +let turnCount = 0; const httpServer = createServer((req, res) => { if (req.url === '/readyz') { res.writeHead(200); res.end('ok'); return; } @@ -28,7 +39,61 @@ wss.on('connection', (ws) => { case 'initialize': return reply({ ok: true }); case 'thread/start': return reply({ thread: { id: 'thread-fake-1' } }); case 'thread/resume': return reply({ thread: { id: msg.params?.threadId ?? 'thread-fake-1' } }); - case 'turn/start': if (HANG_TURN) return; return reply({ accepted: true }); + case 'turn/start': { + turnCount++; + const nativeTurnId = `turn-fake-${turnCount}`; + const started = () => { + ws.send(JSON.stringify({ + jsonrpc: '2.0', + method: 'turn/started', + params: { threadId: msg.params?.threadId, turn: { id: nativeTurnId } }, + })); + }; + const completed = () => { + if (!NO_TURN_TERMINAL) { + const turn = { + id: nativeTurnId, + ...(TURN_STATUS ? { status: TURN_STATUS } : {}), + ...(TURN_STATUS === 'failed' + ? { error: { code: 'fake_failed', message: 'fake failure' } } + : {}), + }; + const completed = JSON.stringify({ + jsonrpc: '2.0', + method: 'turn/completed', + params: { threadId: msg.params?.threadId, turn }, + }); + ws.send(completed); + if (DUPLICATE_TERMINAL) ws.send(completed); + } + }; + const terminal = () => { + started(); + completed(); + }; + if (HANG_TURN) { + if (HANG_TURN_NOTIFY) terminal(); + return; + } + if (ERROR_AFTER_STARTED) { + started(); + ws.send(JSON.stringify({ + jsonrpc: '2.0', + id: msg.id, + error: { code: -32000, message: 'fake response failure after turn/started' }, + })); + setTimeout(completed, 100); + return; + } + if (TERMINAL_BEFORE_RESPONSE) { + terminal(); + reply({ turn: { id: nativeTurnId } }); + } else { + reply({ turn: { id: nativeTurnId } }); + terminal(); + } + return; + } default: return reply({}); } }); diff --git a/test/fleet-shutdown.test.ts b/test/fleet-shutdown.test.ts new file mode 100644 index 000000000..780107ea0 --- /dev/null +++ b/test/fleet-shutdown.test.ts @@ -0,0 +1,955 @@ +import { describe, expect, it, vi } from 'vitest'; +import { + isFleetEntryProvenFreeOfAutorestartTimer, + isFleetEntryProvenTerminalAfterSignal, + signalAndAwaitFleet, + type FleetProcessEntry, +} from '../src/cli/fleet-shutdown.js'; +import { DAEMON_GRACEFUL_EXIT_CODE } from '../src/core/supervisor-shutdown-protocol.js'; + +const entries: FleetProcessEntry[] = [ + { + name: 'botmux-a', pmId: 1, pid: 101, online: true, + autorestart: true, stopExitCodes: [DAEMON_GRACEFUL_EXIT_CODE], + }, + { + name: 'botmux-b', pmId: 2, pid: 202, online: true, + autorestart: true, stopExitCodes: [DAEMON_GRACEFUL_EXIT_CODE], + }, + { + name: 'botmux-offline', pmId: 3, pid: 0, online: false, + status: 'stopped', autorestart: false, + }, +]; + +function gracefulTerminalRows(targets: FleetProcessEntry[]): FleetProcessEntry[] { + return targets + .filter(target => target.online && target.pid > 0) + .map(target => ({ + ...target, + pid: 0, + online: false, + status: 'waiting restart', + exitCode: DAEMON_GRACEFUL_EXIT_CODE, + })); +} + +describe('generation-aware fleet graceful shutdown', () => { + it('never treats PM2 signal-death code 0 as the daemon graceful-stop sentinel', () => { + expect(isFleetEntryProvenFreeOfAutorestartTimer({ + name: 'botmux-a', + pid: 0, + online: false, + status: 'waiting restart', + autorestart: true, + exitCode: 0, + stopExitCodes: [DAEMON_GRACEFUL_EXIT_CODE], + })).toBe(false); + expect(isFleetEntryProvenFreeOfAutorestartTimer({ + name: 'botmux-a', + pid: 0, + online: false, + status: 'waiting restart', + autorestart: true, + exitCode: DAEMON_GRACEFUL_EXIT_CODE, + stopExitCodes: [DAEMON_GRACEFUL_EXIT_CODE], + })).toBe(true); + }); + + it('separates timer-free overlimit admission from post-signal terminal proof', () => { + const abruptOverlimit: FleetProcessEntry = { + name: 'botmux-a', pmId: 1, pid: 0, online: false, + status: 'errored', autorestart: true, exitCode: 0, + stopExitCodes: [DAEMON_GRACEFUL_EXIT_CODE], + }; + expect(isFleetEntryProvenFreeOfAutorestartTimer(abruptOverlimit)).toBe(true); + expect(isFleetEntryProvenTerminalAfterSignal(abruptOverlimit)).toBe(false); + expect(isFleetEntryProvenTerminalAfterSignal({ + ...abruptOverlimit, + exitCode: DAEMON_GRACEFUL_EXIT_CODE, + })).toBe(true); + }); + + it('restores an ACKed daemon whose abrupt exit hits PM2 restart overlimit', () => { + const target = entries[0]!; + const alive = new Set([target.pid]); + let now = 0; + let restored = false; + const signalInitial = vi.fn(() => { alive.delete(target.pid); }); + const startOffline = vi.fn((offlineEntries: FleetProcessEntry[]) => { + expect(offlineEntries).toEqual([ + expect.objectContaining({ name: target.name, status: 'errored', exitCode: 0 }), + ]); + restored = true; + alive.add(303); + }); + + expect(() => signalAndAwaitFleet([target], 'restart', 100, { + signal: vi.fn(), + signalInitial, + assertSignalAuthorityComplete: vi.fn(), + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: () => restored + ? [{ ...target, pid: 303, online: true }] + : [{ + ...target, + pid: 0, + online: false, + status: 'errored', + exitCode: 0, + }], + successorSettleMs: 10, + })).toThrow(/post-signal terminal proof.*exit_code=0.*restored 1 offline PM2 entry/); + expect(signalInitial).toHaveBeenCalledOnce(); + expect(startOffline).toHaveBeenCalledOnce(); + }); + + it('accepts an errored row only when its exit matches the graceful stop policy', () => { + const target = entries[0]!; + const alive = new Set([target.pid]); + let now = 0; + const authority = vi.fn(); + signalAndAwaitFleet([target], 'stop', 100, { + signal(pid) { alive.delete(pid); }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline: vi.fn(), + list: () => [{ + ...target, + pid: 0, + online: false, + status: 'errored', + exitCode: DAEMON_GRACEFUL_EXIT_CODE, + }], + assertSignalAuthorityComplete: authority, + successorSettleMs: 10, + pollMs: 5, + }); + expect(authority).toHaveBeenCalledOnce(); + }); + + it('does not cache a latest terminal proof across a later missing projection', () => { + const target = entries[0]!; + const alive = new Set([target.pid]); + let now = 0; + let listCalls = 0; + const startOffline = vi.fn(); + + expect(() => signalAndAwaitFleet([target], 'restart', 100, { + signal(pid) { alive.delete(pid); }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: () => { + listCalls += 1; + if (listCalls === 1) { + return [{ + ...target, + pid: 0, + online: false, + status: 'waiting restart', + exitCode: DAEMON_GRACEFUL_EXIT_CODE, + }]; + } + return []; + }, + successorSettleMs: 10, + pollMs: 5, + })).toThrow(/fleet is partially stopped.*offline: botmux-a/); + expect(listCalls).toBeGreaterThan(1); + expect(startOffline).not.toHaveBeenCalled(); + }); + + it('does not use a dead replacement exit code to prove its predecessor', () => { + const target = entries[0]!; + const alive = new Set([target.pid]); + let now = 0; + const startOffline = vi.fn(); + + expect(() => signalAndAwaitFleet([target], 'restart', 100, { + signal(pid) { alive.delete(pid); }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: () => [{ + ...target, + pid: 303, + online: false, + status: 'errored', + exitCode: DAEMON_GRACEFUL_EXIT_CODE, + }], + successorSettleMs: 10, + pollMs: 5, + })).toThrow(/post-signal terminal proof.*botmux-a\/101.*fleet is partially stopped/); + expect(startOffline).toHaveBeenCalledOnce(); + }); + + it('refuses before any signal when a non-online PM2 row still owns a live PID', () => { + const signal = vi.fn(); + const list = vi.fn(() => []); + const startOffline = vi.fn(); + expect(() => signalAndAwaitFleet([ + entries[0]!, + { name: 'botmux-transitioning', pmId: 9, pid: 404, online: false, status: 'launching' }, + ], 'restart', 100, { + signal, + isAlive: pid => pid === 101 || pid === 404, + now: () => 0, + sleep: vi.fn(), + startOffline, + list, + })).toThrow(/non-online registry rows still have live PID.*botmux-transitioning:404/); + expect(signal).not.toHaveBeenCalled(); + expect(list).not.toHaveBeenCalled(); + expect(startOffline).not.toHaveBeenCalled(); + }); + + it('refuses duplicate singleton PM2 names before collapsing or signalling either PID', () => { + const signal = vi.fn(); + expect(() => signalAndAwaitFleet([ + { name: 'botmux-a', pmId: 1, pid: 101, online: true }, + { name: 'botmux-a', pmId: 4, pid: 404, online: true }, + ], 'stop', 100, { + signal, + isAlive: () => true, + now: () => 0, + sleep: vi.fn(), + startOffline: vi.fn(), + list: vi.fn(() => []), + })).toThrow(/duplicate registry row.*botmux-a/); + expect(signal).not.toHaveBeenCalled(); + }); + + it('does not signal a later initial target after an earlier signal consumes the deadline', () => { + let now = 0; + const signal = vi.fn(() => { now += 10; }); + const list = vi.fn(() => []); + const startOffline = vi.fn(); + expect(() => signalAndAwaitFleet(entries.slice(0, 2), 'restart', 5, { + signal, + isAlive: () => true, + now: () => now, + sleep: vi.fn(), + startOffline, + list, + })).toThrow(/deadline exhausted during initial daemon signalling.*no later fleet action/); + expect(signal).toHaveBeenCalledTimes(1); + expect(signal).toHaveBeenCalledWith(101); + expect(list).not.toHaveBeenCalled(); + expect(startOffline).not.toHaveBeenCalled(); + }); + + it('propagates a signal authorization failure instead of treating it as already exited', () => { + const list = vi.fn(() => []); + const startOffline = vi.fn(); + expect(() => signalAndAwaitFleet([entries[0]!], 'restart', 100, { + signal: () => { throw new Error('old daemon lacks shutdown capability'); }, + isAlive: () => true, + now: () => 0, + sleep: vi.fn(), + startOffline, + list, + })).toThrow(/old daemon lacks shutdown capability/); + expect(list).not.toHaveBeenCalled(); + expect(startOffline).not.toHaveBeenCalled(); + }); + + it('signals a live non-online successor and never treats it as quiet', () => { + const alive = new Set([202]); + const signalled: number[] = []; + let now = 0; + const startOffline = vi.fn(); + expect(() => signalAndAwaitFleet([entries[1]!], 'restart', 100, { + signal(pid) { + signalled.push(pid); + if (pid === 202) alive.delete(pid); + if (pid === 303) return; // transitional successor refuses + }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: () => { + alive.add(303); + return [{ + name: 'botmux-b', pmId: 2, pid: 303, online: false, status: 'launching', + exitCode: DAEMON_GRACEFUL_EXIT_CODE, + stopExitCodes: [DAEMON_GRACEFUL_EXIT_CODE], + }]; + }, + successorSettleMs: 10, + pollMs: 5, + })).toThrow(/daemon generation\(s\).*live generation untouched/); + expect(signalled).toEqual([202, 303]); + expect(startOffline).not.toHaveBeenCalled(); + }); + + it('requires an initially dormant row to prove that no restart timer can fire', () => { + const list = vi.fn(() => []); + expect(() => signalAndAwaitFleet([ + { name: 'botmux-waiting', pmId: 7, pid: 0, online: false, status: 'waiting restart' }, + ], 'stop', 100, { + signal: vi.fn(), + isAlive: () => false, + now: () => 0, + sleep: vi.fn(), + startOffline: vi.fn(), + list, + })).toThrow(/dormant registry row.*may still restart/); + expect(list).not.toHaveBeenCalled(); + }); + + it('does not trust a bare stopped status as proof that no restart task exists', () => { + expect(() => signalAndAwaitFleet([ + { name: 'botmux-stopped', pmId: 8, pid: 0, online: false, status: 'stopped' }, + ], 'restart', 100, { + signal: vi.fn(), + isAlive: () => false, + now: () => 0, + sleep: vi.fn(), + startOffline: vi.fn(), + list: vi.fn(() => []), + })).toThrow(/dormant registry row.*may still restart/); + }); + + it('signals the whole online fleet before polling and succeeds after the quiet window', () => { + const order: string[] = []; + const alive = new Set([101, 202]); + let now = 0; + signalAndAwaitFleet(entries, 'restart', 100, { + signal(pid) { order.push(`signal:${pid}`); alive.delete(pid); }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { order.push(`sleep:${ms}`); now += ms; }, + startOffline: names => order.push(`start:${names.join(',')}`), + list: () => gracefulTerminalRows(entries), + successorSettleMs: 10, + pollMs: 5, + }); + + expect(order.slice(0, 2)).toEqual(['signal:101', 'signal:202']); + expect(order).not.toContain(expect.stringMatching(/^start:/)); + expect(now).toBeGreaterThanOrEqual(10); + }); + + it('batch-dispatches every initial endpoint and compensates a peer when one refuses', () => { + const alive = new Set([101, 202]); + let now = 0; + let restored = false; + const signalInitial = vi.fn((targets: FleetProcessEntry[]) => { + expect(targets.map(target => target.pid)).toEqual([101, 202]); + // A's endpoint hangs/refuses; B accepted the concurrent request and exits. + alive.delete(202); + }); + const startOffline = vi.fn((offline: FleetProcessEntry[]) => { + expect(offline.map(entry => entry.name)).toEqual(['botmux-b']); + restored = true; + alive.add(303); + }); + expect(() => signalAndAwaitFleet(entries.slice(0, 2), 'restart', 100, { + signal: vi.fn(), + signalInitial, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: () => [ + { name: 'botmux-a', pmId: 1, pid: 101, online: true }, + restored + ? { name: 'botmux-b', pmId: 2, pid: 303, online: true } + : { name: 'botmux-b', pmId: 2, pid: 0, online: false, + status: 'waiting restart', exitCode: 0, stopExitCodes: [0] }, + ], + successorSettleMs: 10, + })).toThrow(/restored 1 offline PM2 entry.*live generation untouched/); + expect(signalInitial).toHaveBeenCalledOnce(); + expect(startOffline).toHaveBeenCalledOnce(); + }); + + it('compensates and fails when an unacked generation later disappears', () => { + const alive = new Set([101, 202]); + let now = 0; + let restored = false; + const startOffline = vi.fn((offline: FleetProcessEntry[]) => { + expect(offline.map(entry => entry.name)).toEqual(['botmux-a', 'botmux-b']); + restored = true; + alive.add(301); + alive.add(302); + }); + expect(() => signalAndAwaitFleet(entries.slice(0, 2), 'restart', 100, { + signal: vi.fn(), + signalInitial: () => { alive.clear(); }, + assertSignalAuthorityComplete: () => { + throw new Error('botmux-a exact IPC ACK missing'); + }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: () => restored + ? [ + { name: 'botmux-a', pmId: 1, pid: 301, online: true }, + { name: 'botmux-b', pmId: 2, pid: 302, online: true }, + ] + : [ + { name: 'botmux-a', pmId: 1, pid: 0, online: false, + status: 'waiting restart', exitCode: 0, stopExitCodes: [0] }, + { name: 'botmux-b', pmId: 2, pid: 0, online: false, + status: 'waiting restart', exitCode: 0, stopExitCodes: [0] }, + ], + successorSettleMs: 10, + })).toThrow(/signal authority: botmux-a exact IPC ACK missing.*restored 2 offline PM2 entries/); + expect(startOffline).toHaveBeenCalledOnce(); + }); + + it('never starts the quiet window while a projected row may still publish a successor', () => { + const alive = new Set([101]); + const startOffline = vi.fn(); + let now = 0; + expect(() => signalAndAwaitFleet([entries[0]!], 'stop', 50, { + signal(pid) { alive.delete(pid); }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: () => [{ + name: 'botmux-a', pmId: 1, pid: 0, online: false, + status: 'waiting restart', exitCode: 1, stopExitCodes: [0], + }], + successorSettleMs: 5, + pollMs: 5, + })).toThrow(/fleet is partially stopped.*offline: botmux-a/); + expect(startOffline).not.toHaveBeenCalled(); + expect(now).toBe(0); + }); + + it('detects and gracefully signals a PM2 successor that appears during the quiet window', () => { + const signalled: number[] = []; + const alive = new Set([101, 202]); + let now = 0; + signalAndAwaitFleet(entries, 'restart', 300, { + signal(pid) { + signalled.push(pid); + alive.delete(pid); + }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline: vi.fn(), + list: () => { + const terminal = gracefulTerminalRows(entries); + if (now < 50 || signalled.includes(303)) return terminal; + alive.add(303); + return [ + terminal.find(entry => entry.name === 'botmux-a')!, + { + ...entries[1]!, + pid: 303, + online: true, + exitCode: DAEMON_GRACEFUL_EXIT_CODE, + }, + ]; + }, + successorSettleMs: 100, + pollMs: 10, + }); + + expect(signalled).toEqual([101, 202, 303]); + expect(now).toBeGreaterThanOrEqual(150); + }); + + it('does not report success when a discovered successor itself refuses to exit', () => { + const signalled: number[] = []; + const alive = new Set([101, 202, 303]); + let now = 0; + const startOffline = vi.fn(); + expect(() => signalAndAwaitFleet([entries[1]!], 'restart', 100, { + signal(pid) { + signalled.push(pid); + if (pid !== 303) alive.delete(pid); + }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: () => [{ + name: 'botmux-b', pmId: 2, pid: 303, online: true, + exitCode: DAEMON_GRACEFUL_EXIT_CODE, + stopExitCodes: [DAEMON_GRACEFUL_EXIT_CODE], + }], + successorSettleMs: 10, + pollMs: 5, + })).toThrow(/1\/1 daemon generation\(s\).*live generation untouched/); + expect(signalled).toEqual([202, 303]); + expect(startOffline).not.toHaveBeenCalled(); + expect(alive.has(303)).toBe(true); + }); + + it('fresh-lists first, starts only a truly offline peer, and leaves the refuser untouched', () => { + const alive = new Set([101, 202]); + const projection = new Map(entries.map(entry => [entry.name, { ...entry }])); + let now = 0; + const startOffline = vi.fn((offlineEntries: FleetProcessEntry[]) => { + for (const entry of offlineEntries) { + projection.set(entry.name, { ...entry, pid: 902, online: true }); + alive.add(902); + } + }); + + expect(() => signalAndAwaitFleet(entries, 'stop', 100, { + signal(pid) { + if (pid === 202) { + alive.delete(pid); + projection.set('botmux-b', { + name: 'botmux-b', pmId: 2, pid: 0, online: false, + status: 'waiting restart', exitCode: 0, stopExitCodes: [0], + }); + } + }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: () => [...projection.values()], + successorSettleMs: 10, + })).toThrow(/restored 1 offline PM2 entry.*live generation untouched/); + + expect(startOffline).toHaveBeenCalledWith( + [expect.objectContaining({ name: 'botmux-b', pmId: 2 })], + expect.any(Number), + ); + expect(startOffline.mock.calls.flatMap(call => call[0]).map(entry => entry.name)) + .not.toContain('botmux-a'); + expect(alive.has(101)).toBe(true); + }); + + it('does not compensate an exact row when a fresh duplicate with the same name exists', () => { + const alive = new Set([101, 202]); + let now = 0; + const startOffline = vi.fn(); + expect(() => signalAndAwaitFleet(entries.slice(0, 2), 'restart', 50, { + signal(pid) { if (pid === 202) alive.delete(pid); }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: () => [ + { name: 'botmux-a', pmId: 1, pid: 101, online: true }, + { + name: 'botmux-b', pmId: 2, pid: 0, online: false, + status: 'waiting restart', exitCode: 0, stopExitCodes: [0], + }, + { + name: 'botmux-b', pmId: 9, pid: 0, online: false, + status: 'waiting restart', exitCode: 1, stopExitCodes: [0], + }, + ], + })).toThrow(/fleet is partially stopped.*offline: botmux-b/); + expect(startOffline).not.toHaveBeenCalled(); + }); + + it('leaves a successor first observed at the refusal boundary untouched and restores an unrelated peer', () => { + const alive = new Set([101, 202]); + const signalled: number[] = []; + let now = 0; + let restored = false; + const startOffline = vi.fn((offlineEntries: FleetProcessEntry[]) => { + expect(offlineEntries.map(entry => entry.name)).toEqual(['botmux-b']); + restored = true; + alive.add(909); + }); + const projection = (): FleetProcessEntry[] => [ + ...(now >= 40 + ? [{ name: 'botmux-a', pmId: 1, pid: 808, online: true }] + : [{ + name: 'botmux-a', pmId: 1, pid: 0, online: false, + status: 'waiting restart', exitCode: 0, stopExitCodes: [0], + }]), + restored + ? { name: 'botmux-b', pmId: 2, pid: 909, online: true } + : { + name: 'botmux-b', pmId: 2, pid: 0, online: false, + status: 'waiting restart', exitCode: 0, stopExitCodes: [0], + }, + ]; + + expect(() => signalAndAwaitFleet(entries.slice(0, 2), 'restart', 50, { + signal(pid) { signalled.push(pid); alive.delete(pid); }, + isAlive(pid) { + if (pid === 808 && now >= 40) return true; + return alive.has(pid); + }, + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: projection, + successorSettleMs: 100, + pollMs: 5, + })).toThrow(/restored 1 offline PM2 entry.*live generation untouched/); + expect(signalled).toEqual([101, 202]); + expect(signalled).not.toContain(808); + expect(startOffline).toHaveBeenCalledOnce(); + }); + + it('does not restart or re-signal a healthy successor discovered during partial refusal', () => { + const alive = new Set([101, 202, 303]); + let now = 0; + const signalled: number[] = []; + const startOffline = vi.fn(); + + expect(() => signalAndAwaitFleet(entries, 'restart', 100, { + signal(pid) { + signalled.push(pid); + if (pid === 202) alive.delete(pid); + }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: () => [ + { name: 'botmux-a', pmId: 1, pid: 101, online: true }, + { name: 'botmux-b', pmId: 2, pid: 303, online: true }, + ], + successorSettleMs: 10, + })).toThrow(/left every live generation untouched/); + + expect(signalled).toEqual([101, 202]); + expect(startOffline).not.toHaveBeenCalled(); + expect(alive.has(101)).toBe(true); + expect(alive.has(303)).toBe(true); + }); + + it('fails closed without a PM2 mutation when fresh-list verification fails', () => { + const alive = new Set([101, 202]); + let now = 0; + const startOffline = vi.fn(); + expect(() => signalAndAwaitFleet(entries, 'stop', 50, { + signal(pid) { if (pid === 202) alive.delete(pid); }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: () => { throw new Error('jlist unavailable'); }, + })).toThrow(/no compensation was attempted.*verification before compensation failed.*jlist unavailable/); + expect(startOffline).not.toHaveBeenCalled(); + }); + + it('fails closed without compensation when successor verification itself cannot be read', () => { + const alive = new Set([101, 202]); + let now = 0; + const startOffline = vi.fn(); + expect(() => signalAndAwaitFleet(entries, 'restart', 100, { + signal(pid) { alive.delete(pid); }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: () => { throw new Error('projection corrupt'); }, + successorSettleMs: 10, + })).toThrow(/state is unverified and no compensation was attempted.*successor verification failed/); + expect(startOffline).not.toHaveBeenCalled(); + }); + + it('reports a genuinely partial fleet when compensation cannot restore an exited peer', () => { + const alive = new Set([101, 202]); + let now = 0; + expect(() => signalAndAwaitFleet(entries, 'restart', 50, { + signal(pid) { if (pid === 202) alive.delete(pid); }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline() { throw new Error('pm2 unavailable'); }, + list: () => [ + { name: 'botmux-a', pmId: 1, pid: 101, online: true }, + { + name: 'botmux-b', pmId: 2, pid: 0, online: false, + status: 'waiting restart', exitCode: 0, stopExitCodes: [0], + }, + ], + })).toThrow(/fleet is partially stopped.*botmux-b.*pm2 unavailable/); + }); + + it('compensates a peer whose PM2 row says online but whose PID is dead', () => { + const alive = new Set([101, 202]); + let now = 0; + let projection: FleetProcessEntry[] = [ + { name: 'botmux-a', pmId: 1, pid: 101, online: true }, + { + name: 'botmux-b', pmId: 2, pid: 303, online: true, + status: 'waiting restart', exitCode: 0, stopExitCodes: [0], + }, // stale: 303 is not alive + ]; + const startOffline = vi.fn((offlineEntries: FleetProcessEntry[]) => { + expect(offlineEntries.map(entry => entry.name)).toEqual(['botmux-b']); + alive.add(404); + projection = [ + { name: 'botmux-a', pmId: 1, pid: 101, online: true }, + { name: 'botmux-b', pmId: 2, pid: 404, online: true }, + ]; + }); + + expect(() => signalAndAwaitFleet(entries, 'stop', 50, { + signal(pid) { if (pid === 202) alive.delete(pid); }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: () => projection, + })).toThrow(/restored 1 offline PM2 entry/); + expect(startOffline).toHaveBeenCalledOnce(); + expect(alive.has(404)).toBe(true); + }); + + it('refuses to claim restoration when PM2 reports an online replacement with a dead PID', () => { + const alive = new Set([101, 202]); + let now = 0; + let projection: FleetProcessEntry[] = [ + { name: 'botmux-a', pmId: 1, pid: 101, online: true }, + { + name: 'botmux-b', pmId: 2, pid: 303, online: true, + status: 'waiting restart', exitCode: 0, stopExitCodes: [0], + }, // stale before compensation + ]; + const startOffline = vi.fn((offlineEntries: FleetProcessEntry[]) => { + expect(offlineEntries.map(entry => entry.name)).toEqual(['botmux-b']); + // PM2 claims it started 404, but OS liveness never confirms that PID. + projection = [ + { name: 'botmux-a', pmId: 1, pid: 101, online: true }, + { name: 'botmux-b', pmId: 2, pid: 404, online: true }, + ]; + }); + + expect(() => signalAndAwaitFleet(entries, 'restart', 50, { + signal(pid) { if (pid === 202) alive.delete(pid); }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: () => projection, + })).toThrow(/fleet is partially stopped.*offline: botmux-b/); + expect(startOffline).toHaveBeenCalledOnce(); + expect(alive.has(404)).toBe(false); + }); + + it('caps a successor projection at the absolute fleet deadline and never acts on its late result', () => { + const alive = new Set([202]); + const signalled: Array<{ pid: number; at: number }> = []; + const listCalls: Array<{ at: number; budgetMs: number }> = []; + const startOffline = vi.fn(); + let now = 0; + + expect(() => signalAndAwaitFleet([entries[1]!], 'restart', 50, { + signal(pid) { + signalled.push({ pid, at: now }); + alive.delete(pid); + }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: budgetMs => { + listCalls.push({ at: now, budgetMs }); + // Even if a non-conforming subprocess returns after its advertised cap, + // no observation at the absolute deadline may trigger another signal. + now = 50; + alive.add(303); + return [{ name: 'botmux-b', pmId: 2, pid: 303, online: true }]; + }, + successorSettleMs: 10, + })).toThrow(/no compensation was attempted.*deadline exhausted during PM2 successor verification/); + + expect(listCalls).toEqual([{ at: 0, budgetMs: 2 }]); + expect(signalled).toEqual([{ pid: 202, at: 0 }]); + expect(startOffline).not.toHaveBeenCalled(); + expect(now).toBe(50); + }); + + it('does not issue another list or signal after a projection consumes the remaining deadline', () => { + const alive = new Set([202]); + const listCalls: Array<{ at: number; budgetMs: number }> = []; + const signalCalls: Array<{ pid: number; at: number }> = []; + const startOffline = vi.fn(); + let now = 0; + + expect(() => signalAndAwaitFleet([entries[1]!], 'stop', 50, { + signal(pid) { signalCalls.push({ pid, at: now }); }, // original refuses + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: budgetMs => { + listCalls.push({ at: now, budgetMs }); + now = 50; + return [{ name: 'botmux-b', pmId: 2, pid: 202, online: true }]; + }, + pollMs: 5, + })).toThrow(/no compensation was attempted.*deadline exhausted during PM2 verification before compensation/); + + expect(listCalls).toEqual([{ at: 40, budgetMs: 2 }]); + expect(signalCalls).toEqual([{ pid: 202, at: 0 }]); + expect(startOffline).not.toHaveBeenCalled(); + expect(now).toBe(50); + }); + + it('partitions one bounded multi-entry compensation inside the absolute fleet deadline', () => { + const targets: FleetProcessEntry[] = [ + { name: 'botmux-a', pmId: 1, pid: 101, online: true }, + { name: 'botmux-b', pmId: 2, pid: 202, online: true }, + { name: 'botmux-c', pmId: 3, pid: 303, online: true }, + ]; + const alive = new Set([101, 202, 303]); + const listCalls: Array<{ at: number; budgetMs: number }> = []; + const compensationCalls: Array<{ names: string[]; at: number; budgetMs: number }> = []; + let now = 0; + let greatestNow = 0; + let compensated = false; + const advance = (ms: number) => { + now += ms; + greatestNow = Math.max(greatestNow, now); + }; + + expect(() => signalAndAwaitFleet(targets, 'restart', 50, { + signal(pid) { if (pid !== 101) alive.delete(pid); }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep: advance, + startOffline(offlineEntries, budgetMs) { + compensationCalls.push({ + names: offlineEntries.map(entry => entry.name), + at: now, + budgetMs, + }); + // One helper gets one shared tail budget for both ids, never a fresh + // per-name timeout. + advance(budgetMs); + compensated = true; + alive.add(404); + alive.add(505); + }, + list: budgetMs => { + listCalls.push({ at: now, budgetMs }); + return [ + { name: 'botmux-a', pmId: 1, pid: 101, online: true }, + compensated ? { name: 'botmux-b', pmId: 2, pid: 404, online: true } : { + name: 'botmux-b', pmId: 2, pid: 0, online: false, + status: 'waiting restart', exitCode: 0, stopExitCodes: [0], + }, + compensated ? { name: 'botmux-c', pmId: 3, pid: 505, online: true } : { + name: 'botmux-c', pmId: 3, pid: 0, online: false, + status: 'waiting restart', exitCode: 0, stopExitCodes: [0], + }, + ]; + }, + })).toThrow(/restored 2 offline PM2 entries.*live generation untouched/); + + expect(compensationCalls).toEqual([{ + names: ['botmux-b', 'botmux-c'], + at: 40, + budgetMs: 5, + }]); + expect(listCalls).toEqual([ + { at: 40, budgetMs: 2 }, + { at: 45, budgetMs: 2 }, + ]); + expect(now).toBe(45); + expect(greatestNow).toBeLessThanOrEqual(50); + }); + + it('reserves production-size projection budgets and retries one transient jlist failure', () => { + const alive = new Set([101, 202]); + let now = 0; + let compensated = false; + const listBudgets: number[] = []; + let listCalls = 0; + + expect(() => signalAndAwaitFleet(entries.slice(0, 2), 'restart', 60_000, { + signal(pid) { + if (pid === 202) alive.delete(pid); + }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline(offline, budgetMs) { + expect(offline.map(entry => entry.name)).toEqual(['botmux-b']); + expect(budgetMs).toBeGreaterThanOrEqual(10_000); + compensated = true; + alive.add(303); + }, + list: budgetMs => { + listBudgets.push(budgetMs); + listCalls += 1; + if (listCalls === 1) throw new Error('transient jlist timeout'); + return [ + { name: 'botmux-a', pmId: 1, pid: 101, online: true }, + compensated + ? { name: 'botmux-b', pmId: 2, pid: 303, online: true } + : { + name: 'botmux-b', pmId: 2, pid: 0, online: false, + status: 'waiting restart', exitCode: 0, stopExitCodes: [0], + }, + ]; + }, + pollMs: 5_000, + })).toThrow(/restored 1 offline PM2 entry.*live generation untouched/); + + expect(listCalls).toBe(3); + expect(listBudgets.every(budget => budget >= 5_000)).toBe(true); + }); + + it('does not conditionally start a row whose PM2 policy may still have a restart timer', () => { + const alive = new Set([101, 202]); + let now = 0; + const startOffline = vi.fn(); + expect(() => signalAndAwaitFleet(entries.slice(0, 2), 'stop', 50, { + signal(pid) { if (pid === 202) alive.delete(pid); }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: () => [ + { name: 'botmux-a', pmId: 1, pid: 101, online: true }, + { + name: 'botmux-b', pmId: 2, pid: 0, online: false, + status: 'waiting restart', exitCode: 1, stopExitCodes: [0], + }, + ], + })).toThrow(/fleet is partially stopped.*offline: botmux-b/); + expect(startOffline).not.toHaveBeenCalled(); + }); + + it.each([ + { label: 'missing exit_code', exitCode: undefined, stopExitCodes: [0] }, + { label: 'null stop code', exitCode: 0, stopExitCodes: [null] }, + { label: 'empty stop code', exitCode: 0, stopExitCodes: [''] }, + ])('does not coerce malformed timer policy into a safe compensation: $label', ({ + exitCode, + stopExitCodes, + }) => { + const alive = new Set([101, 202]); + const startOffline = vi.fn(); + let now = 0; + expect(() => signalAndAwaitFleet(entries.slice(0, 2), 'restart', 50, { + signal(pid) { if (pid === 202) alive.delete(pid); }, + isAlive: pid => alive.has(pid), + now: () => now, + sleep(ms) { now += ms; }, + startOffline, + list: () => [ + { name: 'botmux-a', pmId: 1, pid: 101, online: true }, + { + name: 'botmux-b', pmId: 2, pid: 0, online: false, + status: 'waiting restart', + ...(exitCode !== undefined ? { exitCode } : {}), + stopExitCodes, + }, + ], + })).toThrow(/fleet is partially stopped.*offline: botmux-b/); + expect(startOffline).not.toHaveBeenCalled(); + }); +}); diff --git a/test/herdr-backend.test.ts b/test/herdr-backend.test.ts index 530d583b4..12b562fde 100644 --- a/test/herdr-backend.test.ts +++ b/test/herdr-backend.test.ts @@ -337,6 +337,21 @@ describe('HerdrBackend.spawn', () => { const be = new HerdrBackend(SESSION, { isReattach: true }); be.spawn('claude', [], { cwd: '/work', cols: 80, rows: 24, env: {} }); expect(herdrCall('agent', 'start', 'botmux')).toBeUndefined(); + expect(be.isReattach).toBe(true); + be.kill(); + }); + + it('reports actual fresh start when a predicted reattach has no reusable agent', () => { + setHerdrResponses([ + { match: a => a[0] === 'session' && a[1] === 'list', reply: () => EXISTING_SESSION_REPLY }, + { match: a => a.includes('agent') && a.includes('get'), reply: () => JSON.stringify({ result: {} }) }, + { match: a => a.includes('agent') && a.includes('start'), reply: () => AGENT_GET_REPLY('fresh-2') }, + { match: a => a.includes('read') && (a.includes('agent') || a.includes('pane')), reply: () => PANE_READ_REPLY('') }, + ]); + const be = new HerdrBackend(SESSION, { isReattach: true }); + be.spawn('claude', [], { cwd: '/work', cols: 80, rows: 24, env: {} }); + expect(herdrCall('agent', 'start', 'botmux')).toBeDefined(); + expect(be.isReattach).toBe(false); be.kill(); }); @@ -672,11 +687,21 @@ describe('HerdrBackend message writing', () => { be.kill(); }); + it('reports a rejected pane command instead of claiming raw input acceptance', () => { + const be = spawnBackend('5-5'); + mockedExecFileSync.mockImplementation(() => { + throw new Error('pane disappeared'); + }); + expect(be.sendText('/goal x')).toBe(false); + expect(be.sendSpecialKeys('Enter')).toBe(false); + be.kill(); + }); + it('write() is a no-op after kill()', () => { const be = spawnBackend('5-5'); be.kill(); mockedExecFileSync.mockClear(); - be.sendText('after-exit'); + expect(be.sendText('after-exit')).toBe(false); const call = herdrCall('pane', 'send-text'); expect(call).toBeUndefined(); }); diff --git a/test/hook-runner.test.ts b/test/hook-runner.test.ts index 10fa0bb26..e5277d5f5 100644 --- a/test/hook-runner.test.ts +++ b/test/hook-runner.test.ts @@ -2,10 +2,12 @@ import { spawnSync } from 'node:child_process'; import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { filterMatches, + emitHookEvent, + forwardEmitToDaemon, loadHookConfigs, parseHookCommand, prepareHookPayload, @@ -38,6 +40,76 @@ describe('parseHookCommand', () => { }); }); +describe('managed hook forwarding', () => { + it('uses only the frozen protected port and exact original tuple', async () => { + const originalFetch = globalThis.fetch; + const fetchMock = vi.fn(async () => new Response(null, { status: 202 })); + globalThis.fetch = fetchMock as typeof fetch; + try { + await forwardEmitToDaemon( + 'outbound.send', + { event: 'outbound.send', content: 'old payload' }, + 'poisoned-discovery-app', + { + ipcPort: 4310, + sessionId: 'sid-original', + capability: 'ab'.repeat(32), + turnId: 'turn-original', + dispatchAttempt: 3, + }, + ); + } finally { + globalThis.fetch = originalFetch; + } + + expect(fetchMock).toHaveBeenCalledOnce(); + const [url, init] = fetchMock.mock.calls[0]!; + expect(String(url)).toBe('http://127.0.0.1:4310/api/hooks/emit'); + expect(JSON.parse(String((init as RequestInit).body))).toMatchObject({ + sessionId: 'sid-original', + originCapability: 'ab'.repeat(32), + originTurnId: 'turn-original', + originDispatchAttempt: 3, + }); + }); + + it('forwards managed origin even when session env is blank instead of running local hooks', async () => { + const marker = join(tmpDir, 'must-not-run-local'); + const oldSession = process.env.BOTMUX_SESSION_ID; + const oldApp = process.env.BOTMUX_LARK_APP_ID; + const oldHooks = process.env.BOTMUX_HOOKS_JSON; + const originalFetch = globalThis.fetch; + const fetchMock = vi.fn(async () => new Response(null, { status: 202 })); + process.env.BOTMUX_SESSION_ID = ''; + process.env.BOTMUX_LARK_APP_ID = ''; + process.env.BOTMUX_HOOKS_JSON = JSON.stringify([ + { event: 'outbound.send', command: `/usr/bin/touch ${marker}` }, + ]); + globalThis.fetch = fetchMock as typeof fetch; + try { + emitHookEvent('outbound.send', { content: 'managed' }, { + managedOrigin: { + ipcPort: 4311, + sessionId: 'sid-managed', + capability: 'cd'.repeat(32), + turnId: 'turn-managed', + }, + }); + await vi.waitFor(() => expect(fetchMock).toHaveBeenCalledOnce()); + await new Promise(resolve => setTimeout(resolve, 25)); + expect(existsSync(marker)).toBe(false); + } finally { + globalThis.fetch = originalFetch; + if (oldSession === undefined) delete process.env.BOTMUX_SESSION_ID; + else process.env.BOTMUX_SESSION_ID = oldSession; + if (oldApp === undefined) delete process.env.BOTMUX_LARK_APP_ID; + else process.env.BOTMUX_LARK_APP_ID = oldApp; + if (oldHooks === undefined) delete process.env.BOTMUX_HOOKS_JSON; + else process.env.BOTMUX_HOOKS_JSON = oldHooks; + } + }); +}); + describe('loadHookConfigs', () => { it('loads hooks from hooks.json under the data dir', () => { const hooks: HookConfig[] = [ diff --git a/test/idle-worker-sweeper.test.ts b/test/idle-worker-sweeper.test.ts index 665fb156a..9d6958e64 100644 --- a/test/idle-worker-sweeper.test.ts +++ b/test/idle-worker-sweeper.test.ts @@ -1,4 +1,4 @@ -import { describe, expect, it, vi } from 'vitest'; +import { beforeEach, describe, expect, it, vi } from 'vitest'; vi.mock('../src/services/session-store.js', () => ({ updateSessionPid: vi.fn(), @@ -16,7 +16,15 @@ vi.mock('../src/utils/logger.js', () => ({ }, })); -import { sweepIdleWorkers, DEFAULT_MAX_LIVE_WORKERS } from '../src/core/idle-worker-sweeper.js'; +import { + sweepIdleWorkers, + sweepIdleWorkersAfterTurnDrain, + DEFAULT_MAX_LIVE_WORKERS, +} from '../src/core/idle-worker-sweeper.js'; +import { + __testOnly_resetBotTurnMutationGates, + withBotTurnAdmission, +} from '../src/core/bot-turn-mutation-gate.js'; function ds(sessionId: string, backendType: string, lastMessageAt: number, worker = {}) { return { @@ -42,6 +50,10 @@ function ds(sessionId: string, backendType: string, lastMessageAt: number, worke const now = 1_000_000; describe('sweepIdleWorkers (per-bot count cap)', () => { + beforeEach(() => { + __testOnly_resetBotTurnMutationGates(); + }); + it('falls back to the default cap (30) when the bot has no explicit value', () => { expect(DEFAULT_MAX_LIVE_WORKERS).toBe(30); // DEFAULT_MAX_LIVE_WORKERS + 2 sessions, oldest first by lastMessageAt. @@ -102,6 +114,24 @@ describe('sweepIdleWorkers (per-bot count cap)', () => { expect(activeSessions.get('f').worker).not.toBe(null); }); + it('skips an idle session with durable Codex App dispatch ownership and suspends the next candidate', () => { + const owned = ds('a', 'tmux', now - 90 * 60_000); + owned.session.codexAppDispatchLedger = [ + { dispatchId: 'd-1', turnId: 't-1', state: 'accepted', content: 'owned' }, + ]; + const activeSessions = new Map([ + ['a', owned], + ['b', ds('b', 'tmux', now - 80 * 60_000)], + ['c', ds('c', 'tmux', now - 70 * 60_000)], + ]); + + const suspended = sweepIdleWorkers(activeSessions, { maxLiveWorkers: 2 }); + + expect(suspended.map(entry => entry.sessionId)).toEqual(['b']); + expect(activeSessions.get('a').worker).not.toBe(null); + expect(activeSessions.get('b').worker).toBe(null); + }); + it('is purely count-based: suspends a recently-active session with NO idle-time threshold', () => { // Both sessions are only a couple minutes idle. The old budget had a 30-min // idle gate that would have suspended nothing here; the new policy caps by @@ -188,4 +218,68 @@ describe('sweepIdleWorkers (per-bot count cap)', () => { expect(suspended).toEqual([]); expect(activeSessions.get('a').worker).not.toBe(null); }); + + it('waits for pre-accept inbound turns before choosing an idle cap victim', async () => { + const appId = 'cli_a'; + const a = ds('a', 'tmux', now - 90 * 60_000); + const b = ds('b', 'tmux', now - 10 * 60_000); + const activeSessions = new Map([['a', a], ['b', b]]); + let releaseAdmission!: () => void; + const pausedBeforeAccept = new Promise(resolve => { releaseAdmission = resolve; }); + let admissionStarted!: () => void; + const started = new Promise(resolve => { admissionStarted = resolve; }); + + // Session A has entered the existing-owner message path but is still + // waiting on sender/reaction work, so its old screen snapshot says idle and + // its durable dispatch ledger is not populated yet. + const inboundA = withBotTurnAdmission(appId, async () => { + admissionStarted(); + await pausedBeforeAccept; + // Durable acceptance occurs before the admission is released. + a.session.codexAppDispatchLedger = [ + { dispatchId: 'd-a', turnId: 't-a', state: 'accepted', content: 'message-a' }, + ]; + }); + await started; + + // Spawning B puts the bot over cap. The cap sweep must drain A's + // admission rather than synchronously suspending A from its stale snapshot. + const sweep = sweepIdleWorkersAfterTurnDrain(appId, activeSessions, { maxLiveWorkers: 1 }); + await Promise.resolve(); + expect(a.worker).not.toBe(null); + + releaseAdmission(); + await inboundA; + const suspended = await sweep; + + expect(suspended.map(entry => entry.sessionId)).toEqual(['b']); + expect(a.worker).not.toBe(null); + expect(b.worker).toBe(null); + }); + + it('skips a sweep after a bounded wait instead of freezing the bot mutation gate', async () => { + const appId = 'cli_wedged'; + const activeSessions = new Map([ + ['a', ds('a', 'tmux', now - 90 * 60_000)], + ['b', ds('b', 'tmux', now - 10 * 60_000)], + ]); + let releaseAdmission!: () => void; + let admissionStarted!: () => void; + const started = new Promise(resolve => { admissionStarted = resolve; }); + const admission = withBotTurnAdmission(appId, async () => { + admissionStarted(); + await new Promise(resolve => { releaseAdmission = resolve; }); + }); + await started; + + await expect(sweepIdleWorkersAfterTurnDrain(appId, activeSessions, { + maxLiveWorkers: 1, + mutationAcquireTimeoutMs: 5, + })).resolves.toEqual([]); + expect(activeSessions.get('a').worker).not.toBe(null); + + releaseAdmission(); + await admission; + await expect(withBotTurnAdmission(appId, async () => 'open')).resolves.toBe('open'); + }); }); diff --git a/test/kill-worker-orphaned-backend.test.ts b/test/kill-worker-orphaned-backend.test.ts index e13f85f9e..81462a195 100644 --- a/test/kill-worker-orphaned-backend.test.ts +++ b/test/kill-worker-orphaned-backend.test.ts @@ -16,13 +16,18 @@ * Run: pnpm vitest run test/kill-worker-orphaned-backend.test.ts */ import { describe, it, expect, vi, beforeEach } from 'vitest'; -import type { DaemonSession } from '../src/core/types.js'; +import { EventEmitter } from 'node:events'; +import { mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { activeSessionKey, type DaemonSession } from '../src/core/types.js'; -const { tmuxKill, herdrKill, zellijKill, getBotMock } = vi.hoisted(() => ({ +const { tmuxKill, herdrKill, zellijKill, getBotMock, cancelRiffTaskMock } = vi.hoisted(() => ({ tmuxKill: vi.fn(), herdrKill: vi.fn(), zellijKill: vi.fn(), getBotMock: vi.fn(() => ({ resolvedAllowedUsers: [], config: {} })), + cancelRiffTaskMock: vi.fn(async () => true), })); vi.mock('../src/adapters/backend/tmux-backend.js', () => ({ @@ -37,10 +42,16 @@ vi.mock('../src/adapters/backend/zellij-backend.js', () => ({ vi.mock('../src/bot-registry.js', () => ({ getBot: getBotMock, + getBotBrand: vi.fn(() => 'feishu'), getAllBots: vi.fn(() => []), resolveBrandLabel: vi.fn(() => undefined), })); +vi.mock('../src/adapters/backend/riff-backend.js', () => ({ + hashUrlForLog: vi.fn(() => 'riffhash'), + cancelRiffTaskById: cancelRiffTaskMock, +})); + vi.mock('../src/im/lark/client.js', () => ({ updateMessage: vi.fn(), deleteMessage: vi.fn(), @@ -53,16 +64,28 @@ vi.mock('../src/im/lark/client.js', () => ({ vi.mock('../src/services/frozen-card-store.js', () => ({ loadFrozenCards: vi.fn(() => new Map()), saveFrozenCards: vi.fn(), + deleteFrozenCards: vi.fn(), })); vi.mock('../src/utils/logger.js', () => ({ logger: { info: vi.fn(), warn: vi.fn(), debug: vi.fn(), error: vi.fn() }, })); -import { killWorker } from '../src/core/worker-pool.js'; +import { + __testOnly_setupWorkerHandlers, + closeSession, + initWorkerPool, + killWorker, + sendWorkerInput, + setActiveSessionSafe, + setActiveSessionsRegistry, +} from '../src/core/worker-pool.js'; +import * as sessionStore from '../src/services/session-store.js'; +import { config } from '../src/config.js'; const SID = 'abcd1234-0000-0000-0000-000000000000'; const EXPECTED_NAME = 'bmx-abcd1234'; +let sessionReplyMock: ReturnType; // All stream-card fields left unset on both ds and ds.session so // persistStreamCardState() early-returns (no disk write) during clearUsageLimitState. @@ -80,6 +103,628 @@ const ds = (over: Partial = {}, initOver: any = {}): DaemonSessio beforeEach(() => { vi.clearAllMocks(); getBotMock.mockReturnValue({ resolvedAllowedUsers: [], config: {} } as any); + cancelRiffTaskMock.mockResolvedValue(true); + sessionReplyMock = vi.fn(async () => 'om_reply'); + initWorkerPool({ + sessionReply: sessionReplyMock, + getSessionWorkingDir: () => '/repo', + getActiveCount: () => 1, + closeSession: vi.fn(), + }); +}); + +describe('closeSession — worker-less Riff cancellation is an awaited close precondition', () => { + async function fixture( + cancelled: boolean, + liveWorker = false, + registered = true, + options: { resultTaskId?: string; exitAfterResult?: boolean; holdAbortAck?: boolean } = {}, + ) { + const dataDir = mkdtempSync(join(tmpdir(), 'botmux-riff-close-')); + const previousDataDir = config.session.dataDir; + config.session.dataDir = dataDir; + sessionStore.init('app'); + const session = sessionStore.createSession('oc_riff', 'om_riff', 'riff close', 'group'); + session.larkAppId = 'app'; + session.scope = 'chat'; + session.backendType = 'riff'; + session.riffParentTaskId = 'task-riff-123'; + sessionStore.updateSession(session); + getBotMock.mockReturnValue({ + resolvedAllowedUsers: [], + config: { riff: { baseUrl: 'https://riff.invalid', jwt: 'test' } }, + } as any); + cancelRiffTaskMock.mockResolvedValue(cancelled); + + const worker = liveWorker ? new EventEmitter() as any : null; + if (worker) { + worker.killed = false; + worker.exitCode = null; + worker.signalCode = null; + worker.kill = vi.fn(); + worker.send = vi.fn((message: any) => { + if (message.type === 'close_abort' && message.requestId) { + if (!options.holdAbortAck) { + queueMicrotask(() => worker.emit('message', { + type: 'close_abort_result', + requestId: message.requestId, + ok: true, + })); + } + return; + } + if (message.type !== 'close' || !message.requestId) return; + queueMicrotask(() => worker.emit('message', { + type: 'close_result', + requestId: message.requestId, + ok: cancelled, + ...(options.resultTaskId ? { taskId: options.resultTaskId } : {}), + ...(!cancelled ? { + taskId: options.resultTaskId ?? 'task-riff-123', + error: 'task-cancel HTTP 500', + } : {}), + })); + if (options.exitAfterResult) { + queueMicrotask(() => worker.emit('exit', 0, null)); + } + }); + } + const d = ds({ + chatId: session.chatId, + scope: 'chat', + session, + initConfig: { backendType: 'riff' } as any, + worker, + }); + if (worker) __testOnly_setupWorkerHandlers(d, worker); + const registry = registered + ? new Map([[activeSessionKey(d), d]]) + : new Map(); + setActiveSessionsRegistry(registry); + return { + session, + d, + registry, + cleanup() { + worker?.removeAllListeners(); + setActiveSessionsRegistry(new Map()); + config.session.dataDir = previousDataDir; + sessionStore.init(); + rmSync(dataDir, { recursive: true, force: true }); + }, + }; + } + + it('awaits a confirmed cancel, then clears the task id and closes the row', async () => { + const f = await fixture(true); + try { + const result = await closeSession(f.session.sessionId); + expect(result).toEqual({ ok: true, alreadyClosed: false }); + expect(cancelRiffTaskMock).toHaveBeenCalledWith( + expect.objectContaining({ baseUrl: 'https://riff.invalid' }), + 'task-riff-123', + ); + expect(sessionStore.getSession(f.session.sessionId)).toMatchObject({ status: 'closed' }); + expect(sessionStore.getSession(f.session.sessionId)?.riffParentTaskId).toBeUndefined(); + expect(f.registry.size).toBe(0); + } finally { + f.cleanup(); + } + }); + + it('refuses close on cancel failure and preserves the active row + retry task id', async () => { + const f = await fixture(false); + try { + const result = await closeSession(f.session.sessionId); + expect(result).toEqual({ + ok: false, + alreadyClosed: false, + error: 'riff_cancel_failed', + retryable: true, + taskId: 'task-riff-123', + }); + expect(sessionStore.getSession(f.session.sessionId)).toMatchObject({ + status: 'active', + riffParentTaskId: 'task-riff-123', + }); + expect(f.registry.get(activeSessionKey(f.d))).toBe(f.d); + } finally { + f.cleanup(); + } + }); + + it('refuses worker-less close before cancellation when runtime and durable Riff lineage differ', async () => { + const f = await fixture(true); + try { + f.d.session = { + ...f.session, + riffParentTaskId: 'task-runtime-stale', + }; + + const result = await closeSession(f.session.sessionId); + expect(result).toEqual({ + ok: false, + alreadyClosed: false, + error: 'riff_task_changed', + retryable: true, + taskId: 'task-riff-123', + }); + expect(cancelRiffTaskMock).not.toHaveBeenCalled(); + expect(f.registry.get(activeSessionKey(f.d))).toBe(f.d); + expect(sessionStore.getSession(f.session.sessionId)).toMatchObject({ + status: 'active', + riffParentTaskId: 'task-riff-123', + }); + } finally { + f.cleanup(); + } + }); + + it.each([ + { label: 'exact task', stateTaskId: 'task-fenced-child', expectedTaskId: 'task-fenced-child' }, + { label: 'authoritative null', stateTaskId: null, expectedTaskId: undefined }, + ])('refuses worker-less close during a retained shutdown fence ($label)', async ({ + stateTaskId, + expectedTaskId, + }) => { + const f = await fixture(true); + try { + f.d.riffShutdownState = { + phase: 'prepared', + requestId: 'shutdown-fence-close', + taskId: stateTaskId, + }; + const routeKey = activeSessionKey(f.d); + + const result = await closeSession(f.session.sessionId); + + expect(result).toEqual({ + ok: false, + alreadyClosed: false, + error: 'riff_shutdown_fence_in_progress', + retryable: true, + ...(expectedTaskId ? { taskId: expectedTaskId } : {}), + }); + expect(cancelRiffTaskMock).not.toHaveBeenCalled(); + expect(f.registry.get(routeKey)).toBe(f.d); + expect(f.d.riffShutdownState).toMatchObject({ requestId: 'shutdown-fence-close' }); + expect(sessionStore.getSession(f.session.sessionId)).toMatchObject({ + status: 'active', + riffParentTaskId: 'task-riff-123', + }); + } finally { + f.cleanup(); + } + }); + + it('refuses a live-worker close when its prepare handshake reports remote cancel failure', async () => { + const f = await fixture(false, true); + try { + const worker = f.d.worker as any; + const result = await closeSession(f.session.sessionId); + expect(result).toMatchObject({ + ok: false, + error: 'riff_worker_close_failed', + retryable: true, + taskId: 'task-riff-123', + }); + expect(worker.send).toHaveBeenCalledWith({ + type: 'close', + requestId: expect.any(String), + }); + expect(cancelRiffTaskMock).not.toHaveBeenCalled(); + expect(f.d.worker).toBe(worker); + expect(sessionStore.getSession(f.session.sessionId)).toMatchObject({ + status: 'active', + riffParentTaskId: 'task-riff-123', + }); + } finally { + f.cleanup(); + } + }); + + it('commits a successful live-worker close only after the row is durably closed', async () => { + const f = await fixture(true, true); + try { + const worker = f.d.worker as any; + const result = await closeSession(f.session.sessionId); + expect(result).toEqual({ ok: true, alreadyClosed: false }); + + const closeRequest = worker.send.mock.calls[0]?.[0]; + expect(closeRequest).toEqual({ + type: 'close', + requestId: expect.any(String), + }); + expect(worker.send.mock.calls[1]?.[0]).toEqual({ + type: 'close_commit', + requestId: closeRequest.requestId, + }); + expect(cancelRiffTaskMock).not.toHaveBeenCalled(); + expect(f.d.worker).toBeNull(); + expect(f.registry.size).toBe(0); + expect(sessionStore.getSession(f.session.sessionId)).toMatchObject({ status: 'closed' }); + expect(sessionStore.getSession(f.session.sessionId)?.riffParentTaskId).toBeUndefined(); + } finally { + f.cleanup(); + } + }); + + it('cancels an open persisted Riff row even when it is absent from the active registry', async () => { + const f = await fixture(true, false, false); + try { + const result = await closeSession(f.session.sessionId); + expect(result).toEqual({ ok: true, alreadyClosed: false }); + expect(cancelRiffTaskMock).toHaveBeenCalledWith( + expect.objectContaining({ baseUrl: 'https://riff.invalid' }), + 'task-riff-123', + ); + expect(sessionStore.getSession(f.session.sessionId)).toMatchObject({ status: 'closed' }); + expect(sessionStore.getSession(f.session.sessionId)?.riffParentTaskId).toBeUndefined(); + } finally { + f.cleanup(); + } + }); + + it('rejects a message explicitly during prepare, then restores admission after close failure', async () => { + const f = await fixture(false, true); + try { + const worker = f.d.worker as any; + worker.send = vi.fn(); // hold close_result so the prepare phase is observable + const closeP = closeSession(f.session.sessionId); + await Promise.resolve(); + const closeRequest = worker.send.mock.calls[0]?.[0]; + expect(closeRequest).toMatchObject({ type: 'close', requestId: expect.any(String) }); + expect(f.d.riffCloseState).toMatchObject({ phase: 'preparing', requestId: closeRequest.requestId }); + + expect(sendWorkerInput(f.d, 'racing message', 'om_race')).toBe(false); + await Promise.resolve(); + expect(worker.send.mock.calls.some(([message]: any[]) => message.type === 'message')).toBe(false); + expect(sessionReplyMock).toHaveBeenCalledWith( + expect.any(String), + expect.stringContaining('Riff'), + 'text', + 'app', + 'om_race', + ); + + worker.emit('message', { + type: 'close_result', + requestId: closeRequest.requestId, + ok: false, + taskId: 'task-riff-123', + error: 'task-cancel HTTP 500', + }); + await vi.waitFor(() => expect(worker.send).toHaveBeenCalledWith({ + type: 'close_abort', + requestId: closeRequest.requestId, + })); + // The failed prepare's backend abort may already have completed, but the + // daemon must keep its admission fence until the exact worker ACK lands. + expect(f.d.riffCloseState).toMatchObject({ requestId: closeRequest.requestId }); + expect(sendWorkerInput(f.d, 'during abort ACK wait', 'om_abort_wait')).toBe(false); + worker.emit('message', { + type: 'close_abort_result', + requestId: closeRequest.requestId, + ok: true, + }); + await expect(closeP).resolves.toMatchObject({ ok: false, retryable: true }); + expect(worker.send).toHaveBeenCalledWith({ + type: 'close_abort', + requestId: closeRequest.requestId, + }); + expect(f.d.riffCloseState).toBeUndefined(); + + expect(sendWorkerInput(f.d, 'after failed close', 'om_after')).toBe(true); + expect(worker.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'message', + content: 'after failed close', + turnId: 'om_after', + })); + } finally { + f.cleanup(); + } + }); + + it('retains an uncertain close fence when the exact worker exits before close_result', async () => { + const f = await fixture(true, true); + try { + const worker = f.d.worker as any; + worker.send = vi.fn(); // hold close_result: final remote lineage is unknown + const closeP = closeSession(f.session.sessionId); + await vi.waitFor(() => expect(f.d.riffCloseState).toMatchObject({ + phase: 'preparing', + requestId: expect.any(String), + })); + + worker.emit('exit', 1, null); + await expect(closeP).resolves.toEqual({ + ok: false, + alreadyClosed: false, + error: 'riff_worker_close_failed', + retryable: true, + taskId: 'task-riff-123', + }); + expect(f.d.worker).toBeNull(); + expect(f.d.riffCloseState).toMatchObject({ + phase: 'uncertain', + requestId: expect.any(String), + taskId: 'task-riff-123', + }); + expect(sessionStore.getSession(f.session.sessionId)).toMatchObject({ + status: 'active', + riffParentTaskId: 'task-riff-123', + }); + expect(cancelRiffTaskMock).not.toHaveBeenCalled(); + + // Neither a normal turn nor a second /close may reuse/cancel the stale + // persisted parent while an unreported late child may exist remotely. + expect(sendWorkerInput(f.d, 'must remain fenced', 'om_uncertain')).toBe(false); + await expect(closeSession(f.session.sessionId)).resolves.toEqual({ + ok: false, + alreadyClosed: false, + error: 'riff_close_reconciliation_required', + retryable: true, + taskId: 'task-riff-123', + }); + expect(cancelRiffTaskMock).not.toHaveBeenCalled(); + } finally { + f.cleanup(); + } + }); + + it('aborts a successful prepare when durable close throws, preserves lineage, and retries', async () => { + const f = await fixture(true, true, true, { holdAbortAck: true }); + const closeSpy = vi.spyOn(sessionStore, 'closeSession') + .mockImplementationOnce(() => { throw new Error('disk unavailable'); }); + try { + const worker = f.d.worker as any; + const firstPromise = closeSession(f.session.sessionId); + await vi.waitFor(() => expect(worker.send.mock.calls.some( + ([message]: any[]) => message.type === 'close_abort', + )).toBe(true)); + const closeRequest = worker.send.mock.calls.find(([m]: any[]) => m.type === 'close')?.[0]; + expect(f.d.riffCloseState).toMatchObject({ + phase: 'prepared', + requestId: closeRequest.requestId, + }); + expect(sendWorkerInput(f.d, 'must wait for abort ACK', 'om_abort_race')).toBe(false); + worker.emit('message', { + type: 'close_abort_result', + requestId: closeRequest.requestId, + ok: true, + }); + const first = await firstPromise; + expect(first).toEqual({ + ok: false, + alreadyClosed: false, + error: 'riff_durable_close_failed', + retryable: true, + taskId: 'task-riff-123', + }); + expect(worker.send).toHaveBeenCalledWith({ type: 'close_abort', requestId: closeRequest.requestId }); + expect(sessionStore.getSession(f.session.sessionId)).toMatchObject({ + status: 'active', + riffParentTaskId: 'task-riff-123', + }); + expect(f.registry.get(activeSessionKey(f.d))).toBe(f.d); + expect(sendWorkerInput(f.d, 'after abort', 'om_after_abort')).toBe(true); + + // The retry uses a new request and may ACK normally. + const originalSend = worker.send.getMockImplementation(); + worker.send.mockImplementation((message: any) => { + if (message.type === 'close_abort') { + queueMicrotask(() => worker.emit('message', { + type: 'close_abort_result', + requestId: message.requestId, + ok: true, + })); + return; + } + originalSend?.(message); + }); + const retried = await closeSession(f.session.sessionId); + expect(retried).toEqual({ ok: true, alreadyClosed: false }); + expect(sessionStore.getSession(f.session.sessionId)).toMatchObject({ status: 'closed' }); + } finally { + closeSpy.mockRestore(); + f.cleanup(); + } + }); + + it('aborts when prepared lineage update throws and keeps the new task retryable', async () => { + const f = await fixture(true, true, true, { resultTaskId: 'task-riff-new' }); + const updateSpy = vi.spyOn(sessionStore, 'updateSession') + .mockImplementationOnce(() => { throw new Error('lineage save failed'); }); + try { + const worker = f.d.worker as any; + const result = await closeSession(f.session.sessionId); + expect(result).toEqual({ + ok: false, + alreadyClosed: false, + error: 'riff_durable_close_failed', + retryable: true, + taskId: 'task-riff-new', + }); + const closeRequest = worker.send.mock.calls.find(([m]: any[]) => m.type === 'close')?.[0]; + expect(worker.send).toHaveBeenCalledWith({ type: 'close_abort', requestId: closeRequest.requestId }); + expect(f.d.session.riffParentTaskId).toBe('task-riff-new'); + expect(f.d.riffCloseState).toBeUndefined(); + expect(sendWorkerInput(f.d, 'after lineage failure', 'om_lineage')).toBe(true); + } finally { + updateSpy.mockRestore(); + f.cleanup(); + } + }); + + it('commits the durable close safely when the worker exits after prepare ACK', async () => { + const f = await fixture(true, true, true, { exitAfterResult: true }); + try { + const worker = f.d.worker as any; + const result = await closeSession(f.session.sessionId); + expect(result).toEqual({ ok: true, alreadyClosed: false }); + expect(f.d.worker).toBeNull(); + expect(f.d.riffCloseState).toBeUndefined(); + expect(sessionStore.getSession(f.session.sessionId)).toMatchObject({ status: 'closed' }); + expect(f.registry.size).toBe(0); + expect(worker.send.mock.calls.some(([message]: any[]) => message.type === 'close_commit')).toBe(false); + } finally { + f.cleanup(); + } + }); + + it('removes every exact-object alias while preserving a same-key successor', async () => { + const f = await fixture(true); + try { + const successorSession = sessionStore.createSession('oc_riff', 'om_successor', 'successor', 'group'); + successorSession.larkAppId = 'app'; + successorSession.scope = 'chat'; + sessionStore.updateSession(successorSession); + const successor = ds({ + chatId: successorSession.chatId, + scope: 'chat', + session: successorSession, + initConfig: { backendType: 'pty' } as any, + }); + f.registry.clear(); + f.registry.set('legacy-alias-a', f.d); + f.registry.set('legacy-alias-b', f.d); + f.registry.set(activeSessionKey(f.d), successor); + + const result = await closeSession(f.session.sessionId); + expect(result).toEqual({ ok: true, alreadyClosed: false }); + expect(f.registry.get(activeSessionKey(f.d))).toBe(successor); + expect([...f.registry.values()]).toEqual([successor]); + } finally { + f.cleanup(); + } + }); + + it('fails closed on a worker-less same-key Riff collision and preserves route + task id', async () => { + const f = await fixture(false); + try { + const incomingSession = sessionStore.createSession('oc_riff', 'om_incoming', 'incoming', 'group'); + incomingSession.larkAppId = 'app'; + incomingSession.scope = 'chat'; + sessionStore.updateSession(incomingSession); + const incoming = ds({ + chatId: incomingSession.chatId, + scope: 'chat', + session: incomingSession, + initConfig: { backendType: 'pty' } as any, + }); + const key = activeSessionKey(f.d); + + const result = await setActiveSessionSafe(f.registry, key, incoming); + expect(result).toMatchObject({ + accepted: false, + reason: 'cleanup_failed', + keptSessionId: f.session.sessionId, + preservedIncomingSessionId: incomingSession.sessionId, + cleanupSessionId: f.session.sessionId, + error: 'riff_cancel_failed', + taskId: 'task-riff-123', + }); + expect(f.registry.get(key)).toBe(f.d); + expect(sessionStore.getSession(f.session.sessionId)).toMatchObject({ + status: 'active', + riffParentTaskId: 'task-riff-123', + }); + expect(sessionStore.getSession(incomingSession.sessionId)).toMatchObject({ status: 'active' }); + } finally { + f.cleanup(); + } + }); + + it('uses the live Riff prepare/commit handshake before replacing a collision owner', async () => { + const f = await fixture(true, true); + try { + const worker = f.d.worker as any; + const incomingSession = sessionStore.createSession('oc_riff', 'om_incoming_live', 'incoming', 'group'); + incomingSession.larkAppId = 'app'; + incomingSession.scope = 'chat'; + sessionStore.updateSession(incomingSession); + const incoming = ds({ + chatId: incomingSession.chatId, + scope: 'chat', + session: incomingSession, + initConfig: { backendType: 'pty' } as any, + }); + const key = activeSessionKey(f.d); + + const result = await setActiveSessionSafe(f.registry, key, incoming); + expect(result).toEqual({ accepted: true, closedSessionId: f.session.sessionId }); + const closeRequest = worker.send.mock.calls.find(([m]: any[]) => m.type === 'close')?.[0]; + expect(worker.send).toHaveBeenCalledWith({ type: 'close_commit', requestId: closeRequest.requestId }); + expect(f.registry.get(key)).toBe(incoming); + expect(sessionStore.getSession(f.session.sessionId)).toMatchObject({ status: 'closed' }); + } finally { + f.cleanup(); + } + }); + + it('preserves a live Riff collision owner when prepare cancellation fails', async () => { + const f = await fixture(false, true); + try { + const incomingSession = sessionStore.createSession('oc_riff', 'om_incoming_live_fail', 'incoming', 'group'); + incomingSession.larkAppId = 'app'; + incomingSession.scope = 'chat'; + sessionStore.updateSession(incomingSession); + const incoming = ds({ + chatId: incomingSession.chatId, + scope: 'chat', + session: incomingSession, + initConfig: { backendType: 'pty' } as any, + }); + const key = activeSessionKey(f.d); + + const result = await setActiveSessionSafe(f.registry, key, incoming); + expect(result).toMatchObject({ + accepted: false, + reason: 'cleanup_failed', + cleanupSessionId: f.session.sessionId, + taskId: 'task-riff-123', + }); + expect(f.registry.get(key)).toBe(f.d); + expect(f.d.worker).not.toBeNull(); + expect(sessionStore.getSession(f.session.sessionId)).toMatchObject({ + status: 'active', + riffParentTaskId: 'task-riff-123', + }); + } finally { + f.cleanup(); + } + }); + + it('refuses distinct owners of the same durable session id without cancelling or closing it', async () => { + const f = await fixture(true); + try { + const incoming = ds({ + chatId: f.session.chatId, + scope: 'chat', + session: { ...f.session }, + initConfig: { backendType: 'riff' } as any, + }); + const key = activeSessionKey(f.d); + + const result = await setActiveSessionSafe(f.registry, key, incoming); + expect(result).toEqual({ + accepted: false, + reason: 'cleanup_failed', + keptSessionId: f.session.sessionId, + preservedIncomingSessionId: f.session.sessionId, + cleanupSessionId: f.session.sessionId, + error: 'ambiguous_session_id', + }); + expect(cancelRiffTaskMock).not.toHaveBeenCalled(); + expect(f.registry.get(key)).toBe(f.d); + expect(sessionStore.getSession(f.session.sessionId)).toMatchObject({ + status: 'active', + riffParentTaskId: 'task-riff-123', + }); + } finally { + f.cleanup(); + } + }); }); describe('killWorker — orphaned backing session teardown (no live worker)', () => { @@ -144,4 +789,22 @@ describe('killWorker — with a live worker (unchanged path)', () => { expect(d.worker).toBeNull(); expect(d.managedTurnOrigin).toBeUndefined(); }); + + it('refuses an unprepared live Riff retirement without mutating worker authority or lineage', () => { + const send = vi.fn(); + const worker = { killed: false, send, once: vi.fn() } as any; + const d = ds({ + worker, + managedTurnOrigin: { capability: 'cap-riff-live', turnId: 'om-riff-live' }, + session: { sessionId: SID, backendType: 'riff', riffParentTaskId: 'task-live-riff' } as any, + }, { backendType: 'riff' }); + + killWorker(d); + + expect(send).not.toHaveBeenCalled(); + expect(cancelRiffTaskMock).not.toHaveBeenCalled(); + expect(d.session.riffParentTaskId).toBe('task-live-riff'); + expect(d.worker).toBe(worker); + expect(d.managedTurnOrigin).toEqual({ capability: 'cap-riff-live', turnId: 'om-riff-live' }); + }); }); diff --git a/test/lark-outbound-hook.test.ts b/test/lark-outbound-hook.test.ts index d6ea92722..244562800 100644 --- a/test/lark-outbound-hook.test.ts +++ b/test/lark-outbound-hook.test.ts @@ -65,4 +65,40 @@ describe('Lark outbound hook provider replay suppression', () => { expect(mocks.reply).toHaveBeenCalledOnce(); expect(mocks.emitHookEvent).not.toHaveBeenCalled(); }); + + it('fences the post-provider hook and forwards its frozen managed origin', async () => { + const beforeHook = vi.fn(async () => {}); + const hookOrigin = { + ipcPort: 4310, + sessionId: 'sid', + capability: 'ab'.repeat(32), + turnId: 'turn-1', + dispatchAttempt: 2, + }; + await sendMessage( + 'app', 'oc_chat', 'answer', 'text', undefined, { sessionId: 'sid' }, + { beforeHook, hookOrigin }, + ); + + expect(beforeHook).toHaveBeenCalledOnce(); + expect(mocks.create.mock.invocationCallOrder[0]) + .toBeLessThan(beforeHook.mock.invocationCallOrder[0]!); + expect(beforeHook.mock.invocationCallOrder[0]) + .toBeLessThan(mocks.emitHookEvent.mock.invocationCallOrder[0]!); + expect(mocks.emitHookEvent).toHaveBeenCalledWith( + 'outbound.send', + expect.objectContaining({ messageId: 'om_send', content: 'answer' }), + { managedOrigin: hookOrigin }, + ); + }); + + it('drops only the hook when authority is revoked after provider acceptance', async () => { + const beforeHook = vi.fn(async () => { throw new Error('origin rotated'); }); + await expect(sendMessage( + 'app', 'oc_chat', 'answer', 'text', undefined, { sessionId: 'sid' }, + { beforeHook }, + )).resolves.toBe('om_send'); + expect(beforeHook).toHaveBeenCalledOnce(); + expect(mocks.emitHookEvent).not.toHaveBeenCalled(); + }); }); diff --git a/test/list-chat-bot-members.test.ts b/test/list-chat-bot-members.test.ts index 642befdb4..3c4e0dc65 100644 --- a/test/list-chat-bot-members.test.ts +++ b/test/list-chat-bot-members.test.ts @@ -66,6 +66,7 @@ vi.mock('../src/utils/user-token.js', () => ({ })); vi.mock('../src/bot-registry.js', () => ({ + configureLarkClientHttpTimeout: vi.fn(), loadBotConfigs: vi.fn(() => [ { larkAppId: 'cli_self', larkAppSecret: 's1', cliId: 'codex' }, { larkAppId: 'cli_peer', larkAppSecret: 's2', cliId: 'codex' }, diff --git a/test/managed-origin-attestation.test.ts b/test/managed-origin-attestation.test.ts new file mode 100644 index 000000000..a6e86837d --- /dev/null +++ b/test/managed-origin-attestation.test.ts @@ -0,0 +1,176 @@ +import { execFileSync } from 'node:child_process'; +import { + existsSync, + mkdtempSync, + rmSync, + symlinkSync, + writeFileSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { afterEach, describe, expect, it } from 'vitest'; +import { + attestManagedOrigin, + MANAGED_ORIGIN_PROOF_DOMAIN, + ManagedOriginAttestationError, + writeManagedOriginAttestationProof, +} from '../src/core/managed-origin-attestation.js'; +import { + ensureManagedOriginAttestationDirectory, + managedOriginAttestationProofPath, +} from '../src/core/managed-origin-capability.js'; + +describe('managed-origin host proof sidecar', () => { + const CHANNEL = '77'.repeat(32); + const roots: string[] = []; + const makeRoot = () => { + const root = mkdtempSync(join(tmpdir(), 'botmux-origin-attest-')); + roots.push(root); + return root; + }; + afterEach(() => { + for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); + }); + + it('accepts only the nonce-bound host file and ignores the HTTP body', async () => { + const dataDir = makeRoot(); + const context = { + dataDir, + sessionId: 'session-a', + channelId: CHANNEL, + capability: 'ab'.repeat(32), + ipcPortFallback: 4321, + }; + const nonce = 'cd'.repeat(32); + const result = await attestManagedOrigin({ + context, + nonce, + fetchImpl: async () => { + writeManagedOriginAttestationProof({ + dataDir, + proof: { + domain: MANAGED_ORIGIN_PROOF_DOMAIN, + version: 1, + nonce, + channelId: CHANNEL, + sessionId: context.sessionId, + turnId: 'turn-live', + dispatchAttempt: 4, + requiresCodexAppLedger: true, + issuedAtMs: Date.now(), + }, + }); + return new Response(JSON.stringify({ + // Deliberately forged transport body: it must never win. + turnId: 'turn-forged', + }), { status: 200 }); + }, + }); + expect(result).toEqual({ + sessionId: 'session-a', + turnId: 'turn-live', + dispatchAttempt: 4, + requiresCodexAppLedger: true, + }); + }); + + it('rejects expired proof bytes even when HTTP returns 200', async () => { + const dataDir = makeRoot(); + let now = 50_000; + const nonce = 'ef'.repeat(32); + await expect(attestManagedOrigin({ + context: { + dataDir, + sessionId: 'session-a', + channelId: CHANNEL, + capability: 'ab'.repeat(32), + ipcPortFallback: 4321, + }, + nonce, + timeoutMs: 3, + now: () => now, + wait: async delay => { now += delay; }, + fetchImpl: async () => { + writeManagedOriginAttestationProof({ + dataDir, + proof: { + domain: MANAGED_ORIGIN_PROOF_DOMAIN, + version: 1, + nonce, + channelId: CHANNEL, + sessionId: 'session-a', + turnId: 'turn-old', + requiresCodexAppLedger: false, + issuedAtMs: 1, + }, + }); + return new Response(null, { status: 200 }); + }, + })).rejects.toBeInstanceOf(ManagedOriginAttestationError); + }); + + it('fails closed on a pre-existing symlink leaf without touching its target', () => { + const dataDir = makeRoot(); + const nonce = '12'.repeat(32); + ensureManagedOriginAttestationDirectory(dataDir, 'session-a', CHANNEL); + const target = join(dataDir, 'target'); + writeFileSync(target, 'keep'); + symlinkSync(target, managedOriginAttestationProofPath(dataDir, 'session-a', CHANNEL, nonce)); + expect(() => writeManagedOriginAttestationProof({ + dataDir, + proof: { + domain: MANAGED_ORIGIN_PROOF_DOMAIN, + version: 1, + nonce, + channelId: CHANNEL, + sessionId: 'session-a', + turnId: 'turn', + requiresCodexAppLedger: false, + issuedAtMs: Date.now(), + }, + })).toThrow(); + expect(existsSync(target)).toBe(true); + }); + + it.runIf(process.platform !== 'win32')('opens a FIFO proof nonblocking and times out', async () => { + const dataDir = makeRoot(); + const nonce = '34'.repeat(32); + ensureManagedOriginAttestationDirectory(dataDir, 'session-a', CHANNEL); + execFileSync('mkfifo', [managedOriginAttestationProofPath(dataDir, 'session-a', CHANNEL, nonce)]); + let now = 1_000; + await expect(attestManagedOrigin({ + context: { + dataDir, + sessionId: 'session-a', + channelId: CHANNEL, + capability: 'ab'.repeat(32), + ipcPortFallback: 4321, + }, + nonce, + timeoutMs: 2, + now: () => now, + wait: async delay => { now += delay; }, + fetchImpl: async () => new Response(null, { status: 200 }), + })).rejects.toThrow(/未生成有效/); + }); + + it('removes an oversized proof leaf after rejecting it', () => { + const dataDir = makeRoot(); + const nonce = '56'.repeat(32); + const path = managedOriginAttestationProofPath(dataDir, 'session-a', CHANNEL, nonce); + expect(() => writeManagedOriginAttestationProof({ + dataDir, + proof: { + domain: MANAGED_ORIGIN_PROOF_DOMAIN, + version: 1, + nonce, + channelId: CHANNEL, + sessionId: 'session-a', + turnId: 't'.repeat(9_000), + requiresCodexAppLedger: false, + issuedAtMs: Date.now(), + }, + })).toThrow(/size limit/); + expect(existsSync(path)).toBe(false); + }); +}); diff --git a/test/managed-origin-capability.test.ts b/test/managed-origin-capability.test.ts index 730a36223..3495e38b0 100644 --- a/test/managed-origin-capability.test.ts +++ b/test/managed-origin-capability.test.ts @@ -1,18 +1,32 @@ import { describe, expect, it, afterEach } from 'vitest'; +import { execFileSync } from 'node:child_process'; import { - mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, symlinkSync, writeFileSync, + chmodSync, lstatSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, statSync, + symlinkSync, writeFileSync, } from 'node:fs'; import { tmpdir } from 'node:os'; import { dirname, join } from 'node:path'; import { + ensureManagedOriginAttestationDirectory, + ensureManagedOriginDataRootProbe, + ensureManagedOriginRootLocator, hasMatchingManagedOriginCapability, + hasManagedOriginIsolationMarker, + managedOriginAttestationDirectory, managedOriginCapabilityPath, + managedOriginDataRootProbeAccess, + managedOriginDataRootProbePath, + managedOriginRootLocatorPath, + readManagedOriginAuthorityFile, readManagedOriginCapability, + readManagedOriginRootLocator, RELAY_ORIGIN_CAPABILITY_BASENAME, replaceManagedOriginCapabilityFile, } from '../src/core/managed-origin-capability.js'; describe('managed origin capability transport', () => { + const G1 = '11'.repeat(32); + const G2 = '22'.repeat(32); const dirs: string[] = []; const makeDir = (): string => { const dir = mkdtempSync(join(tmpdir(), 'botmux-origin-cap-')); @@ -27,30 +41,49 @@ describe('managed origin capability transport', () => { it('derives an opaque path and validates a direct per-session claim', () => { const dir = makeDir(); const sessionId = '../session/private'; - const path = managedOriginCapabilityPath(dir, sessionId); + const path = managedOriginCapabilityPath(dir, sessionId, G1); expect(path).toMatch(/\/read-isolation\/origin-[a-f0-9]{64}\.json$/); expect(path).not.toContain(sessionId); - expect(managedOriginCapabilityPath(dir, 'another-session')).not.toBe(path); + expect(managedOriginCapabilityPath(dir, 'another-session', G1)).not.toBe(path); + expect(managedOriginCapabilityPath(dir, sessionId, G2)).not.toBe(path); replaceManagedOriginCapabilityFile(path, JSON.stringify({ sessionId, + channelId: G1, capability: 'ab'.repeat(32), turnId: 'turn-1', dispatchAttempt: 2, })); - expect(readManagedOriginCapability(dir, sessionId)).toEqual({ + expect(readManagedOriginCapability(dir, sessionId, undefined, G1)).toEqual({ sessionId, + channelId: G1, capability: 'ab'.repeat(32), turnId: 'turn-1', dispatchAttempt: 2, }); expect(statSync(path).mode & 0o777).toBe(0o600); - expect(readManagedOriginCapability(dir, 'another-session')).toBeNull(); + expect(readManagedOriginCapability(dir, 'another-session', undefined, G1)).toBeNull(); + expect(readManagedOriginCapability(dir, sessionId, undefined, G2)).toBeNull(); + expect(readManagedOriginCapability(dir, sessionId)).toBeNull(); + expect(hasMatchingManagedOriginCapability( + dir, + sessionId, + 'ab'.repeat(32), + undefined, + G1, + )).toBe(true); + expect(hasMatchingManagedOriginCapability( + dir, + sessionId, + 'ab'.repeat(32), + undefined, + G2, + )).toBe(false); }); it('replaces a planted destination symlink without overwriting its target', () => { const dir = makeDir(); - const path = managedOriginCapabilityPath(dir, 'session-a'); + const path = managedOriginCapabilityPath(dir, 'session-a', G1); mkdirSync(dirname(path), { recursive: true }); const target = join(dir, 'target.txt'); writeFileSync(target, 'sentinel'); @@ -58,11 +91,30 @@ describe('managed origin capability transport', () => { replaceManagedOriginCapabilityFile(path, JSON.stringify({ sessionId: 'session-a', + channelId: G1, capability: 'cd'.repeat(32), })); expect(readFileSync(target, 'utf8')).toBe('sentinel'); - expect(readManagedOriginCapability(dir, 'session-a')?.capability).toBe('cd'.repeat(32)); + expect(readManagedOriginCapability(dir, 'session-a', undefined, G1)?.capability).toBe('cd'.repeat(32)); + }); + + it('recovers a planted attestation-dir symlink without touching its target and writes the static marker', () => { + const dir = makeDir(); + const proofDir = managedOriginAttestationDirectory(dir, 'session-a', G1); + mkdirSync(dirname(proofDir), { recursive: true }); + const target = join(dir, 'attacker-target'); + mkdirSync(target); + writeFileSync(join(target, 'sentinel'), 'keep'); + symlinkSync(target, proofDir); + + expect(ensureManagedOriginAttestationDirectory(dir, 'session-a', G1)).toBe(proofDir); + expect(lstatSync(proofDir).isDirectory()).toBe(true); + expect(lstatSync(proofDir).isSymbolicLink()).toBe(false); + expect(statSync(proofDir).mode & 0o777).toBe(0o700); + expect(hasManagedOriginIsolationMarker(dir, 'session-a', G1)).toBe(true); + expect(hasManagedOriginIsolationMarker(dir, 'session-a', G2)).toBe(false); + expect(readFileSync(join(target, 'sentinel'), 'utf8')).toBe('keep'); }); it('rejects a symlinked parent instead of writing through it', () => { @@ -70,13 +122,14 @@ describe('managed origin capability transport', () => { const targetDir = join(dir, 'attacker-target'); mkdirSync(targetDir); symlinkSync(targetDir, join(dir, 'read-isolation')); - const path = managedOriginCapabilityPath(dir, 'session-a'); + const path = managedOriginCapabilityPath(dir, 'session-a', G1); expect(() => replaceManagedOriginCapabilityFile(path, JSON.stringify({ sessionId: 'session-a', + channelId: G1, capability: 'de'.repeat(32), }))).toThrow(/not a real directory/); - expect(readManagedOriginCapability(dir, 'session-a')).toBeNull(); + expect(readManagedOriginCapability(dir, 'session-a', undefined, G1)).toBeNull(); }); it('reads the Linux relay token but rejects malformed authority', () => { @@ -84,35 +137,100 @@ describe('managed origin capability transport', () => { const relay = join(dir, 'relay'); mkdirSync(relay); const relayPath = join(relay, RELAY_ORIGIN_CAPABILITY_BASENAME); - writeFileSync(relayPath, JSON.stringify({ token: 'ef'.repeat(32) })); + writeFileSync(relayPath, JSON.stringify({ token: 'ef'.repeat(32) }), { mode: 0o600 }); expect(readManagedOriginCapability(dir, 'session-a', relay)).toEqual({ sessionId: 'session-a', capability: 'ef'.repeat(32), }); - writeFileSync(relayPath, JSON.stringify({ token: 'not-a-capability' })); + writeFileSync(relayPath, JSON.stringify({ token: 'not-a-capability' }), { mode: 0o600 }); expect(readManagedOriginCapability(dir, 'session-a', relay)).toBeNull(); }); - it('ready preflight rejects stale, malformed, and non-file relay capabilities', () => { + it('refuses a symlink capability leaf instead of following it', () => { const dir = makeDir(); const relay = join(dir, 'relay'); mkdirSync(relay); - const relayPath = join(relay, RELAY_ORIGIN_CAPABILITY_BASENAME); - const current = '12'.repeat(32); - - writeFileSync(relayPath, JSON.stringify({ token: current })); - expect(hasMatchingManagedOriginCapability(dir, 'session-a', current, relay)).toBe(true); - expect(hasMatchingManagedOriginCapability(dir, 'session-a', '34'.repeat(32), relay)).toBe(false); - - writeFileSync(relayPath, '{broken-json'); - expect(hasMatchingManagedOriginCapability(dir, 'session-a', current, relay)).toBe(false); - - rmSync(relayPath, { force: true }); - mkdirSync(relayPath); - expect(() => replaceManagedOriginCapabilityFile( - relayPath, - JSON.stringify({ token: current }), - )).toThrow(); - expect(hasMatchingManagedOriginCapability(dir, 'session-a', current, relay)).toBe(false); + const target = join(dir, 'target-cap'); + writeFileSync(target, JSON.stringify({ token: 'ef'.repeat(32) }), { mode: 0o600 }); + symlinkSync(target, join(relay, RELAY_ORIGIN_CAPABILITY_BASENAME)); + expect(readManagedOriginCapability(dir, 'session-a', relay)).toBeNull(); + }); + + it('keeps same-session pane generations disjoint when an old worker tears down', () => { + const dir = makeDir(); + const p1 = managedOriginCapabilityPath(dir, 'session-a', G1); + const p2 = managedOriginCapabilityPath(dir, 'session-a', G2); + replaceManagedOriginCapabilityFile(p1, JSON.stringify({ + sessionId: 'session-a', channelId: G1, capability: 'aa'.repeat(32), + })); + replaceManagedOriginCapabilityFile(p2, JSON.stringify({ + sessionId: 'session-a', channelId: G2, capability: 'bb'.repeat(32), + })); + rmSync(p1, { force: true }); + expect(readManagedOriginCapability(dir, 'session-a', undefined, G1)).toBeNull(); + expect(readManagedOriginCapability(dir, 'session-a', undefined, G2)?.capability) + .toBe('bb'.repeat(32)); + }); + + it('reads a root locator through a symlinked ~/.botmux parent and rejects unsafe leaves', () => { + const root = makeDir(); + const osHome = join(root, 'home'); + const actualBotmux = join(root, 'actual-botmux'); + const dataDir = join(root, 'data'); + mkdirSync(osHome); + mkdirSync(actualBotmux, { mode: 0o700 }); + mkdirSync(dataDir); + symlinkSync(actualBotmux, join(osHome, '.botmux')); + + ensureManagedOriginRootLocator(osHome, 'session-a', dataDir); + expect(readManagedOriginRootLocator(osHome, 'session-a')).toEqual({ + sessionId: 'session-a', dataDir: realpathSync(dataDir), + }); + + const locator = managedOriginRootLocatorPath(osHome, 'session-a'); + const target = join(root, 'attacker-locator'); + writeFileSync(target, JSON.stringify({ + domain: 'botmux.managed-origin-root.v1', sessionId: 'session-a', dataDir, + }), { mode: 0o600 }); + rmSync(locator); + symlinkSync(target, locator); + expect(readManagedOriginRootLocator(osHome, 'session-a')).toBeNull(); + }); + + it.runIf(process.platform !== 'win32')('rejects FIFO and oversized authority metadata without blocking', () => { + const dir = makeDir(); + const fifo = join(dir, 'authority.fifo'); + execFileSync('mkfifo', [fifo]); + expect(readManagedOriginAuthorityFile(fifo)).toBeNull(); + const oversized = join(dir, 'oversized.json'); + writeFileSync(oversized, 'x'.repeat(8 * 1024 + 1), { mode: 0o600 }); + expect(readManagedOriginAuthorityFile(oversized)).toBeNull(); + }); + + it('binds the kernel probe to the exact data root and rejects DAC-only denial', () => { + const root = makeDir(); + const dataDir = join(root, 'data'); + const sibling = join(root, 'fake-data'); + mkdirSync(dataDir); + mkdirSync(sibling); + ensureManagedOriginDataRootProbe(dataDir, 'session-a'); + expect(managedOriginDataRootProbeAccess(realpathSync(dataDir), 'session-a')) + .toBe('host_accessible'); + expect(managedOriginDataRootProbePath(realpathSync(dataDir), 'session-a')) + .not.toBe(managedOriginDataRootProbePath(realpathSync(sibling), 'session-a')); + expect(managedOriginDataRootProbeAccess(realpathSync(sibling), 'session-a')) + .toBe('missing_or_unsafe'); + + const fakeProbe = managedOriginDataRootProbePath(realpathSync(sibling), 'session-a'); + replaceManagedOriginCapabilityFile(fakeProbe, JSON.stringify({ + domain: 'botmux.managed-origin-root-probe.v1', + sessionId: 'session-a', + dataDir: realpathSync(sibling), + })); + // A writable fake root can manufacture EACCES with mode 000; only + // Seatbelt's EPERM is accepted as confinement evidence. + chmodSync(fakeProbe, 0o000); + expect(managedOriginDataRootProbeAccess(realpathSync(sibling), 'session-a')) + .toBe('missing_or_unsafe'); }); }); diff --git a/test/overview-card.test.ts b/test/overview-card.test.ts index 04bcb0bc7..7934aeeba 100644 --- a/test/overview-card.test.ts +++ b/test/overview-card.test.ts @@ -130,6 +130,12 @@ describe('buildOverviewCard', () => { ])).toEqual({ active: 1, idle: 1, closed: 1 }); }); + it('does not count a stalled turn as idle in the overview', () => { + expect(countSessions([ + sessionRow({ sessionId: 's1', status: 'stalled' }), + ])).toEqual({ active: 1, idle: 0, closed: 0 }); + }); + it('zh overview localizes all module sections and folder buttons', () => { const json = buildOverviewCard( { sessions: [], schedules: [], settings: makeSettings() }, diff --git a/test/pending-repo-journal.test.ts b/test/pending-repo-journal.test.ts new file mode 100644 index 000000000..22516495b --- /dev/null +++ b/test/pending-repo-journal.test.ts @@ -0,0 +1,163 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest'; +import type { DaemonSession } from '../src/core/types.js'; +import type { Session } from '../src/types.js'; + +vi.mock('../src/services/session-store.js', () => ({ + updateSession: vi.fn(), +})); + +import { + persistPendingRepoCardMessageId, + restorePendingRepoRuntime, + stagePendingRepoSetup, +} from '../src/core/pending-repo-journal.js'; +import * as sessionStore from '../src/services/session-store.js'; + +function makeSession(): Session { + return { + sessionId: 'pending-repo-session', + chatId: 'oc_chat', + rootMessageId: 'om_root', + title: 'durable setup', + status: 'active', + createdAt: new Date('2026-01-01T00:00:00Z').toISOString(), + }; +} + +function makeDs(): DaemonSession { + return { + session: makeSession(), + worker: null, + workerPort: null, + workerToken: null, + larkAppId: 'app_test', + chatId: 'oc_chat', + chatType: 'group', + scope: 'thread', + spawnedAt: 1, + cliVersion: 'test', + lastMessageAt: 1, + hasHistory: false, + initialStartPending: true, + pendingPrompt: 'OPENING_N', + pendingRawInput: '/goal exact raw', + pendingCodexAppText: 'visible opening', + pendingCodexAppApplicationContext: 'app', + pendingCodexAppMessageContext: 'message', + pendingAttachments: [{ type: 'file', path: '/tmp/spec.md', name: 'spec.md' }], + pendingMentions: [{ key: '@_user_1', name: '晓雪', openId: 'ou_owner' }], + pendingSubstituteTrigger: { + target: { name: 'Reviewer', openId: 'ou_reviewer' }, + observedMention: { name: 'Reviewer' }, + disclosure: 'prefix', + }, + pendingSender: { openId: 'ou_owner', type: 'user', name: '晓雪' }, + } as DaemonSession; +} + +beforeEach(() => { + vi.mocked(sessionStore.updateSession).mockReset(); +}); + +describe('pending repository setup journal', () => { + it('atomically captures the complete opening before a picker/worktree can be published', () => { + const ds = makeDs(); + + stagePendingRepoSetup(ds, { + mode: 'auto_worktree', baseDir: '/repos/base', turnId: 'turn-n', + }); + + expect(ds.session.queued).toBe(true); + expect(ds.session.queuedPrompt).toBe('OPENING_N'); + expect(ds.session.queuedCodexAppText).toBe('visible opening'); + expect(ds.session.queuedCodexAppMessageContext).toBe('message'); + expect(ds.session.pendingRepoSetup).toEqual({ + mode: 'auto_worktree', + prompt: 'OPENING_N', + rawInput: '/goal exact raw', + turnId: 'turn-n', + baseDir: '/repos/base', + codexAppText: 'visible opening', + codexAppApplicationContext: 'app', + codexAppMessageContext: 'message', + attachments: [{ type: 'file', path: '/tmp/spec.md', name: 'spec.md' }], + mentions: [{ key: '@_user_1', name: '晓雪', openId: 'ou_owner' }], + substituteTrigger: { + target: { name: 'Reviewer', openId: 'ou_reviewer' }, + observedMention: { name: 'Reviewer' }, + disclosure: 'prefix', + }, + sender: { openId: 'ou_owner', type: 'user', name: '晓雪' }, + }); + expect(sessionStore.updateSession).toHaveBeenCalledWith(ds.session); + }); + + it('rolls every durable field back when setup persistence fails', () => { + const ds = makeDs(); + const oldSetup = { mode: 'picker' as const, prompt: 'OLD' }; + Object.assign(ds.session, { + queued: false, + queuedPrompt: 'old prompt', + queuedCodexAppText: 'old text', + queuedCodexAppMessageContext: 'old context', + pendingRepoSetup: oldSetup, + }); + vi.mocked(sessionStore.updateSession).mockImplementationOnce(() => { + throw new Error('disk unavailable'); + }); + + expect(() => stagePendingRepoSetup(ds, { mode: 'picker' })).toThrow('disk unavailable'); + expect(ds.session).toMatchObject({ + queued: false, + queuedPrompt: 'old prompt', + queuedCodexAppText: 'old text', + queuedCodexAppMessageContext: 'old context', + pendingRepoSetup: oldSetup, + }); + }); + + it('persists card identity transactionally and reconstructs isolated runtime buffers', () => { + const ds = makeDs(); + stagePendingRepoSetup(ds, { mode: 'picker', turnId: 'turn-n' }); + persistPendingRepoCardMessageId(ds, 'om_picker'); + + const restored = { + ...makeDs(), + session: structuredClone(ds.session), + pendingPrompt: undefined, + pendingRawInput: undefined, + pendingAttachments: undefined, + pendingMentions: undefined, + pendingSubstituteTrigger: undefined, + pendingSender: undefined, + initialStartPending: true, + } as DaemonSession; + expect(restorePendingRepoRuntime(restored)).toBe(true); + expect(restored).toMatchObject({ + pendingRepo: true, + pendingPrompt: 'OPENING_N', + pendingRawInput: '/goal exact raw', + pendingCodexAppText: 'visible opening', + pendingCodexAppApplicationContext: 'app', + pendingCodexAppMessageContext: 'message', + repoCardMessageId: 'om_picker', + initialStartPending: false, + }); + expect(restored.pendingAttachments).toEqual(ds.pendingAttachments); + expect(restored.pendingAttachments).not.toBe(ds.session.pendingRepoSetup?.attachments); + expect(restored.pendingMentions).toEqual(ds.pendingMentions); + expect(restored.pendingMentions).not.toBe(ds.session.pendingRepoSetup?.mentions); + }); + + it('restores the previous card id if its persistence fails', () => { + const ds = makeDs(); + stagePendingRepoSetup(ds, { mode: 'picker' }); + ds.session.pendingRepoSetup!.repoCardMessageId = 'om_old'; + vi.mocked(sessionStore.updateSession).mockImplementationOnce(() => { + throw new Error('card id write failed'); + }); + + expect(() => persistPendingRepoCardMessageId(ds, 'om_new')).toThrow('card id write failed'); + expect(ds.session.pendingRepoSetup?.repoCardMessageId).toBe('om_old'); + }); +}); diff --git a/test/persistent-backend-type.test.ts b/test/persistent-backend-type.test.ts index e078abac9..8eddf1587 100644 --- a/test/persistent-backend-type.test.ts +++ b/test/persistent-backend-type.test.ts @@ -93,4 +93,8 @@ describe('shutdownBackendDisposition (shutdown freeze-once)', () => { bot.backendType = 'herdr'; expect(shutdownBackendDisposition(ds({ sessionBackend: 'pty' }))).toBe('close'); }); + it('routes a frozen Riff worker through drain + durable lineage ACK, never direct SIGTERM', () => { + bot.backendType = 'pty'; + expect(shutdownBackendDisposition(ds({ sessionBackend: 'riff' }))).toBe('riff-drain-detach'); + }); }); diff --git a/test/plugin-service-restart-lifecycle.test.ts b/test/plugin-service-restart-lifecycle.test.ts index 9adc922e0..83dff707a 100644 --- a/test/plugin-service-restart-lifecycle.test.ts +++ b/test/plugin-service-restart-lifecycle.test.ts @@ -16,7 +16,7 @@ describe('plugin service restart lifecycle', () => { it('preserves auto services by default and always ensures them after core starts', () => { const source = restartFunctionSource(); const stop = 'if (includePluginServices) await stopPluginServicesForCli(undefined, { autoOnly: true });'; - const coreStart = "runPm2(['start', cfg]);"; + const coreStart = "runPm2(['start', cfg], true, PM2_HOME, timeoutMs);"; const ensure = 'await reconcilePluginServicesForCli(undefined, { autoOnly: true });'; expect(source).toContain(stop); diff --git a/test/pm2-descriptor-guard.test.ts b/test/pm2-descriptor-guard.test.ts new file mode 100644 index 000000000..8ca9b35f3 --- /dev/null +++ b/test/pm2-descriptor-guard.test.ts @@ -0,0 +1,94 @@ +import { describe, expect, it, vi } from 'vitest'; +import { + assertNoUnregisteredLiveDaemonDescriptorsIn, + type Pm2DescriptorGuardRuntime, +} from '../src/cli/pm2-descriptor-guard.js'; + +function runtime( + raw: string, + options: { alive?: boolean; mtime?: number; startIdentity?: string } = {}, +): Pm2DescriptorGuardRuntime { + return { + now: () => 100_000, + exists: () => true, + readdir: () => ['app.json'], + read: () => raw, + mtime: () => options.mtime ?? 100_000, + isAlive: () => options.alive ?? true, + readStartIdentity: () => options.startIdentity ?? 'birth-77', + }; +} + +describe('PM2/live-daemon descriptor reconciliation', () => { + it('blocks a mutation when jlist is empty and a fresh descriptor is semantic garbage', () => { + const mutate = vi.fn(); + expect(() => { + assertNoUnregisteredLiveDaemonDescriptorsIn('start', [], '/registry', runtime('{}')); + mutate(); + }).toThrow(/fresh daemon descriptor.*invalid pid\/app id\/heartbeat/); + expect(mutate).not.toHaveBeenCalled(); + }); + + it('blocks a live descriptor PID absent from the PM2 projection', () => { + const raw = JSON.stringify({ larkAppId: 'app', pid: 77, lastHeartbeat: 100_000 }); + expect(() => assertNoUnregisteredLiveDaemonDescriptorsIn( + 'restart', [], '/registry', runtime(raw), + )).toThrow(/app:77/); + }); + + it('accepts a valid fresh descriptor only when its live PID is registered', () => { + const raw = JSON.stringify({ larkAppId: 'app', pid: 77, lastHeartbeat: 100_000 }); + expect(() => assertNoUnregisteredLiveDaemonDescriptorsIn( + 'stop', [{ pid: 77 }], '/registry', runtime(raw), + )).not.toThrow(); + }); + + it('ignores stale semantic garbage', () => { + expect(() => assertNoUnregisteredLiveDaemonDescriptorsIn( + 'start', [], '/registry', runtime('{}', { mtime: 0 }), + )).not.toThrow(); + }); + + it('blocks a stale descriptor whose live PID still has the exact recorded birth', () => { + const raw = JSON.stringify({ + larkAppId: 'app', + pid: 77, + processStartIdentity: 'birth-77', + lastHeartbeat: 0, + }); + expect(() => assertNoUnregisteredLiveDaemonDescriptorsIn( + 'start', [], '/registry', runtime(raw, { mtime: 0 }), + )).toThrow(/daemon descriptor PID\(s\).*live but absent.*app:77/); + }); + + it('ignores a stale descriptor after proving its recorded PID is dead', () => { + const raw = JSON.stringify({ + larkAppId: 'app', + pid: 77, + processStartIdentity: 'birth-77', + lastHeartbeat: 0, + }); + expect(() => assertNoUnregisteredLiveDaemonDescriptorsIn( + 'start', [], '/registry', runtime(raw, { alive: false, mtime: 0 }), + )).not.toThrow(); + }); + + it('ignores a stale descriptor after proving the PID belongs to a different birth', () => { + const raw = JSON.stringify({ + larkAppId: 'app', + pid: 77, + processStartIdentity: 'birth-77', + lastHeartbeat: 0, + }); + expect(() => assertNoUnregisteredLiveDaemonDescriptorsIn( + 'start', [], '/registry', runtime(raw, { mtime: 0, startIdentity: 'birth-successor' }), + )).not.toThrow(); + }); + + it('fails closed when an old-format stale descriptor still names a live PID', () => { + const raw = JSON.stringify({ larkAppId: 'app', pid: 77, lastHeartbeat: 0 }); + expect(() => assertNoUnregisteredLiveDaemonDescriptorsIn( + 'start', [], '/registry', runtime(raw, { mtime: 0 }), + )).toThrow(/stale daemon descriptor.*live PID 77.*no process-start identity/); + }); +}); diff --git a/test/pm2-exact-start.test.ts b/test/pm2-exact-start.test.ts new file mode 100644 index 000000000..7cf322505 --- /dev/null +++ b/test/pm2-exact-start.test.ts @@ -0,0 +1,92 @@ +import { describe, expect, it, vi } from 'vitest'; +import { + isNonMutatingPm2StartRefusal, + startExactPm2ProcessIds, + type Pm2ExactStartClient, +} from '../src/cli/pm2-exact-start.js'; + +function fakeClient(overrides: Partial = {}): Pm2ExactStartClient { + return { + launchRPC: callback => callback(), + executeRemote: (_method, _id, callback) => callback(), + close: callback => callback(), + ...overrides, + }; +} + +describe('exact PM2 start-if-stopped compensation', () => { + it('uses one RPC connection and dispatches every exact id concurrently', async () => { + const callbacks: Array<(error?: unknown) => void> = []; + const launchRPC = vi.fn((callback: (error?: unknown) => void) => callback()); + const executeRemote = vi.fn(( + _method: 'startProcessId', + _id: number, + callback: (error?: unknown) => void, + ) => { callbacks.push(callback); }); + const close = vi.fn((callback: (error?: unknown) => void) => callback()); + + const pending = startExactPm2ProcessIds( + [1, 2, 3], + fakeClient({ launchRPC, executeRemote, close }), + ); + await Promise.resolve(); + + expect(launchRPC).toHaveBeenCalledOnce(); + expect(executeRemote.mock.calls.map(call => [call[0], call[1]])).toEqual([ + ['startProcessId', 1], + ['startProcessId', 2], + ['startProcessId', 3], + ]); + expect(close).not.toHaveBeenCalled(); + + callbacks.forEach(callback => callback()); + await pending; + expect(close).toHaveBeenCalledOnce(); + }); + + it.each([ + 'process already online', + 'process already started', + 'Process with pid 123 already exists', + '7 id unknown', + ])('treats an atomic non-mutating refusal as a benign no-op: %s', async message => { + expect(isNonMutatingPm2StartRefusal(new Error(message))).toBe(true); + const close = vi.fn((callback: (error?: unknown) => void) => callback()); + await expect(startExactPm2ProcessIds([7], fakeClient({ + executeRemote: (_method, _id, callback) => callback(new Error(message)), + close, + }))).resolves.toBeUndefined(); + expect(close).toHaveBeenCalledOnce(); + }); + + it('aggregates hard start failures and still closes the RPC connection', async () => { + const close = vi.fn((callback: (error?: unknown) => void) => callback()); + await expect(startExactPm2ProcessIds([4, 5], fakeClient({ + executeRemote: (_method, id, callback) => callback( + id === 4 ? new Error('executeApp exploded') : new Error('socket lost'), + ), + close, + }))).rejects.toThrow(/pm_id 4: executeApp exploded; pm_id 5: socket lost/); + expect(close).toHaveBeenCalledOnce(); + }); + + it.each([ + { ids: [1, 1] }, + { ids: [-1] }, + { ids: [1.5] }, + ])('rejects invalid exact id sets before connecting: $ids', async ({ ids }) => { + const launchRPC = vi.fn(); + await expect(startExactPm2ProcessIds(ids, fakeClient({ launchRPC }))) + .rejects.toThrow(/unique non-negative pm_id/); + expect(launchRPC).not.toHaveBeenCalled(); + }); + + it('does not call close when the one RPC connection cannot be opened', async () => { + const close = vi.fn(); + await expect(startExactPm2ProcessIds([1], fakeClient({ + launchRPC: callback => callback(new Error('PM2 unavailable')), + close, + }))).rejects.toThrow(/PM2 unavailable/); + expect(close).not.toHaveBeenCalled(); + }); +}); diff --git a/test/pm2-god-admission.test.ts b/test/pm2-god-admission.test.ts new file mode 100644 index 000000000..2b252ffc0 --- /dev/null +++ b/test/pm2-god-admission.test.ts @@ -0,0 +1,22 @@ +import { describe, expect, it, vi } from 'vitest'; +import { assertIncludePm2RestartAdmission } from '../src/cli/pm2-god-admission.js'; + +describe('restart --include-pm2 admission', () => { + it('admits only an initially zero-God state', () => { + expect(() => assertIncludePm2RestartAdmission([])).not.toThrow(); + }); + + it('rejects a live God before any caller mutation', () => { + const mutate = vi.fn(); + expect(() => { + assertIncludePm2RestartAdmission([101]); + mutate(); + }).toThrow(/cannot be signalled with generation-bound authority.*does not signal or restart.*no process or breadcrumb was changed/); + expect(mutate).not.toHaveBeenCalled(); + }); + + it('rejects duplicate Gods without selecting either generation', () => { + expect(() => assertIncludePm2RestartAdmission([101, 202])) + .toThrow(/multiple PM2 God daemons.*no process or breadcrumb was changed/); + }); +}); diff --git a/test/pm2-jlist.test.ts b/test/pm2-jlist.test.ts new file mode 100644 index 000000000..ba5b19675 --- /dev/null +++ b/test/pm2-jlist.test.ts @@ -0,0 +1,69 @@ +import { describe, expect, it } from 'vitest'; +import { + parseCanonicalPm2Id, + parsePm2JlistOutput, + parsePm2JlistOutputStrict, + parsePm2Integer, +} from '../src/cli/pm2-jlist.js'; + +describe('PM2 jlist projection parsing', () => { + it.each(['{}', 'null', '"not-a-list"'])('rejects non-array shutdown authority: %s', output => { + expect(() => parsePm2JlistOutputStrict(output)).toThrow(/non-array JSON/); + }); + + it('rejects malformed shutdown authority instead of treating it as an empty fleet', () => { + expect(() => parsePm2JlistOutputStrict('[PM2] unavailable')).toThrow(/malformed output/); + }); + + it('accepts an array after PM2 informational output', () => { + const row = { name: 'botmux', pm_id: 0, pid: 42, pm2_env: { status: 'online' } }; + expect(parsePm2JlistOutputStrict(`[PM2] daemon ready\n${JSON.stringify([row])}`)) + .toEqual([row]); + }); + + it('keeps read-only callers backward-compatible with empty fallback', () => { + expect(parsePm2JlistOutput('{}')).toEqual([]); + }); + + it('never coerces absent PM2 identity or exit-code fields to zero', () => { + expect(parsePm2Integer(null)).toBeUndefined(); + expect(parsePm2Integer(undefined)).toBeUndefined(); + expect(parsePm2Integer('')).toBeUndefined(); + expect(parsePm2Integer('0')).toBe(0); + expect(parsePm2Integer(-1, { nonNegative: true })).toBeUndefined(); + }); + + it('never revives a missing canonical pm_id from nested PM2 environment state', () => { + expect(parseCanonicalPm2Id({ pm_id: null, pm2_env: { pm_id: 7 } })).toBeUndefined(); + expect(parseCanonicalPm2Id({ pm2_env: { pm_id: 7 } })).toBeUndefined(); + expect(parseCanonicalPm2Id({ pm_id: 8, pm2_env: { pm_id: 7 } })).toBe(8); + }); + + it.each([ + ['non-object row', '[null]', /row 0 is not an object/], + ['missing name', '[{"pm_id":0,"pid":1,"pm2_env":{"status":"online"}}]', /non-empty name/], + ['missing canonical id', '[{"name":"botmux","pid":1,"pm2_env":{"status":"online"}}]', /canonical non-negative pm_id/], + ['invalid pid', '[{"name":"botmux","pm_id":0,"pid":null,"pm2_env":{"status":"online"}}]', /non-negative pid/], + ['missing status', '[{"name":"botmux","pm_id":0,"pid":1,"pm2_env":{}}]', /pm2_env.status/], + ])('rejects a syntactically valid but semantically unsafe %s', (_label, output, pattern) => { + expect(() => parsePm2JlistOutputStrict(output)).toThrow(pattern as RegExp); + }); + + it('rejects duplicate canonical pm_id values even when the names differ', () => { + const output = JSON.stringify([ + { name: 'botmux-a', pm_id: 4, pid: 41, pm2_env: { status: 'online' } }, + { name: 'botmux-b', pm_id: 4, pid: 42, pm2_env: { status: 'online' } }, + ]); + expect(() => parsePm2JlistOutputStrict(output)) + .toThrow(/duplicate canonical pm_id 4 across botmux-a and botmux-b/); + }); + + it('rejects duplicate positive PIDs even across daemon and dashboard rows', () => { + const output = JSON.stringify([ + { name: 'botmux-a', pm_id: 4, pid: 41, pm2_env: { status: 'online' } }, + { name: 'botmux-dashboard', pm_id: 5, pid: 41, pm2_env: { status: 'online' } }, + ]); + expect(() => parsePm2JlistOutputStrict(output)) + .toThrow(/duplicate positive pid 41 across botmux-a and botmux-dashboard/); + }); +}); diff --git a/test/pm2-preflight.test.ts b/test/pm2-preflight.test.ts new file mode 100644 index 000000000..674f77139 --- /dev/null +++ b/test/pm2-preflight.test.ts @@ -0,0 +1,27 @@ +import { describe, expect, it, vi } from 'vitest'; +import { assertLinuxPm2GodExecutableUsable } from '../src/cli/pm2-preflight.js'; + +describe('PM2 deleted-Node preflight', () => { + it('fails closed when /proc proves the God executable was deleted', () => { + const exists = vi.fn(() => true); + expect(() => assertLinuxPm2GodExecutableUsable(42, { + readlink: () => '/old/node (deleted)', + exists, + })).toThrow(/拒绝自动清理/); + expect(exists).not.toHaveBeenCalled(); + }); + + it('fails closed when the successfully resolved executable no longer exists', () => { + expect(() => assertLinuxPm2GodExecutableUsable(42, { + readlink: () => '/old/node', + exists: () => false, + })).toThrow(/Node 二进制已失效/); + }); + + it('only skips a genuine /proc inspection failure', () => { + expect(() => assertLinuxPm2GodExecutableUsable(42, { + readlink: () => { throw new Error('permission denied'); }, + exists: () => false, + })).not.toThrow(); + }); +}); diff --git a/test/pm2-shutdown-capability.test.ts b/test/pm2-shutdown-capability.test.ts new file mode 100644 index 000000000..2e6385a60 --- /dev/null +++ b/test/pm2-shutdown-capability.test.ts @@ -0,0 +1,99 @@ +import { describe, expect, it, vi } from 'vitest'; +import { SUPERVISOR_SHUTDOWN_PROTOCOL } from '../src/core/supervisor-shutdown-protocol.js'; +import { + assertPm2DaemonShutdownCapabilitiesIn, + type Pm2ShutdownCapabilityRuntime, +} from '../src/cli/pm2-shutdown-capability.js'; + +function descriptor( + pid: number, + protocol: string | null = SUPERVISOR_SHUTDOWN_PROTOCOL, +): string { + return JSON.stringify({ + larkAppId: `app-${pid}`, + ipcPort: 7_900 + pid, + bootInstanceId: `boot-${pid}`, + pid, + processStartIdentity: `birth-${pid}`, + lastHeartbeat: 100_000, + ...(protocol ? { supervisorShutdownProtocol: protocol } : {}), + }); +} + +function runtime( + files: Record, + alive = new Set([101, 202]), +): Pm2ShutdownCapabilityRuntime { + return { + now: () => 100_000, + exists: () => true, + readdir: () => Object.keys(files), + read: path => files[path.split('/').pop()!]!, + mtime: () => 100_000, + isAlive: pid => alive.has(pid), + readStartIdentity: pid => alive.has(pid) ? `birth-${pid}` : undefined, + }; +} + +describe('in-memory daemon shutdown capability rollout fence', () => { + it('accepts one fresh exact-protocol descriptor for every live daemon target', () => { + expect(() => assertPm2DaemonShutdownCapabilitiesIn( + 'restart', + [{ name: 'botmux-a', pid: 101 }, { name: 'botmux-b', pid: 202 }], + '/registry', + runtime({ 'a.json': descriptor(101), 'b.json': descriptor(202) }), + )).not.toThrow(); + }); + + it('blocks an old in-memory daemon lacking the new capability before signal', () => { + const signal = vi.fn(); + expect(() => { + assertPm2DaemonShutdownCapabilitiesIn( + 'restart', + [{ name: 'botmux-a', pid: 101 }], + '/registry', + runtime({ 'a.json': descriptor(101, null) }), + ); + signal(101); + }).toThrow(/does not attest.*first-upgrade boundary.*Session\/Riff workload is idle.*must not be reported as applied/); + expect(signal).not.toHaveBeenCalled(); + }); + + it('fails closed when a live target has no fresh matching descriptor', () => { + expect(() => assertPm2DaemonShutdownCapabilitiesIn( + 'stop', + [{ name: 'botmux-a', pid: 101 }], + '/registry', + runtime({ 'other.json': descriptor(202) }), + )).toThrow(/botmux-a\/101 has no matching fresh daemon descriptor/); + }); + + it('rejects duplicate fresh descriptors for one live PID', () => { + expect(() => assertPm2DaemonShutdownCapabilitiesIn( + 'restart', + [{ name: 'botmux-a', pid: 101 }], + '/registry', + runtime({ 'a.json': descriptor(101), 'copy.json': descriptor(101) }), + )).toThrow(/multiple fresh descriptors claim live PID 101/); + }); + + it('rejects a stale capability when the PID now belongs to a new birth', () => { + const reused = runtime({ 'a.json': descriptor(101) }); + reused.readStartIdentity = () => 'birth-successor'; + expect(() => assertPm2DaemonShutdownCapabilitiesIn( + 'restart-immediately-before-signal', + [{ name: 'botmux-a', pid: 101 }], + '/registry', + reused, + )).toThrow(/process-start identity does not match its descriptor/); + }); + + it('does not require capability from a generation proven already dead', () => { + expect(() => assertPm2DaemonShutdownCapabilitiesIn( + 'restart', + [{ name: 'botmux-a', pid: 101 }], + '/registry', + runtime({}, new Set()), + )).not.toThrow(); + }); +}); diff --git a/test/pm2-start-transaction.test.ts b/test/pm2-start-transaction.test.ts new file mode 100644 index 000000000..14cec2080 --- /dev/null +++ b/test/pm2-start-transaction.test.ts @@ -0,0 +1,278 @@ +import { describe, expect, it, vi } from 'vitest'; +import type { FleetProcessEntry } from '../src/cli/fleet-shutdown.js'; +import { + assertDaemonPm2GracefulExitPolicy, + assertConfiguredPm2FleetOnline, + assertConfiguredPm2FleetReady, + assertExactAttestedDaemonSet, + classifyStartBotFleetAdmission, + normalizeRawPm2StopExitCodes, + reconcileLatePm2StartPublication, + runBoundedPm2StartTransaction, +} from '../src/cli/pm2-start-transaction.js'; +import { DAEMON_GRACEFUL_EXIT_CODE } from '../src/core/supervisor-shutdown-protocol.js'; + +function row( + name: string, + pmId: number, + options: { pid?: number; online?: boolean } = {}, +): FleetProcessEntry { + return { + name, + pmId, + pid: options.pid ?? pmId + 100, + online: options.online ?? true, + status: options.online === false ? 'stopped' : 'online', + }; +} + +const configured = ['botmux-a', 'botmux-b', 'botmux-dashboard']; +const alive = (pid: number) => pid > 0; + +describe('configured PM2 fleet start authority', () => { + it('requires the new daemon PM2 policy instead of trusting capability alone', () => { + expect(() => assertDaemonPm2GracefulExitPolicy('start-idempotent-ready', [{ + ...row('botmux-a', 0), + autorestart: true, + stopExitCodes: [DAEMON_GRACEFUL_EXIT_CODE], + }])).not.toThrow(); + expect(() => assertDaemonPm2GracefulExitPolicy('start-idempotent-ready', [{ + ...row('botmux-a', 0), + autorestart: true, + stopExitCodes: [0], + }])).toThrow( + /does not prove signal-death autorestart.*botmux restart --bootstrap-shutdown-protocol --yes/, + ); + expect(() => assertDaemonPm2GracefulExitPolicy('start-idempotent-ready', [{ + ...row('botmux-a', 0), + autorestart: false, + stopExitCodes: [DAEMON_GRACEFUL_EXIT_CODE], + }])).toThrow(/does not prove signal-death autorestart/); + }); + + it('preserves raw PM2 stop-exit-code elements for exact policy validation', () => { + expect(normalizeRawPm2StopExitCodes([42, '0foo'])).toEqual([42, '0foo']); + expect(normalizeRawPm2StopExitCodes([42, '0x0'])).toEqual([42, '0x0']); + expect(normalizeRawPm2StopExitCodes([42, null])).toEqual([42, null]); + expect(normalizeRawPm2StopExitCodes('42')).toEqual(['42']); + expect(normalizeRawPm2StopExitCodes(null)).toEqual([null]); + }); + + it.each([ + [42, '0foo'], + [42, '0x0'], + [42, null], + ])('rejects restart-suppressing raw stop-exit-code extras: %j', stopExitCodes => { + expect(() => assertDaemonPm2GracefulExitPolicy('start-idempotent-ready', [{ + ...row('botmux-a', 0), + autorestart: true, + stopExitCodes, + }])).toThrow(/does not prove signal-death autorestart.*stop_exit_codes=\[42\]/); + }); + + it('accepts only the numeric sentinel or its canonical decimal string', () => { + for (const stopExitCodes of [[42], ['42']]) { + expect(() => assertDaemonPm2GracefulExitPolicy('start-idempotent-ready', [{ + ...row('botmux-a', 0), + autorestart: true, + stopExitCodes, + }])).not.toThrow(); + } + for (const stopExitCodes of [['042'], ['42foo'], ['0x2a']]) { + expect(() => assertDaemonPm2GracefulExitPolicy('start-idempotent-ready', [{ + ...row('botmux-a', 0), + autorestart: true, + stopExitCodes, + }])).toThrow(/does not prove signal-death autorestart/); + } + }); + + it('accepts only one exact online/live row per configured process', () => { + expect(() => assertConfiguredPm2FleetOnline( + 'start', + [row('botmux-a', 0), row('botmux-b', 1), row('botmux-dashboard', 2)], + configured, + alive, + )).not.toThrow(); + }); + + it('rejects a dead, missing, unexpected, or duplicate-id row', () => { + expect(() => assertConfiguredPm2FleetOnline( + 'start', + [row('botmux-a', 0), row('botmux-b', 1, { pid: 0 }), row('botmux-dashboard', 2)], + configured, + alive, + )).toThrow(/botmux-b/); + expect(() => assertConfiguredPm2FleetOnline( + 'start', + [row('botmux-a', 0), row('botmux-dashboard', 2)], + configured, + alive, + )).toThrow(/botmux-b/); + expect(() => assertConfiguredPm2FleetOnline( + 'start', + [row('botmux-a', 0), row('botmux-b', 1), row('botmux-old', 2)], + configured, + alive, + )).toThrow(/unexpected PM2 core row/); + expect(() => assertConfiguredPm2FleetOnline( + 'start', + [row('botmux-a', 0), row('botmux-b', 0), row('botmux-dashboard', 2)], + configured, + alive, + )).toThrow(/duplicate canonical pm_id 0 across botmux-a and botmux-b/); + expect(() => assertConfiguredPm2FleetOnline( + 'start', + [row('botmux-a', 0, { pid: 777 }), row('botmux-b', 1), + row('botmux-dashboard', 2, { pid: 777 })], + configured, + alive, + )).toThrow(/duplicate positive pid 777 across botmux-a and botmux-dashboard/); + }); + + it.each([ + 'start-idempotent-ready', + 'start-bot-already-online-ready', + ])('does not accept %s PM2-online rows while an old daemon capability is missing', (operation) => { + expect(() => assertConfiguredPm2FleetReady( + operation, + [row('botmux-a', 0), row('botmux-b', 1), row('botmux-dashboard', 2)], + configured, + alive, + () => { throw new Error('botmux-b has no handler-ready shutdown capability'); }, + )).toThrow(/botmux-b has no handler-ready shutdown capability/); + }); + + it('rejects a capability scan that omits a daemon which exited mid-read', () => { + expect(() => assertExactAttestedDaemonSet( + 'restart-after-launch', + [row('botmux-a', 0, { pid: 101 }), row('botmux-b', 1, { pid: 202 })], + [101], + () => true, + )).toThrow(/handler-ready capability set is incomplete.*expected pids: 101, 202.*attested: 101/); + }); +}); + +describe('start-bot exact append-one admission', () => { + it('admits exactly one missing requested bot when all configured peers are live', () => { + expect(classifyStartBotFleetAdmission( + 'start-bot', + [row('botmux-a', 0), row('botmux-dashboard', 2)], + configured, + 'botmux-b', + alive, + )).toEqual({ state: 'start-eligible' }); + }); + + it('is idempotent only for the exact fully-online configured fleet', () => { + expect(classifyStartBotFleetAdmission( + 'start-bot', + [row('botmux-a', 0), row('botmux-b', 1), row('botmux-dashboard', 2)], + configured, + 'botmux-b', + alive, + )).toEqual({ state: 'already-online' }); + }); + + it('rejects dashboard-only and another-missing-bot partial fleets', () => { + expect(() => classifyStartBotFleetAdmission( + 'start-bot', + [row('botmux-dashboard', 2)], + configured, + 'botmux-b', + alive, + )).toThrow(/configured peer.*botmux-a/); + + expect(() => classifyStartBotFleetAdmission( + 'start-bot', + [row('botmux-a', 0), row('botmux-dashboard', 3)], + ['botmux-a', 'botmux-b', 'botmux-c', 'botmux-dashboard'], + 'botmux-c', + alive, + )).toThrow(/configured peer.*botmux-b/); + }); + + it('rejects an existing transitional target instead of routing it through start', () => { + expect(() => classifyStartBotFleetAdmission( + 'start-bot', + [row('botmux-a', 0), row('botmux-b', 1, { online: false }), row('botmux-dashboard', 2)], + configured, + 'botmux-b', + alive, + )).toThrow(/existing non-live\/transitional/); + }); + + it('distinguishes a truly empty fleet from unsafe partial fleets', () => { + expect(classifyStartBotFleetAdmission( + 'start-bot', [], configured, 'botmux-b', alive, + )).toEqual({ state: 'fleet-down' }); + }); +}); + +describe('bounded PM2 start transaction', () => { + it('passes exact budgets and returns only a fresh verified projection', () => { + const order: string[] = []; + const projection = [{ name: 'ready' }]; + const result = runBoundedPm2StartTransaction('start', 30_000, 10_000, { + start: timeout => { order.push(`start:${timeout}`); }, + verifyFresh: timeout => { order.push(`verify:${timeout}`); return projection; }, + rollback: () => { order.push('rollback'); }, + }); + expect(result).toBe(projection); + expect(order).toEqual(['start:30000', 'verify:10000']); + }); + + it('accepts a complete fresh fleet even if the launcher itself timed out', () => { + const rollback = vi.fn(); + expect(runBoundedPm2StartTransaction('start', 30, 10, { + start: () => { throw new Error('ETIMEDOUT'); }, + verifyFresh: () => 'complete-fresh-fleet', + rollback, + })).toBe('complete-fresh-fleet'); + expect(rollback).not.toHaveBeenCalled(); + }); + + it('rolls back a partial launch before exposing verification failure', () => { + const order: string[] = []; + expect(() => runBoundedPm2StartTransaction('restart-start', 30, 10, { + start: () => { order.push('start-partial'); throw new Error('socket closed'); }, + verifyFresh: () => { order.push('verify-fresh'); throw new Error('botmux-b unavailable'); }, + rollback: () => { order.push('rollback-partial'); }, + })).toThrow(/start: socket closed.*verify: botmux-b unavailable.*partial launch was rolled back/); + expect(order).toEqual(['start-partial', 'verify-fresh', 'rollback-partial']); + }); + + it('reports rollback failure without hiding the original start/verify evidence', () => { + expect(() => runBoundedPm2StartTransaction('start-bot', 30, 10, { + start: () => { throw new Error('launch failed'); }, + verifyFresh: () => { throw new Error('target transitional'); }, + rollback: () => { throw new Error('descriptor ambiguous'); }, + })).toThrow(/launch failed.*target transitional.*rollback failed: descriptor ambiguous/); + }); + + it('does not accept an empty first rollback read when a candidate publishes later', () => { + let now = 0; + const observations = [true, false, true, true]; + const reconcileOnce = vi.fn(() => observations.shift() ?? true); + + reconcileLatePm2StartPublication('start-bot', 10, 500, { + now: () => now, + sleep: ms => { now += ms; }, + reconcileOnce, + }); + + // First `true` is the empty projection. The late false observation resets + // the settle clock; only two later restored observations complete it. + expect(reconcileOnce).toHaveBeenCalledTimes(4); + expect(now).toBeGreaterThanOrEqual(120); + }); + + it('returns explicit uncertainty when late publication never settles', () => { + let now = 0; + expect(() => reconcileLatePm2StartPublication('restart-start', 10, 30, { + now: () => now, + sleep: ms => { now += ms; }, + reconcileOnce: () => false, + })).toThrow(/rollback remains uncertain.*late-publication settle window/); + }); +}); diff --git a/test/raw-input-followup-atomicity.test.ts b/test/raw-input-followup-atomicity.test.ts index 7ae5f7722..ebf3c68b1 100644 --- a/test/raw-input-followup-atomicity.test.ts +++ b/test/raw-input-followup-atomicity.test.ts @@ -2,14 +2,17 @@ * Source-level guard for the raw_input + follow-up ATOMIC delivery contract * (PR #157 review blocker, round 2). * - * Why source-level: worker.ts is a process script with no exports, so its - * IPC handler can't be unit-tested directly. The race it guards against: + * The executable queue/composer ordering contract lives in + * test/async-serial-queue.test.ts via runAdoptRawInputSequence. This file keeps + * only the worker/daemon wiring assertions because worker.ts is a process + * script with no exports. The race it guards against: * `process.on('message', async ...)` handlers do NOT serialize — the * raw_input branch awaits 200ms between sendText and Enter, and a separate * `message` IPC handled in that window writes into the PTY first (type-ahead * adapters flush immediately), interleaving the follow-up into the slash * command. The fix makes the follow-up ride on the raw_input IPC itself and - * the worker enqueue it strictly after the Enter. + * the worker write it strictly after the Enter while retaining the same adopt + * queue until the complete adapter lifecycle settles. * * Daemon-side single-IPC behavior is covered in * test/worker-ready-display-mode.test.ts; this file pins the worker-side @@ -18,10 +21,15 @@ * Run: pnpm vitest run test/raw-input-followup-atomicity.test.ts */ import { readFileSync } from 'node:fs'; -import { describe, expect, it } from 'vitest'; +import { describe, expect, it, vi } from 'vitest'; +import { + finalizeRawCommandDelivery, + writeRawCommandLine, +} from '../src/core/raw-command-writer.js'; const workerSrc = readFileSync(new URL('../src/worker.ts', import.meta.url), 'utf-8'); const poolSrc = readFileSync(new URL('../src/core/worker-pool.ts', import.meta.url), 'utf-8'); +const rawWriterSrc = readFileSync(new URL('../src/core/raw-command-writer.ts', import.meta.url), 'utf-8'); function caseRegion(src: string, marker: string, span = 3000): string { const start = src.indexOf(marker); @@ -33,13 +41,16 @@ describe('worker raw_input handler', () => { const region = caseRegion(workerSrc, "case 'raw_input':"); it('queues through an owned restart until the replacement prompt, while preserving normal busy delivery', () => { - const gateIdx = region.indexOf( - 'if (cliRestartInProgress || rawInputRestartGate || sessionRenameInFlight', + const retirementIdx = region.indexOf( + 'if (shutdownDetachRequestId || closeRequestInFlightId || preparedCloseRequestId)', ); + // Gate is multi-line after PR #441 injection fence merge. + const gateIdx = region.indexOf('if (cliRestartInProgress || rawInputRestartGate || sessionRenameInFlight()'); const queueIdx = region.indexOf('pendingRawInputs.push(msg)'); const deliverIdx = region.indexOf('await deliverRawInput(msg)'); - expect(gateIdx).toBeGreaterThanOrEqual(0); + expect(retirementIdx).toBeGreaterThanOrEqual(0); + expect(gateIdx).toBeGreaterThan(retirementIdx); expect(queueIdx).toBeGreaterThan(gateIdx); expect(deliverIdx).toBeGreaterThan(queueIdx); // isPromptReady is false while an active CLI is busy, so gating on it would @@ -60,10 +71,72 @@ describe('worker raw_input handler', () => { expect(gate).toContain('injectionFlushing'); expect(gate).toContain('shouldDeferUserFlush(pendingInjections)'); }); + +}); + +describe('worker adopt/native-rename coordination', () => { + const messageRegion = caseRegion(workerSrc, "case 'message':", 6500); + const flushRegion = caseRegion(workerSrc, 'async function flushPending()', 16000); + + it('parks ordinary adopt messages for the full native-rename settle window', () => { + expect(messageRegion).toContain('pendingAdoptMessages.push(item)'); + expect(messageRegion).toContain('turnId: msg.turnId'); + expect(messageRegion).toContain('dispatchAttempt: msg.dispatchAttempt'); + expect(messageRegion).toContain('cliRestartInProgress || rawInputRestartGate || !backend || sessionRenameInFlight()'); + expect(messageRegion).toContain('await runAdoptMessageForCapturedGeneration(item, () =>'); + expect(flushRegion).toContain('const adoptInputReady = isPromptReady'); + expect(flushRegion).toContain('if (adoptInputReady && pendingAdoptMessages.length > 0)'); + }); + + it('serializes adopt rename and rechecks readiness after older composer writes', () => { + expect(flushRegion).toContain('await runAdoptSessionRenameSequence({'); + expect(flushRegion).toContain('queue: adoptWriteQueue'); + expect(flushRegion).toContain('cliSpawnGeneration === renameGeneration'); + expect(flushRegion).toContain('!rawInputRestartGate'); + expect(flushRegion).toContain('if (!sent)'); + expect(flushRegion).toContain('pendingSessionRename = title'); + expect(flushRegion).toContain('if (!rawInputReady && !supportedSessionRenameReady && !adoptInputReady)'); + expect(flushRegion).toContain("sessionRenamePhase = 'reserved'"); + const beginIdx = flushRegion.indexOf('beginCliWriteCycle();', flushRegion.indexOf('const writeRename')); + const writingIdx = flushRegion.indexOf("sessionRenamePhase = 'writing'", beginIdx); + const commandIdx = flushRegion.indexOf('await sendRawCommandLineSerially(renameBackend', writingIdx); + const sentIdx = flushRegion.indexOf("sessionRenamePhase = 'sent'", commandIdx); + expect(beginIdx).toBeGreaterThanOrEqual(0); + expect(writingIdx).toBeGreaterThan(beginIdx); + expect(commandIdx).toBeGreaterThan(writingIdx); + expect(sentIdx).toBeGreaterThan(commandIdx); + expect(workerSrc).toContain("if (sessionRenamePhase === 'sent') forceClearSessionRenameInFlight()"); + }); + + it('keeps upstream drain priority: raw input, latest rename, then adopt message', () => { + const rawIdx = flushRegion.indexOf('if (rawInputReady && pendingRawInputs.length > 0'); + const renameIdx = flushRegion.indexOf('if (supportedSessionRenameReady && pendingSessionRename !== null'); + const adoptIdx = flushRegion.indexOf('const item = pendingAdoptMessages.shift()!'); + expect(rawIdx).toBeGreaterThanOrEqual(0); + expect(renameIdx).toBeGreaterThan(rawIdx); + expect(adoptIdx).toBeGreaterThan(renameIdx); + }); + + it('fences process-lifetime adopt tasks before transcript mark or replacement-backend write', () => { + const writeRegion = caseRegion(workerSrc, 'async function writeAdoptMessage', 6200); + const runnerRegion = caseRegion(workerSrc, 'async function runAdoptMessageForCapturedGeneration', 1800); + const fenceIdx = writeRegion.indexOf('if (!executionFence || !adoptWriteFenceIsCurrent(executionFence))'); + const rendererIdx = writeRegion.indexOf('renderer?.markNewTurn()'); + const markIdx = writeRegion.indexOf('codexBridgeMarkPendingTurn('); + expect(fenceIdx).toBeGreaterThanOrEqual(0); + expect(rendererIdx).toBeGreaterThan(fenceIdx); + expect(markIdx).toBeGreaterThan(fenceIdx); + expect(writeRegion).toContain("return settleStaleAfterWrite('adopt_generation_changed')"); + expect(writeRegion).toContain("return settleStaleAfterWrite('adopt_generation_changed_before_enter')"); + expect(runnerRegion).toContain('runAdoptQueuedWriteSequence({'); + expect(runnerRegion).toContain('isCurrent: () => adoptWriteFenceIsCurrent(fence)'); + expect(runnerRegion).toContain('onStale: requeueOnce'); + expect(workerSrc).toContain('&& !rawInputRestartGate;'); + }); }); describe('worker raw_input delivery', () => { - const region = caseRegion(workerSrc, 'async function deliverRawInput', 2600); + const region = caseRegion(workerSrc, 'async function deliverRawInput', 7000); it('enqueues followUpContent strictly AFTER the awaited command send (incl. Enter)', () => { const sendIdx = region.indexOf('await sendRawCommandLineSerially(targetBackend, msg.content)'); @@ -73,8 +146,15 @@ describe('worker raw_input delivery', () => { expect(followIdx).toBeGreaterThan(sendIdx); }); - it('routes the follow-up through sendToPty (normal busy-queue semantics)', () => { - expect(region).toContain('sendToPty(msg.followUpContent, msg.followUpTurnId, {'); + it('keeps the exact follow-up identity on sendToPty only in the non-adopt path', () => { + const adoptIdx = region.indexOf('await runAdoptRawInputSequence({'); + const nonAdoptIdx = region.indexOf('const targetBackend = backend;', adoptIdx); + const sendToPtyIdx = region.indexOf( + 'sendToPty(msg.followUpContent!, msg.followUpTurnId, {', + ); + expect(adoptIdx).toBeGreaterThanOrEqual(0); + expect(nonAdoptIdx).toBeGreaterThan(adoptIdx); + expect(sendToPtyIdx).toBeGreaterThan(nonAdoptIdx); expect(region).toContain('codexAppInput: msg.followUpCodexAppInput'); }); @@ -89,12 +169,49 @@ describe('worker raw_input delivery', () => { expect(sendIdx).toBeGreaterThan(capabilityIdx); }); + it('awaits the full adopt follow-up adapter lifecycle in the raw queue transaction', () => { + expect(region).toContain('const writeRawInput = async ('); + expect(region).toContain('targetBackend: SessionBackend,'); + expect(region).toContain('await runAdoptRawInputSequence({'); + expect(region).toContain('queue: adoptWriteQueue'); + expect(region).toContain('isCurrent: () => adoptWriteFenceIsCurrent(fence)'); + expect(region).toContain('onStaleBeforeWrite: () =>'); + expect(region).toContain('onStaleBeforeFollowUp: () =>'); + const staleFollowUp = region.slice( + region.indexOf('onStaleBeforeFollowUp: () =>'), + region.indexOf('writeRawInput:', region.indexOf('onStaleBeforeFollowUp: () =>')), + ); + expect(staleFollowUp).toContain('follow-up was withheld'); + expect(staleFollowUp).not.toContain('pendingAdoptMessages.push'); + expect(region).toContain('const result = await writeAdoptMessage('); + const postRawWriteFollowUp = region.slice( + region.indexOf("if (result === 'stale-before-write')"), + region.indexOf("} else if (result === 'completed')"), + ); + expect(postRawWriteFollowUp).toContain('follow-up was withheld'); + expect(postRawWriteFollowUp).not.toContain('pendingAdoptMessages.push'); + expect(region).toContain('fence,'); + }); + it('holds ordinary prompt flushes only for the text-to-Enter critical window', () => { const flush = caseRegion(workerSrc, 'async function flushPending()', 9000); + expect(flush).toContain( + 'if (shutdownDetachRequestId || closeRequestInFlightId || preparedCloseRequestId) return', + ); expect(flush).toContain('if (commandLineWritesPending > 0) return'); expect(region).not.toContain('if (!isPromptReady)'); expect(region).not.toContain('if (isPromptReady)'); }); + + it('ACKs a durable generic raw opening only after the awaited Enter succeeds', () => { + const sentIdx = region.indexOf('sent = await sendRawCommandLineSerially'); + const ackIdx = region.indexOf("type: 'queued_activation_submitted'", sentIdx); + expect(sentIdx).toBeGreaterThanOrEqual(0); + expect(ackIdx).toBeGreaterThan(sentIdx); + expect(region.slice(sentIdx, ackIdx)).toContain( + "acknowledgeActivation: !!msg.queuedActivationToken && effectiveBackendType !== 'riff'", + ); + }); }); describe('worker command-line write mutex', () => { @@ -110,21 +227,21 @@ describe('worker command-line write mutex', () => { }); describe('worker sendRawCommandLine helper', () => { - const helper = caseRegion(workerSrc, 'async function sendRawCommandLine', 2200); + const helper = caseRegion(rawWriterSrc, 'export async function writeRawCommandLine', 2200); it('generic CLIs: literal text → 200ms beat → Enter in order (slash-picker safe)', () => { const textIdx = helper.indexOf('sendText(content)'); expect(textIdx).toBeGreaterThanOrEqual(0); // Anchor the beat/Enter lookups AFTER the text write so the CoCo branch's own // 200ms beat (which precedes the generic path) can't be mistaken for this one. - const beatIdx = helper.indexOf('setTimeout(r, 200)', textIdx); + const beatIdx = helper.indexOf('delay(beatMs)', textIdx); const enterIdx = helper.indexOf("sendSpecialKeys('Enter')", beatIdx); expect(beatIdx).toBeGreaterThan(textIdx); expect(enterIdx).toBeGreaterThan(beatIdx); }); it('CoCo: types char-by-char (throttled) before a single Enter (paste-coalescing safe)', () => { - const cocoIdx = helper.indexOf("cliId === 'coco'"); + const cocoIdx = helper.indexOf('opts.coco'); expect(cocoIdx, 'CoCo branch present').toBeGreaterThanOrEqual(0); const genericTextIdx = helper.indexOf('sendText(content)'); // The CoCo branch fully precedes the generic one-shot path. @@ -132,23 +249,79 @@ describe('worker sendRawCommandLine helper', () => { // Per-char keystrokes spaced by the throttle — a one-shot write coalesces into // a paste on CoCo, which skips command mode + the slash picker. const charIdx = helper.indexOf('sendText(ch)', cocoIdx); - const throttleIdx = helper.indexOf('COCO_SLASH_TYPE_THROTTLE_MS', cocoIdx); + const throttleIdx = helper.indexOf('opts.cocoThrottleMs', cocoIdx); expect(charIdx).toBeGreaterThan(cocoIdx); expect(charIdx).toBeLessThan(genericTextIdx); expect(throttleIdx).toBeGreaterThan(cocoIdx); // Exactly one Enter, after the beat (a stray 2nd Enter would confirm a /model // selector pick); the branch returns immediately after. const cocoEnterIdx = helper.indexOf("sendSpecialKeys('Enter')", throttleIdx); - const returnIdx = helper.indexOf('return;', throttleIdx); + const returnIdx = helper.indexOf("return sendSpecialKeys('Enter') !== false", throttleIdx); expect(cocoEnterIdx).toBeGreaterThan(throttleIdx); expect(cocoEnterIdx).toBeLessThan(genericTextIdx); - expect(returnIdx).toBeGreaterThan(cocoEnterIdx); + expect(returnIdx).toBeGreaterThan(throttleIdx); + expect(cocoEnterIdx).toBeGreaterThan(returnIdx); expect(returnIdx).toBeLessThan(genericTextIdx); }); }); +describe('raw command backend acceptance', () => { + const immediateDelay = vi.fn(async () => {}); + + it('fails closed when the text write is rejected', async () => { + const sendText = vi.fn(() => false); + const sendSpecialKeys = vi.fn(() => true); + await expect(writeRawCommandLine({ + write: vi.fn(), sendText, sendSpecialKeys, + }, '/goal x', { delay: immediateDelay })).resolves.toBe(false); + expect(sendSpecialKeys).not.toHaveBeenCalled(); + }); + + it('fails closed when Enter is rejected after accepted text', async () => { + const sendText = vi.fn(() => true); + const sendSpecialKeys = vi.fn(() => false); + await expect(writeRawCommandLine({ + write: vi.fn(), sendText, sendSpecialKeys, + }, '/goal x', { delay: immediateDelay })).resolves.toBe(false); + expect(sendSpecialKeys).toHaveBeenCalledWith('Enter'); + }); + + it('fails closed when a PTY-style backend disappears before either write', async () => { + const write = vi.fn(() => false); + await expect(writeRawCommandLine({ write }, '/goal x', { + delay: immediateDelay, + })).resolves.toBe(false); + expect(write).toHaveBeenCalledTimes(1); + }); + + it('does not ACK or enqueue a follower after rejected Enter and retires the durable generation', async () => { + const accepted = await writeRawCommandLine({ + write: vi.fn(), + sendText: vi.fn(() => true), + sendSpecialKeys: vi.fn(() => false), + }, '/goal x', { delay: immediateDelay }); + const onActivationAck = vi.fn(); + const onFollowUp = vi.fn(); + const onDurableFailure = vi.fn(); + + expect(finalizeRawCommandDelivery({ + accepted, + durableActivation: true, + acknowledgeActivation: true, + hasFollowUp: true, + onAccepted: vi.fn(), + onFollowUp, + onActivationAck, + onDurableFailure, + })).toBe(false); + expect(onActivationAck).not.toHaveBeenCalled(); + expect(onFollowUp).not.toHaveBeenCalled(); + expect(onDurableFailure).toHaveBeenCalledOnce(); + }); +}); + describe('daemon prompt_ready dispatch', () => { - const region = caseRegion(poolSrc, "case 'prompt_ready':", 2000); + const region = caseRegion(poolSrc, "case 'prompt_ready':", 5000); it('bundles the follow-up onto the raw_input IPC instead of a second message IPC', () => { expect(region).toContain('followUpContent: followUp?.cliInput'); diff --git a/test/read-isolation.test.ts b/test/read-isolation.test.ts index 57ac6801a..087c2b6ab 100644 --- a/test/read-isolation.test.ts +++ b/test/read-isolation.test.ts @@ -8,6 +8,7 @@ import { deviceCredentialIsolationMarkerPath, isCredentialIsolationReservedBasename, buildCredentialIsolationRules, + isolatedPaneOriginChannel, isolatedPaneReattachSafe, botHomePath, buildV2DenyPaths, @@ -21,10 +22,24 @@ import { assertSafeAppId, normalizeIsolationPath, isolationPaneMarkerContent, + isolationPanePolicyDigest, type V2IsolationContext, type WriteSandboxContext, } from '../src/adapters/cli/read-isolation.js'; -import { managedOriginCapabilityPath } from '../src/core/managed-origin-capability.js'; +import { + managedOriginAttestationDirectory, + managedOriginCapabilityPath, + managedOriginIsolationSentinelPath, + managedOriginRootLocatorPath, +} from '../src/core/managed-origin-capability.js'; + +const G1 = '11'.repeat(32); +const G2 = '22'.repeat(32); +const POLICY1 = isolationPanePolicyDigest({ + readIsolation: true, + writeSandbox: false, + readDenyExtraPaths: ['/private/a'], +}); const ws = (o: Partial = {}): WriteSandboxContext => ({ homeDir: '/Users/bot', @@ -41,6 +56,7 @@ const v2 = (o: Partial = {}): V2IsolationContext => ({ sessionDataDir: '/Users/bot/.botmux/data', currentAppId: 'cli_self', currentSessionId: 'session-self', + currentOriginChannelId: G1, ...o, }); @@ -482,7 +498,9 @@ describe('v2 HYBRID model (buildV2DenyPaths)', () => { '/Users/bot/.lark-cli-bots/cli_self', '/Users/bot/.botmux/data/sessions-cli_self.json', '/Users/bot/.botmux/data/attachments/cli_self', - managedOriginCapabilityPath('/Users/bot/.botmux/data', 'session-self'), + managedOriginCapabilityPath('/Users/bot/.botmux/data', 'session-self', G1), + managedOriginAttestationDirectory('/Users/bot/.botmux/data', 'session-self', G1), + managedOriginRootLocatorPath('/Users/bot', 'session-self'), ]); // traverse shim on each wholesale-denied parent (stat/realpath, not listing) expect(carve.traverseDirs).toEqual([ @@ -517,7 +535,7 @@ describe('v2 HYBRID model (buildV2DenyPaths)', () => { expect(() => assertSafeAppId('...')).toThrow(); expect(() => buildV2CarveOuts(v2({ currentAppId: '..' }))).toThrow(); expect(buildV2CarveOuts(v2({ currentSessionId: '../other' })).allowPaths).toContain( - managedOriginCapabilityPath('/Users/bot/.botmux/data', '../other'), + managedOriginCapabilityPath('/Users/bot/.botmux/data', '../other', G1), ); // botHomePath must also reject an unsafe id (used for own + other BOT_HOMEs) expect(() => botHomePath('/Users/bot/.botmux', '../x')).toThrow(); @@ -564,18 +582,32 @@ describe('buildSeatbeltProfile (verified format)', () => { .toBeGreaterThan(prof.indexOf('(allow file-read* (subpath "/Users/bot/.botmux/bots/cli_self"))')); }); - it('carves only the current session capability and keeps its parent immutable', () => { + it('carves only the current session capability/proof dir and keeps it write-denied', () => { const ownCapability = managedOriginCapabilityPath( '/Users/bot/.botmux/data', 'session-self', + G1, ); const siblingCapability = managedOriginCapabilityPath( '/Users/bot/.botmux/data', 'session-sibling', + G2, + ); + const ownProofDir = managedOriginAttestationDirectory( + '/Users/bot/.botmux/data', + 'session-self', + G1, + ); + const siblingProofDir = managedOriginAttestationDirectory( + '/Users/bot/.botmux/data', + 'session-sibling', + G2, ); const carve = buildV2CarveOuts(v2()); expect(carve.allowPaths).toContain(ownCapability); expect(carve.allowPaths).not.toContain(siblingCapability); + expect(carve.allowPaths).toContain(ownProofDir); + expect(carve.allowPaths).not.toContain(siblingProofDir); const parent = '/Users/bot/.botmux/data/read-isolation'; const protectedWrites = buildReadIsolationProtectedWriteRules(v2()); @@ -593,6 +625,8 @@ describe('buildSeatbeltProfile (verified format)', () => { expect(prof).toContain(readAllow); expect(prof).toContain(writeDeny); expect(prof.indexOf(writeDeny)).toBeGreaterThan(prof.indexOf(readAllow)); + expect(prof).not.toContain(`(allow file-write* (subpath "${ownProofDir}"))`); + expect(prof).toContain(`(deny file-read* (subpath "${managedOriginIsolationSentinelPath('/Users/bot')}"))`); expect(prof).toContain( '(deny file-write* (subpath "/Users/bot/.botmux/.dashboard-secret"))', ); @@ -788,7 +822,11 @@ describe('Linux read isolation (buildLinuxReadIsolationMasks)', () => { // per-sibling), so exclude just those two from the comparison. const macDeny = buildV2DenyPaths(v2()); const linux = buildLinuxReadIsolationMasks({ ctx: v2(), siblingAppIds: [] }).hidePaths; - const wholesalePerBot = new Set(['/Users/bot/.botmux/bots', '/Users/bot/.lark-cli-bots']); + const wholesalePerBot = new Set([ + '/Users/bot/.botmux/bots', + '/Users/bot/.lark-cli-bots', + managedOriginIsolationSentinelPath('/Users/bot'), + ]); const shouldMatch = macDeny.filter(p => !wholesalePerBot.has(p)); for (const p of shouldMatch) expect(linux).toContain(p); }); @@ -857,6 +895,108 @@ describe('isolatedPaneReattachSafe', () => { expect(isolatedPaneReattachSafe('')).toBe(false); expect(isolatedPaneReattachSafe(' ')).toBe(false); }); + + it('binds Darwin warm reattach to the pane channel and exact read/write policy', () => { + const marker = isolationPaneMarkerContent( + 'boot-new', + ['credential', 'read'], + { + originChannelId: G1, + readIsolation: true, + writeSandbox: false, + policyDigest: POLICY1, + }, + ); + expect(isolatedPaneOriginChannel(marker)).toBe(G1); + expect(isolatedPaneReattachSafe(marker, { + requiredCapabilities: ['credential', 'read'], + readIsolation: true, writeSandbox: false, requireOriginChannel: true, + policyDigest: POLICY1, + })).toBe(true); + expect(isolatedPaneReattachSafe(marker, { + requiredCapabilities: ['credential', 'read'], + readIsolation: false, writeSandbox: false, requireOriginChannel: true, + policyDigest: POLICY1, + })).toBe(false); + const broadTmpDigest = isolationPanePolicyDigest({ + readIsolation: true, + writeSandbox: true, + writeAllowExtraPaths: ['/custom/broad-tmp'], + }); + const narrowTmpDigest = isolationPanePolicyDigest({ + readIsolation: true, + writeSandbox: true, + writeAllowExtraPaths: ['/private/var/folders/narrow'], + }); + const broadTmpMarker = isolationPaneMarkerContent('boot-old', ['credential', 'read', 'write'], { + originChannelId: G1, + readIsolation: true, + writeSandbox: true, + policyDigest: broadTmpDigest, + }); + expect(isolatedPaneReattachSafe(broadTmpMarker, { + requiredCapabilities: ['credential', 'read', 'write'], + readIsolation: true, + writeSandbox: true, + requireOriginChannel: true, + policyDigest: narrowTmpDigest, + })).toBe(false); + expect(isolatedPaneReattachSafe(marker, { + requiredCapabilities: ['credential', 'read', 'write'], + readIsolation: true, writeSandbox: true, requireOriginChannel: true, + policyDigest: POLICY1, + })).toBe(false); + expect(isolatedPaneReattachSafe(marker, { + requiredCapabilities: ['credential', 'read'], + readIsolation: true, + writeSandbox: false, + requireOriginChannel: true, + policyDigest: isolationPanePolicyDigest({ + readIsolation: true, + writeSandbox: false, + readDenyExtraPaths: ['/private/b'], + }), + })).toBe(false); + expect(isolatedPaneReattachSafe(JSON.stringify({ + version: 4, + bootId: 'legacy-v4', + readIsolation: true, + writeSandbox: false, + originChannelId: G1, + }), { + requiredCapabilities: ['credential', 'read'], + readIsolation: true, + writeSandbox: false, + requireOriginChannel: true, + policyDigest: POLICY1, + })).toBe(false); + expect(isolatedPaneReattachSafe(isolationPaneMarkerContent( + 'linux-v7', ['credential', 'read'], + ), { + requiredCapabilities: ['credential', 'read'], + requireOriginChannel: false, + })).toBe(true); + expect(isolatedPaneReattachSafe(isolationPaneMarkerContent( + 'credential-only-v7', ['credential'], + ), { + requiredCapabilities: ['credential'], + exactCapabilities: true, + requireOriginChannel: false, + })).toBe(true); + expect(isolatedPaneReattachSafe(isolationPaneMarkerContent( + 'old-broader-policy-v7', ['credential', 'read', 'write'], + ), { + requiredCapabilities: ['credential'], + exactCapabilities: true, + requireOriginChannel: false, + })).toBe(false); + expect(isolatedPaneReattachSafe(isolationPaneMarkerContent( + 'linux-v7', ['credential', 'read'], + ), { + requiredCapabilities: ['credential', 'read'], + requireOriginChannel: true, + })).toBe(false); + }); }); describe('worker capability carve-out ordering', () => { @@ -865,12 +1005,22 @@ describe('worker capability carve-out ordering', () => { it('replaces a planted capability symlink before canonicalizing allow paths', () => { const publishAt = source.indexOf('publishSandboxRelayCapability({ failClosed: true })'); const canonicalizeAt = source.indexOf( - 'carve.allowPaths.map(path => path === capabilityCarvePath ? path : canonical(path))', + '...carve.allowPaths.map(path =>', ); expect(publishAt).toBeGreaterThanOrEqual(0); expect(canonicalizeAt).toBeGreaterThan(publishAt); expect(source).toContain('replaceManagedOriginCapabilityFile(profilePath, buildSeatbeltProfile('); - expect(source).toContain('marker = readRegularHostFileNoFollow('); + expect(source).toContain('marker = readManagedOriginAuthorityFile('); + }); + + it('never lets old-generation teardown unlink the stable mac capability path', () => { + const unlinkStart = source.indexOf('function unlinkManagedOriginCapabilityFiles()'); + const unlinkEnd = source.indexOf('/** Read a host-owned isolation marker', unlinkStart); + const unlink = source.slice(unlinkStart, unlinkEnd); + expect(unlink).toContain('sandboxRelayOutbox'); + expect(unlink).not.toContain('readIsolationOriginCapabilityFile ??'); + expect(source).toContain('ensureManagedOriginIsolationSentinel(osUserHomeDir)'); + expect(source).toContain('osUserHomeDir: canonical('); }); it('denies every same-UID Gateway socket before allowing only the current session socket', () => { @@ -910,7 +1060,8 @@ describe('worker capability carve-out ordering', () => { expect(credentialWrapperAt).toBeLessThan(spawnAt); expect(source).toContain('if (!willReattachPersistent && credentialOnlyBwrap)'); expect(source).toContain('isCredentialIsolationReservedBasename(name)'); - expect(source).toContain('isolatedPaneReattachSafe(marker, appliedIsolationCapabilities)'); + expect(source).toContain('requiredCapabilities: appliedIsolationCapabilities'); + expect(source).toContain('exactCapabilities: true'); }); }); @@ -918,9 +1069,15 @@ describe('CLI protected capability wiring', () => { const cliSource = readFileSync(new URL('../src/cli.ts', import.meta.url), 'utf8'); const vcSource = readFileSync(new URL('../src/cli/vc-agent.ts', import.meta.url), 'utf8'); - it('passes the session id when resolving protected turn snapshots', () => { + it('requires host-file attestation when the fixed sentinel is kernel-denied', () => { + expect(cliSource).toContain( + 'let liveMarkerCtx = findLiveAncestorSessionContext(sendDataDir);', + ); + expect(cliSource).toContain( + 'managedOriginIsolationSentinelAccess(osUserHomeDir)', + ); expect(cliSource).toContain( - 'const liveMarkerCtx = resolveSessionContext(\n resolveDataDir(),\n process.env.BOTMUX_SESSION_ID,', + 'if (!relayDir && isolatedSendRequired && !isolatedCapabilityCtx)', ); expect(cliSource).toContain( 'const liveOrigin = resolveSessionContext(resolveDataDir(), sessionId);', diff --git a/test/relay-picker.test.ts b/test/relay-picker.test.ts index 5aba8d7a9..1baac3ef1 100644 --- a/test/relay-picker.test.ts +++ b/test/relay-picker.test.ts @@ -43,6 +43,7 @@ function makeDs(over: { ownerOpenId?: string; adoptedFrom?: any; cliId?: string | undefined; + backendType?: 'riff' | 'tmux'; worker?: any; scope?: 'thread' | 'chat'; rootMessageId?: string; @@ -63,6 +64,7 @@ function makeDs(over: { ownerOpenId: over.ownerOpenId ?? OWNER, workingDir: '/tmp', cliId: 'cliId' in over ? over.cliId : ('claude-code' as any), + backendType: over.backendType, adoptedFrom: over.adoptedFrom, }, worker: over.worker ?? null, @@ -126,6 +128,26 @@ describe('collectRelayPickerEntries', () => { expect(entries.map(e => e.sessionId)).toEqual(['keep']); }); + it('does not offer Riff sessions whose remote reply route cannot be retargeted', async () => { + const local = makeDs({ sessionId: 'local', chatId: 'oc_local', lastMessageAt: 100 }); + const riff = makeDs({ + sessionId: 'riff', + chatId: 'oc_riff', + lastMessageAt: 9_999, + cliId: 'riff', + backendType: 'riff', + }); + + const entries = await collectRelayPickerEntries( + registryOf(riff, local), + APP, + CURRENT_CHAT, + OWNER, + ); + + expect(entries.map(entry => entry.sessionId)).toEqual(['local']); + }); + it('excludes by ANCHOR, not chatId — same-chat other-topic sessions stay in candidates', async () => { // Two thread-scope sessions in the SAME chat, distinct 话题 roots. The relay // target anchor is om_topicA; only that session is excluded (can't relay diff --git a/test/reply-target-fallback.test.ts b/test/reply-target-fallback.test.ts index 3569ee284..e15238006 100644 --- a/test/reply-target-fallback.test.ts +++ b/test/reply-target-fallback.test.ts @@ -13,7 +13,14 @@ * Run: pnpm vitest run test/reply-target-fallback.test.ts */ import { describe, it, expect } from 'vitest'; -import { beginReplyTargetTurn, fallbackTurnId, isSubstituteTurn, pickTurnReplyTarget, resolveSessionReplyTarget } from '../src/core/reply-target.js'; +import { + beginReplyTargetTurn, + fallbackTurnId, + frozenReplyContextForTurn, + isSubstituteTurn, + pickTurnReplyTarget, + resolveSessionReplyTarget, +} from '../src/core/reply-target.js'; import type { DaemonSession } from '../src/core/types.js'; const NOW = new Date().toISOString(); @@ -196,3 +203,31 @@ describe('per-turn replyTargets — queued/concurrent turns keep their own ancho expect(resolveSessionReplyTarget(ds, 'turn-0')).toEqual({ mode: 'plain', chatId: 'oc_chat' }); }); }); + +describe('frozen reply context', () => { + it('keeps turn A root, quote, and sender after mutable state advances to B', () => { + const ds = makeDs() as DaemonSession; + ds.session.quoteTargetId = 'om_a'; + ds.session.quoteTargetSenderOpenId = 'ou_a'; + ds.session.quoteTargetSenderIsBot = false; + beginReplyTargetTurn(ds, 'om_root_a', 'om_a', NOW); + + ds.session.quoteTargetId = 'om_b'; + ds.session.quoteTargetSenderOpenId = 'ou_b'; + ds.session.quoteTargetSenderIsBot = true; + beginReplyTargetTurn(ds, 'om_root_b', 'om_b', NOW); + + expect(frozenReplyContextForTurn(ds, 'om_a')).toEqual({ + target: { mode: 'thread', rootMessageId: 'om_root_a' }, + quoteTargetId: 'om_a', + replyTargetSenderOpenId: 'ou_a', + replyTargetSenderIsBot: false, + }); + expect(frozenReplyContextForTurn(ds, 'om_b')).toEqual({ + target: { mode: 'thread', rootMessageId: 'om_root_b' }, + quoteTargetId: 'om_b', + replyTargetSenderOpenId: 'ou_b', + replyTargetSenderIsBot: true, + }); + }); +}); diff --git a/test/resource-monitor-runtime.test.ts b/test/resource-monitor-runtime.test.ts index b0f4d321e..3423e31b7 100644 --- a/test/resource-monitor-runtime.test.ts +++ b/test/resource-monitor-runtime.test.ts @@ -53,6 +53,7 @@ describe('runtime monitor helpers', () => { expect(sessionRuntimeBucket(session({ sessionId: 's7', status: 'analyzing' }))).toBe('working'); expect(sessionRuntimeBucket(session({ sessionId: 's8', status: 'active' }))).toBe('working'); expect(sessionRuntimeBucket(session({ sessionId: 's9', status: 'dormant' }))).toBe('idle'); + expect(sessionRuntimeBucket(session({ sessionId: 's10', status: 'stalled' }))).toBe('working'); }); it('builds fresh runtime summary with daemon and session pressure', () => { diff --git a/test/restart-intent-store.test.ts b/test/restart-intent-store.test.ts index f4e5a2150..3a28c60a1 100644 --- a/test/restart-intent-store.test.ts +++ b/test/restart-intent-store.test.ts @@ -11,6 +11,11 @@ import { clearRestartLeaseTo, hasActiveRestartLeaseTo, writeManualIntentIfAbsentTo, + writeRestartAttemptIntentTo, + commitRestartIntentAttemptTo, + claimRestartIntentForReportTo, + hasPreparedRestartIntentTo, + removeRestartIntentAttemptTo, restartIntentPathIn, } from '../src/services/restart-intent-store.js'; @@ -103,4 +108,94 @@ describe('restart-intent store', () => { expect(consumeRestartIntentTo(dir, T0)).toBeNull(); expect(existsSync(restartIntentPathIn(dir))).toBe(false); }); + + it('rolls back only the exact failed start attempt and preserves a newer writer', () => { + writeRestartAttemptIntentTo( + dir, + { kind: 'manual', at: iso(T0) }, + T0, + 'attempt-old', + ); + expect(removeRestartIntentAttemptTo(dir, 'attempt-old')).toBe(true); + expect(existsSync(restartIntentPathIn(dir))).toBe(true); + expect(consumeRestartIntentTo(dir, T0 + 500)).toBeNull(); + + writeRestartAttemptIntentTo( + dir, + { kind: 'manual', at: iso(T0) }, + T0, + 'attempt-old', + ); + writeRestartIntentTo(dir, { + kind: 'update', oldVersion: '1', newVersion: '2', at: iso(T0 + 1_000), + }); + // The newer writer is deferred behind the prepared fence. Aborting the + // partial start keeps it non-consumable until a later verified restart. + expect(consumeRestartIntentTo(dir, T0 + 1_500)).toBeNull(); + expect(removeRestartIntentAttemptTo(dir, 'attempt-old')).toBe(true); + expect(consumeRestartIntentTo(dir, T0 + 2_000)).toBeNull(); + expect(hasPreparedRestartIntentTo(dir, T0 + 2_000)).toBe(false); + + writeRestartAttemptIntentTo( + dir, + { kind: 'manual', at: iso(T0 + 2_000) }, + T0 + 2_000, + 'attempt-new', + ); + expect(commitRestartIntentAttemptTo(dir, 'attempt-new')).toBe(true); + expect(consumeRestartIntentTo(dir, T0 + 3_000)).toMatchObject({ + kind: 'update', oldVersion: '1', newVersion: '2', + }); + }); + + it('does not expose a prepared restart until the exact attempt commits', () => { + writeRestartAttemptIntentTo( + dir, + { kind: 'manual', at: iso(T0) }, + T0, + 'attempt-verified', + ); + expect(hasPreparedRestartIntentTo(dir, T0 + 1_000)).toBe(true); + expect(consumeRestartIntentTo(dir, T0 + 1_000)).toBeNull(); + expect(existsSync(restartIntentPathIn(dir))).toBe(true); + + expect(commitRestartIntentAttemptTo(dir, 'wrong-attempt')).toBe(false); + expect(commitRestartIntentAttemptTo(dir, 'attempt-verified')).toBe(true); + expect(hasPreparedRestartIntentTo(dir, T0 + 2_000)).toBe(false); + expect(consumeRestartIntentTo(dir, T0 + 2_000)).toMatchObject({ + kind: 'manual', + attemptId: 'attempt-verified', + attemptState: 'committed', + }); + }); + + it('atomically claims a commit that lands after a prepared observation', () => { + writeRestartAttemptIntentTo( + dir, + { kind: 'manual', at: iso(T0) }, + T0, + 'attempt-racy-commit', + ); + expect(claimRestartIntentForReportTo(dir, T0 + 1_000)).toEqual({ state: 'prepared' }); + + // This is the old consume(prepared)->commit->hasPrepared(false) gap. The + // next operation now observes+claims committed under one lock. + expect(commitRestartIntentAttemptTo(dir, 'attempt-racy-commit')).toBe(true); + expect(claimRestartIntentForReportTo(dir, T0 + 1_001)).toMatchObject({ + state: 'claimed', + intent: { attemptId: 'attempt-racy-commit', attemptState: 'committed' }, + }); + expect(existsSync(restartIntentPathIn(dir))).toBe(false); + }); + + it('expires an abandoned prepared attempt without ever reporting it', () => { + writeRestartAttemptIntentTo( + dir, + { kind: 'manual', at: iso(T0) }, + T0, + 'attempt-crashed-cli', + ); + expect(consumeRestartIntentTo(dir, T0 + 11 * 60_000)).toBeNull(); + expect(existsSync(restartIntentPathIn(dir))).toBe(false); + }); }); diff --git a/test/restart-report.test.ts b/test/restart-report.test.ts index 34a2ef325..954505262 100644 --- a/test/restart-report.test.ts +++ b/test/restart-report.test.ts @@ -3,8 +3,13 @@ import { existsSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { countActiveSessionsOnDisk } from '../src/services/session-store.js'; -import { buildRestartReportText, sendRestartReportIfPending, fetchChangelog } from '../src/core/restart-report.js'; -import { writeRestartIntentTo, restartIntentPathIn } from '../src/services/restart-intent-store.js'; +import { buildRestartReportText, fetchChangelog, sendRestartReportIfPending } from '../src/core/restart-report.js'; +import { + commitRestartIntentAttemptTo, + restartIntentPathIn, + writeRestartAttemptIntentTo, + writeRestartIntentTo, +} from '../src/services/restart-intent-store.js'; function writeSessions(dir: string, name: string, sessions: Record) { writeFileSync(join(dir, name), JSON.stringify(sessions)); @@ -177,6 +182,30 @@ describe('sendRestartReportIfPending', () => { await sendRestartReportIfPending(w); expect(sent).toHaveLength(1); }); + + it('atomically reclaims a commit that lands after the prepared observation', async () => { + writeRestartAttemptIntentTo( + dir, + { kind: 'manual', at: new Date(T0).toISOString() }, + T0, + 'attempt-full-fleet', + ); + const wait = vi.fn(async () => { + // Commit occurs strictly between two report-claim operations. There is + // no separate hasPrepared read whose false result can lose this commit. + expect(commitRestartIntentAttemptTo(dir, 'attempt-full-fleet')).toBe(true); + }); + const { w, sent } = fakeWiring({ + wait, + preparedCommitWaitMs: 100, + }); + + await sendRestartReportIfPending(w); + + expect(wait).toHaveBeenCalledOnce(); + expect(sent).toHaveLength(1); + expect(existsSync(restartIntentPathIn(dir))).toBe(false); + }); }); describe('fetchChangelog', () => { diff --git a/test/restore-zombie-close.test.ts b/test/restore-zombie-close.test.ts index 1dbd0792f..eeb91c55b 100644 --- a/test/restore-zombie-close.test.ts +++ b/test/restore-zombie-close.test.ts @@ -73,12 +73,46 @@ vi.mock('../src/core/worker-pool.js', () => ({ getActiveSessionsRegistry: vi.fn(() => wp.registry ?? undefined), getCurrentCliVersion: vi.fn(() => '1.0.0-test'), restoreUsageLimitRuntimeState: vi.fn(), + withActiveSessionKeyLock: vi.fn(async (_map: Map, _key: string, action: () => any) => action()), setActiveSessionSafe: vi.fn(async (map: Map, key: string, ds: any) => { const prev = map.get(key); if (prev && prev !== ds) { - for (const [k, v] of map) { if (v === prev) { map.delete(k); break; } } + const prevPending = (prev.session?.codexAppDispatchLedger?.length ?? 0) > 0; + const incomingPending = (ds.session?.codexAppDispatchLedger?.length ?? 0) > 0; + if (prevPending && incomingPending) { + return { + accepted: false, + reason: 'both_pending', + keptSessionId: prev.session.sessionId, + preservedIncomingSessionId: ds.session.sessionId, + }; + } + const closeUnregistered = async (loser: any) => { + for (const [k, v] of map) { + if (v === loser) { map.delete(k); break; } + } + const store = await import('../src/services/session-store.js'); + const persisted = store.getSession(loser.session.sessionId); + if (persisted && persisted.status !== 'closed') { + store.closeSession(loser.session.sessionId); + } + }; + if (prevPending) { + await closeUnregistered(ds); + return { + accepted: false, + reason: 'kept_pending_owner', + keptSessionId: prev.session.sessionId, + closedIncomingSessionId: ds.session.sessionId, + }; + } + await closeUnregistered(prev); } map.set(key, ds); + return { + accepted: true, + ...(prev && prev !== ds ? { closedSessionId: prev.session.sessionId } : {}), + }; }), isRelayableRealSession: (ds: any) => !!ds?.worker || !!ds?.session?.cliId || !!ds?.session?.lastCliInput, @@ -151,7 +185,7 @@ vi.mock('../src/core/session-activity.js', () => ({ markSessionActivity: vi.fn(), })); -import { restoreActiveSessions, closeCliMismatchedSessionsForBot } from '../src/core/session-manager.js'; +import { restoreActiveSessions, closeCliMismatchedSessionsForBot, resumeSession } from '../src/core/session-manager.js'; import { TmuxBackend } from '../src/adapters/backend/tmux-backend.js'; import { forkWorker, closeSession } from '../src/core/worker-pool.js'; import { announceSessionRow } from '../src/core/session-activity.js'; @@ -254,6 +288,34 @@ describe('restoreActiveSessions — persistent-backend zombie-close decision', ( expect(forkWorker).not.toHaveBeenCalled(); }); + it('CLI mismatch on restore preserves and reattaches an unsettled Codex App ledger', async () => { + probe.result = 'exists'; + server.state = 'running'; + const s = makeActivePersistentSession('om_cli_mismatch_pending'); + s.cliId = 'codex-app'; + s.codexAppDispatchLedger = [{ + dispatchId: 'dispatch-pending', + turnId: 'turn-pending', + state: 'prepared', + content: 'prompt', + deliverySink: 'lark', + }]; + sessionStore.updateSession(s); + const map = new Map(); + wp.registry = map; + + await restoreActiveSessions(map); + + expect(closeSession).not.toHaveBeenCalled(); + expect(sessionStore.getSession(s.sessionId)).toMatchObject({ + status: 'active', + codexAppDispatchLedger: [{ dispatchId: 'dispatch-pending' }], + }); + const restored = map.get(sessionKey('om_cli_mismatch_pending', 'app_test')); + expect(restored).toBeDefined(); + expect(forkWorker).toHaveBeenCalledWith(restored, '', true); + }); + it('wrapper mismatch on restore (same cliId) → closes the active record', async () => { // 'aiden x claude' and bare claude-code share cliId='claude-code' but are // distinct launch choices (selectionKeyForBot keys on cliId+wrapperCli). @@ -403,6 +465,27 @@ describe('restoreActiveSessions — persistent-backend zombie-close decision', ( expect(map.get(sessionKey('om_exists', 'app_test'))).toBeDefined(); }); + it('eagerly resumes an unsettled pty session so durable FIFO recovery is not stranded', async () => { + const s = makeActivePersistentSession('om_pty_pending'); + s.backendType = 'pty'; + s.codexAppDispatchLedger = [{ + dispatchId: 'd-pty', + turnId: 't-pty', + state: 'accepted', + content: 'recover me', + }]; + sessionStore.updateSession(s); + const map = new Map(); + wp.registry = map; + + await restoreActiveSessions(map); + + const restored = [...map.values()].find(current => current.session.sessionId === s.sessionId); + expect(restored).toBeDefined(); + expect(forkWorker).toHaveBeenCalledWith(restored, '', true); + expect(closeSession).not.toHaveBeenCalled(); + }); + it('restores only the latest clean Codex App sidecar after a disk reload and re-attaches it', async () => { probe.result = 'exists'; bot.cliId = 'codex-app'; @@ -440,6 +523,85 @@ describe('restoreActiveSessions — persistent-backend zombie-close decision', ( expect(forkWorker).toHaveBeenCalledWith(restored, '', true); expect(sessionStore.getSession(s.sessionId)?.lastCodexAppInput).toEqual(expected); }); + + it('isolates a same-anchor pending collision without aborting startup and preserves both rows', async () => { + probe.result = 'exists'; + bot.cliId = 'codex-app'; + const first = makeActivePersistentSession('om_pending_collision'); + first.codexAppDispatchLedger = [{ + dispatchId: 'dispatch-first', + turnId: 'turn-first', + state: 'prepared', + content: 'first owned output', + deliverySink: 'lark', + }]; + sessionStore.updateSession(first); + const second = makeActivePersistentSession('om_pending_collision'); + second.codexAppDispatchLedger = [{ + dispatchId: 'dispatch-second', + turnId: 'turn-second', + state: 'prepared', + content: 'second owned output', + deliverySink: 'lark', + }]; + sessionStore.updateSession(second); + const map = new Map(); + wp.registry = map; + + await expect(restoreActiveSessions(map)).resolves.toBeUndefined(); + + expect(sessionStore.getSession(first.sessionId)?.status).toBe('active'); + expect(sessionStore.getSession(second.sessionId)?.status).toBe('active'); + expect(closeSession).not.toHaveBeenCalled(); + const canonical = map.get(sessionKey('om_pending_collision', 'app_test'))!; + expect(canonical.session.sessionId).toBe(first.sessionId); + expect(forkWorker).toHaveBeenCalledTimes(1); + expect(forkWorker).toHaveBeenCalledWith(canonical, '', true); + }); +}); + +describe('resumeSession — disk-only legacy anchor collision', () => { + it('refuses a legacy unscoped real owner instead of ghosting it', async () => { + const target = makeActivePersistentSession('om_resume_legacy_conflict'); + sessionStore.closeSession(target.sessionId); + const legacyOwner = makeActivePersistentSession('om_resume_legacy_conflict'); + legacyOwner.larkAppId = undefined; + legacyOwner.lastCliInput = 'existing conversation'; + sessionStore.updateSession(legacyOwner); + const map = new Map(); + wp.registry = map; + + const result = await resumeSession(target.sessionId, map); + + expect(result).toEqual({ + ok: false, + error: 'anchor_occupied', + activeSessionId: legacyOwner.sessionId, + }); + expect(sessionStore.getSession(target.sessionId)?.status).toBe('closed'); + expect(sessionStore.getSession(legacyOwner.sessionId)?.status).toBe('active'); + expect(map.size).toBe(0); + }); + + it('closes a disk-only legacy scratch before reactivating the requested session', async () => { + const target = makeActivePersistentSession('om_resume_legacy_scratch'); + sessionStore.closeSession(target.sessionId); + const legacyScratch = makeActivePersistentSession('om_resume_legacy_scratch'); + legacyScratch.larkAppId = undefined; + legacyScratch.cliId = undefined; + legacyScratch.lastCliInput = undefined; + sessionStore.updateSession(legacyScratch); + const map = new Map(); + wp.registry = map; + + const result = await resumeSession(target.sessionId, map); + + expect(result.ok).toBe(true); + expect(sessionStore.getSession(legacyScratch.sessionId)?.status).toBe('closed'); + expect(sessionStore.getSession(target.sessionId)?.status).toBe('active'); + expect(map.get(sessionKey('om_resume_legacy_scratch', 'app_test'))?.session.sessionId) + .toBe(target.sessionId); + }); }); // ─── Runtime hot-switch sweep (closeCliMismatchedSessionsForBot) ───────────── diff --git a/test/riff-backend.test.ts b/test/riff-backend.test.ts index 26494d483..d1ddec915 100644 --- a/test/riff-backend.test.ts +++ b/test/riff-backend.test.ts @@ -170,6 +170,105 @@ describe('RiffBackend', () => { }); }); + describe('graceful shutdown detach drain', () => { + it('fences new writes, waits for a slow create, returns its exact id, and never cancels or streams it', async () => { + const be = makeBackend({ injectStatusLines: false }); + const ids: Array = []; + be.onTaskId(id => ids.push(id)); + be.spawn('', [], {} as any); + be.write('opening turn'); + await flush(); + + let settled = false; + const prepare = be.prepareShutdownDetach().then(result => { + settled = true; + return result; + }); + await flush(); + expect(settled).toBe(false); + + // This write arrived after the shutdown fence and must not extend the + // drain or create a new remote task. + be.write('must be rejected'); + resolvers.shift()!(taskResponse('task-shutdown-late')); + const result = await prepare; + + expect(result).toEqual({ ok: true, taskId: 'task-shutdown-late' }); + expect(ids).toEqual(['task-shutdown-late']); + expect(calls.filter(c => c.url.includes('/api/task-execute')).length).toBe(1); + expect(calls.filter(c => c.url.includes('/api/task-follow-up')).length).toBe(0); + expect(calls.filter(c => c.url.includes('/api/task-cancel')).length).toBe(0); + expect(calls.filter(c => c.url.includes('/api2/task-stream')).length).toBe(0); + }); + + it('drains two writes accepted before the fence and reports the newest child lineage', async () => { + const be = makeBackend({ injectStatusLines: false }); + const ids: Array = []; + be.onTaskId(id => ids.push(id)); + be.write('first accepted write'); + be.write('second accepted write'); + await flush(); + + const prepare = be.prepareShutdownDetach(); + resolvers.shift()!(taskResponse('task-drain-1')); + await flush(); + const follow = calls.find(c => c.url.includes('/api/task-follow-up')); + expect(follow).toBeDefined(); + expect(JSON.parse(String(follow!.init?.body)).parentTaskId).toBe('task-drain-1'); + resolvers.shift()!(taskResponse('task-drain-2')); + + await expect(prepare).resolves.toEqual({ ok: true, taskId: 'task-drain-2' }); + expect(ids).toEqual(['task-drain-1', 'task-drain-2']); + expect(calls.filter(c => c.url.includes('/api/task-cancel')).length).toBe(0); + expect(calls.filter(c => c.url.includes('/api2/task-stream')).length).toBe(0); + }); + + it('abort restores admission and reconnects the exact drained task', async () => { + const be = makeBackend({ injectStatusLines: false }); + be.write('opening turn'); + await flush(); + const prepare = be.prepareShutdownDetach(); + resolvers.shift()!(taskResponse('task-abort-resume')); + await expect(prepare).resolves.toEqual({ ok: true, taskId: 'task-abort-resume' }); + expect(calls.filter(c => c.url.includes('/api2/task-stream')).length).toBe(0); + + await be.abortShutdownDetach(); + await flush(); + expect(calls.filter(c => c.url.includes('/api2/task-stream?id=task-abort-resume')).length).toBe(1); + + be.write('follow-up after aborted shutdown'); + await flush(); + expect(calls.filter(c => c.url.includes('/api/task-follow-up')).length).toBe(1); + }); + + it('abort invalidates a still-draining prepare and restores admission only after it settles', async () => { + const be = makeBackend({ injectStatusLines: false }); + be.write('accepted before shutdown'); + await flush(); + + const prepare = be.prepareShutdownDetach(); + let abortSettled = false; + const abort = be.abortShutdownDetach().then(result => { + abortSettled = true; + return result; + }); + await flush(); + expect(abortSettled).toBe(false); + expect((be as any).shutdownDetaching).toBe(true); + + resolvers.shift()!(taskResponse('task-abort-during-drain')); + await expect(prepare).resolves.toEqual({ + ok: false, + taskId: 'task-abort-during-drain', + error: 'shutdown_detach_aborted', + }); + await expect(abort).resolves.toEqual({ ok: true, taskId: 'task-abort-during-drain' }); + expect((be as any).shutdownDetachPrepared).toBe(false); + expect((be as any).shutdownDetaching).toBe(false); + expect(calls.filter(c => c.url.includes('/api/task-cancel'))).toHaveLength(0); + }); + }); + describe('SSE clean EOF without done (finding C)', () => { it('treats a clean EOF with no done event as a stream failure — session must not stay busy forever', async () => { const be = makeBackend({ injectStatusLines: false }); @@ -515,7 +614,10 @@ describe('RiffBackend', () => { expect(cancels.length).toBeGreaterThanOrEqual(1); expect(JSON.parse(String(cancels[cancels.length - 1]!.init?.body ?? '{}')).id ?? JSON.parse(String(cancels[0]!.init?.body)).id).toBe('task-late'); expect(calls.filter(c => c.url.includes('/api2/task-stream')).length).toBe(0); - expect((be as any).currentTaskId).not.toBe('task-late'); + // The cancelled child remains the exact lineage anchor until durable + // close commit. An abort can therefore continue from it, not its stale + // parent; it is still never streamed while prepare is fenced. + expect((be as any).currentTaskId).toBe('task-late'); }); it('close during follow-up: the late follow-up task is cancelled', async () => { @@ -536,8 +638,141 @@ describe('RiffBackend', () => { await destroyP; const cancelIds = calls.filter(c => c.url.includes('/api/task-cancel')).map(c => JSON.parse(String(c.init?.body)).id); expect(cancelIds).toContain('task-late-2'); - // late follow-up 不得成为 current，也不得开流 - expect((be as any).currentTaskId).not.toBe('task-late-2'); + // Preserve the cancelled child as retry lineage, but never stream it. + expect((be as any).currentTaskId).toBe('task-late-2'); + expect(calls.filter(c => c.url.includes('/api2/task-stream?id=task-late-2')).length).toBe(0); + }); + }); + + describe('close prepare / abort / retry state contract', () => { + it('restores write admission after cancel failure, then allows a close retry', async () => { + const be = makeBackend({ resumeParentTaskId: 'task-parent', injectStatusLines: false }); + let cancelCalls = 0; + fetchMock.mockImplementation(async (url: string | URL, init?: RequestInit) => { + const u = String(url); + calls.push({ url: u, init }); + if (u.includes('/api/task-cancel')) { + cancelCalls++; + return cancelCalls <= 2 + ? new Response('cancel failed', { status: 500 }) + : Response.json({ success: true, data: {} }); + } + if (u.includes('/api/task-follow-up')) return taskResponse('task-after-failure'); + if (u.includes('/api2/task-stream')) return pendingSseResponse(); + if (u.includes('/api/task-detail')) return Response.json({ success: true, data: { task: {} } }); + return taskResponse('unexpected-create'); + }); + + const failed = await be.destroySession(); + expect(failed).toMatchObject({ ok: false, taskId: 'task-parent' }); + expect((be as any).closing).toBe(false); + + be.write('continue after failed close'); + await flush(); await flush(); + const follow = calls.find(c => c.url.includes('/api/task-follow-up')); + expect(follow).toBeDefined(); + expect(JSON.parse(String(follow!.init?.body)).parentTaskId).toBe('task-parent'); + + const retried = await be.destroySession(); + expect(retried).toEqual({ ok: true, taskId: 'task-after-failure' }); + }); + + it('aborts a successful prepare without losing a late-created child lineage', async () => { + const be = makeBackend({ injectStatusLines: false }); + be.write('opening message'); + await flush(); + const preparedP = be.destroySession(); + resolvers.shift()!(taskResponse('task-late-prepared')); + const prepared = await preparedP; + expect(prepared).toEqual({ ok: true, taskId: 'task-late-prepared' }); + expect((be as any).closing).toBe(true); + + await be.abortDestroySession(); + expect((be as any).closing).toBe(false); + be.write('continue after durable commit failure'); + await flush(); + const follow = calls.find(c => c.url.includes('/api/task-follow-up')); + expect(follow).toBeDefined(); + expect(JSON.parse(String(follow!.init?.body)).parentTaskId).toBe('task-late-prepared'); + }); + + it('does not reopen admission when close timeout wins during an in-flight cancel', async () => { + const be = makeBackend({ resumeParentTaskId: 'task-timeout-parent', injectStatusLines: false }); + (be as any).destroyDeadlineMs = 10; + let resolveCancel!: (response: Response) => void; + fetchMock.mockImplementation(async (url: string | URL, init?: RequestInit) => { + const u = String(url); + calls.push({ url: u, init }); + if (u.includes('/api/task-cancel')) { + return new Promise(resolve => { resolveCancel = resolve; }); + } + if (u.includes('/api/task-follow-up')) return taskResponse('task-after-timeout'); + if (u.includes('/api2/task-stream')) return pendingSseResponse(); + if (u.includes('/api/task-detail')) return Response.json({ success: true, data: { task: {} } }); + return taskResponse('unexpected-create'); + }); + + let settled = false; + const closeP = be.destroySession().then(result => { settled = true; return result; }); + await new Promise(resolve => setTimeout(resolve, 25)); + expect(settled).toBe(false); + be.write('must remain fenced'); + await flush(); + expect(calls.filter(c => c.url.includes('/api/task-follow-up')).length).toBe(0); + + resolveCancel(Response.json({ success: true, data: {} })); + const result = await closeP; + expect(result).toEqual({ ok: false, taskId: 'task-timeout-parent', error: 'close_timeout' }); + expect((be as any).closing).toBe(false); + + be.write('admitted after cancel settled'); + await flush(); + expect(calls.filter(c => c.url.includes('/api/task-follow-up')).length).toBe(1); + }); + + it('cannot publish a prepared close after an abort races a late successful cancel', async () => { + const be = makeBackend({ resumeParentTaskId: 'task-abort-race', injectStatusLines: false }); + let resolveCancel!: (response: Response) => void; + fetchMock.mockImplementation(async (url: string | URL, init?: RequestInit) => { + const u = String(url); + calls.push({ url: u, init }); + if (u.includes('/api/task-cancel')) { + return new Promise(resolve => { resolveCancel = resolve; }); + } + if (u.includes('/api/task-follow-up')) return taskResponse('task-after-abort-race'); + if (u.includes('/api2/task-stream')) return pendingSseResponse(); + if (u.includes('/api/task-detail')) return Response.json({ success: true, data: { task: {} } }); + return taskResponse('unexpected-create'); + }); + + let closeSettled = false; + const close = be.destroySession().then(result => { + closeSettled = true; + return result; + }); + await flush(); + expect(resolveCancel).toBeTypeOf('function'); + + let abortSettled = false; + const abort = be.abortDestroySession().then(() => { abortSettled = true; }); + await flush(); + expect(closeSettled).toBe(false); + expect(abortSettled).toBe(false); + expect((be as any).closing).toBe(true); + + resolveCancel(Response.json({ success: true, data: {} })); + await expect(close).resolves.toEqual({ + ok: false, + taskId: 'task-abort-race', + error: 'close_aborted', + }); + await abort; + expect((be as any).closePrepared).toBe(false); + expect((be as any).closing).toBe(false); + + be.write('admitted after exact abort'); + await flush(); + expect(calls.filter(c => c.url.includes('/api/task-follow-up'))).toHaveLength(1); }); }); diff --git a/test/riff-shutdown-detach.test.ts b/test/riff-shutdown-detach.test.ts new file mode 100644 index 000000000..17ae13d5d --- /dev/null +++ b/test/riff-shutdown-detach.test.ts @@ -0,0 +1,1099 @@ +import { EventEmitter } from 'node:events'; +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { config } from '../src/config.js'; +import type { DaemonSession } from '../src/core/types.js'; +import { + abortRiffShutdownFleet, + abortPreparedRiffShutdown, + canAbortVerifiedExitedRiffPreparation, + collectUniqueDaemonShutdownSessions, + commitPreparedRiffShutdown, + detachRiffWorkerForShutdown, + persistPreparedRiffShutdown, + persistPreparedRiffShutdownFleet, + prepareRiffFleetForShutdown, + prepareRiffSessionForShutdown, + type FencedRiffShutdownParticipant, + type PreparedRiffShutdown, +} from '../src/core/riff-shutdown-detach.js'; +import { sendWorkerInput } from '../src/core/worker-pool.js'; +import * as sessionStore from '../src/services/session-store.js'; + +vi.mock('../src/utils/logger.js', () => ({ + logger: { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn() }, +})); + +type FakeWorker = EventEmitter & { + killed: boolean; + exitCode: number | null; + signalCode: NodeJS.Signals | null; + send: ReturnType; + kill: ReturnType; +}; + +describe('Riff graceful daemon-shutdown detach coordinator', () => { + let dataDir: string; + let previousDataDir: string; + + beforeEach(() => { + dataDir = mkdtempSync(join(tmpdir(), 'botmux-riff-shutdown-')); + previousDataDir = config.session.dataDir; + config.session.dataDir = dataDir; + sessionStore.init('app'); + }); + + afterEach(() => { + vi.restoreAllMocks(); + config.session.dataDir = previousDataDir; + sessionStore.init(); + rmSync(dataDir, { recursive: true, force: true }); + }); + + function fixture( + initialTaskId: string | undefined, + onSend: (worker: FakeWorker, message: any) => void, + ): { ds: DaemonSession; worker: FakeWorker; messages: any[] } { + const session = sessionStore.createSession('oc_riff', 'om_riff', 'riff shutdown', 'group'); + session.larkAppId = 'app'; + session.backendType = 'riff'; + session.riffParentTaskId = initialTaskId; + sessionStore.updateSession(session); + const messages: any[] = []; + const worker = new EventEmitter() as FakeWorker; + worker.killed = false; + worker.exitCode = null; + worker.signalCode = null; + worker.kill = vi.fn(); + worker.send = vi.fn((message: any) => { + messages.push(message); + onSend(worker, message); + }); + const ds = { + larkAppId: 'app', + chatId: session.chatId, + chatType: 'group', + scope: 'chat', + session, + worker, + workerPort: 4100, + workerToken: 'write', + workerViewToken: 'view', + managedTurnOrigin: { capability: 'cap' }, + initConfig: { backendType: 'riff' }, + } as unknown as DaemonSession; + return { ds, worker, messages }; + } + + function asPrepared( + result: Awaited>, + ): PreparedRiffShutdown { + if (!result.ok) throw new Error(`expected prepared Riff shutdown: ${result.error}`); + return result; + } + + function asFenced( + result: Awaited>, + ): FencedRiffShutdownParticipant { + if (result.fence === 'none') throw new Error(`expected fenced Riff shutdown: ${result.error}`); + return result; + } + + it('transactional lineage persistence rolls back its in-memory field when save throws', () => { + const session = sessionStore.createSession('oc_txn', 'om_txn', 'txn', 'group'); + session.backendType = 'riff'; + session.riffParentTaskId = 'task-before'; + sessionStore.updateSession(session); + const blockedDataDir = join(dataDir, 'not-a-directory'); + writeFileSync(blockedDataDir, 'block'); + config.session.dataDir = blockedDataDir; + + expect(() => sessionStore.persistActiveRiffLineageExact(session.sessionId, 'task-after')).toThrow(); + expect(session.riffParentTaskId).toBe('task-before'); + + config.session.dataDir = dataDir; + expect(sessionStore.getSessionFresh(session.sessionId)?.riffParentTaskId).toBe('task-before'); + }); + + it('deduplicates multiple registry aliases to the exact same daemon session object', () => { + const f = fixture('task-parent', () => { /* no IPC */ }); + + expect(collectUniqueDaemonShutdownSessions([f.ds, f.ds, f.ds])).toEqual({ + ok: true, + sessions: [f.ds], + }); + }); + + it('fails closed when distinct daemon session objects claim the same session id', () => { + const f = fixture('task-parent', () => { /* no IPC */ }); + const competing = { + ...f.ds, + session: { ...f.ds.session }, + } as DaemonSession; + + expect(collectUniqueDaemonShutdownSessions([f.ds, competing])).toMatchObject({ + ok: false, + sessionId: f.ds.session.sessionId, + error: expect.stringContaining('distinct daemon session generations'), + }); + }); + + it('retains only the ambiguous session fence and restores an unrelated prepared peer', async () => { + const first = fixture('task-first', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-first-child', + })); + } + }); + const second = fixture('task-second', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-second-child', + })); + } else if (message.type === 'riff_shutdown_abort') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'abort', ok: true, taskId: 'task-second-child', + })); + } + }); + const prepared = await prepareRiffFleetForShutdown([first.ds, second.ds]); + const competingFirst = { + ...first.ds, + session: { ...first.ds.session }, + } as DaemonSession; + const current = collectUniqueDaemonShutdownSessions([ + first.ds, + competingFirst, + second.ds, + ]); + if (current.ok) throw new Error('expected ambiguous daemon session id'); + + const restored = await abortRiffShutdownFleet(prepared + .filter(({ ds }) => ds.session.sessionId !== current.sessionId) + .map(({ ds, result }) => ({ ds, result: asFenced(result) }))); + + expect(restored).toHaveLength(1); + expect(restored[0].result.ok).toBe(true); + expect(first.ds.riffShutdownState).toMatchObject({ phase: 'prepared' }); + expect(first.messages.map(message => message.type)).toEqual(['riff_shutdown_prepare']); + expect(second.ds.riffShutdownState).toBeUndefined(); + expect(second.messages.map(message => message.type)).toEqual([ + 'riff_shutdown_prepare', + 'riff_shutdown_abort', + ]); + }); + + it('refuses an initial durable-read failure before installing any worker fence', async () => { + vi.spyOn(sessionStore, 'getActiveRiffShutdownSnapshotsBatch').mockImplementation(() => { + throw new Error('lock unavailable'); + }); + const f = fixture('task-parent', () => { /* no worker IPC is safe */ }); + + await expect(prepareRiffSessionForShutdown(f.ds)).resolves.toMatchObject({ + ok: false, + taskId: 'task-parent', + error: 'durable_session_read_failed:lock unavailable', + }); + expect(f.ds.riffShutdownState).toBeUndefined(); + expect(f.ds.worker).toBe(f.worker); + expect(f.messages).toEqual([]); + }); + + it('takes one all-owner snapshot before publishing any fleet prepare request', async () => { + const first = fixture('task-first', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-first-child', + })); + } + }); + const second = fixture('task-second', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-second-child', + })); + } + }); + first.ds.session.pid = 101; + second.ds.session.pid = 202; + sessionStore.updateSession(first.ds.session); + sessionStore.updateSession(second.ds.session); + const originalSnapshot = sessionStore.getActiveRiffShutdownSnapshotsBatch; + const snapshot = vi.spyOn(sessionStore, 'getActiveRiffShutdownSnapshotsBatch') + .mockImplementation((sessionIds, options) => { + expect(first.messages).toEqual([]); + expect(second.messages).toEqual([]); + return originalSnapshot(sessionIds, options); + }); + + const results = await prepareRiffFleetForShutdown([first.ds, second.ds]); + + expect(snapshot).toHaveBeenCalledTimes(1); + expect(snapshot).toHaveBeenCalledWith([ + first.ds.session.sessionId, + second.ds.session.sessionId, + ], expect.any(Object)); + expect(results.every(entry => entry.result.ok)).toBe(true); + expect(first.messages.map(message => message.type)).toEqual(['riff_shutdown_prepare']); + expect(second.messages.map(message => message.type)).toEqual(['riff_shutdown_prepare']); + }); + + it('does not inline-abort a timed-out prepare and restores all owners in one concurrent wave', async () => { + let firstAbortRequestId: string | undefined; + let secondAbortRequestId: string | undefined; + const first = fixture('task-first', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-first-child', + })); + } else if (message.type === 'riff_shutdown_abort') { + firstAbortRequestId = message.requestId; + } + }); + const second = fixture('task-second', (_worker, message) => { + if (message.type === 'riff_shutdown_abort') secondAbortRequestId = message.requestId; + // Deliberately never ACK prepare so its fence state is ambiguous. + }); + first.ds.session.pid = 101; + second.ds.session.pid = 202; + sessionStore.updateSession(first.ds.session); + sessionStore.updateSession(second.ds.session); + + const prepared = await prepareRiffFleetForShutdown([first.ds, second.ds], { + drainTimeoutMs: 5, + abortTimeoutMs: 200, + }); + expect(prepared[0].result).toMatchObject({ ok: true, fence: 'prepared' }); + expect(prepared[1].result).toMatchObject({ ok: false, fence: 'possible' }); + expect(first.messages.map(message => message.type)).toEqual(['riff_shutdown_prepare']); + expect(second.messages.map(message => message.type)).toEqual(['riff_shutdown_prepare']); + + const aborting = abortRiffShutdownFleet(prepared.map(({ ds, result }) => ({ + ds, + result: asFenced(result), + })), { abortTimeoutMs: 200 }); + await vi.waitFor(() => { + expect(firstAbortRequestId).toEqual(expect.any(String)); + expect(secondAbortRequestId).toEqual(expect.any(String)); + }); + expect(first.messages.filter(message => message.type === 'riff_shutdown_abort')).toHaveLength(1); + expect(second.messages.filter(message => message.type === 'riff_shutdown_abort')).toHaveLength(1); + + first.worker.emit('message', { + type: 'riff_shutdown_result', requestId: firstAbortRequestId, + phase: 'abort', ok: true, taskId: 'task-first-child', + }); + second.worker.emit('message', { + type: 'riff_shutdown_result', requestId: secondAbortRequestId, + phase: 'abort', ok: true, taskId: 'task-second', + }); + await expect(aborting).resolves.toSatisfy( + (results: Array<{ result: { ok: boolean } }>) => results.every(entry => entry.result.ok), + ); + expect(first.ds.riffShutdownState).toBeUndefined(); + expect(second.ds.riffShutdownState).toBeUndefined(); + + // A prepare ACK arriving after the exact abort wave is inert. + second.worker.emit('message', { + type: 'riff_shutdown_result', requestId: secondAbortRequestId, + phase: 'prepare', ok: true, taskId: 'task-late-ack', + }); + expect(second.ds.riffShutdownState).toBeUndefined(); + expect(second.messages.some(message => message.type === 'riff_shutdown_commit')).toBe(false); + }); + + it.each(['explicit_close_in_progress', 'not_riff_backend'])( + 'classifies exact pre-fence worker refusal %s without sending abort', + async (error) => { + const f = fixture('task-parent', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: false, taskId: null, error, + })); + } + }); + + await expect(detachRiffWorkerForShutdown(f.ds)).resolves.toMatchObject({ + ok: false, + fence: 'none', + error, + }); + expect(f.messages.map(message => message.type)).toEqual(['riff_shutdown_prepare']); + expect(f.ds.riffShutdownState).toBeUndefined(); + expect(f.ds.worker).toBe(f.worker); + }, + ); + + it('clears the synthetic daemon fence when prepare IPC throws before it can be queued', async () => { + const f = fixture('task-parent', () => { /* send is replaced below */ }); + f.worker.send.mockImplementation(() => { throw new Error('IPC channel closed'); }); + + await expect(detachRiffWorkerForShutdown(f.ds)).resolves.toMatchObject({ + ok: false, + fence: 'none', + error: 'riff_shutdown_prepare_send_failed', + }); + expect(f.messages).toEqual([]); + expect(f.ds.riffShutdownState).toBeUndefined(); + expect(f.ds.worker).toBe(f.worker); + }); + + it('does not misclassify an existing older shutdown fence as a pre-fence refusal', async () => { + const f = fixture('task-parent', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: false, taskId: null, + error: 'shutdown detach already prepared as older-request', + })); + } + }); + + await expect(prepareRiffSessionForShutdown(f.ds)).resolves.toMatchObject({ + ok: false, + fence: 'possible', + error: 'shutdown detach already prepared as older-request', + }); + expect(f.ds.riffShutdownState).toMatchObject({ phase: 'preparing' }); + expect(f.messages.map(message => message.type)).toEqual(['riff_shutdown_prepare']); + }); + + it('refuses before a worker fence unless phase-2 plus the abort reserve remain', async () => { + const f = fixture('task-parent', () => { /* no worker IPC is safe */ }); + const now = 10_000; + + await expect(prepareRiffFleetForShutdown([f.ds], { + deadlineMs: now + 1_500, + abortTimeoutMs: 1_000, + now: () => now, + })).resolves.toMatchObject([{ + result: { + ok: false, + fence: 'none', + error: 'insufficient_abort_budget_before_fence', + }, + }]); + expect(f.messages).toEqual([]); + expect(f.ds.riffShutdownState).toBeUndefined(); + }); + + it('persists all prepared owners through one phase-2 batch call', async () => { + const first = fixture('task-first', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-first-child', + })); + } + }); + const second = fixture('task-second', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-second-child', + })); + } + }); + const prepared = await prepareRiffFleetForShutdown([first.ds, second.ds]); + const entries = prepared.map(({ ds, result }) => ({ ds, result: asPrepared(result) })); + const batch = vi.spyOn(sessionStore, 'persistActiveRiffLineagesExactBatch'); + + expect(persistPreparedRiffShutdownFleet(entries)).toEqual({ ok: true }); + expect(batch).toHaveBeenCalledTimes(1); + expect(sessionStore.getSessionFresh(first.ds.session.sessionId)?.riffParentTaskId) + .toBe('task-first-child'); + expect(sessionStore.getSessionFresh(second.ds.session.sessionId)?.riffParentTaskId) + .toBe('task-second-child'); + expect(entries.every(({ result }) => result.lineageVerified)).toBe(true); + }); + + it('retains every fence when verified phase-2 persistence returns after the absolute deadline', async () => { + const f = fixture('task-parent', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-child', + })); + } + }); + const prepared = asPrepared(await prepareRiffSessionForShutdown(f.ds)); + const originalBatch = sessionStore.persistActiveRiffLineagesExactBatch; + let now = 10_000; + const deadlineMs = 10_100; + vi.spyOn(sessionStore, 'persistActiveRiffLineagesExactBatch') + .mockImplementation((updates, options) => { + originalBatch(updates, options); + now = deadlineMs; + }); + + const result = persistPreparedRiffShutdownFleet( + [{ ds: f.ds, result: prepared }], + { deadlineMs, now: () => now }, + ); + + expect(result).toMatchObject({ + ok: false, + error: 'shutdown_deadline_elapsed_after_batch_persist', + rollbackDisposition: 'retain_fence', + sessionIds: [f.ds.session.sessionId], + retainFencedSessionIds: [f.ds.session.sessionId], + }); + expect(prepared.lineageVerified).toBe(true); + expect(f.ds.session.riffParentTaskId).toBe('task-child'); + expect(f.ds.riffShutdownState).toMatchObject({ requestId: prepared.requestId }); + expect(f.messages.map(message => message.type)).toEqual(['riff_shutdown_prepare']); + }); + + it.each([ + ['prewrite_ownership', 'affected'] as const, + ['prewrite_io', 'none'] as const, + ['postrename_ambiguity', 'all'] as const, + ])('maps %s batch failure to the exact retain-fence set', async (stage, expected) => { + const first = fixture('task-first', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-first-child', + })); + } + }); + const second = fixture('task-second', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-second-child', + })); + } + }); + const prepared = await prepareRiffFleetForShutdown([first.ds, second.ds]); + const entries = prepared.map(({ ds, result }) => ({ ds, result: asPrepared(result) })); + const affected = stage === 'postrename_ambiguity' + ? [first.ds.session.sessionId, second.ds.session.sessionId] + : [second.ds.session.sessionId]; + vi.spyOn(sessionStore, 'persistActiveRiffLineagesExactBatch').mockImplementation(() => { + throw new sessionStore.RiffLineageBatchError(stage, affected, `forced ${stage}`); + }); + + const result = persistPreparedRiffShutdownFleet(entries); + expect(result).toMatchObject({ ok: false }); + if (result.ok) throw new Error('expected batch failure'); + expect(result.retainFencedSessionIds).toEqual( + expected === 'all' + ? [first.ds.session.sessionId, second.ds.session.sessionId] + : expected === 'affected' + ? [second.ds.session.sessionId] + : [], + ); + }); + + it('retains the exact fence without writing when runtime owner changes after prepare', async () => { + const f = fixture('task-parent', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-child', + })); + } + }); + const prepared = asPrepared(await prepareRiffSessionForShutdown(f.ds)); + f.ds.session.pid = 999_999; + const batch = vi.spyOn(sessionStore, 'persistActiveRiffLineagesExactBatch'); + + const result = persistPreparedRiffShutdownFleet([{ ds: f.ds, result: prepared }]); + + expect(result).toMatchObject({ + ok: false, + retainFencedSessionIds: [f.ds.session.sessionId], + error: expect.stringContaining('runtime_owner_changed'), + }); + expect(batch).not.toHaveBeenCalled(); + expect(f.ds.riffShutdownState).toMatchObject({ requestId: prepared.requestId }); + }); + + it('persists and fresh-verifies the exact late child before commit', async () => { + const f = fixture('task-parent', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', + requestId: message.requestId, + phase: 'prepare', + ok: true, + taskId: 'task-late-child', + })); + } + }); + + const result = await detachRiffWorkerForShutdown(f.ds); + expect(result).toMatchObject({ + ok: true, + taskId: 'task-late-child', + disposition: 'lineage_persisted', + }); + expect(f.messages.map(message => message.type)).toEqual([ + 'riff_shutdown_prepare', + 'riff_shutdown_commit', + ]); + expect(sessionStore.getSessionFresh(f.ds.session.sessionId)).toMatchObject({ + status: 'active', + riffParentTaskId: 'task-late-child', + }); + expect(f.ds.worker).toBeNull(); + expect(f.worker.kill).not.toHaveBeenCalled(); + }); + + it('treats null as authoritative and clears a stale durable parent before commit', async () => { + const f = fixture('task-stale-parent', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', + requestId: message.requestId, + phase: 'prepare', + ok: true, + taskId: null, + })); + } + }); + + await expect(detachRiffWorkerForShutdown(f.ds)).resolves.toMatchObject({ + ok: true, + taskId: null, + disposition: 'lineage_persisted', + }); + expect(sessionStore.getSessionFresh(f.ds.session.sessionId)?.riffParentTaskId).toBeUndefined(); + expect(f.messages.map(message => message.type)).toEqual([ + 'riff_shutdown_prepare', + 'riff_shutdown_commit', + ]); + }); + + it('restores the prepared worker without cancellation when durable persistence fails', async () => { + vi.spyOn(sessionStore, 'persistActiveRiffLineageExact').mockImplementation(() => { + throw new Error('disk full'); + }); + const f = fixture('task-parent', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', + requestId: message.requestId, + phase: 'prepare', + ok: true, + taskId: 'task-exact-child', + })); + } + if (message.type === 'riff_shutdown_abort') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', + requestId: message.requestId, + phase: 'abort', + ok: true, + taskId: 'task-exact-child', + })); + } + }); + + await expect(detachRiffWorkerForShutdown(f.ds)).resolves.toMatchObject({ + ok: false, + taskId: 'task-exact-child', + error: expect.stringContaining('lineage_persist_failed:disk full'), + }); + expect(f.messages.map(message => message.type)).toEqual([ + 'riff_shutdown_prepare', + 'riff_shutdown_abort', + ]); + // Persistence never advanced, so the accepted remote task and its exact + // worker generation remain live with admission restored. + expect(sessionStore.getSessionFresh(f.ds.session.sessionId)?.riffParentTaskId).toBe('task-parent'); + expect(f.ds.worker).toBe(f.worker); + expect(f.ds.riffShutdownState).toBeUndefined(); + expect(f.messages.some(message => message.type === 'riff_shutdown_commit')).toBe(false); + expect(f.messages.some(message => message.type === 'riff_shutdown_cancel')).toBe(false); + }); + + it('fails closed without commit or signal when persistence fails and abort is not ACKed', async () => { + vi.spyOn(sessionStore, 'persistActiveRiffLineageExact').mockImplementation(() => { + throw new Error('disk full'); + }); + const f = fixture('task-parent', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', + requestId: message.requestId, + phase: 'prepare', + ok: true, + taskId: 'task-uncancellable', + })); + } + if (message.type === 'riff_shutdown_abort') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', + requestId: message.requestId, + phase: 'abort', + ok: false, + taskId: 'task-uncancellable', + error: 'admission restore refused', + })); + } + }); + + await expect(detachRiffWorkerForShutdown(f.ds)).resolves.toMatchObject({ + ok: false, + taskId: 'task-uncancellable', + error: expect.stringContaining('admission_restore_failed:admission restore refused'), + }); + expect(f.messages.map(message => message.type)).toEqual([ + 'riff_shutdown_prepare', + 'riff_shutdown_abort', + ]); + expect(f.messages.some(message => message.type === 'riff_shutdown_commit')).toBe(false); + expect(f.worker.kill).not.toHaveBeenCalled(); + expect(f.ds.worker).toBe(f.worker); + expect(f.ds.riffShutdownState).toMatchObject({ phase: 'prepared' }); + }); + + it('keeps the fence when an abort ACK reports a different task lineage', async () => { + vi.spyOn(sessionStore, 'persistActiveRiffLineageExact').mockImplementation(() => { + throw new Error('disk full'); + }); + const f = fixture('task-parent', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-prepared-child', + })); + } + if (message.type === 'riff_shutdown_abort') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'abort', ok: true, taskId: 'task-unexpected-child', + })); + } + }); + + await expect(detachRiffWorkerForShutdown(f.ds)).resolves.toMatchObject({ + ok: false, + error: expect.stringContaining('abort_task_lineage_mismatch'), + }); + expect(f.ds.worker).toBe(f.worker); + expect(f.ds.riffShutdownState).toMatchObject({ phase: 'prepared' }); + expect(f.messages.map(message => message.type)).toEqual([ + 'riff_shutdown_prepare', + 'riff_shutdown_abort', + ]); + }); + + it('prepares and persists workerless runtime lineage before allowing commit', async () => { + const f = fixture('task-durable-parent', () => { /* workerless: no IPC */ }); + f.ds.worker = null; + f.ds.workerPort = null; + f.ds.workerToken = null; + f.ds.workerViewToken = null; + // Simulate a prior ordinary riff_task_id save failure: runtime owns the + // child while the durable row still names its parent. + f.ds.session.riffParentTaskId = 'task-runtime-child'; + + const prepared = asPrepared(await prepareRiffSessionForShutdown(f.ds)); + expect(prepared.worker).toBeNull(); + expect(f.ds.riffShutdownState).toMatchObject({ phase: 'prepared' }); + expect(sessionStore.getSessionFresh(f.ds.session.sessionId)?.riffParentTaskId) + .toBe('task-durable-parent'); + + expect(persistPreparedRiffShutdown(f.ds, prepared)).toEqual({ ok: true }); + expect(sessionStore.getSessionFresh(f.ds.session.sessionId)?.riffParentTaskId) + .toBe('task-runtime-child'); + expect(commitPreparedRiffShutdown(f.ds, prepared)).toBe(true); + expect(f.ds.riffShutdownState).toBeUndefined(); + expect(f.messages).toEqual([]); + }); + + it('aborts every prepared peer and commits none when one workerless lineage write fails', async () => { + const live = fixture('task-live-parent', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-live-child', + })); + } + if (message.type === 'riff_shutdown_abort') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'abort', ok: true, taskId: 'task-live-child', + })); + } + }); + const workerless = fixture('task-workerless-parent', () => { /* no IPC */ }); + workerless.ds.worker = null; + workerless.ds.session.riffParentTaskId = 'task-workerless-runtime-child'; + + const [livePrepared, workerlessPrepared] = await Promise.all([ + prepareRiffSessionForShutdown(live.ds), + prepareRiffSessionForShutdown(workerless.ds), + ]).then(results => results.map(asPrepared)); + const originalPersist = sessionStore.persistActiveRiffLineageExact; + vi.spyOn(sessionStore, 'persistActiveRiffLineageExact').mockImplementation( + (sessionId, taskId) => { + if (sessionId === workerless.ds.session.sessionId) throw new Error('target disk full'); + return originalPersist(sessionId, taskId); + }, + ); + + const persistence = [ + persistPreparedRiffShutdown(live.ds, livePrepared!), + persistPreparedRiffShutdown(workerless.ds, workerlessPrepared!), + ]; + expect(persistence[0]).toEqual({ ok: true }); + expect(persistence[1]).toMatchObject({ + ok: false, + error: expect.stringContaining('target disk full'), + }); + + await Promise.all([ + abortPreparedRiffShutdown(live.ds, livePrepared!), + abortPreparedRiffShutdown(workerless.ds, workerlessPrepared!), + ]); + expect(live.messages.map(message => message.type)).toEqual([ + 'riff_shutdown_prepare', + 'riff_shutdown_abort', + ]); + expect(live.messages.some(message => message.type === 'riff_shutdown_commit')).toBe(false); + expect(live.ds.worker).toBe(live.worker); + expect(live.ds.riffShutdownState).toBeUndefined(); + expect(workerless.ds.riffShutdownState).toBeUndefined(); + expect(sessionStore.getSessionFresh(workerless.ds.session.sessionId)?.riffParentTaskId) + .toBe('task-workerless-parent'); + }); + + it('retains a fail-closed fence when an unverified prepared worker exits before abort', async () => { + const f = fixture('task-parent', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-unverified-child', + })); + } + }); + const prepared = asPrepared(await prepareRiffSessionForShutdown(f.ds)); + f.worker.exitCode = 1; + // worker-pool clears the exact dead child handle but retains this request's + // shutdown fence for the coordinator. + f.ds.worker = null; + + expect(persistPreparedRiffShutdown(f.ds, prepared)).toMatchObject({ + ok: false, + error: 'stale_worker_generation', + }); + await expect(abortPreparedRiffShutdown(f.ds, prepared)).resolves.toMatchObject({ + ok: false, + error: 'worker_exited_before_admission_restore', + }); + expect(f.ds.worker).toBeNull(); + expect(f.ds.riffShutdownState).toMatchObject({ + phase: 'prepared', + requestId: prepared.requestId, + }); + expect(sendWorkerInput(f.ds, 'must stay fenced', 'turn-exited-unverified')).toBe(false); + }); + + it('classifies an unexpected durable lineage as ownership loss and never overwrites it', async () => { + const f = fixture('task-durable-parent', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => { + // Simulate a different process advancing the durable owner after + // prepare, without changing this daemon's runtime generation. + const sessionsPath = join(dataDir, 'sessions-app.json'); + const projection = JSON.parse(readFileSync(sessionsPath, 'utf8')) as Record; + projection[f.ds.session.sessionId]!.riffParentTaskId = 'task-external-owner'; + writeFileSync(sessionsPath, JSON.stringify(projection, null, 2)); + worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-prepared-child', + }); + }); + } + }); + + await expect(detachRiffWorkerForShutdown(f.ds)).resolves.toMatchObject({ + ok: false, + error: expect.stringContaining('compare-and-set failed'), + rollbackDisposition: 'retain_fence', + }); + expect(sessionStore.getSessionFresh(f.ds.session.sessionId)?.riffParentTaskId) + .toBe('task-external-owner'); + expect(f.ds.riffShutdownState).toMatchObject({ + phase: 'prepared', + }); + expect(f.messages.map(message => message.type)).toEqual(['riff_shutdown_prepare']); + }); + + it('retains the fence when durable worker ownership changes with the same lineage', async () => { + const f = fixture('task-parent', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => { + const sessionsPath = join(dataDir, 'sessions-app.json'); + const projection = JSON.parse(readFileSync(sessionsPath, 'utf8')) as Record; + projection[f.ds.session.sessionId]!.pid = 999_999; + writeFileSync(sessionsPath, JSON.stringify(projection, null, 2)); + worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-prepared-child', + }); + }); + } + }); + + await expect(detachRiffWorkerForShutdown(f.ds)).resolves.toMatchObject({ + ok: false, + error: expect.stringContaining('owner compare-and-set failed'), + rollbackDisposition: 'retain_fence', + }); + expect(f.ds.riffShutdownState).toMatchObject({ phase: 'prepared' }); + expect(f.messages.map(message => message.type)).toEqual(['riff_shutdown_prepare']); + expect(sessionStore.getSessionFresh(f.ds.session.sessionId)).toMatchObject({ + pid: 999_999, + riffParentTaskId: 'task-parent', + }); + }); + + it('retains the fence when ownership is replaced after CAS with the same task id', async () => { + const originalPersist = sessionStore.persistActiveRiffLineageExact; + vi.spyOn(sessionStore, 'persistActiveRiffLineageExact').mockImplementation( + (sessionId, taskId, options) => { + const result = originalPersist(sessionId, taskId, options); + // Exact replacement race: CAS wrote the expected task, then a new + // durable owner published the same task before the separate readback. + const sessionsPath = join(dataDir, 'sessions-app.json'); + const projection = JSON.parse(readFileSync(sessionsPath, 'utf8')) as Record; + projection[sessionId]!.pid = 888_888; + writeFileSync(sessionsPath, JSON.stringify(projection, null, 2)); + return result; + }, + ); + const f = fixture('task-parent', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-same-after-replacement', + })); + } + }); + + await expect(detachRiffWorkerForShutdown(f.ds)).resolves.toMatchObject({ + ok: false, + taskId: 'task-same-after-replacement', + error: expect.stringContaining('fresh_lineage_verification_failed:'), + rollbackDisposition: 'retain_fence', + }); + expect(f.ds.riffShutdownState).toMatchObject({ phase: 'prepared' }); + expect(f.messages.map(message => message.type)).toEqual(['riff_shutdown_prepare']); + expect(sessionStore.getSessionFresh(f.ds.session.sessionId)).toMatchObject({ + pid: 888_888, + riffParentTaskId: 'task-same-after-replacement', + }); + }); + + it('retains the fence when the post-write fresh verification cannot be read', async () => { + const originalFresh = sessionStore.getSessionFresh; + let reads = 0; + vi.spyOn(sessionStore, 'getSessionFresh').mockImplementation(sessionId => { + reads++; + if (reads === 1) throw new Error('lock unavailable'); + return originalFresh(sessionId); + }); + const f = fixture('task-parent', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-written-not-verified', + })); + } + }); + + await expect(detachRiffWorkerForShutdown(f.ds)).resolves.toMatchObject({ + ok: false, + error: 'fresh_lineage_verification_failed:lock unavailable', + rollbackDisposition: 'retain_fence', + }); + expect(f.ds.riffShutdownState).toMatchObject({ phase: 'prepared' }); + expect(f.messages.map(message => message.type)).toEqual(['riff_shutdown_prepare']); + expect(originalFresh(f.ds.session.sessionId)?.riffParentTaskId) + .toBe('task-written-not-verified'); + }); + + it('can release an exited prepared generation only after its lineage was durably verified', async () => { + const f = fixture('task-parent', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-verified-child', + })); + } + }); + const prepared = asPrepared(await prepareRiffSessionForShutdown(f.ds)); + expect(persistPreparedRiffShutdown(f.ds, prepared)).toEqual({ ok: true }); + f.worker.exitCode = 0; + f.ds.worker = null; + + expect(canAbortVerifiedExitedRiffPreparation(f.ds, prepared)).toBe(true); + + await expect(abortPreparedRiffShutdown(f.ds, prepared)).resolves.toEqual({ + ok: true, + taskId: 'task-verified-child', + }); + expect(f.ds.worker).toBeNull(); + expect(f.ds.riffShutdownState).toBeUndefined(); + expect(sessionStore.getSessionFresh(f.ds.session.sessionId)?.riffParentTaskId) + .toBe('task-verified-child'); + }); + + it('does not bless a replacement generation when the verified prepared worker exits', async () => { + const f = fixture('task-parent', (worker, message) => { + if (message.type === 'riff_shutdown_prepare') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', requestId: message.requestId, + phase: 'prepare', ok: true, taskId: 'task-verified-before-replacement', + })); + } + }); + const prepared = asPrepared(await prepareRiffSessionForShutdown(f.ds)); + expect(persistPreparedRiffShutdown(f.ds, prepared)).toEqual({ ok: true }); + f.worker.exitCode = 0; + const replacement = new EventEmitter() as FakeWorker; + replacement.killed = false; + replacement.exitCode = null; + replacement.signalCode = null; + replacement.send = vi.fn(); + replacement.kill = vi.fn(); + f.ds.worker = replacement; + + expect(canAbortVerifiedExitedRiffPreparation(f.ds, prepared)).toBe(false); + + await expect(abortPreparedRiffShutdown(f.ds, prepared)).resolves.toMatchObject({ + ok: false, + taskId: 'task-verified-before-replacement', + error: 'new_worker_generation', + }); + expect(f.ds.worker).toBe(replacement); + expect(f.ds.riffShutdownState).toMatchObject({ + phase: 'prepared', + requestId: prepared.requestId, + }); + expect(replacement.send).not.toHaveBeenCalled(); + }); + + it('times out boundedly and restores the worker instead of falling through to generic kill', async () => { + const f = fixture('task-parent', (worker, message) => { + if (message.type === 'riff_shutdown_abort') { + queueMicrotask(() => worker.emit('message', { + type: 'riff_shutdown_result', + requestId: message.requestId, + phase: 'abort', + ok: true, + taskId: 'task-parent', + })); + } + }); + const result = await detachRiffWorkerForShutdown(f.ds, { + drainTimeoutMs: 10, + abortTimeoutMs: 100, + }); + + expect(result).toMatchObject({ ok: false, error: 'riff_shutdown_prepare_timeout' }); + expect(f.messages.map(message => message.type)).toEqual([ + 'riff_shutdown_prepare', + 'riff_shutdown_abort', + ]); + expect(f.worker.kill).not.toHaveBeenCalled(); + expect(f.ds.worker).toBe(f.worker); + }); + + it('keeps daemon admission fenced until delayed abort restoration is ACKed', async () => { + let abortRequestId: string | undefined; + const f = fixture('task-parent', (_worker, message) => { + if (message.type === 'riff_shutdown_abort') abortRequestId = message.requestId; + }); + const detach = detachRiffWorkerForShutdown(f.ds, { + drainTimeoutMs: 5, + abortTimeoutMs: 200, + }); + await vi.waitFor(() => expect(abortRequestId).toEqual(expect.any(String))); + + expect(f.ds.riffShutdownState).toMatchObject({ phase: 'preparing' }); + expect(sendWorkerInput(f.ds, 'must remain fenced', 'turn-during-abort')).toBe(false); + expect(f.messages.some(message => message.type === 'riff_shutdown_commit')).toBe(false); + expect(f.worker.kill).not.toHaveBeenCalled(); + + f.worker.emit('message', { + type: 'riff_shutdown_result', + requestId: abortRequestId, + phase: 'abort', + ok: true, + taskId: 'task-parent', + }); + await expect(detach).resolves.toMatchObject({ + ok: false, + error: 'riff_shutdown_prepare_timeout', + }); + expect(f.ds.riffShutdownState).toBeUndefined(); + }); + + it('refuses before worker prepare when daemon-owned raw/follow-up input is pending', async () => { + const f = fixture('task-parent', () => { /* no worker IPC is safe */ }); + f.ds.pendingRawInput = '/goal keep this'; + f.ds.pendingFollowUpInput = { + userPrompt: 'follow-up', + cliInput: 'follow-up', + }; + + await expect(detachRiffWorkerForShutdown(f.ds)).resolves.toMatchObject({ + ok: false, + error: expect.stringContaining('daemon_inputs_not_drained:'), + }); + expect(f.messages).toEqual([]); + expect(f.worker.kill).not.toHaveBeenCalled(); + expect(f.ds.worker).toBe(f.worker); + }); + + it('refuses before worker prepare while a durable queued-activation tail is pending', async () => { + const f = fixture('task-parent', () => { /* no worker IPC is safe */ }); + f.ds.session.queuedActivationTail = [{ + id: 'tail-1', + order: 1, + userPrompt: 'accepted successor', + cliInput: { content: 'accepted successor' }, + turnId: 'turn-tail-1', + }]; + + await expect(detachRiffWorkerForShutdown(f.ds)).resolves.toMatchObject({ + ok: false, + error: expect.stringContaining('durable_activation_tail=1'), + }); + expect(f.messages).toEqual([]); + expect(f.worker.kill).not.toHaveBeenCalled(); + expect(f.ds.worker).toBe(f.worker); + }); + + it('retains the fail-closed fence when abort restoration cannot be confirmed', async () => { + const f = fixture('task-parent', () => { /* prepare and abort both held */ }); + const result = await detachRiffWorkerForShutdown(f.ds, { + drainTimeoutMs: 5, + abortTimeoutMs: 5, + }); + + expect(result).toMatchObject({ + ok: false, + error: expect.stringContaining('admission_restore_failed:riff_shutdown_abort_timeout'), + }); + expect(f.ds.riffShutdownState).toMatchObject({ phase: 'preparing' }); + expect(sendWorkerInput(f.ds, 'still blocked', 'turn-fail-closed')).toBe(false); + expect(f.worker.kill).not.toHaveBeenCalled(); + }); +}); diff --git a/test/riff-worker-shutdown-readiness.test.ts b/test/riff-worker-shutdown-readiness.test.ts new file mode 100644 index 000000000..df3c5fda7 --- /dev/null +++ b/test/riff-worker-shutdown-readiness.test.ts @@ -0,0 +1,37 @@ +import { describe, expect, it } from 'vitest'; +import { riffWorkerShutdownInputBlocker } from '../src/core/riff-worker-shutdown-readiness.js'; + +const idle = { + initPromptMaterialized: true, + isFlushing: false, + pendingMessages: 0, + pendingRawInputs: 0, + pendingSessionRename: false, + sessionRenameInFlight: false, + commandLineWritesPending: 0, +}; + +describe('Riff worker shutdown input ownership', () => { + it('allows the current backend task itself when no unsent worker input exists', () => { + expect(riffWorkerShutdownInputBlocker(idle)).toBeNull(); + }); + + it('refuses current-task detach when a follow-up is still in pendingMessages', () => { + expect(riffWorkerShutdownInputBlocker({ ...idle, pendingMessages: 1 })) + .toBe('messages=1'); + }); + + it('accounts for every worker-owned pre-backend input surface', () => { + expect(riffWorkerShutdownInputBlocker({ + initPromptMaterialized: false, + isFlushing: true, + pendingMessages: 2, + pendingRawInputs: 1, + pendingSessionRename: true, + sessionRenameInFlight: true, + commandLineWritesPending: 1, + })).toBe( + 'init=materializing,flushing=1,messages=2,raw=1,rename=1,rename_inflight=1,command_writes=1', + ); + }); +}); diff --git a/test/runner-input.test.ts b/test/runner-input.test.ts index 272e39062..eaf19410b 100644 --- a/test/runner-input.test.ts +++ b/test/runner-input.test.ts @@ -36,12 +36,13 @@ function fakeTmuxPty(opts: { failTextAt?: number; failEnter?: boolean } = {}) { } /** Fake raw-PTY handle (no tmux send methods): exercises the write() fallback. */ -function fakeRawPty(opts: { throwOnWrite?: boolean } = {}) { +function fakeRawPty(opts: { throwOnWrite?: boolean; rejectWrite?: boolean } = {}) { const writes: string[] = []; const pty: PtyHandle = { write(data: string) { if (opts.throwOnWrite) throw new Error('pty gone'); writes.push(data); + return opts.rejectWrite ? false : undefined; }, }; return { pty, writes }; @@ -123,7 +124,7 @@ describe('writeRunnerInput — tmux mode', () => { const res = await writeRunnerInput(pty, MARKER, big); - expect(res).toEqual({ submitted: true }); + expect(res).toEqual({ submitted: true, submissionDisposition: 'submitted' }); expect(textChunks.length).toBeGreaterThan(1); for (const c of textChunks) expect(c.length).toBeLessThanOrEqual(RUNNER_INPUT_CHUNK_BYTES); // pre-flush Enter + submit Enter on the happy path. @@ -177,7 +178,7 @@ describe('writeRunnerInput — tmux mode', () => { const res = await writeRunnerInput(pty, MARKER, big); - expect(res).toEqual({ submitted: false }); + expect(res).toEqual({ submitted: false, submissionDisposition: 'dirty_unknown' }); // Chunks 0 and 1 landed; chunk 2 failed and we bailed. expect(textChunks).toHaveLength(2); // pre-flush Enter + flush-on-failure Enter — the partial line gets terminated. @@ -187,11 +188,37 @@ describe('writeRunnerInput — tmux mode', () => { it('pre-flush gate: when every Enter is dropped, bail submitted:false WITHOUT writing any chunk', async () => { const { pty, textChunks } = fakeTmuxPty({ failEnter: true }); const res = await writeRunnerInput(pty, MARKER, 'short message'); - expect(res).toEqual({ submitted: false }); + expect(res).toEqual({ submitted: false, submissionDisposition: 'untouched' }); // The pre-flush Enter never lands, so we must not write the control line // onto a possibly-dirty buffer. expect(textChunks).toHaveLength(0); }); + + it('treats a false last-chunk result as dirty even when bytes landed and cleanup submits it', async () => { + const runner = makeRunnerSim(); + const content = 'last chunk ambiguity ' + 'x'.repeat(2_000); + const chunkCount = chunkAscii( + MARKER + encodeRunnerInput(content), + RUNNER_INPUT_CHUNK_BYTES, + ).length; + let textCall = 0; + const pty: PtyHandle = { + write() { throw new Error('unused'); }, + sendText(text: string) { + runner.feed(text); + return textCall++ === chunkCount - 1 ? false : true; + }, + sendSpecialKeys() { + runner.feed('\r'); + return true; + }, + }; + + const result = await writeRunnerInput(pty, MARKER, content); + + expect(result).toEqual({ submitted: false, submissionDisposition: 'dirty_unknown' }); + expect(runner.enqueued).toEqual([content]); + }); }); describe('writeRunnerInput — raw PTY fallback', () => { @@ -199,14 +226,21 @@ describe('writeRunnerInput — raw PTY fallback', () => { const content = 'fallback path'; const { pty, writes } = fakeRawPty(); const res = await writeRunnerInput(pty, MARKER, content); - expect(res).toEqual({ submitted: true }); + expect(res).toEqual({ submitted: true, submissionDisposition: 'submitted' }); expect(writes).toEqual([MARKER + encodeRunnerInput(content) + '\r']); }); it('reports submitted:false when the raw write throws (pane gone)', async () => { const { pty } = fakeRawPty({ throwOnWrite: true }); const res = await writeRunnerInput(pty, MARKER, 'x'); - expect(res).toEqual({ submitted: false }); + expect(res).toEqual({ submitted: false, submissionDisposition: 'dirty_unknown' }); + }); + + it('does not promote a raw PTY false return to submitted:true', async () => { + const { pty, writes } = fakeRawPty({ rejectWrite: true }); + const res = await writeRunnerInput(pty, MARKER, 'rejected'); + expect(res).toEqual({ submitted: false, submissionDisposition: 'dirty_unknown' }); + expect(writes).toHaveLength(1); }); }); @@ -218,7 +252,7 @@ describe('writeRunnerInput — runner buffer hygiene (no cross-message contamina // Message A: drop chunk 2 mid-stream. const big = 'A'.repeat(15_000); const resA = await writeRunnerInput(fakeRunnerPty(runner, { dropTextAt: 2 }), MARKER, big); - expect(resA).toEqual({ submitted: false }); + expect(resA).toEqual({ submitted: false, submissionDisposition: 'dirty_unknown' }); // A never enqueues intact; its partial got flushed (discarded as bad input), // so the runner buffer is empty — nothing left to prepend to the next line. expect(runner.enqueued).not.toContain(big); @@ -226,7 +260,7 @@ describe('writeRunnerInput — runner buffer hygiene (no cross-message contamina // Message B through the SAME runner: must arrive intact, not merged with A. const resB = await writeRunnerInput(fakeRunnerPty(runner), MARKER, 'clean message B'); - expect(resB).toEqual({ submitted: true }); + expect(resB).toEqual({ submitted: true, submissionDisposition: 'submitted' }); expect(runner.enqueued).toContain('clean message B'); }); @@ -266,7 +300,7 @@ describe('writeRunnerInput — runner buffer hygiene (no cross-message contamina dropText.add(aChunks.length - 1); dropEnter.add(1); dropEnter.add(2); dropEnter.add(3); const a = await writeRunnerInput(pty, MARKER, aContent); - expect(a).toEqual({ submitted: false }); + expect(a).toEqual({ submitted: false, submissionDisposition: 'dirty_unknown' }); expect(runner.buf).not.toBe(''); // A's partial is stuck in the buffer // Message B: drop ALL of B's pre-flush Enter retries (enters 4,5,6). The gate @@ -274,7 +308,7 @@ describe('writeRunnerInput — runner buffer hygiene (no cross-message contamina dropEnter.add(4); dropEnter.add(5); dropEnter.add(6); const textBefore = textIdx; const b = await writeRunnerInput(pty, MARKER, 'message B'); - expect(b).toEqual({ submitted: false }); // NOT a false success + expect(b).toEqual({ submitted: false, submissionDisposition: 'untouched' }); // NOT a false success expect(textIdx).toBe(textBefore); // zero chunks written for B expect(runner.enqueued).not.toContain('message B'); expect(runner.bad).not.toContain('shape'); // no merged bad line attributed as submitted @@ -283,7 +317,7 @@ describe('writeRunnerInput — runner buffer hygiene (no cross-message contamina // (discarded as bad), then enqueues intact. const cleanRunnerPty = fakeRunnerPty(runner); // fresh counters, no drops const bb = await writeRunnerInput(cleanRunnerPty, MARKER, 'message B prime'); - expect(bb).toEqual({ submitted: true }); + expect(bb).toEqual({ submitted: true, submissionDisposition: 'submitted' }); expect(runner.enqueued).toContain('message B prime'); expect(runner.enqueued).not.toContain('message B'); }); diff --git a/test/runtime-screen-status.test.ts b/test/runtime-screen-status.test.ts new file mode 100644 index 000000000..dd5417b8e --- /dev/null +++ b/test/runtime-screen-status.test.ts @@ -0,0 +1,197 @@ +import { describe, expect, it } from 'vitest'; +import { + beginRuntimeWriteCycle, + isCliBackendGenerationCurrent, + PtyOutputGeneration, + projectRuntimeScreenStatus, + snapshotWithLatestRuntimeStatus, +} from '../src/utils/runtime-screen-status.js'; +import { CodexBridgeQueue } from '../src/services/codex-bridge-queue.js'; +import { IdleDetector } from '../src/utils/idle-detector.js'; +import type { CliAdapter } from '../src/adapters/cli/types.js'; + +function makeIdleAdapter(): CliAdapter { + return { + id: 'runtime-write-cycle-test', + resolvedBin: '/bin/true', + buildArgs: () => [], + writeInput: async () => {}, + systemHints: [], + altScreen: false, + }; +} + +function deferred() { + let resolve!: (value: T) => void; + const promise = new Promise(res => { resolve = res; }); + return { promise, resolve }; +} + +describe('projectRuntimeScreenStatus', () => { + it('keeps an unfinished structured turn working even when a prompt is visible', () => { + expect(projectRuntimeScreenStatus({ + promptReady: true, + analyzing: false, + structuredTurnBlocking: true, + })).toBe('working'); + }); + + it('returns idle only after lifecycle completion and prompt readiness agree', () => { + expect(projectRuntimeScreenStatus({ + promptReady: true, + analyzing: false, + structuredTurnBlocking: false, + })).toBe('idle'); + }); + + it('preserves analyzing while a screen analyzer is active', () => { + expect(projectRuntimeScreenStatus({ + promptReady: true, + analyzing: true, + structuredTurnBlocking: true, + })).toBe('analyzing'); + }); + + it('projects periodic status after an asynchronous snapshot, using a turn that started meanwhile', async () => { + let structuredTurnBlocking = false; + const capture = deferred(); + const pending = snapshotWithLatestRuntimeStatus( + () => capture.promise, + () => projectRuntimeScreenStatus({ + promptReady: true, + analyzing: false, + structuredTurnBlocking, + }), + ); + + // The tick began while idle, but the next turn starts before tmux/observe + // capture resolves. The late update must read the new lifecycle state. + structuredTurnBlocking = true; + capture.resolve('captured screen'); + + await expect(pending).resolves.toEqual({ + snapshot: 'captured screen', + status: 'working', + }); + }); + + it('invalidates rejected-ready evidence for a second PTY chunk in the same millisecond', () => { + const activity = new PtyOutputGeneration(); + const fixedTimestampMs = 123_456; + let lastPtyOutputAtMs = fixedTimestampMs; + + activity.observe(); + const readyEvidence = activity.snapshot(); + expect(activity.isCurrent(readyEvidence)).toBe(true); + + // A second chunk can share the timestamp; the old equality check would + // still trust the first prompt. Generation evidence must reject it. + lastPtyOutputAtMs = fixedTimestampMs; + activity.observe(); + expect(lastPtyOutputAtMs).toBe(fixedTimestampMs); + expect(activity.isCurrent(readyEvidence)).toBe(false); + }); + + it('rejects a grace callback during the same-backend restart gap', () => { + const sameBackend = { id: 'old-backend-object' }; + const fence = { generation: 7, backend: sameBackend }; + + expect(isCliBackendGenerationCurrent(fence, { + generation: 7, + backend: sameBackend, + restartInProgress: false, + })).toBe(true); + // restartCliProcess increments generation and raises the restart fence + // before its jitter/async teardown replaces the backend object. + expect(isCliBackendGenerationCurrent(fence, { + generation: 8, + backend: sameBackend, + restartInProgress: true, + })).toBe(false); + expect(isCliBackendGenerationCurrent(fence, { + generation: 7, + backend: sameBackend, + restartInProgress: true, + })).toBe(false); + }); + + it('stays working when prompt-ready races ahead of in-flight submit verification', () => { + const queue = new CodexBridgeQueue(); + queue.mark('race', 'prompt still polling history', 100); + queue.beginSubmitVerification('race', 200); + + expect(projectRuntimeScreenStatus({ + promptReady: true, + analyzing: false, + structuredTurnBlocking: queue.hasBlockingTurn(201), + })).toBe('working'); + + queue.finishSubmitVerification('race'); + expect(projectRuntimeScreenStatus({ + promptReady: true, + analyzing: false, + structuredTurnBlocking: queue.hasBlockingTurn(202), + })).toBe('idle'); + }); + + it('re-arms before await so a final arriving during submit verification is retained', async () => { + let promptReady = true; + const detector = new IdleDetector(makeIdleAdapter()); + detector.onIdle(() => { + // Mirrors markPromptReady's duplicate-ready guard. + if (promptReady) return; + promptReady = true; + }); + + // Model the previous turn's detector state: it already published idle. + detector.fireIdle(); + const verification = deferred<'confirmed'>(); + beginRuntimeWriteCycle({ + setPromptReady: ready => { promptReady = ready; }, + resetIdleDetector: () => detector.reset(), + }); + const writeResult = verification.promise; + + expect(promptReady).toBe(false); + // The transcript closes before history polling resolves. Because reset + // happened before the await, this edge is delivered instead of no-op'd. + detector.fireIdle(); + expect(promptReady).toBe(true); + + verification.resolve('confirmed'); + await expect(writeResult).resolves.toBe('confirmed'); + // Await completion must not overwrite the final-driven ready state. + expect(promptReady).toBe(true); + detector.dispose(); + }); + + it('re-arms every item when two type-ahead writes both finish before verification returns', async () => { + let promptReady = true; + let completedEdges = 0; + const detector = new IdleDetector(makeIdleAdapter()); + detector.onIdle(() => { + if (promptReady) return; + promptReady = true; + completedEdges++; + }); + detector.fireIdle(); + + for (const result of ['first', 'second'] as const) { + const verification = deferred(); + beginRuntimeWriteCycle({ + setPromptReady: ready => { promptReady = ready; }, + resetIdleDetector: () => detector.reset(), + }); + const writeResult = verification.promise; + + expect(promptReady).toBe(false); + detector.fireIdle(); + expect(promptReady).toBe(true); + verification.resolve(result); + await expect(writeResult).resolves.toBe(result); + } + + expect(completedEdges).toBe(2); + detector.dispose(); + }); +}); diff --git a/test/scheduler-silent-execute.test.ts b/test/scheduler-silent-execute.test.ts index 577956d4c..01567bbdf 100644 --- a/test/scheduler-silent-execute.test.ts +++ b/test/scheduler-silent-execute.test.ts @@ -68,7 +68,12 @@ vi.mock('../src/core/worker-pool.js', () => ({ restoreUsageLimitRuntimeState: vi.fn(), setActiveSessionSafe: vi.fn(async (map: Map, k: string, ds: any) => { map.set(k, ds); }), getActiveSessionsRegistry: vi.fn(() => null), - isRelayableRealSession: vi.fn(() => false), + withActiveSessionKeyLock: vi.fn(async ( + _map: Map, + _key: string, + action: () => any, + ) => action()), + isRelayableRealSession: vi.fn((ds: DaemonSession) => !!ds.worker && !ds.worker.killed), closeSession: vi.fn(), suspendWorker: vi.fn(), })); diff --git a/test/session-adopt.test.ts b/test/session-adopt.test.ts index b7d41243a..8c71c5d95 100644 --- a/test/session-adopt.test.ts +++ b/test/session-adopt.test.ts @@ -46,6 +46,7 @@ vi.mock('../src/bot-registry.js', () => ({ })), getAllBots: vi.fn(() => []), getBotClient: vi.fn(), + getBotBrand: vi.fn(() => 'feishu'), })); vi.mock('../src/config.js', () => ({ @@ -57,6 +58,7 @@ vi.mock('../src/config.js', () => ({ })); vi.mock('../src/services/session-store.js', () => ({ + getSession: vi.fn(), closeSession: vi.fn(), updateSession: vi.fn(), createSession: vi.fn(), @@ -89,6 +91,7 @@ vi.mock('../src/core/session-manager.js', () => ({ ds.lastUserPrompt = userPrompt; ds.lastCliInput = cliInput; }), + persistStreamCardState: vi.fn(), })); vi.mock('../src/services/frozen-card-store.js', () => ({ @@ -106,7 +109,7 @@ vi.mock('@larksuiteoapi/node-sdk', () => ({ // ─── Imports ────────────────────────────────────────────────────────────── import { handleCardAction, type CardHandlerDeps } from '../src/im/lark/card-handler.js'; -import { killWorker, forkWorker } from '../src/core/worker-pool.js'; +import { killWorker, forkWorker, setActiveSessionsRegistry } from '../src/core/worker-pool.js'; import * as sessionStore from '../src/services/session-store.js'; import { deleteMessage } from '../src/im/lark/client.js'; import { getBot } from '../src/bot-registry.js'; @@ -132,7 +135,7 @@ function makeDaemonSession(overrides?: Partial): DaemonSession { pid: null, chatType: 'group', }, - worker: { killed: false, send: vi.fn() } as any, + worker: { killed: false, send: vi.fn(), once: vi.fn() } as any, workerPort: 8080, workerToken: 'tok_secret', larkAppId: APP_ID, @@ -222,18 +225,25 @@ describe('Adopt card actions', () => { const sKey = sessionKey(ROOT_ID, APP_ID); sessions.set(sKey, ds); const deps = makeDeps(sessions); - - await handleCardAction(makeDisconnectEvent(ROOT_ID), deps, APP_ID); - - expect(killWorker).toHaveBeenCalledWith(ds); - expect(sessionStore.closeSession).toHaveBeenCalledWith('uuid-adopt-test'); - expect(sessions.has(sKey)).toBe(false); - expect(deps.sessionReply).toHaveBeenCalledWith( - ROOT_ID, - expect.stringContaining('断开'), - undefined, - APP_ID, - ); + const worker = ds.worker as any; + vi.mocked(sessionStore.getSession).mockReturnValue(ds.session); + setActiveSessionsRegistry(sessions); + + try { + await handleCardAction(makeDisconnectEvent(ROOT_ID), deps, APP_ID); + + expect(worker.send).toHaveBeenCalledWith({ type: 'close' }); + expect(sessionStore.closeSession).toHaveBeenCalledWith('uuid-adopt-test'); + expect(sessions.has(sKey)).toBe(false); + expect(deps.sessionReply).toHaveBeenCalledWith( + ROOT_ID, + expect.stringContaining('断开'), + undefined, + APP_ID, + ); + } finally { + setActiveSessionsRegistry(new Map()); + } }); it('should be a no-op when session does not exist', async () => { diff --git a/test/session-card-model.test.ts b/test/session-card-model.test.ts index b849d6139..221b8e1eb 100644 --- a/test/session-card-model.test.ts +++ b/test/session-card-model.test.ts @@ -53,6 +53,14 @@ describe('session-card-model · composeEntries / statusToDot', () => { const dot = statusToDot('dormant'); expect(dot).toEqual({ tone: 'neutral', pulse: false, label: 'sessions.status.dormant' }); }); + + it('maps stalled Codex App turns to a non-pulsing danger dot', () => { + expect(statusToDot('stalled')).toEqual({ + tone: 'danger', + pulse: false, + label: 'sessions.status.stalled', + }); + }); }); describe('session-card-model · filters', () => { diff --git a/test/session-kanban.test.ts b/test/session-kanban.test.ts index ee23b93ef..22b40e29a 100644 --- a/test/session-kanban.test.ts +++ b/test/session-kanban.test.ts @@ -74,6 +74,7 @@ describe('deriveKanbanColumn', () => { expect(deriveKanbanColumn({ status: 'idle', tuiPromptActive: true })).toBe('in_review'); expect(deriveKanbanColumn({ status: 'idle', agentAttention: { kind: 'x', reason: 'y', at: 1 } })).toBe('in_review'); expect(deriveKanbanColumn({ status: 'limited' })).toBe('in_review'); + expect(deriveKanbanColumn({ status: 'stalled' })).toBe('in_review'); }); it('maps runtime states to default columns', () => { diff --git a/test/session-lifecycle-start.test.ts b/test/session-lifecycle-start.test.ts index 91811ffa5..ebca7ee84 100644 --- a/test/session-lifecycle-start.test.ts +++ b/test/session-lifecycle-start.test.ts @@ -132,10 +132,18 @@ vi.mock('@larksuiteoapi/node-sdk', () => ({ })); import { __testOnly_resetSessionLifecycleHooks } from '../src/services/session-lifecycle-hooks.js'; -import { forkAdoptWorker, forkWorker, initWorkerPool, sendWorkerInput } from '../src/core/worker-pool.js'; +import { + forkAdoptWorker, + forkWorker, + initWorkerPool, + promoteQueuedActivationTail, + sendWorkerInput, +} from '../src/core/worker-pool.js'; import type { DaemonSession } from '../src/core/types.js'; import * as sessionStore from '../src/services/session-store.js'; import { getBot } from '../src/bot-registry.js'; +import { dashboardEventBus } from '../src/core/dashboard-events.js'; +import { retireCodexAppDispatchAfterBackingMissing } from '../src/utils/codex-app-dispatch-ledger.js'; import { mkdtempSync, rmSync, symlinkSync } from 'node:fs'; import { tmpdir, homedir } from 'node:os'; import { join } from 'node:path'; @@ -199,6 +207,7 @@ function defaultBot(overrides: Record = {}) { beforeEach(() => { vi.clearAllMocks(); + vi.mocked(sessionStore.updateSession).mockImplementation(() => undefined); vi.mocked(getBot).mockImplementation(() => defaultBot()); __testOnly_resetSessionLifecycleHooks(); forkMock.mockImplementation(() => makeFakeWorker()); @@ -210,6 +219,82 @@ beforeEach(() => { }); }); +describe('Riff shutdown worker-exit fencing', () => { + it('retains the exact shutdown fence when the worker exits before lineage verification', () => { + const ds = makeDs(); + ds.session.backendType = 'riff'; + forkWorker(ds, 'hello', false); + const worker = forkMock.mock.results.at(-1)!.value; + ds.riffShutdownState = { + phase: 'prepared', + requestId: 'shutdown-request', + taskId: 'task-drained-but-unverified', + }; + worker.killed = true; // suppress unrelated pre-ready user notification + worker.exitCode = 1; + + worker.emit('exit', 1, null); + + expect(ds.worker).toBeNull(); + expect(ds.riffShutdownState).toEqual({ + phase: 'prepared', + requestId: 'shutdown-request', + taskId: 'task-drained-but-unverified', + }); + + Object.assign(ds.session, { + queued: true, + queuedActivationPending: true, + queuedActivationToken: 'exact-retained-token', + queuedActivationInput: { content: 'EXACT_RETAINED_OPENING' }, + queuedActivationTurnId: 'turn-retained', + }); + const journalBefore = structuredClone({ + queued: ds.session.queued, + pending: ds.session.queuedActivationPending, + token: ds.session.queuedActivationToken, + input: ds.session.queuedActivationInput, + turnId: ds.session.queuedActivationTurnId, + }); + vi.mocked(sessionStore.updateSession).mockClear(); + + expect(() => forkWorker(ds, { content: 'EXACT_RETAINED_OPENING' }, { + turnId: 'turn-retained', + })).toThrow(/riff_retirement_fence:shutdown-prepared/); + expect(forkMock).toHaveBeenCalledTimes(1); + expect({ + queued: ds.session.queued, + pending: ds.session.queuedActivationPending, + token: ds.session.queuedActivationToken, + input: ds.session.queuedActivationInput, + turnId: ds.session.queuedActivationTurnId, + }).toEqual(journalBefore); + expect(sessionStore.updateSession).not.toHaveBeenCalled(); + }); + + it('retains the exact worker pointer across error until its later exit is observable', () => { + const ds = makeDs(); + ds.session.backendType = 'riff'; + forkWorker(ds, 'hello', false); + const worker = forkMock.mock.results.at(-1)!.value; + ds.riffShutdownState = { + phase: 'prepared', + requestId: 'shutdown-request-error', + taskId: 'task-verified-before-error', + }; + worker.killed = true; + + worker.emit('error', new Error('ipc failure')); + expect(ds.worker).toBe(worker); + expect(ds.riffShutdownState).toMatchObject({ requestId: 'shutdown-request-error' }); + + worker.exitCode = 1; + worker.emit('exit', 1, null); + expect(ds.worker).toBeNull(); + expect(ds.riffShutdownState).toMatchObject({ requestId: 'shutdown-request-error' }); + }); +}); + describe('Codex App clean-input feature gate', () => { const payload = { content: 'legacy', @@ -264,13 +349,56 @@ describe('Codex App clean-input feature gate', () => { ds.session.vcMeetingImTurnOrigins = { om_vc_im: origin }; expect(sendWorkerInput(ds, payload, 'om_vc_im')).toBe(true); - expect(worker.send).toHaveBeenCalledWith({ + expect(worker.send).toHaveBeenCalledWith(expect.objectContaining({ type: 'message', content: payload.content, codexAppInput: { text: 'clean', clientUserMessageId: 'om_vc_im' }, turnId: 'om_vc_im', + codexAppDispatchId: expect.any(String), vcMeetingImTurnOrigin: origin, - }); + })); + }); + + it('freezes non-Lark delivery sinks into every accepted Codex App entry', () => { + vi.mocked(getBot).mockImplementation(() => defaultBot({ cliId: 'codex-app' })); + const cases = [ + { + sink: 'doc_comment', + prepare: (ds: any, turnId: string) => { + ds.session.docCommentTargets = { + [turnId]: { fileToken: 'doc', fileType: 'docx', commentId: 'comment', turnId }, + }; + }, + }, + { + sink: 'http_wait', + prepare: (ds: any, turnId: string) => { + ds.pendingWaitPromises = new Map([[turnId, { resolve: vi.fn() }]]); + }, + }, + { + sink: 'http_async', + prepare: (ds: any, turnId: string) => { + ds.asyncTriggerResults = new Map([[turnId, { status: 'pending', createdAt: Date.now() }]]); + }, + }, + { + sink: 'suppressed', + prepare: (ds: any, turnId: string) => { + ds.suppressedFinalOutputTurns = new Map([[turnId, 1]]); + }, + }, + ] as const; + + for (const [index, fixture] of cases.entries()) { + const turnId = `turn-sink-${index}`; + const ds = makeDs({ worker: makeFakeWorker() }); + fixture.prepare(ds, turnId); + expect(sendWorkerInput(ds, `payload-${index}`, turnId, { + ...(fixture.sink === 'suppressed' ? { dispatchAttempt: 1 } : {}), + })).toBe(true); + expect(ds.session.codexAppDispatchLedger?.[0]?.deliverySink).toBe(fixture.sink); + } }); it('resolves explicit meeting IM origin while keeping the clean cold-fork sidecar', () => { @@ -326,7 +454,7 @@ describe('Codex App clean-input feature gate', () => { expect(ds.managedTurnOrigin).toBeUndefined(); }); - it('does not lend a previous human turn to a non-empty system prompt', () => { + it('mints a durable identity without borrowing previous-turn routing for a no-id fork', () => { vi.mocked(getBot).mockImplementation(() => defaultBot({ cliId: 'codex-app', codexAppCleanInput: true })); const ds = makeDs({ currentReplyTarget: { @@ -349,12 +477,888 @@ describe('Codex App clean-input feature gate', () => { const init = vi.mocked(worker.send).mock.calls[0][0]; expect(init).toEqual(expect.objectContaining({ prompt: payload.content, - promptCodexAppInput: { text: 'clean' }, + promptCodexAppInput: { + text: 'clean', + clientUserMessageId: expect.stringMatching(/^codex-app-dispatch-/), + }, + turnId: expect.stringMatching(/^codex-app-dispatch-/), + codexAppDispatchId: expect.any(String), })); - expect(init.turnId).toBeUndefined(); + expect(init.promptCodexAppInput.clientUserMessageId).toBe(init.turnId); + expect(init.replyTurnId).toBeUndefined(); expect(init.vcMeetingImTurnOrigin).toBeUndefined(); }); + it('mints distinct persisted identities for consecutive live no-id inputs without losing reply routing', () => { + vi.mocked(getBot).mockImplementation(() => defaultBot({ cliId: 'codex-app', codexAppCleanInput: true })); + const worker = makeFakeWorker(); + const ds = makeDs({ + worker, + currentReplyTarget: { + rootMessageId: 'om_root', + turnId: 'om_route', + updatedAt: new Date().toISOString(), + }, + }); + + expect(sendWorkerInput(ds, { content: 'scheduler one', codexAppInput: { text: 'one' } })).toBe(true); + expect(sendWorkerInput(ds, { content: 'scheduler two', codexAppInput: { text: 'two' } })).toBe(true); + + const messages = vi.mocked(worker.send).mock.calls.map(call => call[0]); + expect(messages).toHaveLength(2); + expect(messages[0]).toEqual(expect.objectContaining({ + turnId: expect.stringMatching(/^codex-app-dispatch-/), + replyTurnId: 'om_route', + codexAppDispatchId: expect.any(String), + })); + expect(messages[1]).toEqual(expect.objectContaining({ + turnId: expect.stringMatching(/^codex-app-dispatch-/), + replyTurnId: 'om_route', + codexAppDispatchId: expect.any(String), + })); + expect(messages[1].turnId).not.toBe(messages[0].turnId); + expect(ds.session.codexAppDispatchLedger).toHaveLength(2); + expect(ds.session.codexAppDispatchLedger?.map(entry => entry.replyTurnId)).toEqual([ + 'om_route', + 'om_route', + ]); + }); + + it('copies the inbound turn registry into the durable ledger after mutable reply state advances', () => { + vi.mocked(getBot).mockImplementation(() => defaultBot({ cliId: 'codex-app' })); + const worker = makeFakeWorker(); + const ds = makeDs({ + worker, + scope: 'chat', + currentReplyTarget: { + rootMessageId: 'om_topic_b', turnId: 'turn-b', updatedAt: new Date().toISOString(), + }, + }); + ds.session.scope = 'chat'; + ds.session.cliId = 'codex-app'; + ds.session.currentReplyTarget = ds.currentReplyTarget; + ds.session.turnReplyContexts = { + 'turn-a': { + target: { mode: 'thread', rootMessageId: 'om_topic_a' }, + quoteTargetId: 'turn-a', + replyTargetSenderOpenId: 'ou_a', + }, + 'turn-b': { target: { mode: 'thread', rootMessageId: 'om_topic_b' } }, + }; + + expect(sendWorkerInput(ds, 'late A input', 'turn-a')).toBe(true); + expect(ds.session.codexAppDispatchLedger?.[0]?.replyTarget) + .toEqual({ mode: 'thread', rootMessageId: 'om_topic_a' }); + expect(ds.session.codexAppDispatchLedger?.[0]).toMatchObject({ + quoteTargetId: 'turn-a', replyTargetSenderOpenId: 'ou_a', + }); + }); + + it('rejects an empty double-fork before mutating or killing a ledger-owned live worker', () => { + vi.mocked(getBot).mockImplementation(() => defaultBot({ cliId: 'codex-app' })); + const worker = makeFakeWorker(); + const ds = makeDs({ worker }); + ds.session.cliId = 'codex-app'; + ds.session.queued = true; + ds.session.codexAppDispatchLedger = [ + { dispatchId: 'old', turnId: 'turn-old', state: 'prepared', content: 'old' }, + ]; + + forkWorker(ds, '', true); + + expect(forkMock).not.toHaveBeenCalled(); + expect(worker.send).not.toHaveBeenCalled(); + expect(worker.kill).not.toHaveBeenCalled(); + expect(ds.worker).toBe(worker); + expect(ds.session.queued).toBe(true); + expect(sessionStore.updateSession).not.toHaveBeenCalled(); + }); + + it('routes a non-empty double-fork into the existing durable FIFO without replacing its worker', () => { + vi.mocked(getBot).mockImplementation(() => defaultBot({ cliId: 'codex-app', codexAppCleanInput: true })); + const worker = makeFakeWorker(); + const ds = makeDs({ worker }); + ds.session.cliId = 'codex-app'; + ds.session.codexAppDispatchLedger = [ + { dispatchId: 'old', turnId: 'turn-old', state: 'prepared', content: 'old' }, + ]; + + forkWorker(ds, { content: 'next', codexAppInput: { text: 'next clean' } }, { + resume: true, + turnId: 'turn-next', + dispatchAttempt: 2, + }); + + expect(forkMock).not.toHaveBeenCalled(); + expect(worker.kill).not.toHaveBeenCalled(); + expect(worker.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'message', + content: 'next', + turnId: 'turn-next', + dispatchAttempt: 2, + codexAppDispatchId: expect.any(String), + })); + expect(ds.session.codexAppDispatchLedger?.map(entry => entry.turnId)) + .toEqual(['turn-old', 'turn-next']); + }); + + it('stages a non-Codex double-fork behind a tokened activation without live IPC', () => { + vi.mocked(getBot).mockImplementation(() => defaultBot({ cliId: 'claude-code' })); + const worker = makeFakeWorker(); + const ds = makeDs({ worker, initialStartPending: true }); + Object.assign(ds.session, { + cliId: 'claude-code', + queuedActivationPending: true, + queuedActivationToken: 'opening-token', + queuedActivationInput: { content: 'OPENING_N' }, + queuedActivationTurnId: 'turn-opening', + }); + + forkWorker(ds, { content: 'FOLLOWER_N1' }, { + resume: true, + turnId: 'turn-follower', + dispatchAttempt: 3, + }); + + expect(forkMock).not.toHaveBeenCalled(); + expect(worker.send).not.toHaveBeenCalled(); + expect(worker.kill).not.toHaveBeenCalled(); + expect(ds.session.queuedActivationTail).toEqual([ + expect.objectContaining({ + order: 1, + turnId: 'turn-follower', + dispatchAttempt: 3, + userPrompt: 'FOLLOWER_N1', + cliInput: { content: 'FOLLOWER_N1' }, + }), + ]); + expect(sessionStore.updateSession).toHaveBeenCalledWith(ds.session); + }); + + it('rejects a gated non-Codex double-fork when exact-tail persistence fails', () => { + vi.mocked(getBot).mockImplementation(() => defaultBot({ cliId: 'claude-code' })); + const worker = makeFakeWorker(); + const ds = makeDs({ worker, initialStartPending: true }); + Object.assign(ds.session, { + cliId: 'claude-code', + queuedActivationPending: true, + queuedActivationToken: 'opening-token', + queuedActivationInput: { content: 'OPENING_N' }, + queuedActivationTurnId: 'turn-opening', + }); + vi.mocked(sessionStore.updateSession).mockImplementationOnce(() => { + throw new Error('tail store unavailable'); + }); + + expect(() => forkWorker(ds, 'FOLLOWER_MUST_NOT_SEND', { turnId: 'turn-follower' })) + .toThrow('tail store unavailable'); + + expect(forkMock).not.toHaveBeenCalled(); + expect(worker.send).not.toHaveBeenCalled(); + expect(worker.kill).not.toHaveBeenCalled(); + expect(ds.session.queuedActivationTail).toBeUndefined(); + expect(ds.session.queuedActivationPending).toBe(true); + }); + + it('rolls back an exact tail promotion when its single durable write fails', () => { + vi.mocked(getBot).mockImplementation(() => defaultBot({ cliId: 'claude-code' })); + const ds = makeDs({ hasHistory: true, initialStartPending: true }); + ds.session.cliId = 'claude-code'; + ds.session.queuedActivationTail = [{ + id: 'tail-promote-1', + order: 1, + userPrompt: 'PROMOTE_ME', + cliInput: { content: 'PROMOTE_ME' }, + turnId: 'turn-promote', + dispatchAttempt: 4, + }]; + ds.session.queuedActivationTailNextOrder = 1; + vi.mocked(sessionStore.updateSession).mockImplementationOnce(() => { + throw new Error('promotion store unavailable'); + }); + + expect(promoteQueuedActivationTail(ds, { send: false })).toBe(false); + + expect(ds.session.queuedActivationPending).toBeUndefined(); + expect(ds.session.queuedActivationToken).toBeUndefined(); + expect(ds.session.queuedActivationInput).toBeUndefined(); + expect(ds.session.queuedActivationTail).toEqual([expect.objectContaining({ + id: 'tail-promote-1', + turnId: 'turn-promote', + dispatchAttempt: 4, + cliInput: { content: 'PROMOTE_ME' }, + })]); + expect(ds.pendingPrompt).toBeUndefined(); + }); + + it('promotes the admitted clean sidecar exactly even after the live config gate flips off', () => { + let cleanInputEnabled = true; + vi.mocked(getBot).mockImplementation(() => defaultBot({ + cliId: 'codex-app', + codexAppCleanInput: cleanInputEnabled, + })); + const worker = makeFakeWorker(); + const ds = makeDs({ worker, initialStartPending: true, hasHistory: true }); + Object.assign(ds.session, { + cliId: 'codex-app', + queuedActivationPending: true, + queuedActivationToken: 'opening-token', + queuedActivationInput: { content: 'OPENING_N' }, + queuedActivationTurnId: 'turn-opening', + }); + const admittedSidecar = { + text: 'FOLLOWER_CLEAN_N1', + additionalContext: { + hidden: { kind: 'application' as const, value: 'exact' }, + }, + }; + + expect(sendWorkerInput(ds, { + content: 'FOLLOWER_LEGACY_N1', + codexAppInput: admittedSidecar, + }, 'turn-follower', { dispatchAttempt: 5 })).toBe(true); + expect(worker.send).not.toHaveBeenCalled(); + expect(ds.session.queuedActivationTail?.[0]?.cliInput.codexAppInput).toEqual({ + ...admittedSidecar, + clientUserMessageId: 'turn-follower', + }); + + // Model the opening ACK, then flip the immediate setting before N+1 is + // promoted. The persisted entry—not current config—is authoritative. + Object.assign(ds.session, { + queuedActivationPending: undefined, + queuedActivationToken: undefined, + queuedActivationInput: undefined, + queuedActivationTurnId: undefined, + queuedActivationDispatchAttempt: undefined, + queuedActivationResume: undefined, + }); + cleanInputEnabled = false; + + expect(promoteQueuedActivationTail(ds)).toBe(true); + + const exactSidecar = { + ...admittedSidecar, + clientUserMessageId: 'turn-follower', + }; + expect(ds.session.queuedActivationInput?.codexAppInput).toEqual(exactSidecar); + expect(ds.session.codexAppDispatchLedger?.at(-1)?.codexAppInput).toEqual(exactSidecar); + expect(worker.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'message', + turnId: 'turn-follower', + dispatchAttempt: 5, + codexAppInput: exactSidecar, + queuedActivationToken: expect.any(String), + })); + }); + + it('retries the exact pre-init activation sidecar after the live gate flips off', () => { + let cleanInputEnabled = true; + vi.mocked(getBot).mockImplementation(() => defaultBot({ + cliId: 'codex-app', + codexAppCleanInput: cleanInputEnabled, + })); + const ds = makeDs(); + Object.assign(ds.session, { + cliId: 'codex-app', + queued: true, + queuedPrompt: 'OPENING_LEGACY', + queuedCodexAppText: 'OPENING_CLEAN', + }); + const exactSidecar = { + text: 'OPENING_CLEAN', + additionalContext: { + hidden: { kind: 'application' as const, value: 'retry-exact' }, + }, + clientUserMessageId: 'turn-exact-retry', + }; + const failingWorker = makeFakeWorker(); + failingWorker.send = vi.fn(() => { + throw new Error('init IPC rejected before acceptance'); + }); + forkMock.mockReturnValueOnce(failingWorker); + + expect(() => forkWorker(ds, { + content: 'OPENING_LEGACY', + codexAppInput: { + text: exactSidecar.text, + additionalContext: exactSidecar.additionalContext, + }, + }, { turnId: 'turn-exact-retry', dispatchAttempt: 7 })) + .toThrow('init IPC rejected before acceptance'); + + expect(ds.session.queued).toBe(true); + expect(ds.session.queuedActivationInput?.codexAppInput).toEqual(exactSidecar); + expect(ds.session.codexAppDispatchLedger).toEqual([]); + + cleanInputEnabled = false; + const retained = ds.session.queuedActivationInput!; + forkWorker(ds, retained, { + turnId: ds.session.queuedActivationTurnId, + dispatchAttempt: ds.session.queuedActivationDispatchAttempt, + }); + + const retryWorker = forkMock.mock.results.at(-1)!.value; + const retryInit = vi.mocked(retryWorker.send).mock.calls[0]![0]; + expect(retryInit.promptCodexAppInput).toEqual(exactSidecar); + expect(ds.session.queuedActivationInput?.codexAppInput).toEqual(exactSidecar); + expect(ds.session.codexAppDispatchLedger?.at(-1)?.codexAppInput).toEqual(exactSidecar); + }); + + it('rolls back only the new double-fork acceptance when existing-worker IPC fails', () => { + vi.mocked(getBot).mockImplementation(() => defaultBot({ cliId: 'codex-app' })); + const worker = makeFakeWorker(); + worker.send = vi.fn(() => { throw new Error('ipc failed'); }); + const ds = makeDs({ worker }); + ds.session.cliId = 'codex-app'; + const oldEntry = { dispatchId: 'old', turnId: 'turn-old', state: 'accepted' as const, content: 'old' }; + ds.session.codexAppDispatchLedger = [oldEntry]; + + forkWorker(ds, 'next', { turnId: 'turn-next' }); + + expect(forkMock).not.toHaveBeenCalled(); + expect(worker.kill).not.toHaveBeenCalled(); + expect(ds.worker).toBe(worker); + expect(ds.session.codexAppDispatchLedger).toEqual([oldEntry]); + }); + + it('persists queued dequeue and Codex App acceptance until the exact submission ACK', async () => { + vi.mocked(getBot).mockImplementation(() => defaultBot({ cliId: 'codex-app' })); + const oldEntry = { + dispatchId: 'dispatch-existing', + turnId: 'turn-existing', + state: 'accepted' as const, + content: 'existing FIFO item', + }; + const ds = makeDs(); + Object.assign(ds.session, { + cliId: 'codex-app', + queued: true, + queuedPrompt: 'queued opening', + queuedCodexAppText: 'queued clean input', + queuedCodexAppMessageContext: 'queued context', + codexAppDispatchLedger: [oldEntry], + }); + const persisted: Array = []; + vi.mocked(sessionStore.updateSession).mockImplementation(session => { + persisted.push(structuredClone(session)); + }); + + forkWorker(ds, 'queued opening', { turnId: 'turn-queued' }); + + const firstDequeued = persisted.find(snapshot => snapshot.queued === false); + expect(firstDequeued).toMatchObject({ + queued: false, + queuedActivationPending: true, + queuedActivationToken: expect.any(String), + queuedActivationInput: { content: 'queued opening' }, + queuedActivationTurnId: 'turn-queued', + queuedPrompt: 'queued opening', + queuedCodexAppText: 'queued clean input', + queuedCodexAppMessageContext: 'queued context', + codexAppDispatchLedger: [ + oldEntry, + expect.objectContaining({ + turnId: 'turn-queued', + state: 'accepted', + queuedActivationToken: expect.any(String), + }), + ], + }); + expect(persisted).not.toContainEqual(expect.objectContaining({ + queued: false, + codexAppDispatchLedger: [oldEntry], + })); + expect(ds.session).toMatchObject({ + queued: false, + queuedActivationPending: true, + queuedActivationToken: expect.any(String), + queuedActivationInput: { content: 'queued opening' }, + queuedPrompt: 'queued opening', + codexAppDispatchLedger: [ + oldEntry, + expect.objectContaining({ turnId: 'turn-queued', state: 'accepted' }), + ], + }); + const worker = forkMock.mock.results.at(-1)!.value; + const init = vi.mocked(worker.send).mock.calls[0]![0]; + expect(init.queuedActivationToken).toBe(ds.session.queuedActivationToken); + worker.emit('message', { + type: 'queued_activation_submitted', + sessionId: ds.session.sessionId, + activationToken: init.queuedActivationToken, + }); + await vi.waitFor(() => expect(ds.session.queuedActivationPending).toBeUndefined()); + expect(ds.session.queuedActivationPending).toBeUndefined(); + expect(ds.session.queuedPrompt).toBeUndefined(); + expect(ds.session.queuedActivationInput).toBeUndefined(); + }); + + it('recovers a post-init crash journal through the accepted Codex FIFO until its ACK', async () => { + vi.mocked(getBot).mockImplementation(() => defaultBot({ cliId: 'codex-app' })); + const accepted = { + dispatchId: 'dispatch-accepted-before-crash', + turnId: 'turn-queued', + state: 'accepted' as const, + content: 'queued opening', + }; + const ds = makeDs(); + Object.assign(ds.session, { + cliId: 'codex-app', + queued: false, + queuedActivationPending: true, + queuedPrompt: 'queued opening', + queuedCodexAppText: 'queued clean input', + queuedCodexAppMessageContext: 'queued context', + codexAppDispatchLedger: [accepted], + }); + const persisted: Array = []; + vi.mocked(sessionStore.updateSession).mockImplementation(session => { + persisted.push(structuredClone(session)); + }); + + forkWorker(ds, '', true); + + const worker = forkMock.mock.results.at(-1)!.value; + const init = vi.mocked(worker.send).mock.calls[0][0]; + expect(init).toMatchObject({ + prompt: '', + resume: true, + queuedActivationToken: expect.any(String), + codexAppRecoveredDispatches: [expect.objectContaining({ + ...accepted, + queuedActivationToken: expect.any(String), + })], + }); + expect(init).not.toHaveProperty('codexAppDispatchId'); + expect(ds.session.codexAppDispatchLedger).toEqual([accepted]); + expect(ds.session.queued).toBe(false); + expect(ds.session.queuedActivationPending).toBe(true); + expect(ds.session.queuedPrompt).toBe('queued opening'); + worker.emit('message', { + type: 'queued_activation_submitted', + sessionId: ds.session.sessionId, + activationToken: init.queuedActivationToken, + }); + await vi.waitFor(() => expect(ds.session.queuedActivationPending).toBeUndefined()); + expect(ds.session.queuedActivationPending).toBeUndefined(); + expect(ds.session.queuedPrompt).toBeUndefined(); + expect(persisted).toContainEqual(expect.objectContaining({ + queued: false, + queuedActivationPending: undefined, + queuedPrompt: undefined, + codexAppDispatchLedger: [accepted], + })); + }); + + it('durably restores a queued payload and preserves the prior FIFO when child fork throws', () => { + vi.mocked(getBot).mockImplementation(() => defaultBot({ + cliId: 'codex-app', + codexAppCleanInput: true, + })); + const oldEntry = { + dispatchId: 'dispatch-existing', + turnId: 'turn-existing', + state: 'accepted' as const, + content: 'existing FIFO item', + }; + const ds = makeDs(); + Object.assign(ds.session, { + cliId: 'codex-app', + queued: true, + queuedPrompt: 'wrapped opening', + queuedCodexAppText: 'clean opening', + queuedCodexAppMessageContext: 'queued', + codexAppDispatchLedger: [oldEntry], + }); + const persisted: Array = []; + vi.mocked(sessionStore.updateSession).mockImplementation(session => { + persisted.push(structuredClone(session)); + }); + forkMock.mockImplementationOnce(() => { throw new Error('synchronous fork failed'); }); + + expect(() => forkWorker(ds, { + content: 'wrapped opening', + codexAppInput: { text: 'clean opening' }, + }, { turnId: 'turn-queued' })).toThrow('synchronous fork failed'); + + expect(ds.worker).toBeNull(); + expect(ds.session).toMatchObject({ + queued: true, + queuedPrompt: 'wrapped opening', + queuedCodexAppText: 'clean opening', + queuedCodexAppMessageContext: 'queued', + }); + expect(ds.session.codexAppDispatchLedger).toEqual([oldEntry]); + expect(persisted.at(-1)).toMatchObject({ + queued: true, + queuedPrompt: 'wrapped opening', + queuedCodexAppText: 'clean opening', + queuedCodexAppMessageContext: 'queued', + codexAppDispatchLedger: [oldEntry], + }); + }); + + it('fences the child and atomically restores queued/FIFO state when init IPC throws', () => { + vi.mocked(getBot).mockImplementation(() => defaultBot({ cliId: 'codex-app' })); + const oldEntry = { + dispatchId: 'dispatch-existing', + turnId: 'turn-existing', + state: 'prepared' as const, + content: 'existing FIFO item', + }; + const ds = makeDs(); + Object.assign(ds.session, { + cliId: 'codex-app', + queued: true, + queuedPrompt: 'queued opening', + queuedCodexAppText: 'queued clean input', + queuedCodexAppMessageContext: 'queued context', + codexAppDispatchLedger: [oldEntry], + }); + const persisted: Array = []; + vi.mocked(sessionStore.updateSession).mockImplementation(session => { + persisted.push(structuredClone(session)); + }); + const worker = makeFakeWorker(); + worker.send = vi.fn(() => { throw new Error('synchronous init IPC failed'); }); + forkMock.mockReturnValueOnce(worker); + + expect(() => forkWorker(ds, 'queued opening', { turnId: 'turn-queued' })) + .toThrow('synchronous init IPC failed'); + + expect(worker.kill).toHaveBeenCalledOnce(); + expect(ds.worker).toBeNull(); + expect(ds.session).toMatchObject({ + queued: true, + queuedPrompt: 'queued opening', + queuedCodexAppText: 'queued clean input', + queuedCodexAppMessageContext: 'queued context', + }); + expect(ds.session.codexAppDispatchLedger).toEqual([oldEntry]); + expect(persisted.at(-1)).toMatchObject({ + queued: true, + queuedPrompt: 'queued opening', + codexAppDispatchLedger: [oldEntry], + }); + }); + + it('retains the exact Codex activation journal after init IPC until submission is proved', async () => { + vi.mocked(getBot).mockImplementation(() => defaultBot({ cliId: 'codex-app' })); + const ds = makeDs(); + Object.assign(ds.session, { + cliId: 'codex-app', + queued: true, + queuedPrompt: 'queued opening', + queuedCodexAppText: 'queued clean input', + codexAppDispatchLedger: [], + }); + + forkWorker(ds, 'queued opening', { turnId: 'turn-queued' }); + const worker = forkMock.mock.results.at(-1)!.value; + worker.emit('error', new Error('async spawn error after send returned')); + await Promise.resolve(); + + expect(ds.session.queued).toBe(false); + expect(ds.session.queuedActivationPending).toBe(true); + expect(ds.session.queuedActivationToken).toEqual(expect.any(String)); + expect(ds.session.queuedActivationInput).toEqual({ content: 'queued opening' }); + expect(ds.session.queuedPrompt).toBe('queued opening'); + expect(ds.session.codexAppDispatchLedger).toEqual([ + expect.objectContaining({ turnId: 'turn-queued', state: 'accepted' }), + ]); + expect(ds.worker).toBeNull(); + expect(ds.initialStartPending).toBe(false); + expect(worker.kill).toHaveBeenCalledOnce(); + }); + + it.each(['error', 'exit'] as const)( + 'retains the exact non-Codex activation journal when its worker emits %s before ACK', + async event => { + const ds = makeDs(); + Object.assign(ds.session, { + cliId: 'codex', + queued: true, + queuedPrompt: 'BACKLOG_AND_TRIGGER_REPLY', + }); + + forkWorker(ds, { content: 'BACKLOG_AND_TRIGGER_REPLY' }, { + turnId: 'turn-trigger-reply', + dispatchAttempt: 3, + }); + const worker = forkMock.mock.results.at(-1)!.value; + if (event === 'error') worker.emit('error', new Error('pre-ACK worker error')); + else worker.emit('exit', 1, null); + await Promise.resolve(); + + expect(ds.worker).toBeNull(); + expect(ds.initialStartPending).toBe(false); + expect(ds.session).toMatchObject({ + queued: false, + queuedPrompt: 'BACKLOG_AND_TRIGGER_REPLY', + queuedActivationPending: true, + queuedActivationToken: expect.any(String), + queuedActivationInput: { content: 'BACKLOG_AND_TRIGGER_REPLY' }, + queuedActivationTurnId: 'turn-trigger-reply', + queuedActivationDispatchAttempt: 3, + queuedActivationResume: false, + }); + expect(vi.mocked(sessionStore.updateSession).mock.calls.map(call => call[0])) + .toContainEqual(expect.objectContaining({ + queued: false, + queuedActivationPending: true, + queuedActivationInput: { content: 'BACKLOG_AND_TRIGGER_REPLY' }, + })); + }, + ); + + it('keeps the worker authoritative and restores its journal when ACK persistence fails', async () => { + vi.useFakeTimers({ now: 0 }); + try { + const ds = makeDs(); + Object.assign(ds.session, { + queued: true, + queuedPrompt: 'queued opening', + }); + forkWorker(ds, 'queued opening', { turnId: 'turn-queued' }); + const worker = forkMock.mock.results.at(-1)!.value; + const init = vi.mocked(worker.send).mock.calls[0]![0]; + vi.mocked(sessionStore.updateSession).mockImplementationOnce(() => { + throw new Error('ACK journal write failed'); + }); + + worker.emit('message', { + type: 'queued_activation_submitted', + sessionId: ds.session.sessionId, + activationToken: init.queuedActivationToken, + }); + await Promise.resolve(); + + expect(ds.worker).toBe(worker); + expect(worker.kill).not.toHaveBeenCalled(); + expect(ds.hasHistory).toBe(false); + expect(ds.initialStartPending).toBe(true); + expect(ds.session).toMatchObject({ + queued: false, + queuedActivationPending: true, + queuedActivationToken: init.queuedActivationToken, + queuedActivationInput: { content: 'queued opening' }, + queuedPrompt: 'queued opening', + }); + + await vi.advanceTimersByTimeAsync(100); + expect(ds.hasHistory).toBe(true); + expect(ds.initialStartPending).toBe(false); + expect(ds.session.queuedActivationPending).toBeUndefined(); + expect(ds.session.queuedActivationToken).toBeUndefined(); + expect(ds.session.queuedActivationInput).toBeUndefined(); + } finally { + vi.useRealTimers(); + } + }); + + it('commits Riff lineage with journal clear atomically before releasing the queued tail', async () => { + vi.useFakeTimers({ now: 0 }); + try { + vi.mocked(getBot).mockImplementation(() => defaultBot({ + cliId: 'riff', + backendType: 'riff', + })); + const release = vi.fn((session: DaemonSession) => { + session.initialStartPending = false; + return true; + }); + initWorkerPool({ + sessionReply: vi.fn(async () => 'om_reply'), + getSessionWorkingDir: () => '/repo', + getActiveCount: () => 1, + closeSession: vi.fn(), + onQueuedActivationSubmitted: release, + }); + const ds = makeDs(); + Object.assign(ds.session, { + cliId: 'riff', + backendType: 'riff', + queued: true, + queuedPrompt: 'queued Riff opening', + riffParentTaskId: 'riff-parent-before', + }); + + forkWorker(ds, 'queued Riff opening', { turnId: 'turn-riff-queued' }); + const worker = forkMock.mock.results.at(-1)!.value; + const init = vi.mocked(worker.send).mock.calls[0]![0]; + vi.mocked(sessionStore.updateSession).mockClear(); + const persisted: Array = []; + + // The lineage notification is staged only: persisting it separately + // would expose new-child + replayable-opening after a daemon crash. + worker.emit('message', { type: 'riff_task_id', taskId: 'riff-child-new' }); + await Promise.resolve(); + expect(sessionStore.updateSession).not.toHaveBeenCalled(); + expect(ds.session.riffParentTaskId).toBe('riff-parent-before'); + expect(ds.pendingRiffActivationTaskId).toBe('riff-child-new'); + + let ackWrites = 0; + vi.mocked(sessionStore.updateSession).mockImplementation(session => { + persisted.push(structuredClone(session)); + ackWrites++; + if (ackWrites === 1) throw new Error('atomic ACK persistence unavailable'); + }); + worker.emit('message', { + type: 'queued_activation_submitted', + sessionId: ds.session.sessionId, + activationToken: init.queuedActivationToken, + riffTaskId: 'riff-child-new', + }); + await Promise.resolve(); + + expect(release).not.toHaveBeenCalled(); + expect(ds.hasHistory).toBe(false); + expect(ds.session).toMatchObject({ + riffParentTaskId: 'riff-parent-before', + queuedActivationPending: true, + queuedActivationToken: init.queuedActivationToken, + queuedActivationInput: { content: 'queued Riff opening' }, + }); + + await vi.advanceTimersByTimeAsync(100); + + expect(release).toHaveBeenCalledOnce(); + expect(ds.hasHistory).toBe(true); + expect(ds.pendingRiffActivationTaskId).toBeUndefined(); + expect(ds.session.riffParentTaskId).toBe('riff-child-new'); + expect(ds.session.queuedActivationPending).toBeUndefined(); + expect(persisted.at(-1)).toMatchObject({ + riffParentTaskId: 'riff-child-new', + queuedActivationPending: undefined, + queuedActivationToken: undefined, + queuedActivationInput: undefined, + }); + } finally { + vi.useRealTimers(); + } + }); + + it('never reports retryable activation failure after init IPC accepted the worker', () => { + const enforceLiveSessionCap = vi.fn(() => { + throw new Error('post-send cap projection failed'); + }); + initWorkerPool({ + sessionReply: vi.fn(async () => 'om_reply'), + getSessionWorkingDir: () => '/repo', + getActiveCount: () => 1, + closeSession: vi.fn(), + enforceLiveSessionCap, + }); + vi.mocked(sessionStore.updateSessionPid).mockImplementationOnce(() => { + throw new Error('post-send pid persistence failed'); + }); + vi.mocked(dashboardEventBus.publish).mockImplementationOnce(() => { + throw new Error('post-send dashboard projection failed'); + }); + const ds = makeDs(); + Object.assign(ds.session, { + queued: true, + queuedPrompt: 'queued opening', + }); + + expect(() => forkWorker(ds, 'queued opening', { turnId: 'turn-queued' })) + .not.toThrow(); + expect(ds.worker).toBe(forkMock.mock.results.at(-1)!.value); + expect(ds.session).toMatchObject({ + queued: false, + queuedActivationPending: true, + queuedActivationInput: { content: 'queued opening' }, + }); + expect(enforceLiveSessionCap).toHaveBeenCalledOnce(); + }); + + it('retries only an unaccepted ACK follow-up with exponential backoff capped at five seconds', async () => { + vi.useFakeTimers({ now: 0 }); + try { + const callTimes: number[] = []; + const release = vi.fn((session: DaemonSession) => { + callTimes.push(Date.now()); + if (callTimes.length < 9) return false; + session.initialStartPending = false; + return true; + }); + initWorkerPool({ + sessionReply: vi.fn(async () => 'om_reply'), + getSessionWorkingDir: () => '/repo', + getActiveCount: () => 1, + closeSession: vi.fn(), + onQueuedActivationSubmitted: release, + }); + const ds = makeDs(); + Object.assign(ds.session, { + queued: true, + queuedPrompt: 'queued opening', + }); + + forkWorker(ds, 'queued opening', { turnId: 'turn-queued' }); + const worker = forkMock.mock.results.at(-1)!.value; + const init = vi.mocked(worker.send).mock.calls[0]![0]; + worker.emit('message', { + type: 'queued_activation_submitted', + sessionId: ds.session.sessionId, + activationToken: init.queuedActivationToken, + }); + await vi.advanceTimersByTimeAsync(0); + for (let i = 0; i < 8; i++) await vi.runOnlyPendingTimersAsync(); + + expect(release).toHaveBeenCalledTimes(9); + expect(callTimes.slice(1).map((time, i) => time - callTimes[i]!)) + .toEqual([100, 200, 400, 800, 1_600, 3_200, 5_000, 5_000]); + expect(ds.initialStartPending).toBe(false); + expect(vi.getTimerCount()).toBe(0); + } finally { + vi.useRealTimers(); + } + }); + + it.each(['accepted', 'prepared'] as const)( + 'does not restore crashed %s N beside hub replay N+1 after exact backing-missing retirement', + state => { + vi.mocked(getBot).mockImplementation(() => defaultBot({ cliId: 'codex-app' })); + const ds = makeDs(); + ds.session.cliId = 'codex-app'; + ds.session.codexAppDispatchLedger = [{ + dispatchId: 'old-dispatch', turnId: 'delivery', dispatchAttempt: 1, + state, content: 'old N', + }]; + const retired = retireCodexAppDispatchAfterBackingMissing( + ds.session.codexAppDispatchLedger, + 'delivery', + 1, + ); + expect(retired.ok).toBe(true); + if (!retired.ok) return; + ds.session.codexAppDispatchLedger = retired.ledger; + sessionStore.updateSession(ds.session); + + forkWorker(ds, 'hub replay N+1', { + resume: true, + turnId: 'delivery', + dispatchAttempt: 2, + }); + + const worker = forkMock.mock.results.at(-1)!.value; + const init = vi.mocked(worker.send).mock.calls[0][0]; + expect(init.codexAppRecoveredDispatches).toBeUndefined(); + expect(init).toEqual(expect.objectContaining({ + turnId: 'delivery', + dispatchAttempt: 2, + codexAppDispatchId: expect.not.stringMatching(/^old-dispatch$/), + })); + expect(ds.session.codexAppDispatchLedger).toEqual([ + expect.objectContaining({ + turnId: 'delivery', dispatchAttempt: 2, state: 'accepted', + }), + ]); + }, + ); + it('starts an old queued activation without a sidecar and a modern one with exactly one', () => { vi.mocked(getBot).mockImplementation(() => defaultBot({ cliId: 'codex-app', codexAppCleanInput: true })); const oldPayload = applyQueuedCodexAppLegacyFallback({ @@ -388,18 +1392,20 @@ describe('Codex App clean-input feature gate', () => { ds.session.agentFrozen = true; expect(sendWorkerInput(ds, payload, 'om_1')).toBe(true); - expect(worker.send).toHaveBeenLastCalledWith({ + expect(worker.send).toHaveBeenLastCalledWith(expect.objectContaining({ type: 'message', content: payload.content, codexAppInput: { text: 'clean', clientUserMessageId: 'om_1' }, turnId: 'om_1', - }); + codexAppDispatchId: expect.any(String), + })); bot.config.codexAppCleanInput = undefined; expect(sendWorkerInput(ds, payload, 'om_2')).toBe(true); - expect(worker.send).toHaveBeenLastCalledWith({ + expect(worker.send).toHaveBeenLastCalledWith(expect.objectContaining({ type: 'message', content: payload.content, turnId: 'om_2', - }); + codexAppDispatchId: expect.any(String), + })); }); it('never applies the sidecar to a frozen non-Codex-App session', () => { @@ -557,6 +1563,53 @@ describe('managed turn authority worker generations', () => { expect(ds.managedTurnOrigin).toBeUndefined(); }); + it('routes a Riff double-fork prompt through the existing worker without replacing it', () => { + vi.mocked(getBot).mockImplementation(() => defaultBot({ + cliId: 'riff', + backendType: 'riff', + riff: { baseUrl: 'https://riff.example' }, + })); + const oldWorker = makeFakeWorker(); + const ds = makeDs({ + worker: oldWorker, + initConfig: { backendType: 'riff' } as any, + }); + ds.session.backendType = 'riff'; + ds.session.riffParentTaskId = 'task-preserved'; + + forkWorker(ds, 'replacement', false); + + expect(oldWorker.send).toHaveBeenCalledWith({ type: 'message', content: 'replacement' }); + expect(oldWorker.send).not.toHaveBeenCalledWith({ type: 'close' }); + expect(oldWorker.kill).not.toHaveBeenCalled(); + expect(ds.worker).toBe(oldWorker); + expect(forkMock).not.toHaveBeenCalled(); + expect(ds.session.riffParentTaskId).toBe('task-preserved'); + }); + + it('refuses an empty Riff refork without touching the existing generation', () => { + vi.mocked(getBot).mockImplementation(() => defaultBot({ + cliId: 'riff', + backendType: 'riff', + riff: { baseUrl: 'https://riff.example' }, + })); + const oldWorker = makeFakeWorker(); + const ds = makeDs({ + worker: oldWorker, + initConfig: { backendType: 'riff' } as any, + }); + ds.session.backendType = 'riff'; + ds.session.riffParentTaskId = 'task-preserved'; + + forkWorker(ds, '', { resume: true }); + + expect(oldWorker.send).not.toHaveBeenCalled(); + expect(oldWorker.kill).not.toHaveBeenCalled(); + expect(ds.worker).toBe(oldWorker); + expect(forkMock).not.toHaveBeenCalled(); + expect(ds.session.riffParentTaskId).toBe('task-preserved'); + }); + it('revokes the old capability immediately when an adopt double-fork replacement fails', () => { const oldWorker = makeFakeWorker(); const ds = makeDs({ @@ -803,12 +1856,13 @@ describe('worker startup failure delivery', () => { content: 'legacy follow-up', codexAppInput: { text: 'clean follow-up' }, }, 'turn-live-clean')).toBe(true); - expect(worker.send).toHaveBeenLastCalledWith({ + expect(worker.send).toHaveBeenLastCalledWith(expect.objectContaining({ type: 'message', content: 'legacy follow-up', codexAppInput: { text: 'clean follow-up', clientUserMessageId: 'turn-live-clean' }, turnId: 'turn-live-clean', - }); + codexAppDispatchId: expect.any(String), + })); worker.emit('message', { type: 'error', message: 'CLI relaunch dependency disappeared', turnId: 'turn-live-clean' }); await Promise.resolve(); diff --git a/test/session-list-liveness.test.ts b/test/session-list-liveness.test.ts index 771feecb9..b067ec0b5 100644 --- a/test/session-list-liveness.test.ts +++ b/test/session-list-liveness.test.ts @@ -16,6 +16,13 @@ describe('botmux list session liveness', () => { )).toBe('prune_real'); }); + it('never auto-prunes an unsettled Codex App owner with no process markers', () => { + expect(sessionListDisposition( + { codexAppDispatchLedger: [{ state: 'prepared' }] }, + { hasPid: false, hasBackingSession: false }, + )).toBe('keep'); + }); + it('keeps live/backed sessions and distinguishes never-started scratch rows', () => { expect(sessionListDisposition({}, { hasPid: true, hasBackingSession: false })).toBe('keep'); expect(sessionListDisposition({}, { hasPid: false, hasBackingSession: true })).toBe('keep'); diff --git a/test/session-manager-auto-recover.test.ts b/test/session-manager-auto-recover.test.ts index c5d703cef..e62e5850c 100644 --- a/test/session-manager-auto-recover.test.ts +++ b/test/session-manager-auto-recover.test.ts @@ -43,7 +43,7 @@ describe('shouldAutoForkOnRestore', () => { describe('staggeredRecoveryFork', () => { const ds = (id: string, worker: unknown = null) => - ({ worker, session: { sessionId: id } } as unknown as DaemonSession); + ({ worker, session: { sessionId: id, status: 'active' } } as unknown as DaemonSession); it('re-forks every queued session', async () => { const forked: string[] = []; @@ -67,6 +67,20 @@ describe('staggeredRecoveryFork', () => { expect(forked).toEqual(['a', 'c']); // 'live' already has a worker — not clobbered }); + it('isolates a synchronous recovery fork failure and continues with later owners', async () => { + const forked: string[] = []; + await expect(staggeredRecoveryFork( + [ds('broken'), ds('healthy')], + current => { + if (current.session.sessionId === 'broken') throw new Error('init IPC rejected'); + forked.push(current.session.sessionId); + }, + 5, + 0, + )).resolves.toBeUndefined(); + expect(forked).toEqual(['healthy']); + }); + it('staggers in batches (delay only kicks in between batches)', async () => { const sessions = Array.from({ length: 5 }, (_, i) => ds(`s${i}`)); const forked: string[] = []; @@ -76,4 +90,25 @@ describe('staggeredRecoveryFork', () => { expect(forked).toHaveLength(5); expect(Date.now() - start).toBeGreaterThanOrEqual(30); }); + + it('rechecks exact ownership after a batch delay and never forks a replaced session', async () => { + const a = ds('a'); + const b = ds('b'); + const owned = new Set([a, b]); + const forked: string[] = []; + setTimeout(() => { + owned.delete(b); + b.session.status = 'closed'; + }, 5); + + await staggeredRecoveryFork( + [a, b], + current => forked.push(current.session.sessionId), + 1, + 20, + current => owned.has(current), + ); + + expect(forked).toEqual(['a']); + }); }); diff --git a/test/session-marker-resolve.test.ts b/test/session-marker-resolve.test.ts index c98d58e41..f69cc44d0 100644 --- a/test/session-marker-resolve.test.ts +++ b/test/session-marker-resolve.test.ts @@ -8,6 +8,8 @@ import { replaceManagedOriginCapabilityFile, } from '../src/core/managed-origin-capability.js'; +const ORIGIN_CHANNEL = 'a'.repeat(64); + // resolveSessionContext is the layer that powers session-id inference for // `botmux send` / history / bots. Regression guard: a detached/backgrounded // invocation breaks the process-tree marker walk, and before the env fallback @@ -24,10 +26,14 @@ describe('resolveSessionContext()', () => { writeFileSync(join(markersDir, String(pid)), body); } - function writeCapability(sessionId: string, body: Record): void { + function writeCapability( + sessionId: string, + body: Record, + channelId = ORIGIN_CHANNEL, + ): void { replaceManagedOriginCapabilityFile( - managedOriginCapabilityPath(dir, sessionId), - JSON.stringify({ sessionId, ...body }), + managedOriginCapabilityPath(dir, sessionId, channelId), + JSON.stringify({ sessionId, channelId, ...body }), ); } @@ -69,7 +75,7 @@ describe('resolveSessionContext()', () => { turnId: 'turn-protected', dispatchAttempt: 3, }); - expect(resolveSessionContext(dir, 'env-sid', process.pid)).toEqual({ + expect(resolveSessionContext(dir, 'env-sid', process.pid, ORIGIN_CHANNEL)).toEqual({ sessionId: 'env-sid', turnId: 'turn-protected', dispatchAttempt: 3, @@ -87,7 +93,7 @@ describe('resolveSessionContext()', () => { turnId: 'turn-residual', dispatchAttempt: 4, }); - expect(resolveSessionContext(dir, 'env-sid', process.pid)).toEqual({ + expect(resolveSessionContext(dir, 'env-sid', process.pid, ORIGIN_CHANNEL)).toEqual({ sessionId: 'env-sid', turnId: 'turn-live', dispatchAttempt: 1, @@ -105,13 +111,28 @@ describe('resolveSessionContext()', () => { turnId: 'turn-protected', dispatchAttempt: 2, }); - expect(resolveSessionContext(dir, 'env-sid', process.pid)).toEqual({ + expect(resolveSessionContext(dir, 'env-sid', process.pid, ORIGIN_CHANNEL)).toEqual({ sessionId: 'marker-sid', turnId: 'turn-marker', dispatchAttempt: 1, }); }); + it('does not read a capability snapshot from another pane channel', () => { + writeCapability('env-sid', { + capability: 'de'.repeat(32), + turnId: 'turn-other-pane', + dispatchAttempt: 5, + }); + + expect(resolveSessionContext( + dir, + 'env-sid', + process.pid, + 'b'.repeat(64), + )).toEqual({ sessionId: 'env-sid' }); + }); + it('falls back to env when the matched marker is empty/legacy (no usable sessionId)', () => { writeMarker(process.pid, ''); // legacy empty marker const ctx = resolveSessionContext(dir, 'env-sid', process.pid); diff --git a/test/session-mutation-guard.test.ts b/test/session-mutation-guard.test.ts new file mode 100644 index 000000000..73761e63f --- /dev/null +++ b/test/session-mutation-guard.test.ts @@ -0,0 +1,79 @@ +import { describe, expect, it } from 'vitest'; +import type { DaemonSession } from '../src/core/types.js'; +import type { Session } from '../src/types.js'; +import { + hasProtectedSessionMutationOwnership, + protectedSessionMutationReasons, +} from '../src/core/session-mutation-guard.js'; + +function session(overrides: Partial = {}): Session { + return { + sessionId: 'guard-session', + chatId: 'oc_chat', + rootMessageId: 'om_root', + title: 'guard', + status: 'active', + createdAt: new Date('2026-01-01T00:00:00Z').toISOString(), + ...overrides, + }; +} + +describe('hasProtectedSessionMutationOwnership', () => { + it.each([ + ['generic activation head', { queuedActivationPending: true }], + ['durable successor tail', { + queuedActivationTail: [{ + id: 'tail-1', order: 1, userPrompt: 'N+1', + cliInput: { content: 'N+1' }, turnId: 'turn-n-plus-1', + }], + }], + ['pending repository setup', { + pendingRepoSetup: { mode: 'picker', prompt: 'OPENING_N' }, + }], + ['Riff activation head', { + cliId: 'riff', backendType: 'riff', queuedActivationPending: true, + }], + ['dashboard backlog', { queued: true, queuedPrompt: 'OPENING_N' }], + ['Codex App ledger', { + cliId: 'codex-app', + codexAppDispatchLedger: [{ + dispatchId: 'dispatch-n', turnId: 'turn-n', state: 'accepted', content: 'N', + }], + }], + ] as const)('protects %s', (_name, overrides) => { + expect(hasProtectedSessionMutationOwnership(session(overrides as Partial))).toBe(true); + }); + + it('protects a runtime-only opening claim but leaves a truly idle row mutable', () => { + const ds = { + session: session(), + initialStartPending: true, + } as DaemonSession; + expect(hasProtectedSessionMutationOwnership(ds)).toBe(true); + expect(hasProtectedSessionMutationOwnership(session())).toBe(false); + }); + + it('classifies backend-neutral ownership separately from Codex App dispatches', () => { + expect(protectedSessionMutationReasons(session({ + cliId: 'traex', + queued: true, + pendingRepoSetup: { mode: 'picker', prompt: 'OPENING_N' }, + }))).toEqual(['queued_todo', 'repository_setup']); + expect(protectedSessionMutationReasons(session({ + cliId: 'codex-app', + codexAppDispatchLedger: [{ + dispatchId: 'dispatch-n', turnId: 'turn-n', state: 'accepted', content: 'N', + }], + }))).toEqual(['codex_app_dispatch']); + }); + + it.each([ + ['Riff close fence', { riffCloseState: { phase: 'prepared', requestId: 'close-1' } }], + ['Riff shutdown fence', { + riffShutdownState: { phase: 'prepared', requestId: 'shutdown-1', taskId: null }, + }], + ])('protects runtime-only %s', (_label, state) => { + const ds = { session: session(), ...state } as unknown as DaemonSession; + expect(hasProtectedSessionMutationOwnership(ds)).toBe(true); + }); +}); diff --git a/test/session-ready-cli.test.ts b/test/session-ready-cli.test.ts index 8b7061c4d..d63bb3a44 100644 --- a/test/session-ready-cli.test.ts +++ b/test/session-ready-cli.test.ts @@ -65,6 +65,7 @@ describe('botmux session-ready — isolated CLI fallback', () => { writeFileSync( join(relayDir, RELAY_ORIGIN_CAPABILITY_BASENAME), JSON.stringify({ token: capability }), + { mode: 0o600 }, ); let receivedBody = ''; diff --git a/test/session-rename-worker.test.ts b/test/session-rename-worker.test.ts index ce54a9316..3ec9ccf68 100644 --- a/test/session-rename-worker.test.ts +++ b/test/session-rename-worker.test.ts @@ -26,14 +26,21 @@ describe('worker native session rename queue', () => { const promptLoopIdx = region.indexOf('while (pendingMessages.length > 0'); expect(region).toContain('const sessionRenameReady = isPromptReady && pendingSessionRename !== null'); - expect(region).toContain('if (sessionRenameInFlight) return'); + expect(region).toContain('if (sessionRenameInFlight()) return'); expect(region).toContain('if (commandLineWritesPending > 0) return'); expect(region).toContain('const rawInputReady = isPromptReady'); - expect(region).toContain('await sendRawCommandLineSerially(backend, buildRename(title))'); + expect(region).toContain('await sendRawCommandLineSerially(renameBackend, buildRename(title))'); + expect(region).toContain("sessionRenamePhase = 'reserved'"); + expect(region).toContain("sessionRenamePhase = 'writing'"); + expect(region).toContain("sessionRenamePhase = 'sent'"); expect(region).toContain('armSessionRenameIdleTimeout()'); expect(region).toContain("effectiveBackendType === 'riff'"); expect(renameIdx).toBeGreaterThanOrEqual(0); expect(renameIdx).toBeLessThan(promptLoopIdx); + expect(region.indexOf("sessionRenamePhase = 'writing'")) + .toBeLessThan(region.indexOf('await sendRawCommandLineSerially(renameBackend')); + expect(region.indexOf('await sendRawCommandLineSerially(renameBackend')) + .toBeLessThan(region.indexOf("sessionRenamePhase = 'sent'")); }); it('blocks type-ahead messages until the rename command returns to prompt', () => { @@ -44,8 +51,10 @@ describe('worker native session rename queue', () => { const readyEnd = workerSource.indexOf('\nfunction persistCliSessionId', readyStart); const readyRegion = workerSource.slice(readyStart, readyEnd); - expect(sendToPtyRegion).toContain('!sessionRenameInFlight && commandLineWritesPending === 0 && shouldWriteNow'); - expect(readyRegion).toContain('clearSessionRenameInFlight()'); + expect(sendToPtyRegion).toContain('!sessionRenameInFlight() && commandLineWritesPending === 0 && shouldWriteNow'); + expect(readyRegion).toContain('settleSessionRenameOnPrompt()'); + expect(workerSource).toContain("if (sessionRenamePhase === 'sent') forceClearSessionRenameInFlight()"); + expect(workerSource).toContain("if (sessionRenamePhase === 'writing') sessionRenamePhase = 'sent'"); expect(workerSource).toContain('Native session rename idle timeout'); }); @@ -69,9 +78,9 @@ describe('worker native session rename queue', () => { it('serializes passthrough writes without changing their busy-delivery semantics', () => { const rawRegion = caseRegion('raw_input'); - // PR #441 起入队条件多了注入围栏（injectionFlushing / barrier），rename 围栏 - // 仍必须在场——只钉本测试关心的三个 restart/rename 因子，不钉整行。 - expect(rawRegion).toContain('if (cliRestartInProgress || rawInputRestartGate || sessionRenameInFlight'); + // PR #441 起入队条件多了注入围栏；rename 仍为函数形式的 phase gate。 + expect(rawRegion).toContain('if (cliRestartInProgress || rawInputRestartGate || sessionRenameInFlight()'); + expect(rawRegion).toContain('injectionFlushing || shouldDeferUserFlush(pendingInjections)'); expect(rawRegion).toContain('pendingRawInputs.push(msg)'); expect(rawRegion).toContain('await deliverRawInput(msg)'); @@ -82,6 +91,6 @@ describe('worker native session rename queue', () => { expect(flushRegion).toContain('await deliverRawInput(raw)'); expect(workerSource).toContain('await sendRawCommandLineSerially(targetBackend, msg.content)'); expect(flushRegion.indexOf('await deliverRawInput(raw)')) - .toBeLessThan(flushRegion.indexOf('await sendRawCommandLineSerially(backend, buildRename(title))')); + .toBeLessThan(flushRegion.indexOf('await runAdoptSessionRenameSequence({')); }); }); diff --git a/test/session-reply-thread-anchor.test.ts b/test/session-reply-thread-anchor.test.ts index a5b029359..7bca6e624 100644 --- a/test/session-reply-thread-anchor.test.ts +++ b/test/session-reply-thread-anchor.test.ts @@ -118,6 +118,18 @@ describe('sessionReply chat-scope chokepoint — shared fold-back anchoring', () expect(mocks.replyMessage).not.toHaveBeenCalled(); }); + it('honors a daemon-frozen turn-A root after mutable session state advances to turn B', async () => { + seedSharedSession({ rootMessageId: 'om_topic_b', turnId: 'turn-b', updatedAt: NOW }); + await sessionReply(CHAT, 'late A', 'text', APP, 'turn-a', { + replyTarget: { mode: 'thread', rootMessageId: 'om_topic_a' }, + }); + + expect(mocks.replyMessage).toHaveBeenCalledWith( + APP, 'om_topic_a', 'late A', 'text', true, undefined, expect.anything(), + ); + expect(mocks.sendMessage).not.toHaveBeenCalled(); + }); + it('plain chat session (no fold-back anchor) keeps replying flat to the chat top-level', async () => { seedSharedSession(undefined); await sessionReply(CHAT, 'hello', 'text', APP); diff --git a/test/session-resume.test.ts b/test/session-resume.test.ts index 8fd9f5b5d..99272f1f2 100644 --- a/test/session-resume.test.ts +++ b/test/session-resume.test.ts @@ -52,6 +52,7 @@ vi.mock('../src/core/worker-pool.js', () => ({ killStalePids: vi.fn(), getCurrentCliVersion: vi.fn(() => '1.0.0-test'), restoreUsageLimitRuntimeState: vi.fn(), + withActiveSessionKeyLock: vi.fn(async (_map: Map, _key: string, action: () => any) => action()), // Faithful: mirror the real setActiveSessionSafe — if a DIFFERENT entry // already holds the key, evict it (close) before setting, instead of a // bare overwrite that would mask a lingering occupant. @@ -61,6 +62,7 @@ vi.mock('../src/core/worker-pool.js', () => ({ for (const [k, v] of map) { if (v === prev) { map.delete(k); break; } } } map.set(key, ds); + return { accepted: true }; }), // Real predicate (same logic as production): worker OR persisted CLI markers. isRelayableRealSession: (ds: any) => @@ -318,6 +320,26 @@ describe('resumeSession', () => { expect(r.ok).toBe(true); }); + it('treats a persisted Riff task id as a real anchor conflict', async () => { + const closed = makeClosedSession({ rootMessageId: 'om_riff_conflict' }); + const sibling = sessionStore.createSession('oc_chat1', 'om_riff_conflict', 'Riff owner'); + sibling.larkAppId = 'app_test'; + sibling.scope = 'thread'; + sibling.riffParentTaskId = 'riff-task-still-live'; + sibling.cliId = undefined; + sibling.lastCliInput = undefined; + sessionStore.updateSession(sibling); + + const result = await resumeSession(closed.sessionId, new Map()); + + expect(result).toEqual({ + ok: false, + error: 'anchor_occupied', + activeSessionId: sibling.sessionId, + }); + expect(sessionStore.getSession(sibling.sessionId)?.status).toBe('active'); + }); + // ── Scratch carve-out (王皓's resume-after-/relay bug) ──────────────────── it('does NOT block on an in-memory daemon-command scratch — evicts it and resumes', async () => { @@ -382,6 +404,51 @@ describe('resumeSession', () => { // Scratch store row should now be closed. expect(sessionStore.getSession(scratch.sessionId)!.status).toBe('closed'); }); + + it('keeps a fresh first owner that appears while resume awaits scratch cleanup', async () => { + const closed = makeClosedSession({ rootMessageId: 'om_resume_race' }); + const scratch = sessionStore.createSession('oc_chat1', 'om_resume_race', '/relay'); + scratch.larkAppId = 'app_test'; + scratch.scope = 'thread'; + scratch.cliId = undefined as any; + scratch.lastCliInput = undefined as any; + sessionStore.updateSession(scratch); + const map = new Map(); + wp.registry = map; + let releaseCleanup!: () => void; + let cleanupStarted!: () => void; + const paused = new Promise(resolve => { releaseCleanup = resolve; }); + const started = new Promise(resolve => { cleanupStarted = resolve; }); + vi.mocked(closeSession).mockImplementationOnce(async (sid: string) => { + cleanupStarted(); + await paused; + sessionStore.closeSession(sid); + return { ok: true, alreadyClosed: false } as any; + }); + + const resuming = resumeSession(closed.sessionId, map); + await started; + const key = sessionKey('om_resume_race', 'app_test'); + const fresh = { + session: { sessionId: 'fresh-first-owner', status: 'active', queued: false }, + worker: null, + initialStartPending: true, + larkAppId: 'app_test', + chatId: 'oc_chat1', + scope: 'thread', + } as any; + map.set(key, fresh); + releaseCleanup(); + + const result = await resuming; + expect(result).toEqual({ + ok: false, + error: 'anchor_occupied', + activeSessionId: 'fresh-first-owner', + }); + expect(map.get(key)).toBe(fresh); + expect(sessionStore.getSession(closed.sessionId)?.status).toBe('closed'); + }); }); describe('success path', () => { @@ -446,6 +513,51 @@ describe('resumeSession', () => { expect(restored?.session.replyThreadAliases?.om_materialized_root).toBeDefined(); }); + it.each([ + ['pending repo setup', (session: any) => { + session.queued = true; + session.queuedPrompt = 'abandoned picker prompt'; + session.pendingRepoSetup = { mode: 'picker', prompt: 'abandoned picker prompt', repoCardMessageId: 'om_old_picker' }; + }], + ['tokened activation head', (session: any) => { + session.queuedActivationPending = true; + session.queuedActivationToken = 'abandoned-token'; + session.queuedActivationInput = { content: 'abandoned head' }; + session.queuedActivationTurnId = 'abandoned-turn'; + session.queuedActivationDispatchAttempt = 3; + }], + ['activation tail', (session: any) => { + session.queuedActivationTail = [{ + id: 'abandoned-tail', order: 1, userPrompt: 'tail', cliInput: { content: 'abandoned tail' }, turnId: 'tail-turn', + }]; + session.queuedActivationTailNextOrder = 2; + }], + ] as const)('never revives legacy %s when a closed row is resumed', async (_label, injectLegacyState) => { + const closed = makeClosedSession({ rootMessageId: `om_legacy_${_label.replaceAll(' ', '_')}` }); + injectLegacyState(closed); + // Simulate a row written by an older release: closed status plus queued + // ownership that the historical close path did not remove. + sessionStore.updateSession(closed); + const map = new Map(); + + const result = await resumeSession(closed.sessionId, map); + + expect(result.ok).toBe(true); + if (!result.ok) return; + const persisted = sessionStore.getSession(closed.sessionId)!; + expect(persisted.status).toBe('active'); + expect(persisted.queued).toBeUndefined(); + expect(persisted.queuedPrompt).toBeUndefined(); + expect(persisted.pendingRepoSetup).toBeUndefined(); + expect(persisted.queuedActivationPending).toBeUndefined(); + expect(persisted.queuedActivationToken).toBeUndefined(); + expect(persisted.queuedActivationInput).toBeUndefined(); + expect(persisted.queuedActivationTail).toBeUndefined(); + expect(persisted.queuedActivationTailNextOrder).toBeUndefined(); + expect(result.ds.initialStartPending).toBeFalsy(); + expect(result.ds.pendingRepo).toBeFalsy(); + }); + it('restores dedicated VC receivers without collapsing them into the ordinary chat slot', async () => { const make = (title: string, receiver?: { meetingId: string; memberId: string }) => { const s = sessionStore.createSession('oc_listener', 'oc_listener', title, 'group'); diff --git a/test/session-store.test.ts b/test/session-store.test.ts index 7315ec950..8c57a14d5 100644 --- a/test/session-store.test.ts +++ b/test/session-store.test.ts @@ -56,11 +56,16 @@ vi.mock('../src/services/frozen-card-store.js', () => ({ // Import the module under test after mocks are set up import { + __testOnly_setAfterRiffBatchRename, init, createSession, getSession, + getActiveRiffShutdownSnapshotsBatch, listSessions, closeSession, + persistActiveRiffLineagesExactBatch, + reactivateClosedSession, + RiffLineageBatchError, updateSession, updateSessionPid, findActiveSessionsByRoot, @@ -84,6 +89,7 @@ beforeEach(() => { }); afterEach(() => { + __testOnly_setAfterRiffBatchRename(undefined); try { rmSync(tempDir, { recursive: true, force: true }); } catch { /* ignore */ } }); @@ -361,6 +367,111 @@ describe('closeSession()', () => { expect(reloaded!.closedAt).toBeDefined(); }); + it('clears the non-generational worker pid atomically on close', () => { + const session = createSession('chat1', 'root1', 'Close PID'); + session.pid = process.pid; + updateSession(session); + closeSession(session.sessionId); + init(); + expect(getSession(session.sessionId)?.pid).toBeUndefined(); + }); + + it('durably abandons Codex App dispatch recovery state on explicit close', () => { + const session = createSession('chat1', 'root1', 'Close Pending Codex'); + session.codexAppDispatchLedger = [{ + dispatchId: 'dispatch-1', + turnId: 'turn-1', + state: 'prepared', + content: 'pending input', + }]; + session.codexAppGenerationCommits = [{ generation: 'generation-1', committedThrough: 3 }]; + updateSession(session); + + closeSession(session.sessionId); + init(); + + const reloaded = getSession(session.sessionId); + expect(reloaded?.status).toBe('closed'); + expect(reloaded?.codexAppDispatchLedger).toBeUndefined(); + expect(reloaded?.codexAppGenerationCommits).toBeUndefined(); + }); + + it('atomically abandons backend-neutral setup, activation head, and tail on explicit close', () => { + const session = createSession('chat1', 'root1', 'Close Generic Queue'); + session.queued = true; + session.queuedPrompt = 'backlog prompt'; + session.queuedCodexAppText = 'visible prompt'; + session.queuedCodexAppMessageContext = ''; + session.pendingRepoSetup = { mode: 'picker', prompt: 'choose repo', repoCardMessageId: 'om_picker' }; + session.queuedActivationPending = true; + session.queuedActivationToken = 'activation-token'; + session.queuedActivationInput = { content: 'prepared head' }; + session.queuedActivationTurnId = 'turn-head'; + session.queuedActivationDispatchAttempt = 2; + session.queuedActivationResume = true; + session.queuedActivationTail = [{ + id: 'tail-1', order: 1, userPrompt: 'tail', cliInput: { content: 'prepared tail' }, turnId: 'turn-tail', + }]; + session.queuedActivationTailNextOrder = 2; + updateSession(session); + + closeSession(session.sessionId); + init(); + + const reloaded = getSession(session.sessionId)!; + expect(reloaded.status).toBe('closed'); + expect(reloaded.queued).toBeUndefined(); + expect(reloaded.queuedPrompt).toBeUndefined(); + expect(reloaded.pendingRepoSetup).toBeUndefined(); + expect(reloaded.queuedActivationPending).toBeUndefined(); + expect(reloaded.queuedActivationToken).toBeUndefined(); + expect(reloaded.queuedActivationInput).toBeUndefined(); + expect(reloaded.queuedActivationTail).toBeUndefined(); + expect(reloaded.queuedActivationTailNextOrder).toBeUndefined(); + }); + + it('rolls back explicit-close state in memory when the atomic save fails', () => { + const session = createSession('chat1', 'root1', 'Close Save Failure'); + session.queuedActivationPending = true; + session.queuedActivationToken = 'keep-token'; + session.queuedActivationInput = { content: 'keep-head' }; + session.pendingRepoSetup = { mode: 'picker', prompt: 'keep setup' }; + updateSession(session); + const durableDir = tempDir; + const nonDirectory = join(durableDir, 'not-a-directory'); + writeFileSync(nonDirectory, 'block writes below this path'); + tempDir = nonDirectory; + + expect(() => closeSession(session.sessionId)).toThrow(); + expect(getSession(session.sessionId)).toMatchObject({ + status: 'active', + queuedActivationPending: true, + queuedActivationToken: 'keep-token', + queuedActivationInput: { content: 'keep-head' }, + pendingRepoSetup: { mode: 'picker', prompt: 'keep setup' }, + }); + + tempDir = durableDir; + init(); + expect(getSession(session.sessionId)).toMatchObject({ + status: 'active', + queuedActivationToken: 'keep-token', + }); + }); + + it('clears Riff lineage atomically with the durable closed row', () => { + const session = createSession('chat1', 'root1', 'Close Riff'); + session.backendType = 'riff'; + session.riffParentTaskId = 'riff-task-prepared'; + updateSession(session); + + closeSession(session.sessionId, { clearRiffParentTaskId: true }); + init(); + + expect(getSession(session.sessionId)).toMatchObject({ status: 'closed' }); + expect(getSession(session.sessionId)?.riffParentTaskId).toBeUndefined(); + }); + it('should call deleteFrozenCards with the sessionId', () => { const session = createSession('chat1', 'root1', 'Frozen'); closeSession(session.sessionId); @@ -388,6 +499,39 @@ describe('closeSession()', () => { }); }); +describe('reactivateClosedSession()', () => { + it('sanitizes queued/setup state left on a legacy closed row', () => { + const session = createSession('chat1', 'root1', 'Legacy Closed Queue'); + closeSession(session.sessionId); + const legacy = getSession(session.sessionId)!; + legacy.queued = true; + legacy.queuedPrompt = 'legacy backlog'; + legacy.pendingRepoSetup = { mode: 'picker', prompt: 'legacy picker' }; + legacy.queuedActivationPending = true; + legacy.queuedActivationToken = 'legacy-token'; + legacy.queuedActivationInput = { content: 'legacy head' }; + legacy.queuedActivationTail = [{ + id: 'legacy-tail', order: 1, userPrompt: 'tail', cliInput: { content: 'legacy tail' }, turnId: 'tail-turn', + }]; + legacy.queuedActivationTailNextOrder = 2; + updateSession(legacy); + + const result = reactivateClosedSession(session.sessionId); + expect(result.ok).toBe(true); + init(); + + const reloaded = getSession(session.sessionId)!; + expect(reloaded.status).toBe('active'); + expect(reloaded.closedAt).toBeUndefined(); + expect(reloaded.queued).toBeUndefined(); + expect(reloaded.pendingRepoSetup).toBeUndefined(); + expect(reloaded.queuedActivationPending).toBeUndefined(); + expect(reloaded.queuedActivationToken).toBeUndefined(); + expect(reloaded.queuedActivationInput).toBeUndefined(); + expect(reloaded.queuedActivationTail).toBeUndefined(); + }); +}); + // ─── updateSession() ───────────────────────────────────────────────────── describe('updateSession()', () => { @@ -479,6 +623,182 @@ describe('updateSessionPid()', () => { }); }); +// ─── Riff fleet shutdown transactions ─────────────────────────────────── + +describe('Riff fleet shutdown transactions', () => { + function createRiffSession(label: string, pid: number, taskId: string) { + const session = createSession(`chat-${label}`, `root-${label}`, `Riff ${label}`); + session.backendType = 'riff'; + session.larkAppId = 'riff-app'; + session.pid = pid; + session.riffParentTaskId = taskId; + updateSession(session); + return session; + } + + it('samples every owner and task from one fresh active projection', () => { + const first = createRiffSession('first', 101, 'task-first'); + const second = createRiffSession('second', 202, 'task-second'); + + expect(getActiveRiffShutdownSnapshotsBatch([ + first.sessionId, + second.sessionId, + ])).toEqual([ + { + sessionId: first.sessionId, + taskId: 'task-first', + owner: { pid: 101, larkAppId: 'riff-app', backendType: 'riff' }, + }, + { + sessionId: second.sessionId, + taskId: 'task-second', + owner: { pid: 202, larkAppId: 'riff-app', backendType: 'riff' }, + }, + ]); + }); + + it('bounds initial snapshot lock contention and reports a prewrite I/O failure', () => { + const session = createRiffSession('locked', 101, 'task-locked'); + const fp = join(tempDir, 'sessions.json'); + writeFileSync(`${fp}.lock`, String(process.pid)); + + const startedAt = Date.now(); + let failure: unknown; + try { + getActiveRiffShutdownSnapshotsBatch([session.sessionId], { maxWaitMs: 10 }); + } catch (error) { + failure = error; + } + + expect(failure).toBeInstanceOf(RiffLineageBatchError); + expect((failure as RiffLineageBatchError).stage).toBe('prewrite_io'); + expect((failure as RiffLineageBatchError).sessionIds).toEqual([session.sessionId]); + expect(Date.now() - startedAt).toBeLessThan(250); + }); + + it('bounds phase-2 lock contention without publishing any lineage', () => { + const session = createRiffSession('locked-phase-two', 101, 'task-before'); + const [snapshot] = getActiveRiffShutdownSnapshotsBatch([session.sessionId]); + const fp = join(tempDir, 'sessions.json'); + const beforeRaw = readFileSync(fp, 'utf-8'); + const beforeInode = statSync(fp).ino; + writeFileSync(`${fp}.lock`, String(process.pid)); + + const startedAt = Date.now(); + let failure: unknown; + try { + persistActiveRiffLineagesExactBatch([{ + ...snapshot, + expectedCurrentTaskIds: [snapshot.taskId], + targetTaskId: 'task-after', + }], { maxWaitMs: 10 }); + } catch (error) { + failure = error; + } + + expect(failure).toBeInstanceOf(RiffLineageBatchError); + expect((failure as RiffLineageBatchError).stage).toBe('prewrite_io'); + expect(readFileSync(fp, 'utf-8')).toBe(beforeRaw); + expect(statSync(fp).ino).toBe(beforeInode); + expect(Date.now() - startedAt).toBeLessThan(250); + }); + + it('rejects a last-row CAS conflict without writing any earlier row', () => { + const first = createRiffSession('first', 101, 'task-first'); + const second = createRiffSession('second', 202, 'task-second'); + const snapshots = getActiveRiffShutdownSnapshotsBatch([ + first.sessionId, + second.sessionId, + ]); + + second.riffParentTaskId = 'task-second-new-owner'; + updateSession(second); + const fp = join(tempDir, 'sessions.json'); + const beforeRaw = readFileSync(fp, 'utf-8'); + const beforeInode = statSync(fp).ino; + + let failure: unknown; + try { + persistActiveRiffLineagesExactBatch(snapshots.map((snapshot, index) => ({ + ...snapshot, + expectedCurrentTaskIds: [snapshot.taskId], + targetTaskId: `detached-${index}`, + }))); + } catch (error) { + failure = error; + } + + expect(failure).toBeInstanceOf(RiffLineageBatchError); + expect((failure as RiffLineageBatchError).stage).toBe('prewrite_ownership'); + expect((failure as RiffLineageBatchError).sessionIds).toEqual([second.sessionId]); + expect(readFileSync(fp, 'utf-8')).toBe(beforeRaw); + expect(statSync(fp).ino).toBe(beforeInode); + const durable = JSON.parse(beforeRaw); + expect(durable[first.sessionId].riffParentTaskId).toBe('task-first'); + expect(durable[second.sessionId].riffParentTaskId).toBe('task-second-new-owner'); + }); + + it('publishes and verifies all target lineages with one batch rename', () => { + const first = createRiffSession('first', 101, 'task-first'); + const second = createRiffSession('second', 202, 'task-second'); + const snapshots = getActiveRiffShutdownSnapshotsBatch([ + first.sessionId, + second.sessionId, + ]); + const fp = join(tempDir, 'sessions.json'); + const beforeInode = statSync(fp).ino; + + const verified = persistActiveRiffLineagesExactBatch(snapshots.map((snapshot, index) => ({ + ...snapshot, + expectedCurrentTaskIds: [snapshot.taskId], + targetTaskId: `detached-${index}`, + }))); + + expect(verified.map(row => row.taskId)).toEqual(['detached-0', 'detached-1']); + expect(verified.map(row => row.owner)).toEqual(snapshots.map(row => row.owner)); + expect(statSync(fp).ino).not.toBe(beforeInode); + const durable = JSON.parse(readFileSync(fp, 'utf-8')); + expect(durable[first.sessionId].riffParentTaskId).toBe('detached-0'); + expect(durable[second.sessionId].riffParentTaskId).toBe('detached-1'); + expect(getSession(first.sessionId)?.riffParentTaskId).toBe('detached-0'); + expect(getSession(second.sessionId)?.riffParentTaskId).toBe('detached-1'); + }); + + it('classifies an exact-owner readback mismatch after rename as ambiguous', () => { + const first = createRiffSession('first', 101, 'task-first'); + const second = createRiffSession('second', 202, 'task-second'); + const snapshots = getActiveRiffShutdownSnapshotsBatch([ + first.sessionId, + second.sessionId, + ]); + const fp = join(tempDir, 'sessions.json'); + __testOnly_setAfterRiffBatchRename(() => { + const projection = JSON.parse(readFileSync(fp, 'utf-8')); + projection[second.sessionId].pid = 303; + writeFileSync(fp, JSON.stringify(projection, null, 2), 'utf-8'); + }); + + let failure: unknown; + try { + persistActiveRiffLineagesExactBatch(snapshots.map((snapshot, index) => ({ + ...snapshot, + expectedCurrentTaskIds: [snapshot.taskId], + targetTaskId: `detached-${index}`, + }))); + } catch (error) { + failure = error; + } + + expect(failure).toBeInstanceOf(RiffLineageBatchError); + expect((failure as RiffLineageBatchError).stage).toBe('postrename_ambiguity'); + expect((failure as RiffLineageBatchError).sessionIds).toEqual([second.sessionId]); + const durable = JSON.parse(readFileSync(fp, 'utf-8')); + expect(durable[first.sessionId].riffParentTaskId).toBe('detached-0'); + expect(durable[second.sessionId].riffParentTaskId).toBe('detached-1'); + expect(durable[second.sessionId].pid).toBe(303); + }); +}); + // ─── Multi-bot isolation (appId scoping) ───────────────────────────────── describe('Multi-bot isolation', () => { diff --git a/test/shutdown-supervisor-contract.test.ts b/test/shutdown-supervisor-contract.test.ts new file mode 100644 index 000000000..3f5945a28 --- /dev/null +++ b/test/shutdown-supervisor-contract.test.ts @@ -0,0 +1,430 @@ +import { readFileSync } from 'node:fs'; +import { describe, expect, it } from 'vitest'; +import { + BOT_TURN_MUTATION_SHUTDOWN_ACQUIRE_TIMEOUT_MS, + DAEMON_SHUTDOWN_MAX_MS, + DAEMON_SHUTDOWN_OVERHEAD_MS, + DAEMON_WORKER_EXIT_GRACE_MS, + FLEET_DAEMON_EXIT_WAIT_MS, + FLEET_SUCCESSOR_SETTLE_MS, + PM2_DAEMON_KILL_TIMEOUT_MS, + PM2_DAEMON_RESTART_DELAY_MS, + RIFF_ADMISSION_RESTORE_TIMEOUT_MS, + RIFF_SHUTDOWN_BATCH_PERSIST_TIMEOUT_MS, + RIFF_SHUTDOWN_DRAIN_TIMEOUT_MS, + RIFF_SHUTDOWN_INITIAL_SNAPSHOT_TIMEOUT_MS, +} from '../src/core/shutdown-budgets.js'; +import { DAEMON_GRACEFUL_EXIT_CODE } from '../src/core/supervisor-shutdown-protocol.js'; + +const cli = readFileSync(new URL('../src/cli.ts', import.meta.url), 'utf8'); +const daemon = readFileSync(new URL('../src/daemon.ts', import.meta.url), 'utf8'); +const fleetShutdown = readFileSync(new URL('../src/cli/fleet-shutdown.ts', import.meta.url), 'utf8'); +const ipcServer = readFileSync(new URL('../src/core/dashboard-ipc-server.ts', import.meta.url), 'utf8'); +const pm2Preflight = readFileSync(new URL('../src/cli/pm2-preflight.ts', import.meta.url), 'utf8'); +const botsStore = readFileSync(new URL('../src/setup/bots-store.ts', import.meta.url), 'utf8'); +const bundledPm2God = readFileSync(new URL('../node_modules/pm2/lib/God.js', import.meta.url), 'utf8'); + +describe('graceful shutdown supervisor contract', () => { + it('uses a nonzero daemon-only graceful sentinel because PM2 maps signal death to zero', () => { + expect(DAEMON_GRACEFUL_EXIT_CODE).toBeGreaterThan(0); + expect(DAEMON_GRACEFUL_EXIT_CODE).toBeLessThan(256); + expect(bundledPm2God).toContain('God.handleExit(clu, code || 0, signal);'); + + const ecosystemStart = cli.indexOf('function ecosystemConfig('); + const daemonPolicy = cli.slice(ecosystemStart, cli.indexOf('const apps:', ecosystemStart)); + const dashboardPolicy = cli.slice( + cli.indexOf("name: 'botmux-dashboard'", ecosystemStart), + cli.indexOf('const cfg = { apps };', ecosystemStart), + ); + expect(daemonPolicy).toContain('stop_exit_codes: [DAEMON_GRACEFUL_EXIT_CODE]'); + expect(daemonPolicy).not.toContain('stop_exit_codes: [0]'); + expect(dashboardPolicy).toContain('stop_exit_codes: [0]'); + + const shutdownStart = daemon.indexOf('const shutdown = async () => {'); + const shutdownEnd = daemon.indexOf("process.on('SIGTERM'", shutdownStart); + const shutdown = daemon.slice(shutdownStart, shutdownEnd); + expect(shutdown).toContain('process.exit(DAEMON_GRACEFUL_EXIT_CODE);'); + expect(shutdown).not.toContain('process.exit(0);'); + expect(cli).toContain('assertDaemonPm2GracefulExitPolicy('); + expect(cli).toContain('`${operation}-handler-ready-pm2-policy`'); + + const projectionStart = cli.indexOf('function toBotmuxPm2ProcessEntry('); + const projectionEnd = cli.indexOf('function readVerifiedBotmuxPm2Projection(', projectionStart); + const projection = cli.slice(projectionStart, projectionEnd); + expect(projection).toContain( + 'stopExitCodes: normalizeRawPm2StopExitCodes(rawStopExitCodes)', + ); + expect(projection).not.toContain('.map(code => parsePm2Integer(code))'); + expect(bundledPm2God).toContain( + "stopExitCodes.map((strOrNum) => typeof strOrNum === 'string' ? parseInt(strOrNum, 10) : strOrNum)", + ); + expect(bundledPm2God).toContain('proc.pm2_env.unstable_restarts >= max_restarts'); + expect(bundledPm2God).toContain('if (!stopping && !overlimit)'); + expect(fleetShutdown).toContain('isFleetEntryProvenTerminalAfterSignal(exactState)'); + expect(fleetShutdown).toContain('post-signal terminal proof'); + expect(fleetShutdown).toContain('latestTrackedPidByName.get(trackedEntry.name) === pid'); + expect(fleetShutdown).toContain('a later missing row is never success'); + expect(fleetShutdown).toContain('liveReplacementPublished'); + expect(fleetShutdown).toContain("replacement's own"); + }); + + it('keeps outer supervisor budgets beyond both success and abort-restore paths', () => { + expect(DAEMON_SHUTDOWN_MAX_MS).toBe( + BOT_TURN_MUTATION_SHUTDOWN_ACQUIRE_TIMEOUT_MS + + RIFF_SHUTDOWN_INITIAL_SNAPSHOT_TIMEOUT_MS + + RIFF_SHUTDOWN_DRAIN_TIMEOUT_MS + + RIFF_SHUTDOWN_BATCH_PERSIST_TIMEOUT_MS + + Math.max(RIFF_ADMISSION_RESTORE_TIMEOUT_MS, DAEMON_WORKER_EXIT_GRACE_MS) + + DAEMON_SHUTDOWN_OVERHEAD_MS, + ); + expect(DAEMON_SHUTDOWN_MAX_MS).toBeLessThanOrEqual(28_000); + expect(PM2_DAEMON_KILL_TIMEOUT_MS).toBeGreaterThan(DAEMON_SHUTDOWN_MAX_MS); + expect(FLEET_DAEMON_EXIT_WAIT_MS).toBeGreaterThan(PM2_DAEMON_KILL_TIMEOUT_MS); + expect(FLEET_SUCCESSOR_SETTLE_MS).toBeGreaterThan(PM2_DAEMON_RESTART_DELAY_MS); + expect(FLEET_DAEMON_EXIT_WAIT_MS) + .toBeGreaterThan(DAEMON_SHUTDOWN_MAX_MS + FLEET_SUCCESSOR_SETTLE_MS); + expect(FLEET_DAEMON_EXIT_WAIT_MS) + .toBeGreaterThan(PM2_DAEMON_KILL_TIMEOUT_MS + FLEET_SUCCESSOR_SETTLE_MS); + }); + + it('public stop signals and polls first, never deletes a possibly refusing entry', () => { + const start = cli.indexOf('async function cmdStop()'); + const end = cli.indexOf('async function cmdRestart()', start); + const stop = cli.slice(start, end); + const signal = stop.indexOf("signalAndAwaitBotmuxProcesses(entries, 'stop')"); + const preMutationProjection = stop.indexOf( + "readVerifiedBotmuxPm2Projection('stop-before-registry-mutation')", + signal, + ); + const pm2Stop = stop.indexOf("runPm2(['stop', String(entry.pmId)])", preMutationProjection); + const justInTime = stop.indexOf( + "revalidateExactQuiescentRowBeforeMutation(\n 'stop-immediately-before-registry-mutation'", + preMutationProjection, + ); + const exactStop = stop.indexOf( + "runPm2(['stop', String(exact.pmId)]", + justInTime, + ); + + expect(signal).toBeGreaterThanOrEqual(0); + expect(preMutationProjection).toBeGreaterThan(signal); + expect(pm2Stop).toBe(-1); + expect(justInTime).toBeGreaterThan(preMutationProjection); + expect(exactStop).toBeGreaterThan(justInTime); + expect(stop).not.toContain("runPm2(['delete'"); + expect(stop).toContain('if (includePluginServices) await stopPluginServicesForCli'); + expect(stop).toContain('const stopErrors: string[] = []'); + expect(stop.indexOf("pm2Capture(['jlist'])", exactStop)).toBeGreaterThan(exactStop); + expect(stop.indexOf("assertNoUnregisteredLiveDaemonDescriptors('stop-after-registry-mutation'", exactStop)) + .toBeGreaterThan(exactStop); + expect(stop).toContain('PM2 registry mutation incomplete'); + }); + + it('restart waits for the fleet decision before any PM2 delete', () => { + const start = cli.indexOf('function deleteAllBotmuxProcesses('); + const end = cli.indexOf('/**\n * One-time migration', start); + const restart = cli.slice(start, end); + const signal = restart.indexOf("signalAndAwaitBotmuxProcesses(entries, 'restart', home"); + const justInTime = restart.indexOf( + "revalidateExactQuiescentRowBeforeMutation(\n 'restart-before-delete'", + signal, + ); + const remove = restart.indexOf("runPm2(['delete', String(exact.pmId)]", justInTime); + expect(signal).toBeGreaterThanOrEqual(0); + expect(justInTime).toBeGreaterThan(signal); + expect(remove).toBeGreaterThan(justInTime); + expect(restart).not.toContain("runPm2(['delete', ...exactIds]"); + expect(restart.indexOf("pm2Capture(['jlist'], home)", remove)).toBeGreaterThan(remove); + expect(restart.indexOf("assertNoUnregisteredLiveDaemonDescriptors(\n 'restart-after-delete'", remove)) + .toBeGreaterThan(remove); + expect(restart).toContain('PM2 delete left registry entries'); + }); + + it('takes one Riff snapshot, batch-persists, then generation-checks and commits before service stop', () => { + const start = daemon.indexOf('const shutdown = async () => {'); + const stop = daemon.indexOf('scheduler.stopScheduler();', start); + const boundedGate = daemon.indexOf('tryWithBotTurnMutation(', start); + const initialUnique = daemon.indexOf( + 'collectUniqueDaemonShutdownSessions(activeSessions.values())', + boundedGate, + ); + const prepareAll = daemon.indexOf('prepareRiffFleetForShutdown(riffCandidates', initialUnique); + const persistAll = daemon.indexOf('persistPreparedRiffShutdownFleet(riffPrepared', prepareAll); + const currentUnique = daemon.indexOf( + 'collectUniqueDaemonShutdownSessions(activeSessions.values())', + initialUnique + 1, + ); + const secondCheck = daemon.indexOf('const riffGenerationMismatch', persistAll); + const commitAll = daemon.indexOf('commitPreparedRiffShutdown(ds, result)', secondCheck); + const teardownUnique = daemon.indexOf( + 'for (const ds of currentShutdownFleet.sessions)', + commitAll, + ); + expect(boundedGate).toBeGreaterThan(start); + expect(initialUnique).toBeGreaterThan(boundedGate); + expect(prepareAll).toBeGreaterThan(initialUnique); + expect(persistAll).toBeGreaterThan(prepareAll); + expect(currentUnique).toBeGreaterThan(persistAll); + expect(secondCheck).toBeGreaterThan(currentUnique); + expect(commitAll).toBeGreaterThan(secondCheck); + expect(teardownUnique).toBeGreaterThan(commitAll); + expect(stop).toBeGreaterThan(commitAll); + expect(daemon.slice(start, stop)).toContain('abortRiffShutdownFleet('); + expect(daemon.slice(start, stop)).toContain('canAbortVerifiedExitedRiffPreparation('); + }); + + it('publishes shutdown capability only after both signal handlers are installed', () => { + const descStart = daemon.indexOf('const desc: DaemonDescriptor = {'); + const firstDescriptorWrite = daemon.indexOf('writeDaemonDescriptor(desc);', descStart); + const sigtermHandler = daemon.indexOf("process.on('SIGTERM'", firstDescriptorWrite); + const sigintHandler = daemon.indexOf("process.on('SIGINT'", sigtermHandler); + const capabilityCommit = daemon.indexOf( + 'desc.supervisorShutdownProtocol = SUPERVISOR_SHUTDOWN_PROTOCOL;', + sigintHandler, + ); + const ipcHandlerReady = daemon.indexOf('setSupervisorShutdownHandler({', sigintHandler); + const attestedWrite = daemon.indexOf('writeDaemonDescriptor(desc);', capabilityCommit); + + expect(descStart).toBeGreaterThanOrEqual(0); + expect(firstDescriptorWrite).toBeGreaterThan(descStart); + expect(daemon.slice(descStart, firstDescriptorWrite)) + .not.toContain('supervisorShutdownProtocol: SUPERVISOR_SHUTDOWN_PROTOCOL'); + expect(sigtermHandler).toBeGreaterThan(firstDescriptorWrite); + expect(sigintHandler).toBeGreaterThan(sigtermHandler); + expect(ipcHandlerReady).toBeGreaterThan(sigintHandler); + expect(capabilityCommit).toBeGreaterThan(ipcHandlerReady); + expect(attestedWrite).toBeGreaterThan(capabilityCommit); + }); + + it('uses exact-id conditional PM2 start for proven-offline compensation, never public start/restart', () => { + const start = cli.indexOf('startOffline: (offlineEntries, timeoutMs) =>'); + const end = cli.indexOf('\n list,', start); + const compensation = cli.slice(start, end); + expect(start).toBeGreaterThanOrEqual(0); + expect(compensation).toContain('runExactPm2Starts(offlineEntries'); + expect(compensation).not.toContain("runPm2(['start'"); + expect(compensation).not.toContain("runPm2(['restart'"); + }); + + it('serializes every core PM2 mutation surface on one async fleet lock', () => { + const regions = [ + ['start', 'async function cmdStart()', '/**\n * Wipe stale dashboard-daemon descriptors'], + ['stop', 'async function cmdStop()', 'async function cmdRestart()'], + ['restart', 'async function cmdRestart()', '/**\n * Bring a SINGLE bot'], + ['start-bot', 'async function ensureBotDaemonStarted(', '/**\n * `botmux start-bot'], + ] as const; + for (const [label, startMarker, endMarker] of regions) { + const start = cli.indexOf(startMarker); + const end = cli.indexOf(endMarker, start); + const region = cli.slice(start, end); + expect(start, label).toBeGreaterThanOrEqual(0); + expect(end, label).toBeGreaterThan(start); + expect(region, label).toContain('withFileLock(PM2_FLEET_MUTATION_LOCK_TARGET'); + expect(region, label).not.toContain('withFileLockSync(PM2_FLEET_MUTATION_LOCK_TARGET'); + } + + const exactHelper = cli.slice( + cli.indexOf('async function cmdInternalPm2StartExact('), + cli.indexOf('function runExactPm2Starts(', cli.indexOf('async function cmdInternalPm2StartExact(')), + ); + expect(exactHelper).toContain('BOTMUX_PM2_FLEET_LOCK_OWNER_PID'); + expect(exactHelper).toContain('PM2_FLEET_MUTATION_LOCK_TARGET}.lock'); + expect(exactHelper).toContain('lockPid !== process.ppid'); + }); + + it('fails closed before PM2 mutation on duplicate Gods, stale preflight, or unregistered descriptors', () => { + const duplicateStart = cli.indexOf('function listSingletonPm2GodDaemonPidsForMutation('); + const duplicateEnd = cli.indexOf('function runPm2(', duplicateStart); + const duplicate = cli.slice(duplicateStart, duplicateEnd); + expect(duplicate).toContain('multiple PM2 God daemons'); + expect(duplicate).not.toContain("process.kill(pid, 'SIGTERM')"); + expect(duplicate).not.toContain("process.kill(pid, 'SIGKILL')"); + + const preflightStart = cli.indexOf('function preflightNodeSanity('); + const preflightEnd = cli.indexOf('async function cmdStart()', preflightStart); + const preflight = cli.slice(preflightStart, preflightEnd); + expect(preflight).toContain('assertLinuxPm2GodExecutableUsable(pm2Pid)'); + expect(preflight).toContain('listPm2GodDaemonPids(home)'); + expect(preflight).not.toContain("join(PM2_HOME, 'pm2.pid')"); + expect(preflight).not.toContain("runPm2(['kill']"); + expect(preflight).not.toContain("'SIGKILL'"); + expect(pm2Preflight).toContain('拒绝自动清理'); + + for (const operation of ['start', 'stop', 'start-bot', 'restart-start']) { + expect(cli).toContain(`assertNoUnregisteredLiveDaemonDescriptors('${operation}'`); + } + }); + + it('publishes manual restart intent only after verified fleet retirement', () => { + const start = cli.indexOf('async function cmdRestart()'); + const end = cli.indexOf('/**\n * Bring a SINGLE bot', start); + const restart = cli.slice(start, end); + const staged = restart.indexOf('consumeRestartIntentTo('); + const preflight = restart.indexOf('assertNoDuplicatePm2GodDaemons()', staged); + const retirement = restart.indexOf('deleteAllBotmuxProcesses()'); + const descriptorCheck = restart.indexOf( + "assertNoUnregisteredLiveDaemonDescriptors('restart-start'", + retirement, + ); + const intent = restart.indexOf('writeRestartAttemptIntentTo(', descriptorCheck); + const transaction = restart.indexOf("runBoundedPm2StartTransaction(\n 'restart-start'", intent); + const newFleet = restart.indexOf("runPm2(['start', cfg]", transaction); + const verify = restart.indexOf("'restart-after-launch'", newFleet); + const compensate = restart.indexOf("rollbackPm2StartAttempt(\n 'restart-start'", verify); + const rollback = restart.indexOf('removeRestartIntentAttemptTo(', compensate); + const commit = restart.indexOf('commitRestartIntentAttemptTo(', rollback); + expect(staged).toBeGreaterThanOrEqual(0); + expect(preflight).toBeGreaterThan(staged); + expect(retirement).toBeGreaterThanOrEqual(0); + expect(descriptorCheck).toBeGreaterThan(retirement); + expect(intent).toBeGreaterThan(descriptorCheck); + expect(transaction).toBeGreaterThan(intent); + expect(newFleet).toBeGreaterThan(transaction); + expect(verify).toBeGreaterThan(newFleet); + expect(compensate).toBeGreaterThan(verify); + expect(rollback).toBeGreaterThan(compensate); + expect(commit).toBeGreaterThan(rollback); + }); + + it('bounds and freshly verifies every public PM2 start surface with compensation', () => { + expect(cli).toContain('const PM2_START_VERIFY_MIN_TIMEOUT_MS = 60_000;'); + expect(cli).toContain('pm2StartVerifyTimeoutMs(configuredNames.length)'); + const regions = [ + cli.slice(cli.indexOf('async function cmdStart()'), cli.indexOf('/**\n * Wipe stale dashboard-daemon descriptors')), + cli.slice(cli.indexOf('async function cmdRestart()'), cli.indexOf('/**\n * Bring a SINGLE bot')), + cli.slice(cli.indexOf('async function ensureBotDaemonStarted('), cli.indexOf('/**\n * `botmux start-bot')), + ]; + for (const region of regions) { + expect(region).toContain('runBoundedPm2StartTransaction('); + expect(region).toContain('PM2_START_COMMAND_TIMEOUT_MS'); + expect(region).toContain('readAndAssertConfiguredFleetOnline('); + expect(region).toContain('rollbackPm2StartAttempt('); + expect(region).toContain('timeoutMs'); + } + }); + + it('holds one bots.json generation from ecosystem rendering through verification/rollback', () => { + expect(botsStore).toContain('withFileLockSync(botsJsonPath'); + const regions = [ + cli.slice(cli.indexOf('async function cmdStart()'), cli.indexOf('/**\n * Wipe stale dashboard-daemon descriptors')), + cli.slice(cli.indexOf('async function cmdRestart()'), cli.indexOf('/**\n * Bring a SINGLE bot')), + cli.slice(cli.indexOf('async function ensureBotDaemonStarted('), cli.indexOf('/**\n * `botmux start-bot')), + ]; + for (const region of regions) { + expect(region).toContain('withFileLock(BOTS_JSON_FILE'); + expect(region).toContain('ecosystemConfig('); + expect(region).toContain('configuredCoreProcessNames('); + expect(region).toContain('assertBotsConfigSnapshotUnchanged('); + } + }); + + it('admits start-bot only through the exact configured fleet classifier', () => { + const start = cli.indexOf('async function ensureBotDaemonStarted('); + const end = cli.indexOf('/**\n * `botmux start-bot', start); + const region = cli.slice(start, end); + expect(region).toContain('classifyStartBotFleetAdmission('); + expect(region).toContain("admission.state === 'already-online'"); + const alreadyOnline = region.slice( + region.indexOf("admission.state === 'already-online'"), + region.indexOf("admission.state === 'fleet-down'"), + ); + expect(alreadyOnline).toContain("'start-bot-already-online-ready'"); + expect(alreadyOnline).toContain('readAndAssertConfiguredFleetOnline('); + expect(region).toContain("admission.state === 'fleet-down'"); + expect(region).toContain("'start-bot-after-launch'"); + expect(region).toContain('preflightNodeSanity()'); + }); + + it('requires fresh handler-ready exact-set verification before idempotent start returns', () => { + const start = cli.indexOf('async function cmdStart()'); + const end = cli.indexOf('/**\n * Wipe stale dashboard-daemon descriptors', start); + const region = cli.slice(start, end); + const liveBranch = region.slice( + region.indexOf('if (liveEntries.length > 0)'), + region.indexOf('const unprovenDormant'), + ); + const verify = liveBranch.indexOf('readAndAssertConfiguredFleetOnline('); + const ready = liveBranch.indexOf("'start-idempotent-ready'", verify); + const returns = liveBranch.indexOf('return;', ready); + expect(verify).toBeGreaterThanOrEqual(0); + expect(ready).toBeGreaterThan(verify); + expect(returns).toBeGreaterThan(ready); + expect(liveBranch).not.toContain('assertConfiguredPm2FleetOnline('); + }); + + it('discovers legacy Gods from the process table and rechecks duplicate Gods before mutation', () => { + const start = cli.indexOf('function cleanupLegacyPm2('); + const end = cli.indexOf('async function cmdStop()', start); + const legacy = cli.slice(start, end); + expect(legacy).toContain('listPm2GodDaemonPids(legacyHome)'); + expect(legacy).not.toContain("join(legacyHome, 'pm2.pid')"); + expect(legacy).toContain('assertNoDuplicatePm2GodDaemons(legacyHome)'); + expect(legacy).toContain('preflightNodeSanity(legacyHome)'); + + expect(cli).not.toContain("runPm2(['kill']"); + }); + + it('exposes an explicit double-confirmed first-upgrade bootstrap without weakening normal shutdown', () => { + const bootstrapStart = cli.indexOf('function bootstrapDeleteAllBotmuxProcesses('); + const bootstrapEnd = cli.indexOf('/**\n * One-time migration', bootstrapStart); + const bootstrap = cli.slice(bootstrapStart, bootstrapEnd); + expect(bootstrap).toContain('readSupervisorProcessStartIdentity(entry.pid)'); + expect(bootstrap).toContain('current.pid !== original.pid'); + expect(bootstrap).toContain("runPm2(\n ['delete', String(current.pmId)]"); + + const restartStart = cli.indexOf('async function cmdRestart()'); + const restartEnd = cli.indexOf('/**\n * Bring a SINGLE bot', restartStart); + const restart = cli.slice(restartStart, restartEnd); + expect(restart).toContain("process.argv.includes('--bootstrap-shutdown-protocol')"); + expect(restart).toContain("process.argv.includes('--yes')"); + expect(restart).toContain("bootstrapDeleteAllBotmuxProcesses('restart')"); + expect(restart).toContain('else deleteAllBotmuxProcesses()'); + expect(cli).toContain('botmux restart --bootstrap-shutdown-protocol --yes'); + }); + + it('rejects include-pm2 before breadcrumb/fleet mutation when a live God exists', () => { + const start = cli.indexOf('async function cmdRestart()'); + const end = cli.indexOf('/**\n * Bring a SINGLE bot', start); + const restart = cli.slice(start, end); + const admission = restart.indexOf( + 'assertIncludePm2RestartAdmission(listPm2GodDaemonPids())', + ); + const consume = restart.indexOf('consumeRestartIntentTo('); + const retire = restart.indexOf('deleteAllBotmuxProcesses()'); + expect(admission).toBeGreaterThanOrEqual(0); + expect(consume).toBeGreaterThan(admission); + expect(retire).toBeGreaterThan(consume); + expect(restart).not.toContain('killPm2GodDaemon'); + expect(cli).toContain('--include-pm2 仅允许“入场时没有 live PM2 God”的干净启动'); + expect(cli).not.toContain('--include-pm2 同时重启 PM2 God'); + }); + + it('attests the whole daemon fleet then uses exact IPC batch/successor requests', () => { + const start = cli.indexOf('function signalAndAwaitBotmuxProcesses('); + const end = cli.indexOf('/** Compensate only rows owned', start); + const helper = cli.slice(start, end); + const preflight = helper.indexOf('`${operation}-shutdown-capability-preflight`'); + const fleet = helper.indexOf('signalAndAwaitFleet(', preflight); + const batch = helper.indexOf('requestAttestedDaemonShutdownBatch(', fleet); + const successor = helper.indexOf('requestAttestedDaemonShutdown(', fleet); + expect(preflight).toBeGreaterThanOrEqual(0); + expect(fleet).toBeGreaterThan(preflight); + expect(batch).toBeGreaterThan(fleet); + expect(successor).toBeGreaterThan(fleet); + expect(helper).toContain('signalInitial: targets =>'); + expect(helper).toContain('processStartByPid'); + expect(cli).toContain("return isBotmuxCoreProcessName(name) && name !== 'botmux-dashboard'"); + }); + + it('keeps supervisor shutdown host-authenticated and exact boot/birth bound', () => { + const route = ipcServer.slice( + ipcServer.indexOf("ipcRoute('POST', SUPERVISOR_SHUTDOWN_ROUTE"), + ipcServer.indexOf('export async function readJsonBody', + ipcServer.indexOf("ipcRoute('POST', SUPERVISOR_SHUTDOWN_ROUTE")), + ); + expect(route).toContain('isTrustedHostIpcRequest(req)'); + expect(route).toContain('isExactSupervisorShutdownRequest(registration, body)'); + expect(route).toContain('jsonRes(res, 202'); + expect(route.indexOf('jsonRes(res, 202')).toBeLessThan(route.indexOf('registration.shutdown()')); + }); +}); diff --git a/test/structured-bridge-clis.test.ts b/test/structured-bridge-clis.test.ts index 6d22f003a..8b8ec3100 100644 --- a/test/structured-bridge-clis.test.ts +++ b/test/structured-bridge-clis.test.ts @@ -11,8 +11,10 @@ import { isStructuredBridgeAdoptCli, isStructuredBridgeAdoptIdleCli, isStructuredBridgeAdoptInputCli, + isStructuredBridgeLifecycleBlockingCli, STRUCTURED_BRIDGE_ALWAYS_CLI_IDS, STRUCTURED_BRIDGE_ADOPT_CLI_IDS, + STRUCTURED_BRIDGE_LIFECYCLE_BLOCKING_CLI_IDS, } from '../src/services/structured-bridge-clis.js'; import { resolveFileBridgePath } from '../src/services/file-bridge-path.js'; @@ -45,9 +47,17 @@ describe('structured-bridge-clis', () => { expect(isStructuredBridgeAdoptIdleCli('coco')).toBe(true); expect(isStructuredBridgeAdoptIdleCli('cursor')).toBe(false); expect(isStructuredBridgeAdoptInputCli('mtr')).toBe(true); - expect(isStructuredBridgeAdoptInputCli('coco')).toBe(false); + expect(isStructuredBridgeAdoptInputCli('coco')).toBe(true); expect(isStructuredBridgeAdoptCli('cursor')).toBe(true); }); + + it('enables the strong status gate only for Codex with a complete terminal contract', () => { + expect(STRUCTURED_BRIDGE_LIFECYCLE_BLOCKING_CLI_IDS).toEqual(['codex']); + expect(isStructuredBridgeLifecycleBlockingCli('codex')).toBe(true); + for (const id of ['traex', 'coco', 'hermes', 'mtr', 'pi', 'grok', 'cursor']) { + expect(isStructuredBridgeLifecycleBlockingCli(id)).toBe(false); + } + }); }); describe('resolveFileBridgePath (grok)', () => { diff --git a/test/submit-confirmation.test.ts b/test/submit-confirmation.test.ts index a0c508944..7d640755f 100644 --- a/test/submit-confirmation.test.ts +++ b/test/submit-confirmation.test.ts @@ -1,5 +1,14 @@ import { describe, expect, it } from 'vitest'; -import { decideSubmitConfirmationAction } from '../src/services/submit-confirmation.js'; +import { + decideSubmitConfirmationAction, + settleDeferredSubmitConfirmation, + settleStaleWriteContinuation, +} from '../src/services/submit-confirmation.js'; +import { + CodexBridgeQueue, + pruneExpiredPreStartHeadsAndEmit, + STRUCTURED_SUBMIT_START_GRACE_MS, +} from '../src/services/codex-bridge-queue.js'; describe('decideSubmitConfirmationAction', () => { it('notifies immediately when the adapter reports a hard failure reason', () => { @@ -42,4 +51,193 @@ describe('decideSubmitConfirmationAction', () => { activityEvidence: undefined, })).toEqual({ kind: 'notify-stuck' }); }); + + it('turns deferred active evidence into a bounded lease, then prunes and emits a buffered successor', async () => { + let now = 0; + const queue = new CodexBridgeQueue(() => now); + queue.mark('active-but-no-start', 'first prompt never reached transcript', 0, 7); + queue.beginSubmitVerification('active-but-no-start', 0, 7); + + const settlement = await settleDeferredSubmitConfirmation(queue, { + turnId: 'active-but-no-start', + dispatchAttempt: 7, + structuredTarget: true, + recheck: async () => false, + usageLimitDetected: () => false, + activityEvidence: () => 'pty-output', + }); + expect(settlement).toMatchObject({ + action: { kind: 'suppress-active', evidence: 'pty-output' }, + lifecycle: 'confirmed', + }); + expect(queue.peek()[0]).toMatchObject({ + turnId: 'active-but-no-start', + dispatchAttempt: 7, + submitConfirmedAtMs: 0, + }); + expect(queue.peek()[0]?.submitVerificationStartedAtMs).toBeUndefined(); + + now = 19_000; + queue.mark('real-successor', 'second prompt really ran', now); + queue.confirmPendingTurn('real-successor', now); + queue.ingest([ + { kind: 'user', uuid: 'u-real-successor', text: 'second prompt really ran', timestampMs: 19_001 }, + { kind: 'assistant_final', uuid: 'a-real-successor', text: 'second prompt done', timestampMs: 19_002 }, + ]); + expect(queue.peek().find(turn => turn.turnId === 'real-successor')?.started).toBe(false); + + now = STRUCTURED_SUBMIT_START_GRACE_MS + 1; + const emitted: Array<{ turnId: string; finalText?: string }> = []; + const ordering: string[] = []; + const dropped = pruneExpiredPreStartHeadsAndEmit(queue, () => { + ordering.push('successor-ready'); + emitted.push(...queue.drainEmittable()); + }, now, turns => { + for (const turn of turns) { + ordering.push(`terminal:${turn.turnId}:${turn.dispatchAttempt ?? '-'}`); + } + }); + + expect(dropped.map(turn => turn.turnId)).toEqual(['active-but-no-start']); + expect(dropped[0]?.dispatchAttempt).toBe(7); + expect(ordering).toEqual([ + 'terminal:active-but-no-start:7', + 'successor-ready', + ]); + expect(emitted).toEqual([ + expect.objectContaining({ turnId: 'real-successor', finalText: 'second prompt done' }), + ]); + }); + + it('still runs Claude deferred rechecks whose bridge ID is not in the structured queue', async () => { + const structuredQueue = new CodexBridgeQueue(); + let recheckCalls = 0; + + const settlement = await settleDeferredSubmitConfirmation(structuredQueue, { + turnId: 'claude-bridge-turn', + recheck: async () => { + recheckCalls++; + return false; + }, + usageLimitDetected: () => false, + activityEvidence: () => undefined, + isCurrent: () => true, + }); + + expect(recheckCalls).toBe(1); + expect(settlement).toMatchObject({ + stale: false, + action: { kind: 'notify-stuck' }, + lifecycle: 'unchanged', + }); + }); + + it('does not run or settle a stale attempt-N timer after attempt N+1 replaced it', async () => { + const queue = new CodexBridgeQueue(); + queue.mark('delivery', 'same durable body', 100, 1); + queue.beginSubmitVerification('delivery', 110, 1); + queue.dropPendingTurn('delivery', 1); + queue.mark('delivery', 'same durable body', 200, 2); + queue.beginSubmitVerification('delivery', 210, 2); + let recheckCalls = 0; + + const settlement = await settleDeferredSubmitConfirmation(queue, { + turnId: 'delivery', + dispatchAttempt: 1, + structuredTarget: true, + recheck: async () => { + recheckCalls++; + return { submitted: true, cliSessionId: 'stale-session' }; + }, + usageLimitDetected: () => false, + activityEvidence: () => 'pty-output', + isCurrent: () => true, + }); + + expect(settlement).toEqual({ + stale: true, + staleReason: 'attempt', + lifecycle: 'unchanged', + }); + expect(recheckCalls).toBe(0); + expect(queue.peek()).toEqual([ + expect.objectContaining({ + turnId: 'delivery', + dispatchAttempt: 2, + submitVerificationStartedAtMs: 210, + }), + ]); + expect(queue.peek()[0]?.submitConfirmedAtMs).toBeUndefined(); + }); + + it('rechecks generation after an awaited callback and returns no stale session side effect', async () => { + const queue = new CodexBridgeQueue(); + queue.mark('delivery', 'durable body', 100, 1); + queue.beginSubmitVerification('delivery', 110, 1); + let current = true; + let release!: (value: { submitted: true; cliSessionId: string }) => void; + const recheck = new Promise<{ submitted: true; cliSessionId: string }>(resolve => { release = resolve; }); + + const pending = settleDeferredSubmitConfirmation(queue, { + turnId: 'delivery', + dispatchAttempt: 1, + structuredTarget: true, + recheck: () => recheck, + usageLimitDetected: () => false, + activityEvidence: () => undefined, + isCurrent: () => current, + }); + await Promise.resolve(); + current = false; + release({ submitted: true, cliSessionId: 'old-generation-session' }); + + const settlement = await pending; + expect(settlement).toEqual({ + stale: true, + staleReason: 'generation', + lifecycle: 'unchanged', + }); + expect(settlement).not.toHaveProperty('cliSessionId'); + expect(queue.peek()[0]).toMatchObject({ + dispatchAttempt: 1, + submitVerificationStartedAtMs: 110, + }); + expect(queue.peek()[0]?.submitConfirmedAtMs).toBeUndefined(); + }); +}); + +describe('settleStaleWriteContinuation', () => { + it('leaves a replacement ordinary mark untouched and relies on automatic carryover', () => { + const replacementQueue = new CodexBridgeQueue(); + replacementQueue.mark('same-ordinary-turn', 'replayed body', 200); + const terminals: Array<{ turnId: string; attempt: number }> = []; + + const disposition = settleStaleWriteContinuation( + { turnId: 'same-ordinary-turn' }, + 'write_generation_changed', + (turnId, _errorCode, attempt) => terminals.push({ turnId, attempt }), + ); + + expect(disposition).toBe('ordinary-carryover'); + expect(terminals).toEqual([]); + expect(replacementQueue.peek()).toEqual([ + expect.objectContaining({ turnId: 'same-ordinary-turn', started: false }), + ]); + }); + + it('emits an exact ambiguous terminal for a durable stale continuation', () => { + const terminals: Array<{ turnId: string; errorCode: string; attempt: number }> = []; + const disposition = settleStaleWriteContinuation( + { turnId: 'durable-turn', dispatchAttempt: 4 }, + 'write_generation_changed', + (turnId, errorCode, attempt) => terminals.push({ turnId, errorCode, attempt }), + ); + + expect(disposition).toBe('ambiguous-terminal'); + expect(terminals).toEqual([{ + turnId: 'durable-turn', + errorCode: 'write_generation_changed', + attempt: 4, + }]); + }); }); diff --git a/test/supervisor-shutdown-client.test.ts b/test/supervisor-shutdown-client.test.ts new file mode 100644 index 000000000..e2eb47721 --- /dev/null +++ b/test/supervisor-shutdown-client.test.ts @@ -0,0 +1,75 @@ +import { describe, expect, it, vi } from 'vitest'; +import { + requestAttestedDaemonShutdown, + requestAttestedDaemonShutdownBatch, +} from '../src/cli/supervisor-shutdown-client.js'; +import type { AttestedPm2DaemonShutdownTarget } from '../src/cli/pm2-shutdown-capability.js'; + +const target: AttestedPm2DaemonShutdownTarget = { + name: 'botmux-a', + pid: 101, + larkAppId: 'cli_a', + ipcPort: 7901, + bootInstanceId: 'boot-a', + processStartIdentity: 'birth-a', +}; + +describe('attested daemon supervisor shutdown client', () => { + it('accepts only an exact 202 ACK from the target boot/birth', () => { + const postMany = vi.fn(() => [{ + status: 202, + bodyRaw: JSON.stringify({ + ok: true, + accepted: true, + larkAppId: 'cli_a', + bootInstanceId: 'boot-a', + processStartIdentity: 'birth-a', + }), + }]); + expect(() => requestAttestedDaemonShutdown(target, 'secret', { + readStartIdentity: () => 'birth-a', + postMany, + })).not.toThrow(); + expect(postMany).toHaveBeenCalledOnce(); + }); + + it('never contacts a same-PID successor with a different birth', () => { + const postMany = vi.fn(); + expect(() => requestAttestedDaemonShutdown(target, 'secret', { + readStartIdentity: () => 'birth-b', + postMany, + })).toThrow(/process generation changed/); + expect(postMany).not.toHaveBeenCalled(); + }); + + it('fails closed on a generation-mismatch response', () => { + expect(() => requestAttestedDaemonShutdown(target, 'secret', { + readStartIdentity: () => 'birth-a', + postMany: () => [{ + status: 409, + bodyRaw: JSON.stringify({ ok: false, error: 'supervisor_shutdown_generation_mismatch' }), + }], + })).toThrow(/rejected exact supervisor shutdown.*status 409/); + }); + + it('dispatches the initial fleet in one batch so a timeout does not suppress peers', () => { + const peer = { ...target, name: 'botmux-b', pid: 202, larkAppId: 'cli_b', ipcPort: 7902, + bootInstanceId: 'boot-b', processStartIdentity: 'birth-b' }; + const postMany = vi.fn((inputs) => { + expect(inputs).toHaveLength(2); + return [ + { error: 'supervisor shutdown request timed out' }, + { status: 202, bodyRaw: JSON.stringify({ + ok: true, accepted: true, larkAppId: 'cli_b', bootInstanceId: 'boot-b', + processStartIdentity: 'birth-b', + }) }, + ]; + }); + const attempts = requestAttestedDaemonShutdownBatch([target, peer], 'secret', { + readStartIdentity: pid => pid === 101 ? 'birth-a' : 'birth-b', + postMany, + }); + expect(postMany).toHaveBeenCalledOnce(); + expect(attempts.map(attempt => attempt.ok)).toEqual([false, true]); + }); +}); diff --git a/test/supervisor-shutdown-ipc.test.ts b/test/supervisor-shutdown-ipc.test.ts new file mode 100644 index 000000000..bd4bb0362 --- /dev/null +++ b/test/supervisor-shutdown-ipc.test.ts @@ -0,0 +1,25 @@ +import { describe, expect, it } from 'vitest'; +import { isExactSupervisorShutdownRequest } from '../src/core/supervisor-shutdown-ipc.js'; + +const identity = { + larkAppId: 'cli_a', + bootInstanceId: 'boot-a', + processStartIdentity: 'birth-a', +}; + +describe('generation-bound supervisor shutdown request', () => { + it('accepts only the exact in-process app/boot/birth tuple', () => { + expect(isExactSupervisorShutdownRequest(identity, { ...identity })).toBe(true); + }); + + it('rejects a successor on the same port or reused PID', () => { + expect(isExactSupervisorShutdownRequest(identity, { + ...identity, + bootInstanceId: 'boot-b', + })).toBe(false); + expect(isExactSupervisorShutdownRequest(identity, { + ...identity, + processStartIdentity: 'birth-b', + })).toBe(false); + }); +}); diff --git a/test/supervisor-shutdown-loopback.integration.test.ts b/test/supervisor-shutdown-loopback.integration.test.ts new file mode 100644 index 000000000..3179a3ab9 --- /dev/null +++ b/test/supervisor-shutdown-loopback.integration.test.ts @@ -0,0 +1,83 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { daemonIpcAuthHeaders } from '../src/core/daemon-ipc-auth.js'; +import { + setIpcAuthSecret, + setSupervisorShutdownHandler, + startIpcServer, + type IpcServerHandle, +} from '../src/core/dashboard-ipc-server.js'; +import { SUPERVISOR_SHUTDOWN_ROUTE } from '../src/core/supervisor-shutdown-ipc.js'; + +const SECRET = 'supervisor-loopback-test-secret'; +const IDENTITY = { + larkAppId: 'cli_loopback', + bootInstanceId: 'boot-loopback', + processStartIdentity: 'birth-loopback', +}; + +let server: IpcServerHandle | null = null; + +afterEach(async () => { + setSupervisorShutdownHandler(null); + setIpcAuthSecret(null); + if (server) await server.close(); + server = null; +}); + +async function signedPost(body: unknown): Promise { + if (!server) throw new Error('test IPC server is not running'); + return fetch(`http://127.0.0.1:${server.port}${SUPERVISOR_SHUTDOWN_ROUTE}`, { + method: 'POST', + headers: daemonIpcAuthHeaders({ + secret: SECRET, + port: server.port, + method: 'POST', + path: SUPERVISOR_SHUTDOWN_ROUTE, + headers: { 'content-type': 'application/json' }, + }), + body: JSON.stringify(body), + }); +} + +describe('supervisor shutdown signed loopback handshake', () => { + it('authenticates the real route and accepts only the registered boot/birth generation', async () => { + setIpcAuthSecret(SECRET); + server = await startIpcServer({ + port: 0, + host: '127.0.0.1', + authRequired: true, + ready: Promise.resolve(), + }); + + const unsigned = await fetch( + `http://127.0.0.1:${server.port}${SUPERVISOR_SHUTDOWN_ROUTE}`, + { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify(IDENTITY), + }, + ); + expect(unsigned.status).toBe(401); + + const beforeRegistration = await signedPost(IDENTITY); + expect(beforeRegistration.status).toBe(503); + + const shutdown = vi.fn(async () => {}); + setSupervisorShutdownHandler({ ...IDENTITY, shutdown }); + + const wrongBoot = await signedPost({ ...IDENTITY, bootInstanceId: 'boot-successor' }); + expect(wrongBoot.status).toBe(409); + const wrongBirth = await signedPost({ ...IDENTITY, processStartIdentity: 'birth-successor' }); + expect(wrongBirth.status).toBe(409); + expect(shutdown).not.toHaveBeenCalled(); + + const accepted = await signedPost(IDENTITY); + expect(accepted.status).toBe(202); + expect(await accepted.json()).toEqual({ + ok: true, + accepted: true, + ...IDENTITY, + }); + await vi.waitFor(() => expect(shutdown).toHaveBeenCalledOnce()); + }); +}); diff --git a/test/tmux-backend-env.test.ts b/test/tmux-backend-env.test.ts index bf7a5b6b9..69721f998 100644 --- a/test/tmux-backend-env.test.ts +++ b/test/tmux-backend-env.test.ts @@ -133,14 +133,17 @@ describe('buildBotmuxEnvAssignments()', () => { expect(out).not.toContain('PATH=/usr/bin'); }); - it('forwards the worker-owned MCP relay capability into the CLI pane', () => { + it('forwards only a Codex App bootstrap path and strips the retired shared-secret env', () => { + const retiredSharedSecret = 'A'.repeat(43); + const bootstrapPath = '/private/bot-home/control.bootstrap'; const out = buildBotmuxEnvAssignments({ BOTMUX: '1', - BOTMUX_MCP_GATEWAY_SOCKET: '/tmp/botmux-mcp/session/gateway.sock', - BOTMUX_MCP_GATEWAY_REQUIRED: '1', + BOTMUX_CODEX_APP_CONTROL_NONCE: retiredSharedSecret, + BOTMUX_CODEX_APP_CONTROL_BOOTSTRAP: bootstrapPath, }); - expect(out).toContain('BOTMUX_MCP_GATEWAY_SOCKET=/tmp/botmux-mcp/session/gateway.sock'); - expect(out).toContain('BOTMUX_MCP_GATEWAY_REQUIRED=1'); + expect(out).toContain(`BOTMUX_CODEX_APP_CONTROL_BOOTSTRAP=${bootstrapPath}`); + expect(out.join(' ')).not.toContain(retiredSharedSecret); + expect(out.some(value => value.startsWith('BOTMUX_CODEX_APP_CONTROL_NONCE='))).toBe(false); }); it('forwards Hermes profile paths so the pane and transcript reader use the same state DB', () => { diff --git a/test/tmux-reattach-backend.test.ts b/test/tmux-reattach-backend.test.ts index 80432f53a..7c4033e6a 100644 --- a/test/tmux-reattach-backend.test.ts +++ b/test/tmux-reattach-backend.test.ts @@ -88,4 +88,14 @@ describe('selectSessionBackend', () => { expect(selected.backend.constructor.name).toBe('MockZellijBackend'); expect((selected.backend as any).opts).toEqual({ ownsSession: true, isReattach: false }); }); + + it('marks an existing zellij session as reattach without making it pipe mode', () => { + vi.mocked(ZellijBackend.hasSession).mockReturnValue(true); + + const selected = selectSessionBackend({ sessionId: '9cfa0024-197d-4781-845b-c541dceb8980', backendType: 'zellij' }); + + expect(selected.isZellijMode).toBe(true); + expect(selected.isPipeMode).toBe(false); + expect((selected.backend as any).opts).toEqual({ ownsSession: true, isReattach: true }); + }); }); diff --git a/test/transfer-session.test.ts b/test/transfer-session.test.ts index 6d8264a2c..785d194ad 100644 --- a/test/transfer-session.test.ts +++ b/test/transfer-session.test.ts @@ -16,6 +16,7 @@ import { describe, it, expect, beforeEach, vi } from 'vitest'; vi.mock('../src/services/session-store.js', () => ({ updateSession: vi.fn(), getSession: vi.fn(), + listSessions: vi.fn(() => []), closeSession: vi.fn(), })); @@ -53,6 +54,10 @@ import { dashboardEventBus } from '../src/core/dashboard-events.js'; import { sessionKey } from '../src/core/types.js'; import type { DaemonSession } from '../src/core/types.js'; import type { Session } from '../src/types.js'; +import { + __testOnly_resetBotTurnMutationGates, + withBotTurnAdmission, +} from '../src/core/bot-turn-mutation-gate.js'; function makeDs(overrides: Partial = {}): DaemonSession { const session: Session = { @@ -94,6 +99,13 @@ function makeDs(overrides: Partial = {}): DaemonSession { } as DaemonSession; } +function deferred() { + let resolve!: (value: T) => void; + let reject!: (reason?: unknown) => void; + const promise = new Promise((res, rej) => { resolve = res; reject = rej; }); + return { promise, resolve, reject }; +} + describe('transferSession', () => { let registry: Map; @@ -112,6 +124,8 @@ describe('transferSession', () => { beforeEach(() => { vi.clearAllMocks(); + __testOnly_resetBotTurnMutationGates(); + vi.mocked(sessionStore.listSessions).mockReturnValue([]); registry = new Map(); setActiveSessionsRegistry(registry); }); @@ -177,6 +191,105 @@ describe('transferSession', () => { expect(adoptDs.chatId).toBe('oc_source'); }); + it('refuses a Riff transfer before worker, store, registry, or reply-route mutation', async () => { + const worker = { killed: false, send: vi.fn(), kill: vi.fn() } as any; + const riffDs = makeDs({ + worker, + initConfig: { backendType: 'riff' } as any, + }); + riffDs.session.cliId = 'riff'; + riffDs.session.backendType = 'riff'; + riffDs.session.riffParentTaskId = 'task-riff-route'; + registry.set(sessionKey('om_source_root', 'cli_app_test'), riffDs); + + const result = await callTransfer(riffDs.session.sessionId, 'oc_target', 'om_M1_target'); + + expect(result).toEqual({ ok: false, error: 'riff_not_relayable' }); + expect(killWorkerSpy).not.toHaveBeenCalled(); + expect(forkWorkerSpy).not.toHaveBeenCalled(); + expect(sessionStore.updateSession).not.toHaveBeenCalled(); + expect(updateMessageMock).not.toHaveBeenCalled(); + expect(riffDs.worker).toBe(worker); + expect(riffDs.session).toMatchObject({ + chatId: 'oc_source', + rootMessageId: 'om_source_root', + riffParentTaskId: 'task-riff-route', + }); + expect(registry.get(sessionKey('om_source_root', 'cli_app_test'))).toBe(riffDs); + expect(registry.has(sessionKey('oc_target', 'cli_app_test'))).toBe(false); + }); + + it('refuses transfer while the durable Codex App dispatch ledger is non-empty', async () => { + const ds = makeDs(); + ds.session.codexAppDispatchLedger = [ + { dispatchId: 'd-1', turnId: 't-1', state: 'prepared', content: 'owned' }, + ]; + registry.set(sessionKey('om_source_root', 'cli_app_test'), ds); + + const result = await callTransfer(ds.session.sessionId, 'oc_target', 'om_target_root'); + + expect(result).toEqual({ ok: false, error: 'codex_app_dispatch_pending' }); + expect(forkWorkerSpy).not.toHaveBeenCalled(); + expect(killWorkerSpy).not.toHaveBeenCalled(); + expect(sessionStore.updateSession).not.toHaveBeenCalled(); + expect(ds.chatId).toBe('oc_source'); + expect(ds.session.rootMessageId).toBe('om_source_root'); + expect(registry.get(sessionKey('om_source_root', 'cli_app_test'))).toBe(ds); + }); + + it('drains a pre-accept turn and rechecks durable ownership before transfer', async () => { + const ds = makeDs(); + registry.set(sessionKey('om_source_root', 'cli_app_test'), ds); + let release!: () => void; + let markStarted!: () => void; + const paused = new Promise(resolve => { release = resolve; }); + const started = new Promise(resolve => { markStarted = resolve; }); + const inbound = withBotTurnAdmission(ds.larkAppId, async () => { + markStarted(); + await paused; + ds.session.codexAppDispatchLedger = [{ + dispatchId: 'd-raced', + turnId: 't-raced', + state: 'accepted', + content: 'accepted before transfer', + }]; + }); + await started; + + const transfer = callTransfer(ds.session.sessionId, 'oc_target', 'om_target_root'); + await Promise.resolve(); + expect(killWorkerSpy).not.toHaveBeenCalled(); + release(); + await inbound; + + expect(await transfer).toEqual({ ok: false, error: 'codex_app_dispatch_pending' }); + expect(killWorkerSpy).not.toHaveBeenCalled(); + expect(forkWorkerSpy).not.toHaveBeenCalled(); + expect(registry.get(sessionKey('om_source_root', ds.larkAppId))).toBe(ds); + }); + + it('does not kill or overwrite a source successor discovered after cleanup awaits', async () => { + const ds = makeDs(); + const sourceKey = sessionKey('om_source_root', 'cli_app_test'); + registry.set(sourceKey, ds); + const successor = makeDs({ + session: { ...ds.session, sessionId: 'source-successor' }, + }); + vi.mocked(sessionStore.listSessions).mockImplementationOnce(() => { + ds.session.status = 'closed'; + registry.set(sourceKey, successor); + return []; + }); + + const result = await callTransfer(ds.session.sessionId, 'oc_target', 'om_target_root'); + + expect(result).toEqual({ ok: false, error: 'session_not_active' }); + expect(registry.get(sourceKey)).toBe(successor); + expect(killWorkerSpy).not.toHaveBeenCalled(); + expect(forkWorkerSpy).not.toHaveBeenCalled(); + expect(ds.chatId).toBe('oc_source'); + }); + it('returns same_anchor when a chat-scope source targets its own chat (chat→chat)', async () => { const ds = makeDs({ scope: 'chat' }); ds.session.scope = 'chat'; @@ -407,6 +520,72 @@ describe('transferSession', () => { expect(registry.get(sessionKey('oc_target', 'cli_app_test'))).toBe(existingDs); }); + it('refuses a disk-only legacy target owner with an unsettled Codex App dispatch', async () => { + const movingDs = makeDs(); + registry.set(sessionKey('om_source_root', 'cli_app_test'), movingDs); + const legacyConflict: Session = { + ...movingDs.session, + sessionId: 'legacy-disk-only-owner', + chatId: 'oc_target', + rootMessageId: 'om_legacy_target', + scope: 'chat', + larkAppId: undefined, + codexAppDispatchLedger: [{ + dispatchId: 'legacy-dispatch', + turnId: 'legacy-turn', + state: 'prepared', + content: 'owned output', + deliverySink: 'lark', + }], + }; + vi.mocked(sessionStore.listSessions).mockReturnValue([legacyConflict]); + + const result = await callTransfer( + movingDs.session.sessionId, + 'oc_target', + 'om_M1_target', + ); + + expect(result).toEqual({ ok: false, error: 'target_chat_has_session' }); + expect(sessionStore.closeSession).not.toHaveBeenCalled(); + expect(killWorkerSpy).not.toHaveBeenCalled(); + expect(forkWorkerSpy).not.toHaveBeenCalled(); + expect(movingDs.chatId).toBe('oc_source'); + expect(registry.get(sessionKey('om_source_root', 'cli_app_test'))).toBe(movingDs); + }); + + it('retires a disk-only legacy scratch before claiming the target anchor', async () => { + const movingDs = makeDs(); + registry.set(sessionKey('om_source_root', 'cli_app_test'), movingDs); + const legacyScratch: Session = { + ...movingDs.session, + sessionId: 'legacy-disk-only-scratch', + chatId: 'oc_target', + rootMessageId: 'om_legacy_scratch', + scope: 'chat', + larkAppId: undefined, + cliId: undefined, + lastCliInput: undefined, + queued: false, + codexAppDispatchLedger: [], + }; + vi.mocked(sessionStore.listSessions).mockReturnValue([legacyScratch]); + vi.mocked(sessionStore.getSession).mockImplementation((sid: string) => + sid === legacyScratch.sessionId ? legacyScratch : undefined, + ); + + const result = await callTransfer( + movingDs.session.sessionId, + 'oc_target', + 'om_M1_target', + ); + + expect(result).toEqual({ ok: true }); + expect(sessionStore.closeSession).toHaveBeenCalledWith(legacyScratch.sessionId); + expect(registry.get(sessionKey('oc_target', 'cli_app_test'))).toBe(movingDs); + expect(forkWorkerSpy).toHaveBeenCalledTimes(1); + }); + it('closes the daemon-command scratch session occupying the target chat slot', async () => { // Regression: a /relay command in the target chat creates a placeholder // session record with `worker: null`. Previously the pre-flight scan @@ -427,6 +606,8 @@ describe('transferSession', () => { rootMessageId: 'om_relay_cmd_msg', scope: 'chat', title: '/relay', + cliId: undefined, + lastCliInput: undefined, }, worker: null, // command-time placeholder, no real worker chatId: 'oc_target', @@ -500,6 +681,28 @@ describe('transferSession', () => { expect(body).toMatch(/"img_key":\s*"old_image_key"/); }); + it('reattaches at the routing commit before awaiting the source-card patch', async () => { + const ds = makeDs(); + registry.set(sessionKey('om_source_root', 'cli_app_test'), ds); + const patch = deferred(); + updateMessageMock.mockImplementationOnce(() => patch.promise); + + const transferring = callTransfer(ds.session.sessionId, 'oc_target', 'om_M1_target'); + await vi.waitFor(() => expect(updateMessageMock).toHaveBeenCalledTimes(1)); + + // The target owner is already runnable before the best-effort Lark PATCH. + // A close that wins during that await must not be followed by a stale fork. + expect(forkWorkerSpy).toHaveBeenCalledTimes(1); + expect(registry.get(sessionKey('oc_target', 'cli_app_test'))).toBe(ds); + registry.delete(sessionKey('oc_target', 'cli_app_test')); + ds.session.status = 'closed'; + + patch.resolve(); + await expect(transferring).resolves.toEqual({ ok: true }); + expect(forkWorkerSpy).toHaveBeenCalledTimes(1); + expect(registry.has(sessionKey('oc_target', 'cli_app_test'))).toBe(false); + }); + it('frozen card renders no extra element when no currentImageKey is set (hidden mode)', async () => { // Sessions in hidden / collapsed display mode never produced a server- // rendered screenshot, so currentImageKey is undefined. We deliberately @@ -640,4 +843,64 @@ describe('setActiveSessionSafe', () => { expect(registry.get(key)).toBe(ds); expect(sessionStore.closeSession).not.toHaveBeenCalled(); }); + + it('keeps an unsettled incumbent and closes a ledger-empty incoming collision', async () => { + const incumbent = makeSimpleDs('pending-incumbent'); + incumbent.session.codexAppDispatchLedger = [{ + dispatchId: 'dispatch-incumbent', + turnId: 'turn-incumbent', + state: 'prepared', + content: 'owned', + deliverySink: 'lark', + }]; + const incoming = makeSimpleDs('ledger-empty-incoming'); + vi.mocked(sessionStore.getSession).mockImplementation((sid: string) => + sid === incoming.session.sessionId ? incoming.session : undefined, + ); + const key = sessionKey('oc_c', 'cli_app_test'); + registry.set(key, incumbent); + + const result = await setActiveSessionSafe(registry, key, incoming); + + expect(result).toEqual({ + accepted: false, + reason: 'kept_pending_owner', + keptSessionId: incumbent.session.sessionId, + closedIncomingSessionId: incoming.session.sessionId, + }); + expect(registry.get(key)).toBe(incumbent); + expect(sessionStore.closeSession).toHaveBeenCalledWith(incoming.session.sessionId); + }); + + it('fails closed and preserves both rows when both colliding owners are unsettled', async () => { + const incumbent = makeSimpleDs('pending-incumbent'); + incumbent.session.codexAppDispatchLedger = [{ + dispatchId: 'dispatch-incumbent', + turnId: 'turn-incumbent', + state: 'prepared', + content: 'owned incumbent', + deliverySink: 'lark', + }]; + const incoming = makeSimpleDs('pending-incoming'); + incoming.session.codexAppDispatchLedger = [{ + dispatchId: 'dispatch-incoming', + turnId: 'turn-incoming', + state: 'prepared', + content: 'owned incoming', + deliverySink: 'lark', + }]; + const key = sessionKey('oc_c', 'cli_app_test'); + registry.set(key, incumbent); + + const result = await setActiveSessionSafe(registry, key, incoming); + + expect(result).toEqual({ + accepted: false, + reason: 'both_pending', + keptSessionId: incumbent.session.sessionId, + preservedIncomingSessionId: incoming.session.sessionId, + }); + expect(registry.get(key)).toBe(incumbent); + expect(sessionStore.closeSession).not.toHaveBeenCalled(); + }); }); diff --git a/test/trigger-session-root-message.test.ts b/test/trigger-session-root-message.test.ts index 85d7196c7..9f2002277 100644 --- a/test/trigger-session-root-message.test.ts +++ b/test/trigger-session-root-message.test.ts @@ -29,9 +29,11 @@ vi.mock('../src/services/oncall-store.js', () => ({ const mockCreateSession = vi.fn(); const mockUpdateSession = vi.fn(); +const mockCloseSession = vi.fn(); vi.mock('../src/services/session-store.js', () => ({ createSession: (...args: any[]) => mockCreateSession(...args), updateSession: (...args: any[]) => mockUpdateSession(...args), + closeSession: (...args: any[]) => mockCloseSession(...args), })); vi.mock('../src/services/message-queue.js', () => ({ @@ -39,10 +41,18 @@ vi.mock('../src/services/message-queue.js', () => ({ })); const mockForkWorker = vi.fn(); +const mockQueuedTailAdmission = vi.fn(); +const activeKeyLocks = vi.hoisted(() => ({ + byMap: new WeakMap, Map>>(), +})); vi.mock('../src/core/worker-pool.js', () => ({ forkWorker: (...args: any[]) => mockForkWorker(...args), sendWorkerInput: (ds: any, payload: any, turnId?: string, opts: any = {}) => { if (!ds.worker || ds.worker.killed) return false; + const gated = ds.session.queuedActivationPending === true + || (ds.session.queuedActivationTail?.length ?? 0) > 0 + || (ds.initialStartPending === true && ds.session.queuedActivationInput !== undefined); + if (gated) return mockQueuedTailAdmission(ds, payload, turnId, opts); ds.worker.send({ type: 'message', content: typeof payload === 'string' ? payload : payload.content, @@ -56,10 +66,32 @@ vi.mock('../src/core/worker-pool.js', () => ({ }); return true; }, + hasQueuedActivationAdmissionGate: (ds: any) => ds.session.queuedActivationPending === true + || (ds.session.queuedActivationTail?.length ?? 0) > 0 + || (ds.initialStartPending === true && ds.session.queuedActivationInput !== undefined), getCurrentCliVersion: vi.fn(() => 'test-cli-version'), + withActiveSessionKeyLock: vi.fn(async (map: Map, key: string, action: () => any) => { + let locks = activeKeyLocks.byMap.get(map); + if (!locks) { + locks = new Map(); + activeKeyLocks.byMap.set(map, locks); + } + const previous = locks.get(key) ?? Promise.resolve(); + let release!: () => void; + const hold = new Promise(resolve => { release = resolve; }); + const tail = previous.catch(() => {}).then(() => hold); + locks.set(key, tail); + await previous.catch(() => {}); + try { return await action(); } + finally { + release(); + if (locks.get(key) === tail) locks.delete(key); + } + }), })); const mockRememberLastCliInput = vi.fn(); +const mockGetAvailableBots = vi.fn(async () => []); const mockBuildFollowUpCliInput = vi.fn((prompt: string, _sessionId?: string, opts?: any) => ({ content: `follow:${prompt}`, codexAppInput: opts?.cliId === 'codex-app' && opts?.codexAppText ? { text: opts.codexAppText } : undefined, @@ -74,7 +106,7 @@ vi.mock('../src/core/session-manager.js', () => ({ buildNewTopicPrompt: vi.fn((prompt: string) => `new:${prompt}`), buildNewTopicCliInput: (...args: any[]) => mockBuildNewTopicCliInput(...args), ensureSessionWhiteboard: vi.fn(), - getAvailableBots: vi.fn(async () => []), + getAvailableBots: (...args: any[]) => mockGetAvailableBots(...args), rememberLastCliInput: (...args: any[]) => mockRememberLastCliInput(...args), })); @@ -90,6 +122,7 @@ vi.mock('../src/im/lark/card-handler.js', () => ({ import { buildExternalEventTopicMessage, triggerSessionTurn } from '../src/core/trigger-session.js'; import { sessionKey } from '../src/core/types.js'; +import { withActiveSessionKeyLock } from '../src/core/worker-pool.js'; const APP = 'app1'; const CHAT = 'oc_root_chat'; @@ -136,6 +169,25 @@ describe('triggerSessionTurn rootMessageId target', () => { botOpenId: 'ou_bot', }); mockGetMessageChatId.mockResolvedValue(CHAT); + mockQueuedTailAdmission.mockImplementation((ds: any, payload: any, turnId?: string, opts: any = {}) => { + const order = (ds.session.queuedActivationTailNextOrder ?? 0) + 1; + ds.session.queuedActivationTailNextOrder = order; + ds.session.queuedActivationTail = [ + ...(ds.session.queuedActivationTail ?? []), + { + id: `tail-${order}`, + order, + userPrompt: typeof payload === 'string' ? payload : payload.content, + cliInput: typeof payload === 'string' ? { content: payload } : payload, + turnId: turnId ?? `tail-turn-${order}`, + ...(opts.dispatchAttempt !== undefined + ? { dispatchAttempt: opts.dispatchAttempt } + : {}), + }, + ]; + mockUpdateSession(ds.session); + return true; + }); mockCreateSession.mockImplementation((chatId: string, rootMessageId: string, title: string, chatType: 'group' | 'p2p') => ({ sessionId: 'sess_new', chatId, @@ -224,6 +276,126 @@ describe('triggerSessionTurn rootMessageId target', () => { expect(send).toHaveBeenCalledWith({ type: 'message', content: expect.stringContaining('follow:') }); }); + it.each([ + ['normal opening', undefined], + ['raw text-to-Enter opening', '/goal OPENING_RAW_N'], + ])('durably queues an external trigger behind a live %s activation', async (_label, pendingRawInput) => { + const send = vi.fn(); + const ds = existingDs({ + worker: { killed: false, send } as any, + initialStartPending: true, + ...(pendingRawInput ? { pendingRawInput } : {}), + }); + Object.assign(ds.session, { + cliId: 'claude-code', + queuedActivationPending: true, + queuedActivationToken: 'opening-token', + queuedActivationInput: { content: pendingRawInput ? '' : 'OPENING_N' }, + queuedActivationTurnId: 'turn-opening', + }); + const beforeDispatch = vi.fn(() => ({ dispatchAttempt: 4 })); + + const res = await triggerSessionTurn( + request(), + { larkAppId: APP, activeSessions: new Map([[sessionKey(ROOT, APP), ds]]) }, + { stableTurnId: 'external-follower-n1', beforeDispatch }, + ); + + expect(res).toMatchObject({ + ok: true, + triggerId: 'external-follower-n1', + action: 'queued', + }); + expect(send).not.toHaveBeenCalled(); + expect(mockQueuedTailAdmission).toHaveBeenCalledWith( + ds, + expect.objectContaining({ content: expect.stringContaining('follow:') }), + 'external-follower-n1', + { dispatchAttempt: 4 }, + ); + expect(ds.session.queuedActivationTail).toEqual([ + expect.objectContaining({ + turnId: 'external-follower-n1', + dispatchAttempt: 4, + cliInput: expect.objectContaining({ content: expect.stringContaining('follow:') }), + }), + ]); + expect(mockRememberLastCliInput).toHaveBeenCalled(); + }); + + it('reports failure and does not send or persist input history when gated tail admission fails', async () => { + const send = vi.fn(); + const ds = existingDs({ + worker: { killed: false, send } as any, + initialStartPending: true, + }); + Object.assign(ds.session, { + cliId: 'claude-code', + queuedActivationPending: true, + queuedActivationToken: 'opening-token', + queuedActivationInput: { content: 'OPENING_N' }, + }); + mockQueuedTailAdmission.mockReturnValueOnce(false); + + const res = await triggerSessionTurn(request(), { + larkAppId: APP, + activeSessions: new Map([[sessionKey(ROOT, APP), ds]]), + }); + + expect(res).toMatchObject({ ok: false, errorCode: 'trigger_failed' }); + expect(send).not.toHaveBeenCalled(); + expect(ds.session.queuedActivationTail).toBeUndefined(); + expect(mockRememberLastCliInput).not.toHaveBeenCalled(); + }); + + it('does not report delivery or persist input while a retained Riff abort fence is active', async () => { + const send = vi.fn(); + const ds = existingDs({ + worker: { killed: false, send } as any, + riffShutdownState: { + phase: 'preparing', + requestId: 'shutdown-abort-awaiting-ack', + taskId: 'task-current', + }, + }); + const res = await triggerSessionTurn(request(), { + larkAppId: APP, + activeSessions: new Map([[sessionKey(ROOT, APP), ds]]), + }); + + expect(res).toMatchObject({ + ok: false, + errorCode: 'trigger_failed', + error: expect.stringContaining('shutdown-preparing'), + }); + expect(send).not.toHaveBeenCalled(); + expect(mockRememberLastCliInput).not.toHaveBeenCalled(); + }); + + it('does not refork or ACK a trigger after the exact fenced worker has exited', async () => { + const ds = existingDs({ + worker: null, + riffShutdownState: { + phase: 'prepared', + requestId: 'shutdown-worker-exited', + taskId: 'task-drained-unverified', + }, + }); + const res = await triggerSessionTurn(request(), { + larkAppId: APP, + activeSessions: new Map([[sessionKey(ROOT, APP), ds]]), + }); + + expect(res).toMatchObject({ + ok: false, + errorCode: 'trigger_failed', + error: expect.stringContaining('shutdown-prepared'), + }); + expect(mockForkWorker).not.toHaveBeenCalled(); + expect(mockRememberLastCliInput).not.toHaveBeenCalled(); + expect(ds.riffShutdownState).toMatchObject({ requestId: 'shutdown-worker-exited' }); + }); + it('uses an internal stable turn id without changing the public trigger schema', async () => { const send = vi.fn(); const ds = existingDs({ worker: { killed: false, send } as any, workerGeneration: 7 }); @@ -334,6 +506,31 @@ describe('triggerSessionTurn rootMessageId target', () => { expect(mockForkWorker).toHaveBeenCalledWith(ds, { content: expect.stringContaining('follow:') }, { resume: true, turnId: expect.stringMatching(/^trg_/) }); }); + it('fails closed for a dormant ledger-only owner instead of reforking over it', async () => { + const ds = existingDs(); + ds.session.cliId = 'codex-app'; + ds.session.codexAppDispatchLedger = [{ + dispatchId: 'dispatch-owned', + turnId: 'turn-owned', + state: 'prepared', + content: 'durable existing turn', + deliverySink: 'lark', + }]; + + const res = await triggerSessionTurn(request(), { + larkAppId: APP, + activeSessions: new Map([[sessionKey(ROOT, APP), ds]]), + }); + + expect(res).toMatchObject({ + ok: false, + errorCode: 'trigger_failed', + error: expect.stringContaining('durable_owner'), + }); + expect(mockForkWorker).not.toHaveBeenCalled(); + expect(mockRememberLastCliInput).not.toHaveBeenCalled(); + }); + it('preserves the clean split when an external event reforks a stopped Codex App session', async () => { mockGetBot.mockReturnValue({ config: { larkAppId: APP, cliId: 'codex-app', codexAppCleanInput: true, workingDir: '/tmp' }, @@ -379,6 +576,37 @@ describe('triggerSessionTurn rootMessageId target', () => { expect(mockRunAutoWorktreeCommit).toHaveBeenCalledWith(expect.objectContaining({ ds })); }); + it('closes the unpublished trigger row when auto-worktree setup persistence fails', async () => { + mockBotAutoWorktreeEnabled.mockReturnValue(true); + mockUpdateSession + .mockImplementationOnce(() => undefined) + .mockImplementationOnce(() => { throw new Error('setup store unavailable'); }); + const activeSessions = new Map(); + + await expect(triggerSessionTurn(request(), { larkAppId: APP, activeSessions })) + .rejects.toThrow('setup store unavailable'); + + expect(activeSessions.has(sessionKey(ROOT, APP))).toBe(false); + expect(mockCloseSession).toHaveBeenCalledTimes(1); + expect(mockCloseSession).toHaveBeenCalledWith('sess_new'); + expect(mockRunAutoWorktreeCommit).not.toHaveBeenCalled(); + expect(mockForkWorker).not.toHaveBeenCalled(); + }); + + it('never closes an incumbent when the trigger creation path loses the key claim', async () => { + mockBotAutoWorktreeEnabled.mockReturnValue(true); + const send = vi.fn(); + const incumbent = existingDs({ worker: { killed: false, send } as any }); + const activeSessions = new Map([[sessionKey(ROOT, APP), incumbent]]); + + const result = await triggerSessionTurn(request(), { larkAppId: APP, activeSessions }); + + expect(result).toMatchObject({ ok: true, target: { sessionId: incumbent.session.sessionId } }); + expect(activeSessions.get(sessionKey(ROOT, APP))).toBe(incumbent); + expect(mockCloseSession).not.toHaveBeenCalled(); + expect(mockCreateSession).not.toHaveBeenCalled(); + }); + it('passes the clean split into a new Codex App session without worktree staging', async () => { mockGetBot.mockReturnValue({ config: { larkAppId: APP, cliId: 'codex-app', codexAppCleanInput: true, workingDir: '/tmp' }, @@ -477,6 +705,72 @@ describe('triggerSessionTurn rootMessageId target', () => { expect(mockCreateSession).not.toHaveBeenCalled(); }); + it('shares first-owner locking with a paused resume claim and reuses the resume winner', async () => { + const activeSessions = new Map(); + const key = sessionKey(ROOT, APP); + const resumed = existingDs(); + resumed.session.cliId = 'claude-code'; + let resumeEntered!: () => void; + let releaseResume!: () => void; + const entered = new Promise(resolve => { resumeEntered = resolve; }); + const paused = new Promise(resolve => { releaseResume = resolve; }); + + const resumeClaim = withActiveSessionKeyLock(activeSessions, key, async () => { + expect(activeSessions.has(key)).toBe(false); + resumeEntered(); + await paused; + activeSessions.set(key, resumed); + }); + await entered; + + const triggering = triggerSessionTurn(request(), { larkAppId: APP, activeSessions }); + await new Promise(resolve => setTimeout(resolve, 0)); + expect(mockCreateSession).not.toHaveBeenCalled(); + + releaseResume(); + await resumeClaim; + const result = await triggering; + + expect(result).toMatchObject({ ok: true, target: { sessionId: 'sess_existing' } }); + expect(activeSessions.get(key)).toBe(resumed); + expect(mockCreateSession).not.toHaveBeenCalled(); + expect(mockForkWorker).toHaveBeenCalledWith(resumed, expect.anything(), expect.objectContaining({ resume: true })); + }); + + it('keeps the opening reservation and buffers when new-session fork pre-accept throws', async () => { + mockForkWorker.mockImplementationOnce(() => { throw new Error('fork preaccept failed'); }); + const activeSessions = new Map(); + + await expect(triggerSessionTurn(request(), { larkAppId: APP, activeSessions })) + .rejects.toThrow('fork preaccept failed'); + + const ds = activeSessions.get(sessionKey(ROOT, APP)); + expect(ds?.initialStartPending).toBe(true); + expect(ds?.pendingPrompt).toContain(''); + expect(ds?.pendingCodexAppText).toBe('外部事件触发'); + }); + + it('cleans the wait registry but retains the opening reservation when write-ahead throws', async () => { + const activeSessions = new Map(); + const req = request(); + req.options = { waitForFinalOutput: true, timeoutMs: 1000 }; + + const result = await triggerSessionTurn( + req, + { larkAppId: APP, activeSessions }, + { + stableTurnId: 'stable_write_ahead_failure', + beforeDispatch: () => { throw new Error('write-ahead failed'); }, + }, + ); + + expect(result).toMatchObject({ ok: false, errorCode: 'trigger_failed', error: 'write-ahead failed' }); + const ds = activeSessions.get(sessionKey(ROOT, APP)); + expect(ds?.initialStartPending).toBe(true); + expect(ds?.pendingWaitPromises?.size ?? 0).toBe(0); + expect(mockForkWorker).not.toHaveBeenCalled(); + }); + it('requires chatId when rootMessageId is specified', async () => { const activeSessions = new Map(); const res = await triggerSessionTurn(request({ chatId: undefined }), { larkAppId: APP, activeSessions }); diff --git a/test/v3-session-relay-client.test.ts b/test/v3-session-relay-client.test.ts index 2e82aa35d..6e4e57444 100644 --- a/test/v3-session-relay-client.test.ts +++ b/test/v3-session-relay-client.test.ts @@ -15,6 +15,7 @@ import { } from '../src/workflows/v3/session-relay-client.js'; const CAPABILITY = 'c'.repeat(64); +const ORIGIN_CHANNEL = 'a'.repeat(64); describe('readWorkflowSessionRelayContext', () => { let root: string; @@ -49,6 +50,7 @@ describe('readWorkflowSessionRelayContext', () => { writeFileSync( join(relayDir, RELAY_ORIGIN_CAPABILITY_BASENAME), JSON.stringify({ token: CAPABILITY }), + { mode: 0o600 }, ); const context = readWorkflowSessionRelayContext({ env: { @@ -72,41 +74,59 @@ describe('readWorkflowSessionRelayContext', () => { // host session (whose live marker is visible) must keep the strictly // stronger marker + signed-envelope path instead of being hijacked onto // the relay with a stale token. - const carveOut = managedOriginCapabilityPath(dataDir, 'sess-1'); + const carveOut = managedOriginCapabilityPath(dataDir, 'sess-1', ORIGIN_CHANNEL); mkdirSync(dirname(carveOut), { recursive: true }); - writeFileSync(carveOut, JSON.stringify({ sessionId: 'sess-1', capability: CAPABILITY })); + writeFileSync(carveOut, JSON.stringify({ + sessionId: 'sess-1', + channelId: ORIGIN_CHANNEL, + capability: CAPABILITY, + }), { mode: 0o600 }); expect(readWorkflowSessionRelayContext({ - env: { BOTMUX_SESSION_ID: 'sess-1' }, + env: { + BOTMUX_SESSION_ID: 'sess-1', + BOTMUX_ORIGIN_CHANNEL_ID: ORIGIN_CHANNEL, + }, dataDir, findMarker: () => ({ sessionId: 'sess-1', turnId: 'turn-1' }), })).toBeNull(); // A corrupt marker ({sessionId: ''}) does not count as live — same // precedence as resolveSessionContext. expect(readWorkflowSessionRelayContext({ - env: { BOTMUX_SESSION_ID: 'sess-1' }, + env: { + BOTMUX_SESSION_ID: 'sess-1', + BOTMUX_ORIGIN_CHANNEL_ID: ORIGIN_CHANNEL, + }, dataDir, findMarker: () => ({ sessionId: '' }), })).not.toBeNull(); }); it('detects a macOS read-isolated session via the per-session carve-out file', () => { - const carveOut = managedOriginCapabilityPath(dataDir, 'sess-1'); + const carveOut = managedOriginCapabilityPath(dataDir, 'sess-1', ORIGIN_CHANNEL); mkdirSync(dirname(carveOut), { recursive: true }); writeFileSync(carveOut, JSON.stringify({ sessionId: 'sess-1', + channelId: ORIGIN_CHANNEL, capability: CAPABILITY, turnId: 'turn-7', dispatchAttempt: 2, - })); + ipcPort: 4321, + }), { mode: 0o600 }); const context = readWorkflowSessionRelayContext({ - env: { BOTMUX_SESSION_ID: 'sess-1' }, + env: { + BOTMUX_SESSION_ID: 'sess-1', + BOTMUX_ORIGIN_CHANNEL_ID: ORIGIN_CHANNEL, + BOTMUX_DAEMON_IPC_PORT: '4999', + }, dataDir, }); expect(context).toEqual({ sessionId: 'sess-1', capability: CAPABILITY, + originChannelId: ORIGIN_CHANNEL, turnId: 'turn-7', dispatchAttempt: 2, + ipcPortFallback: 4321, }); }); @@ -114,6 +134,7 @@ describe('readWorkflowSessionRelayContext', () => { writeFileSync( join(relayDir, RELAY_ORIGIN_CAPABILITY_BASENAME), JSON.stringify({ token: CAPABILITY }), + { mode: 0o600 }, ); for (const bad of ['abc', '0', '-4310', '4310.5']) { const context = readWorkflowSessionRelayContext({ @@ -133,6 +154,7 @@ describe('postWorkflowSessionRunMutation', () => { const context: WorkflowSessionRelayContext = { sessionId: 'sess-1', capability: CAPABILITY, + originChannelId: ORIGIN_CHANNEL, turnId: 'turn-7', dispatchAttempt: 2, larkAppId: 'cli_owner', @@ -156,11 +178,14 @@ describe('postWorkflowSessionRunMutation', () => { expect(response).toEqual({ ok: true, status: 200, bodyRaw: JSON.stringify({ ok: true }) }); expect(fetchImpl).toHaveBeenCalledOnce(); const [url, init] = fetchImpl.mock.calls[0]! as unknown as [string, RequestInit]; - expect(url).toBe('http://127.0.0.1:4999/api/v3/session-runs/run-1/cancel'); + // The protected capability snapshot's port is authoritative routing data; + // mutable discovery descriptors cannot override it. + expect(url).toBe('http://127.0.0.1:4310/api/v3/session-runs/run-1/cancel'); expect(JSON.parse(String(init.body))).toEqual({ reason: 'stop', sessionId: 'sess-1', originCapability: CAPABILITY, + originChannelId: ORIGIN_CHANNEL, originTurnId: 'turn-7', originDispatchAttempt: 2, }); @@ -182,9 +207,9 @@ describe('postWorkflowSessionRunMutation', () => { }); }); - it('prefers discovery, falls back to the env port marker, then fails closed', async () => { + it('prefers the protected capability port, falls back to discovery, then fails closed', async () => { const fetchImpl = fetchOk(); - const resolveIpcPort = vi.fn(() => undefined); + const resolveIpcPort = vi.fn(() => 4999); await postWorkflowSessionRunMutation({ context, runId: 'run-1', @@ -195,6 +220,20 @@ describe('postWorkflowSessionRunMutation', () => { expect(resolveIpcPort).toHaveBeenCalledWith('cli_owner'); expect(String(fetchImpl.mock.calls[0]![0])).toContain(':4310/'); + const discoveryFetch = fetchOk(); + await postWorkflowSessionRunMutation({ + context: { + sessionId: 'sess-1', + capability: CAPABILITY, + larkAppId: 'cli_owner', + }, + runId: 'run-1', + mutation: 'start', + resolveIpcPort, + fetchImpl: discoveryFetch, + }); + expect(String(discoveryFetch.mock.calls[0]![0])).toContain(':4999/'); + await expect(postWorkflowSessionRunMutation({ context: { sessionId: 'sess-1', capability: CAPABILITY }, runId: 'run-1', diff --git a/test/vc-meeting-daemon-session.test.ts b/test/vc-meeting-daemon-session.test.ts index 205c671fb..75e929ab2 100644 --- a/test/vc-meeting-daemon-session.test.ts +++ b/test/vc-meeting-daemon-session.test.ts @@ -614,22 +614,27 @@ async function waitForConsumerApplyFinalCard(afterIndex = patchedMessages.length } /** 新交互流程：下拉选 agent 只暂存，点"确认"才生效。返回确认后的卡片响应。 */ -async function selectConsumerAgentViaCard(label: string, operatorOpenId = TARGET_OPEN_ID): Promise { - await __vcMeetingAgentTest.handleCardAction({ - operator: { open_id: operatorOpenId }, - action: lastInteractiveCardSelectOption(label), - }, APP_ID); +async function confirmLatestConsumerCard(operatorOpenId = TARGET_OPEN_ID): Promise { const patchIndex = patchedMessages.length; const result = await __vcMeetingAgentTest.handleCardAction({ operator: { open_id: operatorOpenId }, action: { value: lastInteractiveCardButton('确认') }, }, APP_ID); - if (result?.header?.title?.content === '会议处理设置中') { + if (result?.header?.title?.content === '会议处理设置中' + || result?.toast?.content === '会议 agent 选择正在处理中') { return (await waitForConsumerApplyFinalCard(patchIndex)) ?? result; } return result; } +async function selectConsumerAgentViaCard(label: string, operatorOpenId = TARGET_OPEN_ID): Promise { + await __vcMeetingAgentTest.handleCardAction({ + operator: { open_id: operatorOpenId }, + action: lastInteractiveCardSelectOption(label), + }, APP_ID); + return confirmLatestConsumerCard(operatorOpenId); +} + async function selectConsumerProfilesViaCard( profileIds: readonly string[], initialCard?: any, @@ -5654,7 +5659,14 @@ describe('VC meeting daemon session lifecycle', () => { expect(triggerSessionCalls).toHaveLength(0); __vcMeetingAgentTest.setConsumerPendingItemLimitForTest(undefined); - await selectConsumerAgentViaCard('Claude Loopy'); + for (let i = 0; i < 20 + && sentMessages.filter(message => message.msgType === 'interactive').length < 2; + i += 1) await Promise.resolve(); + expect(sentMessages.filter(message => message.msgType === 'interactive')).toHaveLength(2); + // The overflow recovery card already stages the current agent; confirming + // it resumes the same durable member/cursor without starting a second + // dropdown apply in parallel. + await confirmLatestConsumerCard(); await __vcMeetingAgentTest.injectConsumer(APP_ID, 'm_joined_464646464', { force: true }); expect(triggerSessionCalls).toHaveLength(1); expect(triggerSessionCalls[0].req.envelope.payload.entries.filter((entry: any) => entry.kind === 'item')) diff --git a/test/vc-meeting-receiver-recovery-lifecycle.test.ts b/test/vc-meeting-receiver-recovery-lifecycle.test.ts index 1a4e12ec6..642cc1731 100644 --- a/test/vc-meeting-receiver-recovery-lifecycle.test.ts +++ b/test/vc-meeting-receiver-recovery-lifecycle.test.ts @@ -116,6 +116,40 @@ describe('VC meeting receiver boot-recovery lifecycle', () => { expect(recovery.snapshot(key)).toMatchObject({ ready: true, pending: false, timerArmed: false }); }); + it('keeps the boot fence until exact dispatch retirement persists after backing is missing', async () => { + recovery.setBackingMissingProbe(() => true); + let retirementWorks = false; + const retired: Array<[string, string, number]> = []; + recovery.setDispatchRetirement((sessionId, turnId, dispatchAttempt) => { + retired.push([sessionId, turnId, dispatchAttempt]); + return retirementWorks; + }); + const key = recovery.start('sess_retire', 'delivery_retire', 5, { + memberId: 'member_retire', + }); + recovery.finishScheduling(); + + recovery.acknowledge('sess_retire', 'delivery_retire', 5); + expect(recovery.snapshot(key)).toMatchObject({ ready: false, pending: true, timerArmed: true }); + expect(retired).toEqual([['sess_retire', 'delivery_retire', 5]]); + + await vi.advanceTimersByTimeAsync(8_000); + expect(recovery.snapshot(key)).toMatchObject({ ready: false, pending: true, timerArmed: true }); + expect(retired).toEqual([ + ['sess_retire', 'delivery_retire', 5], + ['sess_retire', 'delivery_retire', 5], + ]); + + retirementWorks = true; + await vi.advanceTimersByTimeAsync(5_000); + expect(retired).toEqual([ + ['sess_retire', 'delivery_retire', 5], + ['sess_retire', 'delivery_retire', 5], + ['sess_retire', 'delivery_retire', 5], + ]); + expect(recovery.snapshot(key)).toMatchObject({ ready: true, pending: false, timerArmed: false }); + }); + it('keeps the boot gate pending when reset IPC send throws', () => { const failureLog = daemonSource.indexOf('failed to fence boot-ambiguous receiver'); expect(failureLog).toBeGreaterThanOrEqual(0); diff --git a/test/vc-meeting-runtime-lease-recovery.test.ts b/test/vc-meeting-runtime-lease-recovery.test.ts index 18f350db6..bbbc9d2ee 100644 --- a/test/vc-meeting-runtime-lease-recovery.test.ts +++ b/test/vc-meeting-runtime-lease-recovery.test.ts @@ -86,6 +86,7 @@ function harness(input: { probe?: (backend: 'tmux' | 'herdr' | 'zellij', sessionName: string) => 'exists' | 'missing' | 'unknown'; missingPersistentScope?: FakeSession['testPersistentScope']; backendAvailable?: (backend: 'tmux' | 'herdr' | 'zellij') => boolean; + retireDispatch?: (sessionId: string, turnId: string, dispatchAttempt: number) => boolean; } = {}) { const sessions = new Map((input.sessions ?? []).map(ds => [ds.session.sessionId, ds])); const sent: Array<{ sessionId: string; turnId: string; dispatchAttempt: number }> = []; @@ -94,6 +95,7 @@ function harness(input: { const probes: string[] = []; const warnings: string[] = []; const errors: string[] = []; + const retired: Array<{ sessionId: string; turnId: string; dispatchAttempt: number }> = []; const recovery = createRecovery({ findSession: (sessionId: string) => sessions.get(sessionId), sendExpiry: (ds: FakeSession, message: { turnId: string; dispatchAttempt: number }) => { @@ -117,10 +119,14 @@ function harness(input: { probes.push(`${backend}:${sessionName}`); return input.probe?.(backend, sessionName) ?? 'missing'; }, + retireDispatch: (sessionId: string, turnId: string, dispatchAttempt: number) => { + retired.push({ sessionId, turnId, dispatchAttempt }); + return input.retireDispatch?.(sessionId, turnId, dispatchAttempt) ?? true; + }, warn: (message: string) => warnings.push(message), error: (message: string) => errors.push(message), } as any); - return { recovery, sessions, sent, killed, backingKills, probes, warnings, errors }; + return { recovery, sessions, sent, killed, backingKills, probes, retired, warnings, errors }; } describe('VC meeting runtime lease recovery', () => { @@ -196,6 +202,26 @@ describe('VC meeting runtime lease recovery', () => { expect(h.killed).toEqual(['session_a']); expect(h.backingKills).toEqual(['tmux:bmx-session_']); expect(h.probes).toEqual(['tmux:bmx-session_']); + expect(h.retired).toEqual([{ + sessionId: 'session_a', turnId: 'delivery_a', dispatchAttempt: 1, + }]); + expect(h.recovery.snapshot()).toEqual([]); + }); + + it('keeps a workerless receiver fenced when exact ledger retirement cannot persist', async () => { + let retirementWorks = false; + const h = harness({ + sessions: [fakeSession({ worker: false, persistentScope: 'tmux' })], + retireDispatch: () => retirementWorks, + }); + h.recovery.arm(ref(), 'agent_test'); + + expect(h.recovery.snapshot()).toMatchObject([{ phase: 'blocked', timerArmed: true }]); + expect(h.errors.some(message => message.includes('dispatch retirement failed'))).toBe(true); + + retirementWorks = true; + await vi.advanceTimersByTimeAsync(5_000); + expect(h.retired).toHaveLength(2); expect(h.recovery.snapshot()).toEqual([]); }); diff --git a/test/voice-provider-fence.test.ts b/test/voice-provider-fence.test.ts new file mode 100644 index 000000000..ab7f907c6 --- /dev/null +++ b/test/voice-provider-fence.test.ts @@ -0,0 +1,190 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +const { FakeWebSocket } = vi.hoisted(() => { + class FakeWebSocket { + static instances: FakeWebSocket[] = []; + + readonly url: string; + readonly send = vi.fn(); + readonly close = vi.fn(); + private readonly listeners = new Map unknown>>(); + + constructor(url: string) { + this.url = url; + FakeWebSocket.instances.push(this); + } + + on(event: string, listener: (...args: any[]) => unknown): this { + const listeners = this.listeners.get(event) ?? []; + listeners.push(listener); + this.listeners.set(event, listeners); + return this; + } + + async emit(event: string, ...args: any[]): Promise { + for (const listener of [...(this.listeners.get(event) ?? [])]) { + await listener(...args); + } + } + } + + return { FakeWebSocket }; +}); + +vi.mock('ws', () => ({ default: FakeWebSocket })); + +import { openaiSynthesizePcm } from '../src/services/voice/openai.js'; +import { mintSamiToken, samiSynthesizePcm } from '../src/services/voice/sami.js'; + +const SAMI_CREDS = { + accessKey: 'access', + secretKey: 'secret', + appkey: 'app', + tokenUrl: 'https://token.example.test', + wsUrl: 'wss://speech.example.test', +}; + +function tokenResponse(): Response { + return new Response(JSON.stringify({ token: 'short-lived-token' }), { + status: 200, + headers: { 'content-type': 'application/json' }, + }); +} + +function deferred(): { promise: Promise; resolve: () => void } { + let resolve!: () => void; + const promise = new Promise((done) => { resolve = done; }); + return { promise, resolve }; +} + +describe('voice provider effect fences', () => { + beforeEach(() => { + FakeWebSocket.instances.length = 0; + }); + + afterEach(() => { + vi.unstubAllGlobals(); + vi.restoreAllMocks(); + }); + + it('fences SAMI token minting before fetch and fails closed when revoked', async () => { + const fetchMock = vi.fn(async () => tokenResponse()); + vi.stubGlobal('fetch', fetchMock); + + await expect(mintSamiToken(SAMI_CREDS, 60, { + beforeProviderEffect: () => { throw new Error('origin revoked before token'); }, + })).rejects.toThrow('origin revoked before token'); + + expect(fetchMock).not.toHaveBeenCalled(); + }); + + it('fences SAMI WebSocket construction after token minting', async () => { + const fetchMock = vi.fn(async () => tokenResponse()); + vi.stubGlobal('fetch', fetchMock); + let fenceCall = 0; + const beforeProviderEffect = vi.fn(async () => { + fenceCall += 1; + if (fenceCall === 2) throw new Error('origin revoked before connect'); + }); + + await expect(samiSynthesizePcm( + SAMI_CREDS, + 'hello', + { speaker: 'voice' }, + { beforeProviderEffect }, + )).rejects.toThrow('origin revoked before connect'); + + expect(fetchMock).toHaveBeenCalledTimes(1); + expect(beforeProviderEffect).toHaveBeenCalledTimes(2); + expect(FakeWebSocket.instances).toHaveLength(0); + }); + + it('awaits a fresh fence after async WebSocket open before sending', async () => { + vi.stubGlobal('fetch', vi.fn(async () => tokenResponse())); + const sendFence = deferred(); + let fenceCall = 0; + const beforeProviderEffect = vi.fn(async () => { + fenceCall += 1; + if (fenceCall === 3) await sendFence.promise; + }); + + const synthesis = samiSynthesizePcm( + SAMI_CREDS, + ' hello ', + { speaker: 'voice' }, + { beforeProviderEffect }, + ); + await vi.waitFor(() => expect(FakeWebSocket.instances).toHaveLength(1)); + const socket = FakeWebSocket.instances[0]!; + + const opening = socket.emit('open'); + await vi.waitFor(() => expect(beforeProviderEffect).toHaveBeenCalledTimes(3)); + expect(socket.send).not.toHaveBeenCalled(); + + sendFence.resolve(); + await opening; + expect(socket.send).toHaveBeenCalledTimes(1); + expect(JSON.parse(String(socket.send.mock.calls[0]![0]))).toMatchObject({ + token: 'short-lived-token', + appkey: 'app', + namespace: 'TTS', + event: 'StartTask', + }); + + await socket.emit('message', Buffer.from([1, 2, 3]), true); + await socket.emit('message', Buffer.from(JSON.stringify({ + status_code: 20000000, + event: 'TaskFinished', + })), false); + await expect(synthesis).resolves.toMatchObject({ + data: Buffer.from([1, 2, 3]), + sampleRate: 24000, + channels: 1, + }); + }); + + it('does not send on an opened SAMI socket when the last-moment fence revokes', async () => { + vi.stubGlobal('fetch', vi.fn(async () => tokenResponse())); + let fenceCall = 0; + const beforeProviderEffect = vi.fn(async () => { + fenceCall += 1; + if (fenceCall === 3) throw new Error('origin revoked before send'); + }); + + const synthesis = samiSynthesizePcm( + SAMI_CREDS, + 'hello', + { speaker: 'voice' }, + { beforeProviderEffect }, + ); + await vi.waitFor(() => expect(FakeWebSocket.instances).toHaveLength(1)); + const socket = FakeWebSocket.instances[0]!; + + await socket.emit('open'); + + await expect(synthesis).rejects.toThrow('origin revoked before send'); + expect(beforeProviderEffect).toHaveBeenCalledTimes(3); + expect(socket.send).not.toHaveBeenCalled(); + expect(socket.close).toHaveBeenCalledTimes(1); + }); + + it('runs the OpenAI fence immediately before the provider fetch', async () => { + const order: string[] = []; + const fetchMock = vi.fn(async () => { + order.push('fetch'); + return new Response(new Uint8Array([4, 5, 6]), { status: 200 }); + }); + vi.stubGlobal('fetch', fetchMock); + + const pcm = await openaiSynthesizePcm( + { baseUrl: 'https://openai.example.test/v1/', apiKey: 'key', model: 'tts-model' }, + 'hello', + { speaker: 'alloy' }, + { beforeProviderEffect: () => { order.push('fence'); } }, + ); + + expect(order).toEqual(['fence', 'fetch']); + expect(fetchMock).toHaveBeenCalledTimes(1); + expect(pcm).toMatchObject({ data: Buffer.from([4, 5, 6]), sampleRate: 24000, channels: 1 }); + }); +}); diff --git a/test/worker-app-runner-control-wiring.test.ts b/test/worker-app-runner-control-wiring.test.ts index 38ac3530e..eab412f82 100644 --- a/test/worker-app-runner-control-wiring.test.ts +++ b/test/worker-app-runner-control-wiring.test.ts @@ -1,5 +1,9 @@ import { readFileSync } from 'node:fs'; -import { describe, expect, it } from 'vitest'; +import { describe, expect, it, vi } from 'vitest'; +import { + CodexAppControlProofDeadline, + codexAppSignedStateReadiness, +} from '../src/utils/codex-app-control.js'; const workerSource = readFileSync(new URL('../src/worker.ts', import.meta.url), 'utf8'); @@ -11,10 +15,218 @@ describe('worker app-runner control-channel wiring', () => { expect(workerSource).not.toContain('codexAppOscPending'); }); - it('rejects marker identity mismatches and keeps dispatch authority worker-owned', () => { - expect(workerSource).toContain('if (!identity.ok)'); - expect(workerSource).toContain('payload.dispatchAttempt !== currentBotmuxDispatchAttempt'); - expect(workerSource).toContain('const dispatchAttempt = currentBotmuxDispatchAttempt;'); - expect(workerSource).not.toContain('const dispatchAttempt = payload.dispatchAttempt'); + it('reserves Codex App attribution before writes and settles finals only from the worker FIFO', () => { + const flushStart = workerSource.indexOf('async function flushPending'); + const flushEnd = workerSource.indexOf('function sendToPty', flushStart); + const flush = workerSource.slice(flushStart, flushEnd); + const reserveIdx = flush.indexOf('codexAppTurnDispatchQueue.reserve('); + const writeIdx = flush.indexOf('await writeAdapter.writeStructuredInput('); + expect(reserveIdx).toBeGreaterThan(-1); + expect(writeIdx).toBeGreaterThan(reserveIdx); + expect(flush).toContain("result.submissionDisposition === 'untouched'"); + expect(flush).toContain("result.submissionDisposition === 'flushed_invalid'"); + expect(flush).toContain('input buffer is not provably clean'); + expect(flush).toContain('if (backend && dispatchStillPending) scheduleSubmitFailureNotify('); + expect(flush).toContain('if (backend && dispatchStillPending) {'); + const safeRetryStart = flush.indexOf('const retryQueuedActivation ='); + const retryTransition = flush.indexOf("retryQueuedActivation ? 'retry' : 'cancel'", safeRetryStart); + const requeue = flush.indexOf('requeueUnsubmittedQueuedActivation(item);', retryTransition); + const submittedAck = flush.indexOf("type: 'queued_activation_submitted'", retryTransition); + expect(safeRetryStart).toBeGreaterThan(writeIdx); + expect(retryTransition).toBeGreaterThan(safeRetryStart); + expect(requeue).toBeGreaterThan(retryTransition); + expect(submittedAck).toBeGreaterThan(requeue); + expect(flush.slice(safeRetryStart, submittedAck)).toContain('codexAppSafeNonSubmission'); + expect(flush.slice(safeRetryStart, submittedAck)).not.toContain("type: 'queued_activation_submitted'"); + + const markerStart = workerSource.indexOf('function handleTrustedCodexAppMarker('); + const markerEnd = workerSource.indexOf('function handleAppRunnerOscMarker(', markerStart); + const marker = workerSource.slice(markerStart, markerEnd); + expect(marker).toContain('const settlement = codexAppTurnDispatchQueue.settleFinal(payload, false);'); + expect(marker).toContain('codexAppTurnDispatchQueue.commitExactHead(codexAppDispatchHandle)'); + expect(workerSource).toContain('codexAppControlRecordApplicationGate.run('); + expect(workerSource).toContain('codexAppControlReplayWindow.commit(identity.generation, record.seq);'); + expect(workerSource.indexOf('codexAppControlRecordApplicationGate.run(')) + .toBeLessThan(workerSource.indexOf('codexAppControlReplayWindow.commit(identity.generation, record.seq);')); + const codexSettlementStart = marker.indexOf('const settlement = codexAppTurnDispatchQueue.settleFinal(payload, false);'); + const miraFallbackStart = marker.indexOf('} else {\n // Mira/Mir', codexSettlementStart); + expect(marker.slice(codexSettlementStart, miraFallbackStart)).not.toContain('currentBotmuxTurnId'); + expect(marker.slice(codexSettlementStart, miraFallbackStart)).not.toContain('currentBotmuxDispatchAttempt'); + expect(marker).not.toContain('const dispatchAttempt = payload.dispatchAttempt'); + expect(marker).toContain('empty final settled for botmux turn'); + expect(workerSource).not.toContain('settleLegacyCodexAppEmptyFinal'); + expect(marker).toContain('published idle before the required final transaction'); + expect(marker).toContain('submitted the next turn before the required final transaction'); + }); + + it('ACKs a fresh RPC queued activation only after confirmed turn/start acceptance', () => { + const engageStart = workerSource.indexOf('async function engageCodexRpc('); + const engageEnd = workerSource.indexOf('/** RPC panes have NO terminal input path', engageStart); + const engage = workerSource.slice(engageStart, engageEnd); + const firstTurn = engage.indexOf('await engine.sendFirstTurn('); + const accepted = engage.indexOf("if (first.outcome === 'accepted' && cfg.queuedActivationToken)", firstTurn); + const ack = engage.indexOf("type: 'queued_activation_submitted'", accepted); + expect(firstTurn).toBeGreaterThan(-1); + expect(accepted).toBeGreaterThan(firstTurn); + expect(ack).toBeGreaterThan(accepted); + expect(engage.slice(firstTurn, accepted)).toContain("if (first.outcome === 'not-sent')"); + }); + + it('restores the durable FIFO but never treats warm signed idle as proof that prepared input was unwritten', () => { + const activateStart = workerSource.indexOf('function activateCodexAppControlConnection('); + const activateEnd = workerSource.indexOf('function handleCodexAppControlLine(', activateStart); + const activate = workerSource.slice(activateStart, activateEnd); + expect(activate).not.toContain('markPromptReady()'); + expect(workerSource).toContain('codexAppTurnDispatchQueue.restore('); + expect(workerSource).not.toContain('requeueUnwrittenRecoveredCodexAppPrefix'); + expect(workerSource).not.toContain("requestCodexAppDispatchTransition('reset'"); + + const markerStart = workerSource.indexOf('async function handleTrustedCodexAppMarker('); + const markerEnd = workerSource.indexOf('function handleAppRunnerOscMarker(', markerStart); + const marker = workerSource.slice(markerStart, markerEnd); + expect(marker).toContain('codexAppTurnDispatchQueue.recoveredPrefix().length > 0'); + expect(marker).toContain('Codex App signed idle cannot prove the recovered prepared frame was never buffered'); + + const stopStart = workerSource.indexOf('function stopCodexAppControlChannel('); + const stopEnd = workerSource.indexOf('function failCodexAppControlGeneration(', stopStart); + const stop = workerSource.slice(stopStart, stopEnd); + expect(stop).toContain('if (!opts.preserveDispatchRecovery) {'); + expect(stop.indexOf('codexAppTurnDispatchQueue.clear();')) + .toBeGreaterThan(stop.indexOf('if (!opts.preserveDispatchRecovery) {')); + + const prepareStart = workerSource.indexOf('async function prepareCodexAppControlGeneration('); + const prepareEnd = workerSource.indexOf('async function rotateCodexAppControlEndpoint(', prepareStart); + expect(workerSource.slice(prepareStart, prepareEnd)) + .toContain('stopCodexAppControlChannel({ preserveDispatchRecovery: true });'); + }); + + it('keeps auth-to-state proof armed and permits type-ahead only after signed runner readiness', () => { + const activateStart = workerSource.indexOf('function activateCodexAppControlConnection('); + const activateEnd = workerSource.indexOf('function handleCodexAppControlLine(', activateStart); + const activate = workerSource.slice(activateStart, activateEnd); + expect(activate).not.toContain('codexAppProofDeadline.clear();'); + expect(activate).toContain('Authenticated Codex App runner did not publish signed state'); + + const markerStart = workerSource.indexOf('async function handleTrustedCodexAppMarker('); + const markerEnd = workerSource.indexOf('function handleAppRunnerOscMarker(', markerStart); + const marker = workerSource.slice(markerStart, markerEnd); + expect(marker.indexOf('codexAppSignedStateObserved = true;')).toBeGreaterThan( + marker.indexOf('if (!state.accepted)'), + ); + expect(marker).toContain('codexAppProofDeadline.clear();'); + expect(marker).toContain("if (readiness === 'invalid')"); + expect(marker).toContain("if (readiness === 'waiting')"); + expect(marker).toContain('codexAppInputReady = true;'); + const invalidStart = marker.indexOf("if (readiness === 'invalid')"); + const waitingStart = marker.indexOf("if (readiness === 'waiting')"); + const applyStart = marker.indexOf('const state = applyTrustedCodexAppStateMarker(', waitingStart); + expect(marker.slice(invalidStart, waitingStart)).toContain('failCodexAppControlGeneration('); + expect(marker.slice(waitingStart, applyStart)).toContain('codexAppProofDeadline.armed'); + expect(marker.slice(waitingStart, applyStart)).not.toContain('codexAppProofDeadline.clear();'); + expect(marker.indexOf('codexAppProofDeadline.clear();')).toBeGreaterThan(applyStart); + + const runtimeGateStart = workerSource.indexOf('function codexAppRuntimeTypeAheadReady()'); + const runtimeGateEnd = workerSource.indexOf('async function flushPending()', runtimeGateStart); + const runtimeGate = workerSource.slice(runtimeGateStart, runtimeGateEnd); + expect(runtimeGate).toContain('codexAppControlProven'); + expect(runtimeGate).toContain('codexAppSignedStateObserved'); + expect(runtimeGate).toContain('codexAppInputReady'); + expect(workerSource).toContain('projectCodexAppControlReadinessStatus(base, {'); + const firstPromptTimeout = workerSource.slice( + workerSource.indexOf('const releaseFirstPromptTimeout'), + workerSource.indexOf('// Riff (and other remote HTTP backends)'), + ); + expect(firstPromptTimeout).toContain( + "if (decideHardTimeoutAction(cliAdapter?.supportsTypeAhead === true) === 'flush')", + ); + expect(firstPromptTimeout).not.toContain('codexAppRuntimeTypeAheadReady()'); + }); + + it('keeps the proof timer armed for acceptingInput:false and clears it only for true', async () => { + vi.useFakeTimers(); + const deadline = new CodexAppControlProofDeadline(); + try { + const falseTimedOut = vi.fn(); + deadline.arm(falseTimedOut, 100); + expect(codexAppSignedStateReadiness({ busy: false, acceptingInput: false })).toBe('waiting'); + await vi.advanceTimersByTimeAsync(100); + expect(falseTimedOut).toHaveBeenCalledTimes(1); + + const missingTimedOut = vi.fn(); + deadline.arm(missingTimedOut, 100); + expect(codexAppSignedStateReadiness({ busy: false })).toBe('invalid'); + await vi.advanceTimersByTimeAsync(100); + expect(missingTimedOut).toHaveBeenCalledTimes(1); + + const readyTimedOut = vi.fn(); + deadline.arm(readyTimedOut, 100); + expect(codexAppSignedStateReadiness({ busy: false, acceptingInput: true })).toBe('ready'); + deadline.clear(); + await vi.advanceTimersByTimeAsync(100); + expect(readyTimedOut).not.toHaveBeenCalled(); + } finally { + deadline.clear(); + vi.useRealTimers(); + } + }); + + it('rejects fresh authentication with recovered prepared ownership before activation or publication', () => { + const activateStart = workerSource.indexOf('function activateCodexAppControlConnection('); + const activateEnd = workerSource.indexOf('async function handleCodexAppControlLine(', activateStart); + const activate = workerSource.slice(activateStart, activateEnd); + const recoveredGuard = activate.indexOf("proofKind === 'fresh runner'"); + const fail = activate.indexOf('failCodexAppControlGeneration(', recoveredGuard); + const persist = activate.indexOf('persistCodexAppControlState(', recoveredGuard); + const accepted = activate.indexOf('encodeCodexAppControlAccepted(', recoveredGuard); + const published = activate.indexOf("type: 'codex_app_generation_active'", recoveredGuard); + + expect(recoveredGuard).toBeGreaterThan(-1); + expect(fail).toBeGreaterThan(recoveredGuard); + expect(persist).toBeGreaterThan(fail); + expect(accepted).toBeGreaterThan(fail); + expect(published).toBeGreaterThan(fail); + }); + + it('fails the worker generation before publishing terminal or exit signals when the real runner exits with prepared ownership', () => { + const callbackStart = workerSource.lastIndexOf('backend.onExit((code, signal) => {'); + const callbackEnd = workerSource.indexOf('backend.onError(', callbackStart); + const callback = workerSource.slice(callbackStart, callbackEnd); + const fatalIdx = callback.indexOf("lastInitConfig?.cliId === 'codex-app' && codexAppControlFatal"); + const fatalReturnIdx = callback.indexOf('return;', fatalIdx); + const preparedIdx = callback.indexOf('const codexAppPreparedAtExit'); + const failIdx = callback.indexOf('failCodexAppControlGeneration(', preparedIdx); + const terminalIdx = callback.indexOf('emitTurnTerminal(', preparedIdx); + const exitIdx = callback.indexOf("send({ type: 'claude_exit'", preparedIdx); + + expect(fatalIdx).toBeGreaterThan(-1); + expect(fatalReturnIdx).toBeGreaterThan(fatalIdx); + expect(preparedIdx).toBeGreaterThan(fatalReturnIdx); + expect(preparedIdx).toBeGreaterThan(-1); + expect(failIdx).toBeGreaterThan(preparedIdx); + expect(terminalIdx).toBeGreaterThan(failIdx); + expect(exitIdx).toBeGreaterThan(failIdx); + }); + + it('destroys incomplete final transactions before cumulative commit or ACK', () => { + const handlerStart = workerSource.indexOf('function handleCodexAppControlLine('); + const handlerEnd = workerSource.indexOf('function acceptCodexAppControlSocket(', handlerStart); + const handler = workerSource.slice(handlerStart, handlerEnd); + const assembleIdx = handler.indexOf('const finalResult = connection.finalAssembler.accept('); + const rejectIdx = handler.indexOf("if (finalResult.status === 'reject')", assembleIdx); + const destroyIdx = handler.indexOf('connection.socket.destroy();', rejectIdx); + const applicationIdx = handler.indexOf('codexAppControlRecordApplicationGate.run(', assembleIdx); + const commitIdx = handler.indexOf('codexAppControlReplayWindow.commit(', assembleIdx); + const ackIdx = handler.indexOf('encodeCodexAppControlAck(', commitIdx); + const semanticRejectIdx = handler.indexOf('if (!applied)', assembleIdx); + + expect(assembleIdx).toBeGreaterThan(-1); + expect(rejectIdx).toBeGreaterThan(assembleIdx); + expect(destroyIdx).toBeGreaterThan(rejectIdx); + expect(semanticRejectIdx).toBeGreaterThan(destroyIdx); + expect(applicationIdx).toBeGreaterThan(assembleIdx); + expect(commitIdx).toBeGreaterThan(semanticRejectIdx); + expect(commitIdx).toBeGreaterThan(destroyIdx); + expect(ackIdx).toBeGreaterThan(commitIdx); + expect(handler).toContain("if (finalResult.status === 'accepted') return;"); }); }); diff --git a/test/worker-codex-app-turn-routing.integration.test.ts b/test/worker-codex-app-turn-routing.integration.test.ts new file mode 100644 index 000000000..1ce6f9efc --- /dev/null +++ b/test/worker-codex-app-turn-routing.integration.test.ts @@ -0,0 +1,598 @@ +import { execFileSync, spawn, type ChildProcess } from 'node:child_process'; +import { randomBytes } from 'node:crypto'; +import { + chmodSync, + copyFileSync, + existsSync, + mkdtempSync, + readFileSync, + rmSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join, resolve } from 'node:path'; +import { afterEach, describe, expect, it } from 'vitest'; +import type { DaemonToWorker, WorkerToDaemon } from '../src/types.js'; + +const children = new Set(); +const tempDirs = new Set(); +const tmuxSessions = new Set(); + +async function stopChild(child: ChildProcess): Promise { + if (child.exitCode !== null || child.signalCode !== null) return; + await new Promise(resolvePromise => { + const timer = setTimeout(() => child.kill('SIGKILL'), 3_000); + child.once('exit', () => { + clearTimeout(timer); + resolvePromise(); + }); + if (child.connected) child.send({ type: 'close' } satisfies DaemonToWorker); + else child.kill('SIGTERM'); + }); +} + +async function hardKillWorkerOnly(child: ChildProcess): Promise { + if (child.exitCode !== null || child.signalCode !== null) return; + await new Promise(resolvePromise => { + child.once('exit', () => resolvePromise()); + child.kill('SIGKILL'); + }); +} + +function waitForChildExit( + child: ChildProcess, + logs: string[], + timeoutMs = 10_000, +): Promise<{ code: number | null; signal: NodeJS.Signals | null }> { + if (child.exitCode !== null || child.signalCode !== null) { + return Promise.resolve({ code: child.exitCode, signal: child.signalCode }); + } + return new Promise((resolvePromise, rejectPromise) => { + const timer = setTimeout(() => { + child.off('exit', onExit); + rejectPromise(new Error(`worker did not exit after CLI crash\n${logs.join('')}`)); + }, timeoutMs); + const onExit = (code: number | null, signal: NodeJS.Signals | null) => { + clearTimeout(timer); + resolvePromise({ code, signal }); + }; + child.once('exit', onExit); + }); +} + +afterEach(async () => { + await Promise.all([...children].map(stopChild)); + children.clear(); + for (const name of tmuxSessions) { + try { execFileSync('tmux', ['kill-session', '-t', name], { stdio: 'ignore' }); } catch { /* gone */ } + } + tmuxSessions.clear(); + for (const dir of tempDirs) rmSync(dir, { recursive: true, force: true }); + tempDirs.clear(); +}); + +function replacementSessionId(label: string): string { + return `${randomBytes(4).toString('hex')}-${label}-${process.pid}-${Date.now()}`; +} + +function spawnWorker( + root: string, + sessionId: string, + fakeCodex: string, + requestLog: string, + behavior: string, + logs: string[], + messages: WorkerToDaemon[], + onMessage?: (message: WorkerToDaemon, child: ChildProcess) => void, +): ChildProcess { + const child = spawn(process.execPath, ['--import', 'tsx', resolve('src/worker.ts')], { + cwd: resolve('.'), + env: { + ...process.env, + HOME: root, + NODE_ENV: 'test', + NODE_OPTIONS: [process.env.NODE_OPTIONS, '--import=tsx'].filter(Boolean).join(' '), + // tmux launch scripts intentionally sanitize NODE_OPTIONS, so use the + // built JS runner rather than relying on the parent's tsx loader. + BOTMUX_TEST_CODEX_APP_RUNNER_PATH: resolve('dist/codex-app-runner.js'), + SESSION_DATA_DIR: root, + BOTMUX_SESSION_ID: sessionId, + LARK_APP_ID: 'app_worker_replacement', + LARK_APP_SECRET: 'secret', + FAKE_CODEX_LOG: requestLog, + FAKE_CODEX_VERSION: '0.136.0', + FAKE_CODEX_BEHAVIOR: behavior, + }, + stdio: ['ignore', 'pipe', 'pipe', 'ipc'], + }); + children.add(child); + child.stdout?.on('data', chunk => logs.push(chunk.toString())); + child.stderr?.on('data', chunk => logs.push(chunk.toString())); + child.on('message', raw => { + const message = raw as WorkerToDaemon; + messages.push(message); + if (message.type === 'error') logs.push(`[worker-ipc-error] ${message.message}\n`); + if (message.type === 'final_output') { + logs.push(`[worker-ipc-final] turn=${message.turnId} dispatch=${message.codexAppSettlement?.dispatchId ?? '-'} content=${JSON.stringify(message.content)}\n`); + } + onMessage?.(message, child); + }); + return child; +} + +function replacementInit( + sessionId: string, + fakeCodex: string, + prompt: string, + extra: Partial> = {}, +): Extract { + return { + type: 'init', + sessionId, + chatId: 'oc_worker_replacement', + rootMessageId: 'om_worker_replacement_root', + workingDir: resolve('.'), + cliId: 'codex-app', + cliPathOverride: fakeCodex, + backendType: 'tmux', + prompt, + larkAppId: 'app_worker_replacement', + larkAppSecret: 'secret', + ...extra, + }; +} + +function waitFor( + child: ChildProcess, + logs: string[], + predicate: () => boolean, + timeoutMs = 20_000, +): Promise { + if (predicate()) return Promise.resolve(); + return new Promise((resolvePromise, rejectPromise) => { + const poll = setInterval(() => { + if (!predicate()) return; + cleanup(); + resolvePromise(); + }, 20); + const timer = setTimeout(() => { + cleanup(); + rejectPromise(new Error(`worker routing timeout\n${logs.join('')}`)); + }, timeoutMs); + const onExit = (code: number | null, signal: NodeJS.Signals | null) => { + cleanup(); + rejectPromise(new Error(`worker exited early (${code ?? signal})\n${logs.join('')}`)); + }; + const cleanup = () => { + clearInterval(poll); + clearTimeout(timer); + child.off('exit', onExit); + }; + child.once('exit', onExit); + }); +} + +function readRequests(path: string): Array> { + if (!existsSync(path)) return []; + return readFileSync(path, 'utf8').split('\n').filter(Boolean).map(line => JSON.parse(line)); +} + +describe('Codex App worker queued-turn attribution', () => { + it('routes and ACKs turn N before N+1 after both inputs were written in one flush', async () => { + const root = mkdtempSync(join(tmpdir(), 'botmux-worker-codex-routing-')); + tempDirs.add(root); + const fakeCodex = join(root, 'fake-codex'); + const requestLog = join(root, 'requests.jsonl'); + copyFileSync(resolve('test/fixtures/fake-codex-app-server.mjs'), fakeCodex); + chmodSync(fakeCodex, 0o755); + + const sessionId = `codex-routing-${process.pid}-${Date.now()}`; + const logs: string[] = []; + const messages: WorkerToDaemon[] = []; + const nodeOptions = [process.env.NODE_OPTIONS, '--import=tsx'].filter(Boolean).join(' '); + const child = spawn(process.execPath, ['--import', 'tsx', resolve('src/worker.ts')], { + cwd: resolve('.'), + env: { + ...process.env, + HOME: root, + NODE_ENV: 'test', + NODE_OPTIONS: nodeOptions, + BOTMUX_TEST_CODEX_APP_RUNNER_PATH: resolve('src/codex-app-runner.ts'), + SESSION_DATA_DIR: root, + BOTMUX_SESSION_ID: sessionId, + LARK_APP_ID: 'app_worker_routing', + LARK_APP_SECRET: 'secret', + FAKE_CODEX_LOG: requestLog, + FAKE_CODEX_VERSION: '0.136.0', + // Hold turn 1 open so flushPending deterministically writes turn 2 and + // overwrites its legacy singleton globals before final 1 arrives. + FAKE_CODEX_BEHAVIOR: 'delayed-first', + }, + stdio: ['ignore', 'pipe', 'pipe', 'ipc'], + }); + children.add(child); + child.stdout?.on('data', chunk => logs.push(chunk.toString())); + child.stderr?.on('data', chunk => logs.push(chunk.toString())); + let followUpSent = false; + child.on('message', raw => { + const message = raw as WorkerToDaemon; + messages.push(message); + if (message.type === 'error') logs.push(`[worker-ipc-error] ${message.message}\n`); + // `ready` means the adapter/backend are installed, while Codex App's + // authenticated runner still has to initialize and render its first ›. + // Queue turn 2 in that deterministic gap. + if (message.type === 'ready' && !followUpSent) { + followUpSent = true; + child.send(followUp); + } + }); + + const init: DaemonToWorker = { + type: 'init', + sessionId, + chatId: 'oc_worker_routing', + rootMessageId: 'om_worker_routing_root', + // Keep Node package resolution anchored at the checkout so the nested + // source runner can load the tsx hook supplied through NODE_OPTIONS. + workingDir: resolve('.'), + cliId: 'codex-app', + cliPathOverride: fakeCodex, + backendType: 'pty', + prompt: 'turn one legacy', + promptCodexAppInput: { + text: 'turn one', + clientUserMessageId: 'om_worker_turn_1', + }, + larkAppId: 'app_worker_routing', + larkAppSecret: 'secret', + turnId: 'om_worker_turn_1', + }; + const followUp: DaemonToWorker = { + type: 'message', + content: 'turn two legacy', + codexAppInput: { + text: 'turn two', + clientUserMessageId: 'om_worker_turn_2', + }, + turnId: 'om_worker_turn_2', + }; + + try { + child.send(init); + await waitFor(child, logs, () => ( + messages.filter(message => message.type === 'final_output').length >= 2 + && messages.filter(message => message.type === 'turn_terminal').length >= 2 + )); + + const finals = messages.filter( + (message): message is Extract => message.type === 'final_output', + ); + expect(finals.map(final => ({ + turnId: final.turnId, + content: final.content, + dispatchAttempt: final.dispatchAttempt, + }))).toEqual([ + { turnId: 'om_worker_turn_1', content: 'fake answer 1', dispatchAttempt: undefined }, + { turnId: 'om_worker_turn_2', content: 'fake answer 2', dispatchAttempt: undefined }, + ]); + const terminals = messages.filter( + (message): message is Extract => message.type === 'turn_terminal', + ); + expect(terminals.map(terminal => ({ + turnId: terminal.turnId, + status: terminal.status, + }))).toEqual([ + { turnId: 'om_worker_turn_1', status: 'completed' }, + { turnId: 'om_worker_turn_2', status: 'completed' }, + ]); + + const turnStarts = readRequests(requestLog).filter(request => request.method === 'turn/start'); + expect(turnStarts.map(request => request.params.clientUserMessageId)).toEqual([ + 'om_worker_turn_1', + 'om_worker_turn_2', + ]); + const joinedLogs = logs.join(''); + const secondWrite = joinedLogs.indexOf('turn two legacy'); + const firstFinalMap = joinedLogs.indexOf('mapped to botmux turn om_worker_tu'); + expect(secondWrite).toBeGreaterThan(-1); + expect(firstFinalMap).toBeGreaterThan(secondWrite); + expect(joinedLogs).not.toContain('rejected final marker'); + } finally { + await stopChild(child); + } + }, 25_000); +}); + +describe('Codex App worker replacement durable handoff', () => { + it('turns a real tmux runner exit with prepared durable ownership into a Node worker exit', async () => { + const root = mkdtempSync(join(tmpdir(), 'botmux-worker-codex-cli-exit-')); + tempDirs.add(root); + const fakeCodex = join(root, 'fake-codex'); + const requestLog = join(root, 'requests.jsonl'); + copyFileSync(resolve('test/fixtures/fake-codex-app-server.mjs'), fakeCodex); + chmodSync(fakeCodex, 0o755); + + const sessionId = replacementSessionId('cli-exit'); + const tmuxSession = `bmx-${sessionId.slice(0, 8)}`; + tmuxSessions.add(tmuxSession); + const entry = { + dispatchId: 'dispatch-cli-exit-1', + turnId: 'turn-cli-exit-1', + dispatchAttempt: 17, + content: 'crash runner after prepared', + codexAppInput: { text: 'crash runner after prepared', clientUserMessageId: 'turn-cli-exit-1' }, + }; + + const logs: string[] = []; + const messages: WorkerToDaemon[] = []; + const prematureAdmissionSignals: WorkerToDaemon[] = []; + const worker = spawnWorker( + root, sessionId, fakeCodex, requestLog, 'success', logs, messages, + (message, child) => { + if (message.type === 'codex_app_dispatch_transition') { + child.send({ + type: 'codex_app_dispatch_persisted', + requestId: message.requestId, + ok: true, + } satisfies DaemonToWorker); + } + // Withhold settlement durability so the exact dispatch remains + // prepared when the runner process exits. + if (message.type === 'claude_exit' + || (message.type === 'turn_terminal' && message.status === 'ambiguous')) { + // Either edge would let a durable receiver consider N complete and + // admit N+1 before the worker-exit recovery fence is armed. + prematureAdmissionSignals.push(message); + } + }, + ); + worker.send(replacementInit(sessionId, fakeCodex, entry.content, { + env: { FAKE_CODEX_LOG: requestLog, FAKE_CODEX_BEHAVIOR: 'success' }, + turnId: entry.turnId, + dispatchAttempt: entry.dispatchAttempt, + codexAppDispatchId: entry.dispatchId, + promptCodexAppInput: entry.codexAppInput, + })); + + await waitFor(worker, logs, () => messages.some(message => + message.type === 'final_output' + && message.codexAppSettlement?.dispatchId === entry.dispatchId)); + + const workerExit = waitForChildExit(worker, logs); + // Kill the actual runner pane while leaving the Node worker alive. This + // exercises backend.onExit's direct prepared-generation fail-close path, + // unlike SIGKILLing the worker process as the replacement tests below do. + execFileSync('tmux', ['send-keys', '-t', tmuxSession, 'C-c'], { stdio: 'ignore' }); + const exited = await workerExit; + + expect(prematureAdmissionSignals).toEqual([]); + expect(exited).toEqual({ code: null, signal: 'SIGKILL' }); + expect(messages).toContainEqual(expect.objectContaining({ + type: 'error', + message: expect.stringContaining('prepared dispatch'), + turnId: entry.turnId, + dispatchAttempt: entry.dispatchAttempt, + })); + expect(logs.join('')).toContain('worker replacement requires exact recovery'); + expect(readRequests(requestLog).filter(request => request.method === 'turn/start')) + .toHaveLength(1); + }, 35_000); + + it('reattaches a surviving tmux runner and settles its unacked final with an empty replacement prompt', async () => { + const root = mkdtempSync(join(tmpdir(), 'botmux-worker-codex-replace-')); + tempDirs.add(root); + const fakeCodex = join(root, 'fake-codex'); + const requestLog = join(root, 'requests.jsonl'); + copyFileSync(resolve('test/fixtures/fake-codex-app-server.mjs'), fakeCodex); + chmodSync(fakeCodex, 0o755); + + const sessionId = replacementSessionId('replay'); + tmuxSessions.add(`bmx-${sessionId.slice(0, 8)}`); + const entry = { + dispatchId: 'dispatch-replay-1', + turnId: 'turn-replay-1', + state: 'prepared' as const, + content: 'survive worker kill', + codexAppInput: { text: 'survive worker kill', clientUserMessageId: 'turn-replay-1' }, + }; + + const logs1: string[] = []; + const messages1: WorkerToDaemon[] = []; + const worker1 = spawnWorker( + root, sessionId, fakeCodex, requestLog, 'success', logs1, messages1, + (message, child) => { + if (message.type === 'codex_app_dispatch_transition') { + child.send({ + type: 'codex_app_dispatch_persisted', + requestId: message.requestId, + ok: true, + } satisfies DaemonToWorker); + } + // Intentionally withhold the final settlement ACK. The runner keeps + // final-end unacked while this worker is SIGKILLed. + }, + ); + worker1.send(replacementInit(sessionId, fakeCodex, entry.content, { + env: { FAKE_CODEX_LOG: requestLog, FAKE_CODEX_BEHAVIOR: 'success' }, + turnId: entry.turnId, + codexAppDispatchId: entry.dispatchId, + promptCodexAppInput: entry.codexAppInput, + })); + + await waitFor(worker1, logs1, () => messages1.some(message => + message.type === 'final_output' + && message.codexAppSettlement?.dispatchId === entry.dispatchId)); + await hardKillWorkerOnly(worker1); + + const logs2: string[] = []; + const messages2: WorkerToDaemon[] = []; + const worker2 = spawnWorker( + root, sessionId, fakeCodex, requestLog, 'success', logs2, messages2, + (message, child) => { + if (message.type === 'codex_app_dispatch_transition') { + child.send({ + type: 'codex_app_dispatch_persisted', + requestId: message.requestId, + ok: true, + } satisfies DaemonToWorker); + } + if (message.type === 'final_output' && message.codexAppSettlement) { + child.send({ + type: 'codex_app_dispatch_persisted', + requestId: message.codexAppSettlement.requestId, + ok: true, + } satisfies DaemonToWorker); + } + }, + ); + worker2.send(replacementInit(sessionId, fakeCodex, '', { + env: { FAKE_CODEX_LOG: requestLog, FAKE_CODEX_BEHAVIOR: 'success' }, + resume: true, + cliSessionId: 'thread-fake', + codexAppRecoveredDispatches: [entry], + })); + + await waitFor(worker2, logs2, () => messages2.some(message => + message.type === 'turn_terminal' + && message.turnId === entry.turnId + && message.status === 'completed')); + + const replayed = messages2.find((message): message is Extract => + message.type === 'final_output' && message.codexAppSettlement?.dispatchId === entry.dispatchId); + expect(replayed).toMatchObject({ + sessionId, + turnId: entry.turnId, + content: 'fake answer 1', + }); + expect(logs2.join('')).toContain('warm reattach'); + expect(readRequests(requestLog).filter(request => request.method === 'turn/start')).toHaveLength(1); + }, 35_000); + + it('holds N+1 behind replayed empty N until daemon durability ACKs final-end', async () => { + const root = mkdtempSync(join(tmpdir(), 'botmux-worker-codex-empty-replace-')); + tempDirs.add(root); + const fakeCodex = join(root, 'fake-codex'); + const requestLog = join(root, 'requests.jsonl'); + copyFileSync(resolve('test/fixtures/fake-codex-app-server.mjs'), fakeCodex); + chmodSync(fakeCodex, 0o755); + + const sessionId = replacementSessionId('empty'); + tmuxSessions.add(`bmx-${sessionId.slice(0, 8)}`); + const first = { + dispatchId: 'dispatch-empty-1', + turnId: 'turn-empty-1', + state: 'prepared' as const, + content: 'empty first', + codexAppInput: { text: 'empty first', clientUserMessageId: 'turn-empty-1' }, + }; + const second = { + dispatchId: 'dispatch-empty-2', + turnId: 'turn-empty-2', + content: 'second after empty', + codexAppInput: { text: 'second after empty', clientUserMessageId: 'turn-empty-2' }, + }; + + const logs1: string[] = []; + const messages1: WorkerToDaemon[] = []; + const worker1 = spawnWorker( + root, sessionId, fakeCodex, requestLog, 'empty-first', logs1, messages1, + (message, child) => { + if (message.type === 'codex_app_dispatch_transition') { + child.send({ + type: 'codex_app_dispatch_persisted', + requestId: message.requestId, + ok: true, + } satisfies DaemonToWorker); + } + }, + ); + worker1.send(replacementInit(sessionId, fakeCodex, first.content, { + env: { FAKE_CODEX_LOG: requestLog, FAKE_CODEX_BEHAVIOR: 'empty-first' }, + turnId: first.turnId, + codexAppDispatchId: first.dispatchId, + promptCodexAppInput: first.codexAppInput, + })); + await waitFor(worker1, logs1, () => messages1.some(message => + message.type === 'final_output' + && message.codexAppSettlement?.dispatchId === first.dispatchId + && message.content === '')); + await hardKillWorkerOnly(worker1); + + const logs2: string[] = []; + const messages2: WorkerToDaemon[] = []; + let secondSent = false; + let firstAcked = false; + let secondSubmittedBeforeFirstAck = false; + const order: string[] = []; + const worker2 = spawnWorker( + root, sessionId, fakeCodex, requestLog, 'empty-first', logs2, messages2, + (message, child) => { + if (message.type === 'ready' && !secondSent) { + secondSent = true; + child.send({ + type: 'message', + content: second.content, + codexAppInput: second.codexAppInput, + turnId: second.turnId, + codexAppDispatchId: second.dispatchId, + } satisfies DaemonToWorker); + } + if (message.type === 'codex_app_dispatch_transition') { + if (message.entries[0]?.dispatchId === second.dispatchId) { + secondSubmittedBeforeFirstAck = !firstAcked; + order.push('submit-second'); + } + child.send({ + type: 'codex_app_dispatch_persisted', + requestId: message.requestId, + ok: true, + } satisfies DaemonToWorker); + } + if (message.type === 'final_output' && message.codexAppSettlement) { + if (message.codexAppSettlement.dispatchId === first.dispatchId) { + order.push('final-first'); + setTimeout(() => { + firstAcked = true; + order.push('ack-first'); + if (child.connected) child.send({ + type: 'codex_app_dispatch_persisted', + requestId: message.codexAppSettlement!.requestId, + ok: true, + } satisfies DaemonToWorker); + }, 250); + } else if (message.codexAppSettlement.dispatchId === second.dispatchId) { + order.push('final-second'); + child.send({ + type: 'codex_app_dispatch_persisted', + requestId: message.codexAppSettlement.requestId, + ok: true, + } satisfies DaemonToWorker); + } + } + }, + ); + worker2.send(replacementInit(sessionId, fakeCodex, '', { + env: { FAKE_CODEX_LOG: requestLog, FAKE_CODEX_BEHAVIOR: 'empty-first' }, + resume: true, + cliSessionId: 'thread-fake', + codexAppRecoveredDispatches: [first], + })); + + await waitFor(worker2, logs2, () => messages2.filter(message => + message.type === 'turn_terminal' + && (message.turnId === first.turnId || message.turnId === second.turnId) + && message.status === 'completed').length === 2, 30_000); + + const finals = messages2.filter( + (message): message is Extract => + message.type === 'final_output' && !!message.codexAppSettlement, + ); + expect(finals.map(final => ({ turnId: final.turnId, content: final.content }))).toEqual([ + { turnId: first.turnId, content: '' }, + { turnId: second.turnId, content: 'fake answer 2' }, + ]); + expect(secondSubmittedBeforeFirstAck).toBe(false); + expect(order).toEqual(['final-first', 'ack-first', 'submit-second', 'final-second']); + expect(readRequests(requestLog).filter(request => request.method === 'turn/start')) + .toHaveLength(2); + }, 40_000); +}); diff --git a/test/worker-durable-expiry-order.test.ts b/test/worker-durable-expiry-order.test.ts index b18dc3b0c..521b7c522 100644 --- a/test/worker-durable-expiry-order.test.ts +++ b/test/worker-durable-expiry-order.test.ts @@ -8,7 +8,7 @@ import { describe, expect, it } from 'vitest'; const workerSource = readFileSync(new URL('../src/worker.ts', import.meta.url), 'utf8'); describe('worker durable lease expiry ordering', () => { - it('removes exact queued attempt N behind an ordinary current turn before ACKing', () => { + it('durably retires exact queued attempt N behind an ordinary current turn before ACKing', () => { const start = workerSource.indexOf("case 'expire_durable_turn':"); const end = workerSource.indexOf("case 'reset_ambiguous_receiver':", start); expect(start).toBeGreaterThanOrEqual(0); @@ -16,20 +16,26 @@ describe('worker durable lease expiry ordering', () => { const branch = workerSource.slice(start, end); const currentExact = branch.indexOf('const currentExact = durableTurnInFlight'); - const pendingLoop = branch.indexOf('for (let i = pendingMessages.length - 1; i >= 0; i--)'); - const exactTurn = branch.indexOf('item.turnId === msg.turnId', pendingLoop); - const exactAttempt = branch.indexOf('item.dispatchAttempt === msg.dispatchAttempt', pendingLoop); - const remove = branch.indexOf('pendingMessages.splice(i, 1)', pendingLoop); - const pendingAck = branch.indexOf("acknowledge('queued_removed');", remove); + const pendingCount = branch.indexOf('const removedPending = pendingMessages.filter('); + const retire = branch.indexOf('await retireCodexAppDispatchForDurableReplay(', pendingCount); + const pendingAck = branch.indexOf("acknowledge('queued_removed');", retire); const noProof = branch.indexOf('withholding ACK for daemon fencing', pendingAck); expect(currentExact).toBeGreaterThanOrEqual(0); - expect(pendingLoop).toBeGreaterThan(currentExact); - expect(exactTurn).toBeGreaterThan(pendingLoop); - expect(exactAttempt).toBeGreaterThan(exactTurn); - expect(remove).toBeGreaterThan(exactAttempt); - expect(pendingAck).toBeGreaterThan(remove); + expect(pendingCount).toBeGreaterThan(currentExact); + expect(retire).toBeGreaterThan(pendingCount); + expect(pendingAck).toBeGreaterThan(retire); expect(noProof).toBeGreaterThan(pendingAck); + + const retireStart = workerSource.indexOf('async function retireCodexAppDispatchForDurableReplay('); + const retireEnd = workerSource.indexOf('\nfunction ', retireStart + 1); + const retireHelper = workerSource.slice(retireStart, retireEnd); + const persist = retireHelper.indexOf("requestCodexAppDispatchTransition('cancel'"); + const localQueueCancel = retireHelper.indexOf('codexAppTurnDispatchQueue.cancelExact(', persist); + const localPendingRemove = retireHelper.indexOf('pendingMessages.splice(index, 1)', persist); + expect(persist).toBeGreaterThanOrEqual(0); + expect(localQueueCancel).toBeGreaterThan(persist); + expect(localPendingRemove).toBeGreaterThan(persist); }); it('ACKs active exact expiry only after synchronous owned-CLI restart fencing', () => { @@ -37,11 +43,13 @@ describe('worker durable lease expiry ordering', () => { const end = workerSource.indexOf("case 'reset_ambiguous_receiver':", start); const branch = workerSource.slice(start, end); const exactBranch = branch.indexOf('if (currentExact)'); - const restart = branch.indexOf("restartCliProcess('durable lease expiry'", exactBranch); + const retire = branch.indexOf('await retireCodexAppDispatchForDurableReplay(', exactBranch); + const restart = branch.indexOf("restartCliProcess('durable lease expiry'", retire); const ack = branch.indexOf("acknowledge('cli_fenced');", restart); expect(exactBranch).toBeGreaterThanOrEqual(0); - expect(restart).toBeGreaterThan(exactBranch); + expect(retire).toBeGreaterThan(exactBranch); + expect(restart).toBeGreaterThan(retire); expect(ack).toBeGreaterThan(restart); }); @@ -102,7 +110,7 @@ describe('worker durable lease expiry ordering', () => { // PR #441 起入队条件还含注入围栏（injectionFlushing / barrier）——本测试只钉 // restart/rename 三因子仍在场且顺序不变，不钉整行。 const rawGate = rawInput.indexOf( - 'if (cliRestartInProgress || rawInputRestartGate || sessionRenameInFlight', + 'if (cliRestartInProgress || rawInputRestartGate || sessionRenameInFlight()', ); const rawQueue = rawInput.indexOf('pendingRawInputs.push(msg)', rawGate); const rawDeliver = rawInput.indexOf('await deliverRawInput(msg)', rawQueue); diff --git a/test/worker-pipe-initial-screen-order.test.ts b/test/worker-pipe-initial-screen-order.test.ts index 343bd9f34..e7ab608bb 100644 --- a/test/worker-pipe-initial-screen-order.test.ts +++ b/test/worker-pipe-initial-screen-order.test.ts @@ -15,9 +15,31 @@ describe('worker pipe initial screen ordering', () => { expect(captureIdx).toBeGreaterThan(idleIdx); }); + it('starts Codex App warm liveness only after old-key challenge proof, not from a backend flag', () => { + const source = readFileSync(join(process.cwd(), 'src/worker.ts'), 'utf8'); + const activationIdx = source.indexOf('function activateCodexAppControlConnection('); + const proofKindIdx = source.indexOf("const proofKind = identity.generation === codexAppFreshCandidateGeneration", activationIdx); + const beginIdx = source.indexOf('codexAppTurnLiveness.beginReattachObservation();', proofKindIdx); + const pipeGateIdx = source.indexOf('if (isPipeMode && backend && isPersistentBackendReattach)'); + const seedIdx = source.indexOf('seedBackendScreen(`${effectiveBackendType} reattach`, backend);', pipeGateIdx); + + expect(activationIdx).toBeGreaterThan(-1); + expect(proofKindIdx).toBeGreaterThan(activationIdx); + expect(beginIdx).toBeGreaterThan(proofKindIdx); + expect(seedIdx).toBeGreaterThan(pipeGateIdx); + expect(source).not.toContain('shouldBeginCodexAppReattachObservation({'); + }); + it('runs a busy-pattern idle probe after each submitted input', () => { const source = readFileSync(join(process.cwd(), 'src/worker.ts'), 'utf8'); - const writeIdx = source.indexOf('result = await cliAdapter.writeStructuredInput(backend, msg, item.codexAppInput);'); + // After hybrid RPC merge, structured write may go through writeAdapter path. + let writeIdx = source.indexOf('result = item.codexAppInput && writeAdapter.writeStructuredInput'); + if (writeIdx < 0) { + writeIdx = source.indexOf('await writeAdapter.writeStructuredInput(writeBackend, msg, item.codexAppInput)'); + } + if (writeIdx < 0) { + writeIdx = source.indexOf('result = await cliAdapter.writeStructuredInput(backend, msg, item.codexAppInput);'); + } const probeIdx = source.indexOf('scheduleBusyPatternIdleProbe(`${cliName()} post-submit`);'); const helperIdx = source.indexOf('function scheduleBusyPatternIdleProbe(source: string): void'); @@ -173,6 +195,236 @@ describe('worker pipe initial screen ordering', () => { expect(markReadyIdx).toBeGreaterThan(flushIdx); }); + it('rejects Codex App prompts before proof and while the explicit queue is active', () => { + const source = readFileSync(join(process.cwd(), 'src/worker.ts'), 'utf8'); + const markStart = source.indexOf('function markPromptReady'); + const markEnd = source.indexOf('function persistCliSessionId', markStart); + const mark = source.slice(markStart, markEnd); + + const livenessGuardIdx = mark.indexOf( + "if (lastInitConfig?.cliId === 'codex-app' && !codexAppTurnLiveness.notePrompt())", + ); + const proofGuardIdx = mark.indexOf( + "if (lastInitConfig?.cliId === 'codex-app' && !codexAppControlProven)", + ); + const signedIdleGuardIdx = mark.indexOf( + "if (lastInitConfig?.cliId === 'codex-app' && !codexAppReadyAuthority.canPublishPromptReady())", + ); + const readySetIdx = mark.indexOf('isPromptReady = true;'); + const promptReadySendIdx = mark.indexOf("send({ type: 'prompt_ready' });"); + const idleUpdateIdx = mark.indexOf( + 'usageLimitTracker.classify(content, projectedRuntimeScreenStatus())', + ); + + expect(proofGuardIdx).toBeGreaterThan(-1); + expect(proofGuardIdx).toBeLessThan(livenessGuardIdx); + expect(livenessGuardIdx).toBeGreaterThan(-1); + expect(signedIdleGuardIdx).toBeGreaterThan(livenessGuardIdx); + // The explicit runner queue wins before any immediate daemon/card status + // projection; returning from the guard therefore suppresses both paths. + // The shared projector composes the structured lifecycle gate with signed + // Codex App liveness instead of hard-coding an idle card update here. + expect(livenessGuardIdx).toBeLessThan(readySetIdx); + expect(signedIdleGuardIdx).toBeLessThan(readySetIdx); + expect(readySetIdx).toBeLessThan(promptReadySendIdx); + expect(promptReadySendIdx).toBeLessThan(idleUpdateIdx); + }); + + it('lets explicit Codex App activity override a stale idle screen heuristic', () => { + const source = readFileSync(join(process.cwd(), 'src/worker.ts'), 'utf8'); + const helperStart = source.indexOf('function codexAppLivenessStatus'); + const helperEnd = source.indexOf('// Per-turn usage-limit state machine', helperStart); + const helper = source.slice(helperStart, helperEnd); + const activityStart = source.indexOf("if (kind === 'activity' && lastInitConfig?.cliId === 'codex-app')"); + const activityEnd = source.indexOf("if (kind === 'final'", activityStart); + const activity = source.slice(activityStart, activityEnd); + + expect(helper).toContain("liveness.active && base === 'idle' ? 'working' : base"); + expect(activity).toContain('applyTrustedCodexAppActivityMarker('); + expect(activity).toContain('if (!activity.accepted)'); + expect(activity).toContain('isPromptReady = false;'); + }); + + it('re-drives a deferred Codex App prompt after a queued submit cancellation', () => { + const source = readFileSync(join(process.cwd(), 'src/worker.ts'), 'utf8'); + const flushStart = source.indexOf('async function flushPending'); + const flushEnd = source.indexOf('function sendToPty', flushStart); + const flush = source.slice(flushStart, flushEnd); + + expect(flush.match(/codexAppPromptReplay\.cancelSubmission\(\s*codexAppTurnLiveness,\s*codexAppReadyAuthority,\s*codexAppLivenessHandle,?\s*\)/g)).toHaveLength(2); + expect(flush).toContain('codexAppPromptReplay.consumeAfterFlush(codexAppTurnLiveness)'); + expect(flush.lastIndexOf('markPromptReady();')).toBeGreaterThan(flush.lastIndexOf('isFlushing = false;')); + }); + + it('persists a late-created public candidate before spawn and never trusts Codex App OSC', () => { + const source = readFileSync(join(process.cwd(), 'src/worker.ts'), 'utf8'); + const prepareIdx = source.indexOf('await prepareCodexAppControlGeneration(cfg, willReattachPersistent, !!persistentSessionName);'); + const candidateIdx = source.indexOf('prepareFreshCodexAppControlBootstrap(cfg, !!persistentSessionName);'); + const injectIdx = source.indexOf('childEnv[CODEX_APP_CONTROL_BOOTSTRAP_ENV] = codexAppControlBootstrapPathForSpawn;'); + const spawnIdx = source.indexOf('backend.spawn(spawnBin, spawnArgs'); + const finalizeIdx = source.indexOf('finalizeCodexAppControlGeneration(', spawnIdx); + const onDataIdx = source.indexOf('backend.onData(onPtyData);', spawnIdx); + const finalizeStart = source.indexOf('function finalizeCodexAppControlGeneration('); + const finalizeEnd = source.indexOf('function rejectCodexAppControlMarker', finalizeStart); + const finalize = source.slice(finalizeStart, finalizeEnd); + + expect(prepareIdx).toBeGreaterThan(-1); + expect(candidateIdx).toBeGreaterThan(prepareIdx); + expect(injectIdx).toBeGreaterThan(candidateIdx); + expect(spawnIdx).toBeGreaterThan(injectIdx); + expect(finalizeIdx).toBeGreaterThan(spawnIdx); + expect(onDataIdx).toBeGreaterThan(finalizeIdx); + expect(finalize).toContain("codexAppControlProven && codexAppControlStateValue?.status === 'active'"); + expect(source).toContain("const APP_RUNNER_OSC_CLI_IDS = new Set(['mira', 'mir']);"); + expect(source).not.toContain('CODEX_APP_CONTROL_NONCE_ENV'); + expect(source).not.toContain('codexAppControlNonceForSpawn'); + }); + + it('awaits bind and locator publication before every backend spawn path', () => { + const source = readFileSync(join(process.cwd(), 'src/worker.ts'), 'utf8'); + const spawnStart = source.indexOf('async function spawnCli('); + const prepareIdx = source.indexOf('await prepareCodexAppControlGeneration(', spawnStart); + const pluginPrepareIdx = source.indexOf('await prepareCliPluginGenerationAndGateway(cfg, cliAdapter)', prepareIdx); + const backendSpawnIdx = source.indexOf('backend.spawn(spawnBin, spawnArgs', prepareIdx); + + expect(spawnStart).toBeGreaterThan(-1); + expect(prepareIdx).toBeGreaterThan(spawnStart); + expect(backendSpawnIdx).toBeGreaterThan(prepareIdx); + expect(source.match(/await spawnCli\(/g)).toHaveLength(3); + expect(source.slice(spawnStart, prepareIdx)).toContain('const spawnGeneration = ++cliSpawnGeneration;'); + expect(source.slice(prepareIdx, backendSpawnIdx)) + .toContain('if (spawnGeneration !== cliSpawnGeneration) throw new CliSpawnSupersededError();'); + expect(source.slice(pluginPrepareIdx, backendSpawnIdx) + .match(/if \(spawnGeneration !== cliSpawnGeneration\) throw new CliSpawnSupersededError\(\);/g)) + .toHaveLength(2); + expect(source).toContain('function killCli(opts: { preservePending?: boolean } = {}): void {\n cliSpawnGeneration++;'); + const restartHandler = source.slice( + source.indexOf('async function restartCliProcess('), + source.indexOf('// ─── HTTP + WebSocket Server'), + ); + const initHandler = source.slice( + source.indexOf("case 'init':"), + source.indexOf("case 'codex_app_dispatch_persisted':"), + ); + const messageHandler = source.slice( + source.indexOf("case 'message':"), + source.indexOf("case 'raw_input':"), + ); + for (const handler of [restartHandler, initHandler, messageHandler]) { + expect(handler).toContain('if (err instanceof CliSpawnSupersededError) return;'); + } + }); + + it('uses hardened locators, random endpoints, and process-lifetime publisher leases', () => { + const source = readFileSync(join(process.cwd(), 'src/worker.ts'), 'utf8'); + const prepareStart = source.indexOf('async function prepareCodexAppControlGeneration('); + const prepareEnd = source.indexOf('/** Late-create the only secret-bearing file', prepareStart); + const prepare = source.slice(prepareStart, prepareEnd); + const bootstrapStart = prepareEnd; + const bootstrapEnd = source.indexOf('function finalizeCodexAppControlGeneration(', bootstrapStart); + const bootstrap = source.slice(bootstrapStart, bootstrapEnd); + const acceptStart = source.indexOf('function acceptCodexAppControlSocket('); + const acceptEnd = source.indexOf('function removeStaleCodexAppSocket', acceptStart); + const accept = source.slice(acceptStart, acceptEnd); + const stopStart = source.indexOf('function stopCodexAppControlChannel('); + const stopEnd = source.indexOf('function failCodexAppControlGeneration', stopStart); + const stop = source.slice(stopStart, stopEnd); + + expect(prepare).toContain("process.platform === 'win32' ? codexAppWindowsControlRoot()"); + const leaseIdx = prepare.indexOf('await ensureCodexAppWindowsOwnerLease(cfg.sessionId)'); + const locatorIdx = prepare.indexOf('codexAppControlLocatorPath(controlRoot, cfg.sessionId)'); + expect(leaseIdx).toBeGreaterThan(-1); + expect(locatorIdx).toBeGreaterThan(leaseIdx); + expect(prepare).toContain('else await ensureCodexAppPosixOwnerLease(controlRoot, cfg.sessionId);'); + expect(prepare).toContain('codexAppControlLocatorPath(controlRoot, cfg.sessionId)'); + expect(prepare).toContain('const started = await startCodexAppControlEndpoint(cfg, channelId);'); + expect(source).toContain('await bindThenPublishCodexAppControlLocator({'); + expect(source).toContain('generateCodexAppWindowsPipeEndpoint()'); + expect(source).toContain('generateCodexAppPosixSocketEndpoint(codexAppControlSocketDirectory)'); + expect(bootstrap).toContain("{ kind: 'locator', locatorPath: codexAppControlLocatorPathValue }"); + expect(accept).toContain("rotateCodexAppControlEndpoint('active socket closed')"); + expect(accept).toContain('if (wasActive'); + expect(stop).toContain("if (socketPath && process.platform !== 'win32')"); + expect(stop).not.toContain('unlinkSync(codexAppControlLocatorPathValue)'); + expect(prepare).toContain('Deleting first would'); + const killStart = source.indexOf('function killCli('); + const killEnd = source.indexOf('function cleanup()', killStart); + const cleanupStart = source.indexOf('function cleanup()'); + const cleanupEnd = source.indexOf("process.on('SIGTERM'", cleanupStart); + expect(source.slice(killStart, killEnd)).not.toContain('releaseCodexAppPosixOwnerLease()'); + expect(source.slice(cleanupStart, cleanupEnd)).toContain('releaseCodexAppPosixOwnerLease();'); + }); + + it('prevents retired async endpoints from publishing or failing a newer channel', () => { + const source = readFileSync(join(process.cwd(), 'src/worker.ts'), 'utf8'); + const start = source.slice( + source.indexOf('async function startCodexAppControlEndpoint('), + source.indexOf('function installCodexAppControlEndpoint('), + ); + const rotateStart = source.indexOf('function rotateCodexAppControlEndpoint('); + const rotate = source.slice( + rotateStart, + source.indexOf('async function prepareCodexAppControlGeneration(', rotateStart), + ); + + expect(start.indexOf('channelId !== codexAppControlChannelId')).toBeLessThan( + start.indexOf('writeCodexAppControlLocator(locatorPath, locator);'), + ); + expect(start).toContain('if (codexAppControlServer === server)'); + expect(rotate).toContain('if (shouldFailCodexAppControlChannel({'); + expect(rotate).toContain('if (codexAppControlRotation === rotation) codexAppControlRotation = undefined;'); + }); + + it('keeps an unproved endpoint bound until the shared 90-second fail-close deadline', () => { + const source = readFileSync(join(process.cwd(), 'src/worker.ts'), 'utf8'); + const acceptStart = source.indexOf('function acceptCodexAppControlSocket('); + const acceptEnd = source.indexOf('function removeStaleCodexAppSocket', acceptStart); + const accept = source.slice(acceptStart, acceptEnd); + const finalizeStart = source.indexOf('function finalizeCodexAppControlGeneration('); + const finalizeEnd = source.indexOf('function rejectCodexAppControlMarker', finalizeStart); + const finalize = source.slice(finalizeStart, finalizeEnd); + + expect(accept).toContain('Rotate only after the authenticated runner closes.'); + expect(accept).toContain('if (wasActive'); + expect(accept).toContain('codexAppProofDeadline.arm(() => {'); + expect(accept).toContain('did not re-authenticate within'); + expect(accept).not.toContain('pre-auth socket closed'); + expect(finalize).toContain('codexAppProofDeadline.arm(() => {'); + }); + + it('shares the 90-second first-prompt cap with bootstrap cleanup and runner proof', () => { + const source = readFileSync(join(process.cwd(), 'src/worker.ts'), 'utf8'); + const bootstrapStart = source.indexOf('function prepareFreshCodexAppControlBootstrap('); + const bootstrapEnd = source.indexOf('function finalizeCodexAppControlGeneration(', bootstrapStart); + const bootstrap = source.slice(bootstrapStart, bootstrapEnd); + const proofStart = bootstrapEnd; + const proofEnd = source.indexOf('function rejectCodexAppControlMarker', proofStart); + const proof = source.slice(proofStart, proofEnd); + + expect(source).toContain('const FIRST_PROMPT_HARD_TIMEOUT_MS = CODEX_APP_CONTROL_STARTUP_TIMEOUT_MS;'); + expect(bootstrap).toContain('armCodexAppControlStartupTimeout(cleanupCodexAppControlBootstrap)'); + expect(proof).toContain('codexAppProofDeadline.arm(() => {'); + expect(bootstrap).not.toContain('30_000'); + expect(proof).not.toContain('30_000'); + }); + + it('kills legacy/no-public-key reattach and fail-closes candidate setup before PTY listeners attach', () => { + const source = readFileSync(join(process.cwd(), 'src/worker.ts'), 'utf8'); + const preflightIdx = source.indexOf('shouldColdStartCodexAppReattach({'); + const preflightKillIdx = source.indexOf('killPersistentSession(', preflightIdx); + const prepareIdx = source.indexOf('prepareCodexAppControlGeneration(', preflightIdx); + const finalizeIdx = source.indexOf('finalizeCodexAppControlGeneration(', prepareIdx); + const failureKillIdx = source.indexOf('killPersistentSession(', finalizeIdx); + const onDataIdx = source.indexOf('backend.onData(onPtyData);', finalizeIdx); + + expect(preflightIdx).toBeGreaterThan(-1); + expect(preflightKillIdx).toBeGreaterThan(preflightIdx); + expect(preflightKillIdx).toBeLessThan(prepareIdx); + expect(finalizeIdx).toBeGreaterThan(prepareIdx); + expect(failureKillIdx).toBeGreaterThan(finalizeIdx); + expect(failureKillIdx).toBeLessThan(onDataIdx); + }); + it('honors a true ready signal that arrives AFTER the timeout fallback (slow cold start)', () => { // ReadyGate.receive() is one-shot: once the 45s fallback fires, a later // releaseReadyGate from the real signal is skipped entirely. A CLI whose diff --git a/test/worker-ready-display-mode.test.ts b/test/worker-ready-display-mode.test.ts index b7c5e43f6..3ae7ed61a 100644 --- a/test/worker-ready-display-mode.test.ts +++ b/test/worker-ready-display-mode.test.ts @@ -127,6 +127,7 @@ vi.mock('@larksuiteoapi/node-sdk', () => ({ // ─── Imports under test ──────────────────────────────────────────────────── import { CARD_POSTING_SENTINEL, initWorkerPool, __testOnly_setupWorkerHandlers } from '../src/core/worker-pool.js'; +import { MessageWithdrawnError } from '../src/im/lark/client.js'; import type { DaemonSession } from '../src/core/types.js'; import { getBot } from '../src/bot-registry.js'; @@ -286,6 +287,32 @@ describe('Worker ready: set_display_mode re-sync', () => { expect(displayModeCalls).toHaveLength(0); }); + it('preserves worker and pending Codex FIFO when the root is withdrawn during ready POST', async () => { + sessionReplyMock.mockRejectedValueOnce(new MessageWithdrawnError('om_root')); + const fakeWorker = makeFakeWorker(); + const ds = makeDs({ + streamCardPending: true, + streamCardId: undefined, + worker: fakeWorker, + }); + ds.session.codexAppDispatchLedger = [{ + dispatchId: 'dispatch-pending', + turnId: 'turn-pending', + state: 'prepared', + content: 'pending', + }]; + + __testOnly_setupWorkerHandlers(ds, fakeWorker); + fakeWorker.emit('message', { + type: 'ready', port: 9999, token: 'tok_abc', turnId: 'turn-pending', + }); + await flush(); + + expect(closeSessionMock).not.toHaveBeenCalled(); + expect(fakeWorker.kill).not.toHaveBeenCalled(); + expect(ds.session.codexAppDispatchLedger).toHaveLength(1); + }); + it('PATCH path sends set_display_mode when displayMode is screenshot', async () => { const fakeWorker = makeFakeWorker(); // Existing card + streamCardPending=false → PATCH path @@ -517,6 +544,51 @@ describe('Worker ready: set_display_mode re-sync', () => { ); }); + it('replays a restored raw opening with its durable token and releases only on the matching ACK', async () => { + const submitted = vi.fn(async () => true); + initWorkerPool({ + sessionReply: sessionReplyMock, + getSessionWorkingDir: () => '/tmp', + getActiveCount: () => 1, + closeSession: closeSessionMock, + onQueuedActivationSubmitted: submitted, + }); + const fakeWorker = makeFakeWorker(); + const ds = makeDs({ + worker: fakeWorker, + pendingRawInput: '/goal RESTORED_RAW_N', + initialStartPending: true, + } as Partial); + Object.assign(ds.session, { + queuedActivationPending: true, + queuedActivationToken: 'raw-activation-token', + queuedActivationTurnId: 'turn-raw-n', + pendingRepoSetup: { + mode: 'picker', prompt: '', rawInput: '/goal RESTORED_RAW_N', turnId: 'turn-raw-n', + }, + }); + + __testOnly_setupWorkerHandlers(ds, fakeWorker); + fakeWorker.emit('message', { type: 'prompt_ready' }); + await flush(); + + expect(fakeWorker.send).toHaveBeenCalledWith({ + type: 'raw_input', + content: '/goal RESTORED_RAW_N', + queuedActivationToken: 'raw-activation-token', + turnId: 'turn-raw-n', + }); + expect(submitted).not.toHaveBeenCalled(); + + fakeWorker.emit('message', { + type: 'queued_activation_submitted', + sessionId: ds.session.sessionId, + activationToken: 'raw-activation-token', + }); + await flush(); + expect(submitted).toHaveBeenCalledWith(ds, 'raw-activation-token'); + }); + it('prompt_ready bundles the buffered follow-up ONTO the raw_input IPC (single atomic message)', async () => { // Two separate IPCs would race inside the worker: its async message // handlers don't serialize, and raw_input awaits 200ms between sendText diff --git a/test/worker-riff-retirement-protocol.test.ts b/test/worker-riff-retirement-protocol.test.ts new file mode 100644 index 000000000..671c2fede --- /dev/null +++ b/test/worker-riff-retirement-protocol.test.ts @@ -0,0 +1,134 @@ +import { readFileSync } from 'node:fs'; +import { describe, expect, it } from 'vitest'; + +const workerSource = readFileSync(new URL('../src/worker.ts', import.meta.url), 'utf8'); +const workerPoolSource = readFileSync(new URL('../src/core/worker-pool.ts', import.meta.url), 'utf8'); + +describe('worker Riff retirement protocol', () => { + it('refuses Riff generation restart before the local restart helper can run', () => { + const start = workerSource.indexOf("case 'restart':"); + const end = workerSource.indexOf("case 'expire_durable_turn':", start); + const restart = workerSource.slice(start, end); + + const riffGuard = restart.indexOf("if (effectiveBackendType === 'riff')"); + const refusal = restart.indexOf('Refused Riff generation restart', riffGuard); + const guardBreak = restart.indexOf('break;', refusal); + const replacement = restart.indexOf('await restartCliProcess(', guardBreak); + + expect(start).toBeGreaterThanOrEqual(0); + expect(end).toBeGreaterThan(start); + expect(riffGuard).toBeGreaterThanOrEqual(0); + expect(refusal).toBeGreaterThan(riffGuard); + expect(guardBreak).toBeGreaterThan(refusal); + expect(replacement).toBeGreaterThan(guardBreak); + }); + + it('refuses request-less Riff close before destroy or process exit', () => { + const start = workerSource.indexOf("case 'close':"); + const end = workerSource.indexOf("case 'close_commit':", start); + const close = workerSource.slice(start, end); + + const riffBranch = close.indexOf("if (effectiveBackendType === 'riff')"); + const requestlessGuard = close.indexOf('if (!msg.requestId)', riffBranch); + const refusal = close.indexOf('Refused unsafe request-less Riff close', requestlessGuard); + const guardBreak = close.indexOf('break;', refusal); + const localDestroy = close.lastIndexOf('backend?.destroySession?.()'); + const localExit = close.lastIndexOf('process.exit(0)'); + + expect(start).toBeGreaterThanOrEqual(0); + expect(end).toBeGreaterThan(start); + expect(riffBranch).toBeGreaterThanOrEqual(0); + expect(requestlessGuard).toBeGreaterThan(riffBranch); + expect(refusal).toBeGreaterThan(requestlessGuard); + expect(guardBreak).toBeGreaterThan(refusal); + expect(localDestroy).toBeGreaterThan(guardBreak); + expect(localExit).toBeGreaterThan(localDestroy); + }); + + it('refuses request-less Riff suspend before teardown or process exit', () => { + const start = workerSource.indexOf("case 'suspend':"); + const end = workerSource.indexOf('\n }\n});', start); + const suspend = workerSource.slice(start, end); + + const riffGuard = suspend.indexOf("if (effectiveBackendType === 'riff')"); + const refusal = suspend.indexOf('Refused unsafe Riff suspend', riffGuard); + const guardBreak = suspend.indexOf('break;', refusal); + const localDestroy = suspend.indexOf('(backend?.destroySession ?? backend?.kill)', guardBreak); + const localExit = suspend.indexOf('process.exit(0)', localDestroy); + + expect(start).toBeGreaterThanOrEqual(0); + expect(end).toBeGreaterThan(start); + expect(riffGuard).toBeGreaterThanOrEqual(0); + expect(refusal).toBeGreaterThan(riffGuard); + expect(guardBreak).toBeGreaterThan(refusal); + expect(localDestroy).toBeGreaterThan(guardBreak); + expect(localExit).toBeGreaterThan(localDestroy); + }); + + it('checks every unsent input buffer before fencing the backend or allowing commit', () => { + const prepareStart = workerSource.indexOf("case 'riff_shutdown_prepare':"); + const prepareEnd = workerSource.indexOf("case 'riff_shutdown_commit':", prepareStart); + const prepare = workerSource.slice(prepareStart, prepareEnd); + const readiness = prepare.indexOf('riffWorkerShutdownInputBlocker({'); + const queueCount = prepare.indexOf('pendingMessages: pendingMessages.length', readiness); + const rawCount = prepare.indexOf('pendingRawInputs: pendingRawInputs.length', readiness); + const initFence = prepare.indexOf('initPromptMaterialized', readiness); + const refusal = prepare.indexOf('worker_inputs_not_drained:', readiness); + const backendPrepare = prepare.indexOf('backend?.prepareShutdownDetach?.()', refusal); + + expect(readiness).toBeGreaterThanOrEqual(0); + expect(initFence).toBeGreaterThan(readiness); + expect(queueCount).toBeGreaterThan(readiness); + expect(rawCount).toBeGreaterThan(queueCount); + expect(refusal).toBeGreaterThan(rawCount); + expect(backendPrepare).toBeGreaterThan(refusal); + + const commitStart = workerSource.indexOf("case 'riff_shutdown_commit':", prepareEnd); + const commitEnd = workerSource.indexOf("case 'riff_shutdown_abort':", commitStart); + const commit = workerSource.slice(commitStart, commitEnd); + expect(commit).toContain("shutdownDetachPhase !== 'prepared'"); + expect(commit.indexOf("shutdownDetachPhase !== 'prepared'")) + .toBeLessThan(commit.indexOf('process.exit(0)')); + }); + + it('has no shutdown cancellation command that can discard accepted Riff work', () => { + expect(workerSource).not.toContain("case 'riff_shutdown_cancel':"); + expect(workerSource).not.toContain('cancelShutdownDetach'); + }); + + it('ACKs shutdown and explicit-close abort only after backend admission restoration', () => { + const shutdownStart = workerSource.indexOf("case 'riff_shutdown_abort':"); + const shutdownEnd = workerSource.indexOf("case 'close_commit':", shutdownStart); + const shutdown = workerSource.slice(shutdownStart, shutdownEnd); + const shutdownRestore = shutdown.indexOf('await backend?.abortShutdownDetach?.()'); + expect(shutdownRestore).toBeGreaterThanOrEqual(0); + expect(shutdown.indexOf("phase: 'abort'", shutdownRestore)) + .toBeGreaterThan(shutdownRestore); + + const closeStart = workerSource.indexOf("case 'close_abort':"); + const closeEnd = workerSource.indexOf("case 'suspend':", closeStart); + const close = workerSource.slice(closeStart, closeEnd); + const closeRestore = close.indexOf('await backend?.abortDestroySession?.()'); + expect(closeRestore).toBeGreaterThanOrEqual(0); + expect(close.indexOf("type: 'close_abort_result'", closeRestore)) + .toBeGreaterThan(closeRestore); + }); + + it('retains close and shutdown generations across worker error and preserves close fence on exit', () => { + const errorStart = workerPoolSource.indexOf("worker.on('error', (err) => {"); + const errorEnd = workerPoolSource.indexOf("worker.stdout?.on('data'", errorStart); + const errorHandler = workerPoolSource.slice(errorStart, errorEnd); + expect(errorStart).toBeGreaterThanOrEqual(0); + expect(errorHandler).toContain('ds.riffShutdownState !== undefined'); + expect(errorHandler).toContain('|| ds.riffCloseState !== undefined'); + expect(errorHandler.indexOf('if (!retainExactRetirementGeneration)')) + .toBeLessThan(errorHandler.indexOf('ds.riffCloseState = undefined')); + + const exitStart = workerPoolSource.indexOf("worker.on('exit', (code, signal) => {"); + const exitEnd = workerPoolSource.indexOf('\n return worker;', exitStart); + const exitHandler = workerPoolSource.slice(exitStart, exitEnd); + expect(exitStart).toBeGreaterThanOrEqual(0); + expect(exitHandler).toContain("phase: 'uncertain'"); + expect(exitHandler).not.toContain('ds.riffCloseState = undefined'); + }); +}); diff --git a/test/worker-structured-lifecycle-status.test.ts b/test/worker-structured-lifecycle-status.test.ts new file mode 100644 index 000000000..ea17610fe --- /dev/null +++ b/test/worker-structured-lifecycle-status.test.ts @@ -0,0 +1,459 @@ +import { readFileSync } from 'node:fs'; +import { describe, expect, it } from 'vitest'; + +const source = readFileSync(new URL('../src/worker.ts', import.meta.url), 'utf8'); + +function functionSlice(name: string, nextName: string): string { + const start = source.indexOf(`function ${name}`); + const end = source.indexOf(`function ${nextName}`, start + 1); + expect(start).toBeGreaterThanOrEqual(0); + expect(end).toBeGreaterThan(start); + return source.slice(start, end); +} + +describe('worker structured-turn status wiring', () => { + it('rejects a prompt heuristic before publishing ready or clearing in-flight input', () => { + const body = functionSlice('markPromptReady', 'persistCliSessionId'); + const lifecycleGate = body.indexOf('hasStructuredLifecycleBlock()'); + expect(lifecycleGate).toBeGreaterThanOrEqual(0); + expect(body.indexOf('idleDetector?.reset()', lifecycleGate)).toBeGreaterThan(lifecycleGate); + expect(body.indexOf('isPromptReady = true')).toBeGreaterThan(lifecycleGate); + expect(body.indexOf("send({ type: 'prompt_ready' })")).toBeGreaterThan(lifecycleGate); + expect(body).toContain('projectedRuntimeScreenStatus()'); + }); + + it('re-drives a rejected confirmed-start lease after its bounded grace expires', () => { + const scheduler = functionSlice('redriveRejectedStructuredReady', 'markPromptReady'); + expect(scheduler.indexOf("pruneExpiredStructuredHeadsAndEmit('structured pre-start gate')")) + .toBeLessThan(scheduler.indexOf('hasStructuredLifecycleBlock()')); + expect(scheduler).toContain('hasStructuredLifecycleBlock()'); + expect(scheduler).toContain('idleDetector?.fireIdle()'); + expect(scheduler).toContain('captureBackendScreen(backend)'); + const grace = functionSlice('scheduleStructuredStartGraceRecheck', 'markPromptReady'); + const fence = grace.indexOf('isCliBackendGenerationCurrent('); + const redrive = grace.indexOf('redriveRejectedStructuredReady()'); + expect(grace).toContain('const cliGenerationAtSchedule = cliSpawnGeneration'); + expect(grace).toContain('{ generation: cliGenerationAtSchedule, backend: backendAtSchedule }'); + expect(grace).toContain('restartInProgress: cliRestartInProgress'); + expect(fence).toBeGreaterThanOrEqual(0); + expect(redrive).toBeGreaterThan(fence); + expect(scheduler).toContain('ptyOutputGeneration.isCurrent(readyEvidenceGeneration)'); + + const body = functionSlice('markPromptReady', 'persistCliSessionId'); + expect(body).toContain('codexBridgeQueue.preStartLeaseRemainingMs()'); + expect(body).toContain('scheduleStructuredStartGraceRecheck(remainingMs)'); + expect(body).toContain('ptyOutputGeneration.snapshot()'); + }); + + it('records authoritative submit confirmation in fresh and adopt write paths', () => { + const flush = functionSlice('flushPending', 'sendToPty'); + const verification = flush.indexOf('codexBridgeQueue.beginSubmitVerification(bridgeTurnId, undefined, item.dispatchAttempt)'); + const write = flush.indexOf('await writeAdapter.writeInput(writeBackend, msg)'); + expect(verification).toBeGreaterThanOrEqual(0); + expect(write).toBeGreaterThan(verification); + expect(flush).toContain('result?.submitted === true && bridgeTurnId'); + expect(flush).toContain('codexBridgeQueue.confirmPendingTurn(bridgeTurnId, undefined, item.dispatchAttempt)'); + expect(flush).toContain('codexBridgeQueue.finishSubmitVerification(bridgeTurnId, undefined, item.dispatchAttempt)'); + + const adopt = functionSlice('writeAdoptMessage', 'isWorkflowWorker'); + const adoptCycle = adopt.indexOf('beginCliWriteCycle()'); + const adoptWrite = adopt.indexOf('await cliAdapter.writeInput(adoptBackend as unknown as PtyHandle, content)'); + expect(adoptCycle).toBeGreaterThanOrEqual(0); + expect(adoptWrite).toBeGreaterThan(adoptCycle); + expect(adopt).toContain('result?.submitted === true && adoptStructuredBridgeTurnId'); + expect(adopt).toContain('codexBridgeQueue.confirmPendingTurn(adoptStructuredBridgeTurnId, undefined, dispatchAttempt)'); + }); + + it('re-arms readiness for every fresh type-ahead item and never resets adopt readiness after await', () => { + const flush = functionSlice('flushPending', 'sendToPty'); + const loop = flush.indexOf('while (pendingMessages.length > 0 && backend && cliAdapter)'); + const perItemCycle = flush.indexOf('beginCliWriteCycle()', loop); + const itemWrite = flush.indexOf('await writeAdapter.writeInput(writeBackend, msg)', loop); + expect(perItemCycle).toBeGreaterThan(loop); + expect(itemWrite).toBeGreaterThan(perItemCycle); + + const adopt = functionSlice('writeAdoptMessage', 'isWorkflowWorker'); + const adoptWrite = adopt.indexOf('await cliAdapter.writeInput(adoptBackend as unknown as PtyHandle, content)'); + expect(adoptWrite).toBeGreaterThanOrEqual(0); + expect(adopt.slice(adoptWrite)).not.toContain('isPromptReady = false;'); + expect(adopt.slice(adoptWrite)).not.toContain('idleDetector?.reset();'); + }); + + it('uses the same lifecycle-aware projection for periodic and screenshot updates', () => { + const screenshot = functionSlice('captureAndUpload', 'applyDisplayMode'); + const periodic = functionSlice('startScreenUpdates', 'stopScreenUpdates'); + expect(screenshot).toContain('const status = projectedRuntimeScreenStatus()'); + expect(periodic).toContain('snapshotWithLatestRuntimeStatus(async () =>'); + expect(periodic).toContain('}, projectedRuntimeScreenStatus)'); + expect(periodic.indexOf('usageLimitTracker.classify(snapshot.content, status)')) + .toBeGreaterThan(periodic.indexOf('}, projectedRuntimeScreenStatus)')); + }); + + it('limits the strong ready gate to drivers with a complete terminal contract', () => { + const gate = functionSlice('hasStructuredLifecycleBlock', 'structuredBridgeIsCodex'); + const awaitingGate = gate.indexOf('rpcTurnsAwaitingActivation.size > 0'); + const rpcGate = gate.indexOf('turn.rpcActive === true'); + const transcriptAllowlist = gate.indexOf('isStructuredBridgeLifecycleBlockingCli(lastInitConfig?.cliId)'); + expect(awaitingGate).toBeGreaterThanOrEqual(0); + expect(awaitingGate).toBeLessThan(rpcGate); + expect(rpcGate).toBeGreaterThanOrEqual(0); + expect(transcriptAllowlist).toBeGreaterThan(rpcGate); + expect(gate).toContain('isStructuredBridgeLifecycleBlockingCli(lastInitConfig?.cliId)'); + expect(gate).toContain('codexBridgeQueue.hasBlockingTurn()'); + + const projector = functionSlice('projectedRuntimeScreenStatus', 'beginCliWriteCycle'); + expect(projector).toContain('structuredTurnBlocking: hasStructuredLifecycleBlock()'); + }); + + it('re-drives idle for both normal-final and interrupted terminal events', () => { + const ingest = functionSlice('codexBridgeIngest', 'codexBridgeMarkPendingTurn'); + expect(ingest).toContain('result.events.some(isStructuredTerminalEvent)'); + expect(ingest).toContain('idleDetector?.fireIdle()'); + + const attach = functionSlice('codexBridgeAttach', 'cursorLateAttachMode'); + expect(attach).toContain('live.some(isStructuredTerminalEvent)'); + expect(attach).toContain('idleDetector?.fireIdle()'); + }); + + it('settles empty normal/abort terminals without publishing a fake final response', () => { + const emit = functionSlice('emitReadyCodexTurns', 'stopCodexBridge'); + const outputGuard = emit.indexOf('if (!turn.finalText) continue;'); + const terminalLoop = emit.indexOf('for (const turn of ready)', outputGuard + 1); + expect(outputGuard).toBeGreaterThanOrEqual(0); + expect(terminalLoop).toBeGreaterThan(outputGuard); + expect(emit.slice(terminalLoop)).toContain("turn.terminalStatus ?? 'completed'"); + expect(emit.slice(terminalLoop)).toContain('turn.dispatchAttempt'); + }); + + it('fences stale deferred-attempt effects before persistence, redrive, reschedule, or terminal', () => { + const schedule = functionSlice('scheduleSubmitFailureNotify', 'detectBareShellLaunch'); + expect(schedule).toContain('const cliGenerationAtSchedule = cliSpawnGeneration'); + expect(schedule).toContain('backend === backendAtSchedule'); + expect(schedule).toContain('dispatchAttempt: turnIdentity?.dispatchAttempt'); + expect(schedule).toContain('structuredTarget,'); + expect(schedule).toContain('isCurrent: deferredAttemptIsCurrent'); + const staleGuard = schedule.indexOf('if (settlement.stale)'); + expect(staleGuard).toBeGreaterThanOrEqual(0); + expect(schedule.indexOf('persistCliSessionId(cliSessionId)', staleGuard)).toBeGreaterThan(staleGuard); + expect(schedule.indexOf('redriveRejectedStructuredReady()', staleGuard)).toBeGreaterThan(staleGuard); + expect(schedule.indexOf('scheduleSubmitFailureNotify(', staleGuard)).toBeGreaterThan(staleGuard); + expect(schedule.indexOf('emitDurableFailure(', staleGuard)).toBeGreaterThan(staleGuard); + + const restart = functionSlice('restartCliProcess', 'startWebServer'); + expect(restart.indexOf('cliSpawnGeneration += 1')) + .toBeLessThan(restart.indexOf('cliRestartInProgress = true')); + }); + + it('generation-fences the normal flush continuation immediately after writeInput settles', () => { + const flush = functionSlice('flushPending', 'sendToPty'); + const capture = flush.indexOf('const writeGeneration = cliSpawnGeneration'); + const write = flush.indexOf('await writeAdapter.writeInput(writeBackend, msg)', capture); + const resolvedGuard = flush.indexOf('if (!writeContinuationIsCurrent())', write); + const busyProbe = flush.indexOf('scheduleBusyPatternIdleProbe(', write); + const catchStart = flush.indexOf('} catch (err: any) {', resolvedGuard); + const rejectedGuard = flush.indexOf('if (!writeContinuationIsCurrent())', catchStart); + const catchLog = flush.indexOf('log(`writeInput threw:', catchStart); + const persistence = flush.indexOf('persistCliSessionId(result.cliSessionId)', catchStart); + const redrive = flush.indexOf('redriveRejectedStructuredReady()', catchStart); + const deferredTimer = flush.indexOf('scheduleSubmitFailureNotify(', catchStart); + + expect(capture).toBeGreaterThanOrEqual(0); + expect(write).toBeGreaterThan(capture); + expect(resolvedGuard).toBeGreaterThan(write); + expect(busyProbe).toBeGreaterThan(resolvedGuard); + expect(rejectedGuard).toBeGreaterThan(catchStart); + expect(catchLog).toBeGreaterThan(rejectedGuard); + expect(persistence).toBeGreaterThan(rejectedGuard); + expect(redrive).toBeGreaterThan(rejectedGuard); + expect(deferredTimer).toBeGreaterThan(rejectedGuard); + const staleSettlement = flush.slice( + flush.indexOf('const handleStaleWriteContinuation ='), + flush.indexOf('let result:', flush.indexOf('const handleStaleWriteContinuation =')), + ); + expect(staleSettlement).toContain('settleStaleWriteContinuation('); + expect(staleSettlement).toContain("emitTurnTerminal(turnId, 'ambiguous', code, dispatchAttempt)"); + expect(staleSettlement).not.toContain('dropPendingTurn('); + expect(staleSettlement).not.toContain('send({'); + }); + + it('generation-fences an RPC ack before creating or activating bridge state', () => { + const flush = functionSlice('flushPending', 'sendToPty'); + const rpcBranch = flush.slice(flush.indexOf('if (writeRpcEngine) {')); + const sendAck = rpcBranch.indexOf('await writeRpcEngine.sendTurn(msg, rpcTurnIdentity)'); + const generationFence = rpcBranch.indexOf('if (!writeContinuationIsCurrent())', sendAck); + const assignBridge = rpcBranch.indexOf('bridgeTurnId = rpcTurnIdentity.turnId', generationFence); + const activate = rpcBranch.indexOf('activateRpcTurnLifecycle(', assignBridge); + expect(sendAck).toBeGreaterThanOrEqual(0); + expect(generationFence).toBeGreaterThan(sendAck); + expect(assignBridge).toBeGreaterThan(generationFence); + expect(activate).toBeGreaterThan(assignBridge); + expect(flush).toContain('codexBridgeActive && !writeRpcEngine'); + expect(rpcBranch.slice(0, sendAck)).not.toContain('codexBridgeMarkPendingTurn('); + + const ingest = functionSlice('codexBridgeIngest', 'codexBridgeMarkPendingTurn'); + expect(ingest).toContain('rpcTranscriptIngestBlockedByAwaitingActivation('); + expect(ingest).toContain('rpcTurnsAwaitingActivation.keys()'); + expect(ingest).toContain('opts.hydrationOwnerKey'); + }); + + it('anchors delayed RPC replay to the exact owner send-start time instead of ACK time', () => { + const install = functionSlice('installAwaitingRpcActivation', 'clearRpcLifecycleFailClosedOwner'); + expect(install).toContain('rpcTurnsAwaitingActivationReplayAnchors.set(ownerKey, Date.now())'); + expect(install).toContain('awaitingRpcActivationReplayAnchorMs('); + expect(install).toContain('sameRpcGeneration(rpcTurnsAwaitingActivation.get(ownerKey), generation)'); + + const activate = functionSlice('activateRpcTurnLifecycle', 'releaseRpcTurnTerminalDeferral'); + const readAnchor = activate.indexOf('const replayAnchorMs = awaitingRpcActivationReplayAnchorMs('); + const mark = activate.indexOf('codexBridgeMarkPendingTurn(', readAnchor); + expect(readAnchor).toBeGreaterThanOrEqual(0); + expect(mark).toBeGreaterThan(readAnchor); + expect(activate.slice(mark, activate.indexOf(');', mark) + 2)).toContain('replayAnchorMs'); + + const flush = functionSlice('flushPending', 'sendToPty'); + const rpcCatch = flush.indexOf('if (rpcTurnIdentity && rpcTurnGeneration && writeRpcEngine)'); + const catchAnchor = flush.indexOf('const replayAnchorMs = awaitingRpcActivationReplayAnchorMs(', rpcCatch); + const clear = flush.indexOf('clearAwaitingRpcActivation(', catchAnchor); + const catchMark = flush.indexOf('codexBridgeMarkPendingTurn(', clear); + expect(catchAnchor).toBeGreaterThan(rpcCatch); + expect(clear).toBeGreaterThan(catchAnchor); + expect(catchMark).toBeGreaterThan(clear); + expect(flush.slice(catchMark, flush.indexOf(');', catchMark) + 2)).toContain('replayAnchorMs'); + }); + + it('fails closed on an ambiguous follow-up RPC response before generic submit-failure cleanup', () => { + const flush = source.slice( + source.indexOf('async function flushPending'), + source.indexOf('function sendToPty'), + ); + const rpcCatch = flush.indexOf('if (rpcTurnIdentity && rpcTurnGeneration && writeRpcEngine)'); + const install = flush.indexOf('installRpcLifecycleFailClosedOwner(rpcTurnIdentity, rpcTurnGeneration)', rpcCatch); + const genericCleanup = flush.indexOf('codexAppPromptReplay.cancelSubmission(', rpcCatch); + expect(rpcCatch).toBeGreaterThanOrEqual(0); + expect(install).toBeGreaterThan(rpcCatch); + expect(install).toBeLessThan(genericCleanup); + expect(flush.slice(rpcCatch, genericCleanup)).toContain('codexBridgeMarkPendingTurn('); + expect(flush.slice(rpcCatch, genericCleanup)).toContain('break;'); + }); + + it('registers fresh accepted RPC turns by exact id+attempt and fails closed on lifecycle registration loss', () => { + const engage = source.slice( + source.indexOf('async function engageCodexRpc'), + source.indexOf('function armRpcStartupDialogDismiss'), + ); + expect(engage).toContain('turnId: cfg.turnId ??'); + expect(engage).toContain('dispatchAttempt: cfg.dispatchAttempt'); + expect(engage).toContain('installAwaitingRpcActivation('); + expect(engage).toContain('if (!first.nativeTurnId)'); + expect(engage).toContain('installRpcLifecycleFailClosedOwner(firstIdentity, firstGeneration)'); + expect(engage).toContain('activateRpcTurnLifecycle('); + expect(engage).toContain('firstGeneration,'); + expect(engage).toContain('deferredFreshRpcTurn = {'); + expect(engage).toContain('A dispatched-but-unconfirmed first turn is never re-sent'); + const notSent = engage.indexOf("if (first.outcome === 'not-sent')"); + const durableClaim = engage.indexOf('durableTurnInFlight = true', notSent); + expect(durableClaim).toBeGreaterThan(notSent); + expect(engage.slice(notSent, durableClaim)).toContain("return 'not-engaged'"); + expect(engage.match(/installRpcLifecycleFailClosedOwner\(firstIdentity, firstGeneration\)/g)) + .toHaveLength(2); + expect(engage.match(/awaitingRpcActivationReplayAnchorMs\(firstIdentity, firstGeneration\)/g)) + .toHaveLength(2); + expect(engage).toContain('codexBridgeMarkPendingTurn('); + }); + + it('generation-fences every awaited RPC engagement stage and never paste-falls back after supersession', () => { + const engage = source.slice( + source.indexOf('async function engageCodexRpc'), + source.indexOf('function armRpcStartupDialogDismiss'), + ); + const start = engage.indexOf('await engine.start()'); + const startFence = engage.indexOf('assertRpcEngagementCurrent()', start); + const thread = engage.indexOf('await engine.resumeThread', startFence); + const threadFence = engage.indexOf('assertRpcEngagementCurrent()', thread); + const first = engage.indexOf('await engine.sendFirstTurn', threadFence); + const firstFence = engage.indexOf('assertRpcEngagementCurrent()', first); + const durable = engage.indexOf('durableTurnInFlight = true', firstFence); + expect(start).toBeGreaterThanOrEqual(0); + expect(startFence).toBeGreaterThan(start); + expect(thread).toBeGreaterThan(startFence); + expect(threadFence).toBeGreaterThan(thread); + expect(first).toBeGreaterThan(threadFence); + expect(firstFence).toBeGreaterThan(first); + expect(firstFence).toBeLessThan(durable); + expect(engage).toContain('!rpcEngagementFence.isCurrent(engagementLease)'); + expect(engage).toContain('throw err instanceof CliSpawnSupersededError'); + expect(engage).toContain('clearRpcEnginePidMarker(enginePidMarker)'); + + const stop = functionSlice('stopCodexRpcEngine', 'persistentPaneInfo'); + expect(stop).toContain('rpcEngagementFence.invalidate()'); + }); + + it('keeps stale RPC write ownership until exact teardown instead of ordinary carryover replay', () => { + const flush = source.slice( + source.indexOf('async function flushPending'), + source.indexOf('function sendToPty'), + ); + const ackStale = flush.indexOf('if (!writeContinuationIsCurrent())', flush.indexOf('await writeRpcEngine.sendTurn')); + const ackBreak = flush.indexOf('break;', ackStale); + const rpcCatch = flush.indexOf('} catch (err: any) {', ackBreak); + const errorStale = flush.indexOf('if (!writeContinuationIsCurrent())', rpcCatch); + const errorBreak = flush.indexOf('break;', errorStale); + expect(flush.slice(ackStale, ackBreak)).toContain('Deferred stale Codex RPC continuation to engine teardown'); + expect(flush.slice(ackStale, ackBreak)).not.toContain('clearAwaitingRpcActivation('); + expect(flush.slice(ackStale, ackBreak)).not.toContain('handleStaleWriteContinuation('); + expect(flush.slice(errorStale, errorBreak)).toContain('Deferred stale failed Codex RPC continuation to engine teardown'); + expect(flush.slice(errorStale, errorBreak)).not.toContain('clearAwaitingRpcActivation('); + }); + + it('retires a no-native RPC fail-closed owner when its bounded pre-start attribution lease expires', () => { + const prune = functionSlice('pruneExpiredStructuredHeadsAndEmit', 'codexBridgeDrainAndMaybeEmit'); + const lookup = prune.indexOf('rpcLifecycleFailClosedOwners.get(ownerKey)'); + const clear = prune.indexOf('clearRpcLifecycleFailClosedOwner(identity, failedClosedGeneration)', lookup); + const terminal = prune.indexOf('emitTurnTerminal(', clear); + const restart = prune.indexOf('void restartCliProcess(', clear); + const pruneReturn = prune.indexOf('if (dropped.length === 0) return false'); + const redrive = prune.indexOf('redriveRejectedStructuredReady()', terminal); + expect(lookup).toBeGreaterThanOrEqual(0); + expect(clear).toBeGreaterThan(lookup); + expect(terminal).toBeGreaterThan(clear); + expect(restart).toBeGreaterThan(clear); + expect(terminal).toBeLessThan(restart); + expect(restart).toBeLessThan(pruneReturn); + expect(redrive).toBeGreaterThan(terminal); + expect(prune).toContain('clearAwaitingRpcActivation(identity, failedClosedGeneration)'); + expect(prune).toContain('deferredFreshRpcTurn = undefined'); + expect(prune.indexOf('if (turn.dispatchAttempt === undefined) continue')).toBe(-1); + expect(prune).toContain("'rpc_delivery_ambiguous_timeout'"); + }); + + it('buffers a fast native terminal until exact activation and hydrates output before terminal retirement', () => { + const handle = functionSlice('handleRpcTurnTerminal', 'activateRpcTurnLifecycle'); + expect(handle).toContain('rpcTurnsAwaitingActivation.get(ownerKey)'); + expect(handle).toContain('pendingRpcTurnTerminals.set(ownerKey, { terminal, generation })'); + expect(handle).toContain('sameRpcGeneration(awaiting, generation)'); + + const activate = functionSlice('activateRpcTurnLifecycle', 'releaseRpcTurnTerminalDeferral'); + expect(activate).toContain('codexBridgeQueue.markRpcActive('); + expect(activate).toContain('identity.turnId,'); + expect(activate).toContain('identity.dispatchAttempt,'); + expect(activate).toContain('codexBridgeQueue.hasTerminalTurn(identity.turnId, identity.dispatchAttempt)'); + expect(activate).toContain('if (transcriptTerminalObserved) emitReadyCodexTurns()'); + expect(activate).toContain('settleRpcTurnTerminal(pendingTerminal.terminal, generation)'); + + const hydrate = functionSlice('hydrateCompletedRpcTurn', 'settleRpcTurnTerminal'); + expect(hydrate).toContain('hydrationOwnerKey: ownerKey'); + const hydrationDrain = hydrate.indexOf('codexBridgeDrainAndMaybeEmit({'); + const finalize = hydrate.indexOf('finalizeRpcTurnTerminal(', hydrationDrain); + expect(hydrationDrain).toBeGreaterThanOrEqual(0); + expect(finalize).toBeGreaterThan(hydrationDrain); + expect(hydrate).toContain('CODEX_RPC_TERMINAL_HYDRATION_DELAYS_MS'); + + const init = source.slice(source.indexOf('await spawnCli(msg'), source.indexOf('// Queue the initial prompt')); + expect(init.indexOf('await spawnCli(msg')).toBeLessThan(init.indexOf('releaseRpcTurnTerminalDeferral(')); + }); + + it('attaches fresh Codex-family RPC rollouts from offset zero so a fast first terminal remains hydratable', () => { + const spawn = source.slice( + source.indexOf('async function spawnCli'), + source.indexOf('async function restartCliProcess'), + ); + const codexBranch = spawn.slice( + spawn.indexOf("} else if (cfg.cliId === 'codex')"), + spawn.indexOf("} else if (cfg.cliId === 'traex')"), + ); + const traexBranch = spawn.slice( + spawn.indexOf("} else if (cfg.cliId === 'traex')"), + spawn.indexOf("} else if (cfg.cliId === 'coco')"), + ); + for (const branch of [codexBranch, traexBranch]) { + expect(branch).toContain("effectiveResume ? 'baseline-existing' : 'fresh-empty'"); + expect(branch).not.toContain("codexBridgeAttach(rolloutPath, 'baseline-existing')"); + } + }); + + it('holds every successor behind an unresolved fail-closed RPC owner even when Codex type-ahead is enabled', () => { + const flush = source.slice( + source.indexOf('async function flushPending'), + source.indexOf('function sendToPty'), + ); + const preflightGate = flush.indexOf('if (rpcLifecycleFailClosedOwners.size > 0)'); + const typeAheadDecision = flush.indexOf('const typeAheadAllowed = pendingInputAllowsTypeAhead'); + const loopStart = flush.indexOf('while (pendingMessages.length > 0'); + const postWriteBreak = flush.indexOf('if (rpcLifecycleFailClosedOwners.size > 0) break', loopStart); + expect(preflightGate).toBeGreaterThanOrEqual(0); + expect(preflightGate).toBeLessThan(typeAheadDecision); + expect(postWriteBreak).toBeGreaterThan(loopStart); + }); + + it('makes native terminal settlement idempotent and generation-owned across abort/death/stop cleanup', () => { + const settle = functionSlice('settleRpcTurnTerminal', 'handleRpcTurnTerminal'); + expect(settle).toContain('const existingSettlement = settlingRpcTerminalOwners.get(ownerKey)'); + expect(settle).toContain('sameRpcGeneration(existingSettlement, generation)'); + expect(settle).toContain('codexBridgeQueue.stopRpcActive('); + + const finalize = functionSlice('finalizeRpcTurnTerminal', 'hydrateCompletedRpcTurn'); + expect(finalize).toContain('settlingRpcTerminalOwners.get(ownerKey)'); + expect(finalize).toContain('codexBridgeQueue.dropPendingTurn(identity.turnId, identity.dispatchAttempt, true)'); + expect(finalize).toContain("terminal.status === 'failed' || terminal.status === 'aborted'"); + expect(finalize).toContain("terminal.status === 'engine-dead'"); + expect(finalize).toContain("terminal.status === 'stopped'"); + + const stop = functionSlice('stopCodexRpcEngine', 'persistentPaneInfo'); + expect(stop.indexOf('engine?.stop()')).toBeLessThan(stop.indexOf('codexRpcEngine = undefined')); + expect(stop.indexOf('for (const [ownerKey, pending] of [...pendingRpcTurnTerminals])')) + .toBeLessThan(stop.indexOf('pendingRpcTurnTerminals.clear()')); + expect(stop).toContain('settleRpcTurnTerminal(pending.terminal, pending.generation)'); + const pendingLoop = stop.slice( + stop.indexOf('for (const [ownerKey, pending] of [...pendingRpcTurnTerminals])'), + stop.indexOf('for (const [ownerKey, identity] of [...rpcTurnsAwaitingActivationIdentities])'), + ); + expect(pendingLoop).toContain('notifyRpcTeardownBeforeActivation(identity, pending.terminal.status)'); + expect(stop).toContain('for (const [ownerKey, identity] of [...rpcTurnsAwaitingActivationIdentities])'); + expect(stop).toContain("'rpc_engine_teardown_before_turn_start_ack'"); + const notify = functionSlice('notifyRpcTeardownBeforeActivation', 'stopCodexRpcEngine'); + expect(notify).toContain('为避免重复执行未自动重发'); + expect(notify).toContain('兜底输出可能未被捕获'); + expect(stop).toContain('turn.finalText === undefined'); + expect(stop).toContain('turn.rpcActive'); + expect(stop).toContain('ownedRpcTurns.has(rpcTurnOwnerKey({'); + expect(stop).toContain('codexBridgeQueue.dropPendingTurn(turn.turnId, turn.dispatchAttempt, true)'); + }); + + it('does not synthesize prompt-ready after a conclusively failed unstarted submit', () => { + const cleanup = functionSlice('dropFailedBridgeMark', 'scheduleSubmitFailureNotify'); + expect(cleanup).toContain('codexBridgeQueue.dropPendingTurn(bridgeTurnId, dispatchAttempt)'); + expect(cleanup).toContain('emitReadyCodexTurns()'); + expect(cleanup).not.toContain('idleDetector?.fireIdle()'); + }); + + it('drains completions in every explicit expired-head mutation path', () => { + const prune = functionSlice('pruneExpiredStructuredHeadsAndEmit', 'codexBridgeDrainAndMaybeEmit'); + expect(prune).toContain('pruneExpiredPreStartHeadsAndEmit('); + expect(prune).toContain('emitReadyCodexTurns,'); + expect(prune).toContain('if (retiredRpcOwner || turn.dispatchAttempt !== undefined)'); + expect(prune).toContain("'ambiguous',"); + expect(prune).toContain("'structured_start_timeout',"); + expect(prune).toContain('turn.dispatchAttempt,'); + // Hermes, MTR attach+ingest, generic split-live+ingest, the 1s bridge tick, + // and the rejected-ready lease timer all funnel through the helper. + expect(source.match(/pruneExpiredStructuredHeadsAndEmit\('/g)).toHaveLength(7); + + const ticker = functionSlice('codexBridgeStartTimer', 'hermesBridgeAttach'); + const ingest = ticker.indexOf('codexBridgeIngest()'); + const finallyBlock = ticker.indexOf('} finally {'); + const tickPrune = ticker.indexOf("pruneExpiredStructuredHeadsAndEmit('structured bridge tick')"); + expect(ingest).toBeGreaterThanOrEqual(0); + expect(finallyBlock).toBeGreaterThan(ingest); + expect(tickPrune).toBeGreaterThan(finallyBlock); + }); + + it('carries the structured mark through adopt submit confirmation and exception cleanup', () => { + const adopt = functionSlice('writeAdoptMessage', 'isWorkflowWorker'); + const handler = source.slice(source.indexOf("case 'message':"), source.indexOf("case 'raw_input':")); + expect(handler).toContain('await runAdoptMessageForCapturedGeneration(item, () =>'); + expect(handler).toContain('msg.dispatchAttempt'); + expect(adopt).toContain('adoptStructuredBridgeTurnId = codexBridgeMarkPendingTurn(content, turnId, dispatchAttempt)'); + expect(adopt).toContain('scheduleSubmitFailureNotify('); + expect(adopt).toContain("'submit history'"); + expect(adopt).toContain('dropFailedBridgeMark(adoptStructuredBridgeTurnId, dispatchAttempt)'); + }); +}); diff --git a/test/worker-suspend.test.ts b/test/worker-suspend.test.ts index bcbe29287..416c037a2 100644 --- a/test/worker-suspend.test.ts +++ b/test/worker-suspend.test.ts @@ -123,6 +123,30 @@ describe('suspendWorker', () => { expect(ds.managedTurnOrigin).toEqual({ capability: 'cap-pty', turnId: 'om-pty' }); }); + it('refuses suspension while the durable Codex App dispatch ledger is non-empty', () => { + const worker = fakeWorker(); + const ds: any = { + session: { + sessionId: 'sid-owned', + status: 'active', + codexAppDispatchLedger: [ + { dispatchId: 'd-1', turnId: 't-1', state: 'prepared', content: 'owned' }, + ], + }, + initConfig: { backendType: 'tmux' }, + worker, + workerPort: 3456, + workerToken: 'token', + lastScreenStatus: 'idle', + }; + + expect(suspendWorker(ds, 'manual_suspend')).toBe(false); + expect(worker.send).not.toHaveBeenCalled(); + expect(ds.worker).toBe(worker); + expect(ds.workerPort).toBe(3456); + expect(ds.workerToken).toBe('token'); + }); + it.each([ ['missing', null], ['already killed', { killed: true }], diff --git a/test/write-input.test.ts b/test/write-input.test.ts index c077c8474..f16c72699 100644 --- a/test/write-input.test.ts +++ b/test/write-input.test.ts @@ -224,6 +224,15 @@ describe('writeInput: single-line, tmux mode', () => { expect(pty.pasteText).not.toHaveBeenCalled(); }); + it.each([ + ['hermes', createHermesAdapter('/bin/hermes')], + ['pi', createPiAdapter('/bin/pi')], + ['mtr', createMtrAdapter('/bin/mtr')], + ] satisfies AdapterEntry[])('%s: returns undefined without authoritative submit evidence', async (_name, adapter) => { + const result = await adapter.writeInput(makeTmuxPty(), 'silent submit path'); + expect(result).toBeUndefined(); + }); + it.each(PASTE_BUFFER_ADAPTERS)('%s: pasteText + delayed Enter, no sendText', async (_name, adapter) => { const pty = makeTmuxPty(); await adapter.writeInput(pty, 'hello world'); @@ -1203,7 +1212,7 @@ describe('codex writeInput submission confirmation', () => { const adapter = createCodexAdapter('/bin/codex'); const result = await adapter.writeInput(pty, MULTILINE); - expect(result).toBeUndefined(); + expect(result).toEqual({ submitted: true }); expect(pty.pasteText).toHaveBeenCalledWith(MULTILINE); expect(pty.sendText).not.toHaveBeenCalled(); expect(pty.sendSpecialKeys).toHaveBeenCalledTimes(1); @@ -1297,6 +1306,24 @@ describe('codex writeInput submission confirmation', () => { expect(pty.sendText).not.toHaveBeenCalled(); expect(pty.sendSpecialKeys).toHaveBeenCalledTimes(4); }); + + it('does not crash and reports failure when history.jsonl is absent', async () => { + // Codex has no fresh-install short-wait branch like CoCo. When + // history.jsonl does not exist at submit time, currentFileSize returns 0 + // and waitForHistoryAppend polls until the budget expires, then retries + // Enter and finally surfaces { submitted: false, recheck } — it must NOT + // throw or silently return undefined (which would let a missing submit + // look like a confirmed one). + const { rmSync } = await import('node:fs'); + try { rmSync(codexHistoryPath()); } catch { /* may not exist */ } + const pty = makeTmuxPty({ confirmCodexSubmit: false }); + const adapter = createCodexAdapter('/bin/codex'); + const result = await adapter.writeInput(pty, MULTILINE); + + expect(result).toMatchObject({ submitted: false }); + expect(typeof (result as any)?.recheck).toBe('function'); + expect((result as any).recheck()).toBe(false); + }); }); describe('coco writeInput submission confirmation', () => { @@ -1330,8 +1357,9 @@ describe('coco writeInput submission confirmation', () => { const pty = makeCocoPasteTmuxPty(); const result = await adapter.writeInput(pty, MULTILINE); - // Successful submit returns undefined (no warning needed) - expect(result).toBeUndefined(); + // Verified history append is authoritative for the worker's bounded + // structured-turn start lease. + expect(result).toEqual({ submitted: true }); // tmux paste-buffer path: single pasteText with the whole content, then // exactly one Enter (no retries — the mock confirmed via history.jsonl). expect(pty.pasteText).toHaveBeenCalledWith(MULTILINE); @@ -1393,8 +1421,8 @@ describe('coco writeInput submission confirmation', () => { const result = await adapter.writeInput(pty, angled); // Success path: JSON-decode + startsWith finds the Go-escaped content, - // so writeInput returns undefined (no warning queued). - expect(result).toBeUndefined(); + // so writeInput returns an authoritative submit confirmation. + expect(result).toEqual({ submitted: true }); }); it('skips verification on fresh install with no history.jsonl yet', async () => { @@ -1410,6 +1438,49 @@ describe('coco writeInput submission confirmation', () => { expect(result).toBeUndefined(); }); + it('fresh install: returns { submitted: true } when history.jsonl appears with our marker during the short wait', async () => { + // Fresh-install branch 1: history.jsonl is absent at submit time, but CoCo + // creates it and appends our marker within the 1.2s short-wait window. + // Adapter must return authoritative { submitted: true }, not undefined. + const { rmSync } = await import('node:fs'); + try { rmSync(COCO_HISTORY_PATH); } catch { /* may not exist */ } + const adapter = createCocoAdapter('/bin/coco'); + // The mock confirms on the first Enter by writing a coco-shaped history + // line — but the file does not exist yet when baseByte is sampled, so we + // exercise the fresh-install short-wait path rather than the normal loop. + const pty = makeCocoPasteTmuxPty({ confirmCocoSubmit: true }); + const result = await adapter.writeInput(pty, MULTILINE); + expect(result).toEqual({ submitted: true }); + }); + + it('fresh install: falls through to retry loop when history.jsonl appears without our marker', async () => { + // Fresh-install branch 3: history.jsonl is absent at submit time, appears + // during the short wait, but does NOT contain our marker (e.g. another + // session's line landed first). Adapter must NOT silently return undefined; + // it must fall through to the normal retry/failure loop and surface + // { submitted: false, recheck } so the worker can warn rather than mask a + // real submit failure on a new install. + const { rmSync } = await import('node:fs'); + try { rmSync(COCO_HISTORY_PATH); } catch { /* may not exist */ } + const adapter = createCocoAdapter('/bin/coco'); + let submittedOnce = false; + const pty: PtyHandle = { + write: vi.fn(), + sendText: vi.fn(), + sendSpecialKeys: vi.fn((key: string) => { + if (key !== 'Enter') return; + if (submittedOnce) return; + submittedOnce = true; + // File appears, but with an unrelated line — our marker is absent. + appendCocoHistory('some other session\'s submit, not ours'); + }), + pasteText: vi.fn(), + }; + const result = await adapter.writeInput(pty, MULTILINE); + expect(result).toMatchObject({ submitted: false }); + expect(typeof (result as any)?.recheck).toBe('function'); + }); + it('confirms submit when baseByte lands mid-line (non-atomic history append)', async () => { // CoCo/Trae 0.120.32 appends history.jsonl non-atomically, so the file size // captured as baseByte can fall in the MIDDLE of the JSONL line that ends up @@ -1445,8 +1516,8 @@ describe('coco writeInput submission confirmation', () => { const adapter = createCocoAdapter('/bin/coco'); const result = await adapter.writeInput(pty, prompt); - // Confirmed → no warning, and no spurious retry Enters. - expect(result).toBeUndefined(); + // Confirmed → authoritative success, no warning or spurious retry Enters. + expect(result).toEqual({ submitted: true }); const enterCalls = (pty.sendSpecialKeys as any).mock.calls.filter((c: string[]) => c[0] === 'Enter').length; expect(enterCalls).toBe(1); }); diff --git a/test/zellij-observe-backend.test.ts b/test/zellij-observe-backend.test.ts index 06fac6556..b672c4f5e 100644 --- a/test/zellij-observe-backend.test.ts +++ b/test/zellij-observe-backend.test.ts @@ -6,9 +6,11 @@ import { describe, it, expect, vi, beforeEach } from 'vitest'; // transiently disappears and comes back. const calls: string[][] = []; let listPanesResult: () => string = () => JSON.stringify([{ id: 2, is_plugin: false }]); +let failActions = false; vi.mock('node:child_process', () => ({ execFileSync: (bin: string, args: string[]) => { calls.push([bin, ...args]); + if (failActions && args.includes('action')) throw new Error('pane unavailable'); if (args.includes('dump-screen')) return 'line one\nline two\nline three\n'; if (args.includes('list-panes')) return listPanesResult(); return ''; @@ -26,6 +28,7 @@ describe('ZellijObserveBackend input encoding', () => { let be: ZellijObserveBackend; beforeEach(() => { calls.length = 0; + failActions = false; be = new ZellijObserveBackend(S, P, { cliPid: 999 }); }); @@ -44,6 +47,12 @@ describe('ZellijObserveBackend input encoding', () => { expect(actionArgs('write')).toEqual(['write', '--pane-id', P, '3']); }); + it('returns false when the targeted pane action is unavailable', () => { + failActions = true; + expect(be.sendText('/goal x')).toBe(false); + expect(be.sendSpecialKeys('Enter')).toBe(false); + }); + it('pasteText wraps text in bracketed-paste markers', () => { be.pasteText('x'); // captured call = ['zellij','--session',S,'action','write','--pane-id',P,...bytes]