Skip to content

feat(shutdown): reap wedged ssh transport children on exit — exact-PID registry (issue #6)#9

Merged
bcourbage merged 5 commits into
mainfrom
feat/ssh-reaper
Jul 19, 2026
Merged

feat(shutdown): reap wedged ssh transport children on exit — exact-PID registry (issue #6)#9
bcourbage merged 5 commits into
mainfrom
feat/ssh-reaper

Conversation

@bcourbage

@bcourbage bcourbage commented Jul 19, 2026

Copy link
Copy Markdown
Owner

Reaps a wedged sync's orphaned ssh transport child on app exit, via a pure-C exact-child-PID registry. Follow-up to #7; passed cumulative adversarial review and the full live-validation matrix.

Defect

The engine spawns one ssh child per remote connection (remote.ml). Normal OCaml termination reaps it via at_exit end_ssh, but the AppKit host can terminate without running OCaml at_exit (applicationWillTerminate → unison_bridge_shutdown is a bounded C helper), and a sync wedged on a frozen remote is blocked on the transport socket, so closing fds never makes it exit. The child survived as an orphan (observed alive to t+230s; died only when the remote was killed). Characterized during #7 / TC10.

Design — exact-child-PID registry (needs no OCaml/Lwt progress at shutdown)

A C-owned, mutex-protected registry with three operations:

  • track(pid) — OCaml records the exact pid right after spawn. Deduplicated under the mutex, so a repeat track is a no-op and one retire fully removes it.
  • retire(pid) — teardown calls this before it waitpid/close_sessions the child. Under the mutex, and only while the pid is still registered, it SIGKILLs the exact pid and removes it — atomically. SIGKILL is issued before removal: the child is thereafter irrevocably terminating (SIGKILL can't be caught/blocked) though it may not have exited the instant kill() returns; its pid stays reserved until the following waitpid. Idempotent; never signals a pid it no longer tracks (so it can't hit a reused pid).
  • reap() — the pure-C shutdown pass (called first in unison_bridge_shutdown): under the mutex it sets closing, SIGKILLs every still-registered pid, then clears the set.

Every kill(2) is issued while holding the mutex, before removal. Because OCaml can only waitpid a child after retire removed it under the same mutex, no concurrent reap can free a pid mid-signal → no PID-reuse race and no unregistered-but-still-alive window. No process enumeration — exact registered pids only. A spawn racing closing, or a full registry, is SIGKILLed at once so it can't escape. Full adversarial proof in docs/ssh-reaper-design.md.

Both spawn paths are covered by patches/0004: the synchronous buildShellConnection (CLI) and the async openConnectionStart/End/Cancel path the macOS GUI actually uses. Hooks are overridable (int -> unit) ref no-ops in remote.ml; the macOS bridge (uimacbridge.ml) installs the real ones. CLI/GTK builds link with zero bridge symbols (verified) and keep their prior at_exit behavior.

Toolchain / blob

The blob's runtime ABI is version-locked, so OCaml 5.5.0 is pinned: make check-ocaml-version fail-fast on build/vendor-blob; CI + release link via ocaml/setup-ocaml 5.5.0 (not bare brew install ocaml). vendor-blob builds only unison-blob.o (upstream's own app link can't resolve our C symbols).

Blob rebuilt at base 91421d0 + patches 0002/0003/0004, OCaml 5.5.0: sha256 a57f5c4ec18d96277ac2cde58a7d8f703b012daffdefa42877638671eb062b03, 5462536 bytes.

Validation

  • 479 tests, 0 failures, incl. test_concurrentRetireAndReap_sameChild_isSafe — a genuinely concurrent retire vs reap on the same tracked child, synchronized start, 50 iterations, outcome-based assertions (child SIGKILLed+reaped, registry empty, unrelated survives, no hang/leak) — plus dedup, idempotent-retire, late-registration-after-shutdown, exact-kill, unrelated-survives, no-children.
  • Debug + Release build; Release carries no test-hook strings; the test-only reset is UNISON_DEBUG_HOOKS-gated and absent from Release; git diff --check clean.
  • Live (key auth, Demeter): normal-quit → no orphan (0 ssh, 0 remote, 0 errors); frozen-remote-server wedge → app quit → exact orphan reaped.
  • Live (password auth, 192.168.2.241): open/scan/sync/leave/reopen; adaptive hold after sync with same-session Rescan and no re-prompt, and re-prompt only after an actual close; credential-prompt cancellation → pending ssh reaped, no stale callback, clean replacement profile; interactive-auth connection wedged during sync (stopped local transport child) → app quit → exact ssh child reaped; an unrelated ssh session survived untouched (exact-PID scoping). Password entered by the maintainer directly; never inspected/logged.

Limitations

  • Teardown-time reaping only — this reaps transport children when the connection tears down or the app exits; it is not in-process core scan/connect cancellation (a separate, out-of-scope item).
  • Blob not bit-reproducible — inputs (source/patches/toolchain/command) are pinned, but the .o is not byte-identical across clean rebuilds (OCaml/ld nondeterminism, observed); documented in vendor/README.md.
  • Fixed 64-child registry (MAX_TRANSPORT_CHILDREN) — far beyond any real profile set; a pid that would overflow it is SIGKILLed rather than left untracked.
  • The password-auth wedge was induced by stopping the local transport child (the remote is password-only, so its server can't be SIGSTOPed without the password); the frozen-remote-server variant is covered over key auth.

Refs #6 (not closed by this PR).

🤖 Generated with Claude Code

bcourbage and others added 5 commits July 19, 2026 13:50
A sync wedged on a frozen remote leaves the ssh child blocked on the transport
socket (not stdio). On app quit the AppKit host terminates without running
OCaml at_exit, so remote.ml's at_exit `end_ssh` never fires and pipe closure
can't wake the blocked child — it survives as an orphan (observed alive to
t+230s; died only when the remote was killed).

Fix: a pure-C, mutex-protected, exact-child-PID registry consumed by C during
unison_bridge_shutdown, whose guarantee needs no OCaml/Lwt progress at exit.
- patches/0004: overridable Remote.register/unregisterTransportChild hooks
  (default no-ops → CLI/GTK unchanged; verified the text UI links with zero
  bridge symbols). The macOS bridge (uimacbridge.ml) installs real hooks.
  Both spawn paths covered: buildShellConnection (CLI) and the async
  openConnectionStart/End/Cancel path the GUI actually uses. Register right
  after spawn; unregister strictly before every reap (waitpid/close_session),
  while the PID is still reserved — so the registry never holds a reusable PID.
- UnisonBridgeC.c: track/untrack/reap. reap atomically flips a `closing` flag +
  detaches the set under a leaf mutex, then SIGKILLs the snapshot outside it
  (SIGKILL, not SIGTERM, so a stopped/wedged child can't defer it; no waitpid,
  so shutdown never blocks and there's no double-reap race). A register racing
  `closing` kills the just-spawned child immediately — a spawn can't escape.
  unison_bridge_shutdown calls the reaper first, before the bounded root
  release. Adversarial race proof in docs/ssh-reaper-design.md.

Toolchain: the blob's runtime ABI is version-locked, so pin OCaml 5.5.0 —
`make check-ocaml-version` fail-fast on build/vendor-blob; CI/release link via
ocaml/setup-ocaml 5.5.0 (not bare `brew install ocaml`). vendor-blob now builds
just unison-blob.o (upstream's own app link can't resolve our C symbols).

Blob rebuilt at base 91421d0 + patches 0002/0003/0004, OCaml 5.5.0:
sha256 4292c9a3…, 5462536 bytes. Reproducibility recorded honestly in
vendor/README: inputs pinned, but the .o is NOT byte-identical across clean
rebuilds (OCaml/ld nondeterminism observed).

Tests: +7 registry tests (lifecycle, idempotent, no-children, exact-kill,
unrelated-survives, late-registration-race, unregister-before-reap/PID-reuse),
using real forked children (477 total, 0 failures). Debug+Release build;
Release carries no test-hook strings; the test-only reset is Debug-gated
(UNISON_DEBUG_HOOKS) and absent from Release.

Live-verified: wedged-remote quit now SIGKILLs the exact orphan (reap n=1, ssh
gone); normal quit reaps via the normal path (0 ssh, 0 remote servers left).

Refs #6.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
setup-ocaml installs the 5.5.0 switch but doesn't add its bin to PATH for
plain run: steps, so ocamlc was not found. Append $(opam var bin) to
$GITHUB_PATH so ocamlc (and make's ocamlc -where / check-ocaml-version)
resolves in Verify/Build/Test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…/live windows

Adversarial-review corrections to the SSH-reaper:

1. PID-reuse race in the reaper: it cleared+unlocked before SIGKILL, so a
   concurrent OCaml waitpid could reap+free the pid before the signal landed.
   Now every kill(2) is issued WHILE HOLDING the mutex (reap signals all
   registered pids under the lock, then clears).

2. Untracked-live-child window: unregister ran before waitpid/close_session,
   leaving a child alive-but-untracked (up to ~2s, or indefinitely for an
   unresponsive ssh) and invisible to a racing shutdown. Replaced with
   `retireTransportChild`: under the mutex, SIGKILL the exact child AND remove
   it, only while still registered — the child is dead before it leaves the
   registry. OCaml then waitpid/close_sessions it. Default hook stays a no-op
   so CLI/GTK are unchanged (text UI links with zero bridge symbols). On the
   GUI, normal teardown now SIGKILLs the child too (pty is app-internal; the
   remote exits on the connection drop as on any close) — verified no
   close-path regression (normal quit: 0 ssh, 0 remote, 0 errors).

3. track() deduplicates under the mutex, so duplicate track + one retire leaves
   no stale entry.

4. Rewrote docs/ssh-reaper-design.md proof + code comments for the retire
   contract; removed the incorrect "closing stdio proves the child is no longer
   live" claim.

5. Tests rewritten: retire-kills-exact-child, duplicate-track→one-retire,
   retire idempotent (never signals an untracked pid), retire-then-reap no
   double-target, reap-exact / unrelated-survives / no-children, late-
   registration-after-shutdown. No test describes a live untracked child.
   (478 total, 0 failures.)

6. INSTALL.md + vendor/README: every maintainer path now requires OCaml 5.5.0
   (opam switch), not an unversioned `brew install ocaml`.

Blob rebuilt (base 91421d0 + 0002/0003/0004, OCaml 5.5.0): sha256 a57f5c4e…,
5462536 bytes. Live-verified: wedged-remote quit reaps the exact orphan;
normal quit leaves no ssh/remote and no errors.

Refs #6.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. Add test_concurrentRetireAndReap_sameChild_isSafe: runs retire(pid) and
   reap() on two threads with a synchronized start, racing for the registry
   mutex, over 50 iterations. Asserts outcomes (not timing): the tracked child
   is SIGKILLed + reaped, the registry ends empty, an unrelated child survives,
   and no iteration hangs or leaks. Replaces the sequential-only coverage of
   the shutdown-vs-retire ordering. (479 tests, 0 failures.)

2. Documentation-only corrections (no runtime change, blob unchanged):
   - UnisonBridgeC.h: track/untrack -> track/retire wording.
   - UnisonBridgeC.c + docs/ssh-reaper-design.md: replace "child is already
     dead when kill returns" with the precise guarantee — SIGKILL is issued
     before removal, the child is irrevocably terminating (SIGKILL can't be
     caught/blocked), and its pid stays reserved until the following waitpid.
   - INSTALL.md: the linked OCaml 5.5.0 runtime comes from the selected
     toolchain (preferred: the opam 5.5.0 switch), not necessarily Homebrew.

Refs #6.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace 'the child is already dead from retire' with the precise guarantee:
the child has already been sent SIGKILL by retire and is irrevocably
terminating. Comment-only; no runtime change, blob unchanged.

Refs #6.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@bcourbage bcourbage changed the title feat(shutdown): reap wedged ssh transport children on exit (SSH-reaper) feat(shutdown): reap wedged ssh transport children on exit — exact-PID registry (issue #6) Jul 19, 2026
@bcourbage
bcourbage marked this pull request as ready for review July 19, 2026 21:59
@bcourbage
bcourbage merged commit 5228641 into main Jul 19, 2026
1 check passed
bcourbage added a commit that referenced this pull request Jul 20, 2026
Safety-correction pass for PR #17 (included-root attribution).

1. Existing-but-unreadable target is no longer treated as absent, even
   when OPTIONAL. `include?`/`source?` skip only a proven-`.missing` file;
   an unreadable one is `.unreadable` -> unreliable. Unison reads profiles
   as bytes, so a file we fail to decode as UTF-8 may still carry roots we
   can't see; treating it as absent would misattribute an archive as an
   orphan.

2. filesystemRead classification made precise. A directory, or a
   non-regular object (FIFO/device/socket), is `.unreadable` and NOT
   `.missing`: its existence must suppress the `.prf` fallback for
   `include`, and a FIFO/device is classified via stat WITHOUT being opened
   (an open could block indefinitely). A leading UTF-8 BOM is stripped so
   the first line never mis-parses as `.raw`.

3. Enumeration failure propagates GLOBAL uncertainty. When the Unison
   directory can't be enumerated, the controller no longer returns an empty
   profile set (which would mark every archive a certain orphan and
   preselect the local-only ones for deletion). Every row becomes uncertain
   and none are preselected.

4. Uncertainty is surfaced. A new orange banner (pure, tested
   `attributionWarning`) explains an unreadable directory and names
   profiles with unresolved/unreadable includes; the per-row tooltip now
   lists the includes case too.

5. Tests: optional invalid-UTF-8 -> unreliable; exact-path directory
   alongside a valid <name>.prf -> no fallback + unreliable; FIFO ->
   unreadable without hanging; missing/regular classification; BOM parity;
   global-uncertainty forces no destructive preselect; warning-text content
   + no em-dash. Full suite 580 tests, 0 failures (1 pre-existing env skip).

Debug build + Release compile both succeed. Swift-only; vendored blob
a57f5c4e and vendor/ patches/ untouched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant