execve does not terminate sibling vCPU threads before guest_reset (address-space teardown race)

[jserv]

Summary

sys_execve rebuilds the guest address space (guest_reset -> reload ELF -> rebuild page tables) without first terminating the other threads in the thread group. On a multithreaded guest, any sibling vCPU still running while guest_reset zeroes memory and recycles the page-table pool races directly against that teardown. Latent correctness bug, independent of and pre-dating PR #123.

Linux semantics

execve(2) on a multithreaded process terminates every other thread in the group before the new image is installed (de_thread() / zap_other_threads(), fs/exec.c). The calling thread is the sole survivor and assumes the group leader's PID/TID.

Current elfuse behavior

sys_execve (src/syscall/exec.c) marks its point of no return at exec.c:581 (before closing CLOEXEC fds) and tears the address space down at guest_reset(g) (exec.c:689) with no sibling teardown anywhere in the function -- no thread_interrupt_all, thread_join_workers, or thread_quiesce_siblings. guest_reset (src/core/guest.c:1584), on the calling thread:

• memsets every tracked guest region to zero (guest.c:1602-1611),
• zeroes and recycles the page-table pool in place (pt_pool_next -> pt_pool_base, guest.c:1613-1616),
• zeroes shim code + data (guest.c:1618-1622).

Guest threads are 1:1 with host pthreads, each owning its own HVF vCPU (MAX_THREADS = 64, thread.h:29).

Race window and failure modes

While the execing thread runs guest_reset + rebuild, a sibling vCPU may be:

1. Executing guest code inside hv_vcpu_run() with in-flight page-table walks into pages being zeroed/rewritten -- the same hazard PR Re-enter execve through the MMU-off shim boot path #123 fixed for the calling thread, on a thread that never gets the MMU-off re-entry.
2. Blocked in a host syscall handler that reads/writes guest memory.
3. Walking shim_data on a fast path -- guest_reset zeroes the shim block underneath it (the EL1 abort-recover handler only covers the urandom write ranges).

Consequences: sibling translation fault (crash/exit 128), silent corruption of the freshly loaded image, or a sibling executing instructions from a different program after the swap.

Why it has not been caught

The exec test corpus is dominated by single-threaded callers (env, true, shells that fork then exec). The trigger is a genuinely multithreaded process calling execve from any thread while siblings are still active.

Relationship to PR #123

PR #123 fixed the calling thread's own stale-TLB instruction abort via the MMU-off _start re-entry (exec_stage_mmu_off_reentry, exec.c:85), scoped to the one vCPU that runs execve. This issue is the complementary gap: the sibling vCPUs that PR #123 does not touch and that today are never stopped.

Existing machinery -- and why it is not enough as-is

• exit_group: thread_interrupt_all() -> hv_vcpus_exit(), then thread_join_workers() (thread.c:316,379). But thread_join_workers waits ~100 ms then detaches still-active workers (thread.c:338,353). That is safe for a dying process; for exec it is not a barrier -- a detached sibling can later wake and reuse the same guest_t after reset.
• fork: thread_quiesce_siblings() / thread_resume_siblings() (thread.c:426, forkipc.c:1420) -- quiesce-and-resume, wrong shape for exec.

execve needs a new exec-specific terminate-and-prove-gone path: stop every sibling, confirm it can no longer touch guest_t, then guest_reset. Reusing the exit_group helper as-is is incomplete.

What the fix must cover

1. Terminate all CLONE_THREAD siblings before guest_reset and prove they are gone (no detach-on-timeout fallthrough past the point of no return).
2. Host-side blocking: hv_vcpus_exit only breaks hv_vcpu_run, not a sibling blocked in read/poll/nanosleep (time.c:342, poll.c:186). Need either a host signal to force EINTR or a defined fatal-timeout policy; exec must not cross reset while a sibling can still reach guest memory.
3. Run normal thread-exit cleanup for each killed sibling: robust-futex list walk and CLONE_CHILD_CLEARTID clear+wake (forkipc.c:882,889; VM-clone forkipc.c:1137). Force-kill must not bypass this.
4. Signal/thread-table state: masks are per-thread, pending is process-wide (signal_reset_for_exec, signal.c:327). Reap dead sibling thread entries so thread_signal_deliverable() / signal_pending() do not scan obsolete masks.
5. Lock ordering: execve already holds mmap_lock (order 1) and takes fd_lock (order 3, exec.c:596). Teardown must preserve mmap_lock -> fd_lock -> sig_lock -> thread_lock and must not join while holding thread_lock, or it deadlocks against a sibling holding those locks.
6. Reassign the calling thread as the sole surviving thread / group leader.
7. Regression test: a multithreaded guest with busy sibling threads (spinning on shared memory, and one blocked in a host syscall) that then calls execve from a non-leader thread.

Related hazards surfaced during review

• CLONE_VM-without-CLONE_THREAD inferiors (VM-clone / ptrace, forkipc.c:1286) share guest_t* but are not thread-group siblings. Recycling the address space in place still wipes their memory. This is a semantic-policy question (conditionally unshare, or refuse), not just "don't tear them down." posix_spawn/IPC fork children are separate host processes and must be excluded.
• MAP_SHARED / memfd overlays (mem.c:1631, forkipc.c:1437): guest_reset memsets tracked backing unconditionally, so a MAP_SHARED region backed by a host file has zeroes written straight through to the file. Exec teardown must not corrupt shared backing or leave stale host overlays -- possibly a distinct bug worth its own issue.