Fork Sync: Update from parent repository#174
Open
github-actions[bot] wants to merge 51 commits into
Open
Conversation
The contrib/dockerswarm harness baked PRRTE into its image from a git-archive of the *committed* tree. That froze the image at build-time HEAD, so testing an uncommitted change meant either committing first or hand-copying files into all ten containers and rebuilding in place. It also only ever built and ran Linux, leaving macOS-specific build and runtime regressions uncaught. This reworks the harness to build the live working tree, out of tree (VPATH), so the source stays pristine and can feed two independent builds. build.sh distcleans and autogens the tree once, then compiles it in a builder container that bind-mounts the source read-only and installs into a shared volume the nodes mount; a `macos` mode does the same build natively on the host. The image no longer contains PRRTE -- it provides only the toolchain, a baked PMIx default, SSH wiring, and an entrypoint that makes the shared-volume install loadable and puts its binaries on PATH (without which plm/ssh cannot find prted on the other nodes). Point PMIX_SRC at an openpmix checkout to build PMIx from source too, covering both code bases. A new run-tests.sh drives the suite and reports pass/fail: the full multi-node set on Linux (prterun, elastic grow/shrink, and a radix-2 deep-tree relay), and a bounded single-host set on macOS that verifies the Darwin build and single-host launch while degrading cleanly when the native DVM is unavailable. The result eliminates the stale-image and commit-only-code problems -- edit, rerun build.sh, and the swarm runs your change -- and adds native macOS coverage from the same sources. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The rmaps (Resource MAPping) subsystem carries a lot of implicit structure that is hard to reconstruct from the code alone: the priority-ordered "keep every module and let each accept or defer" selection model, the packed mapping/ranking/binding policy bits, the per-app (MPMD) dispatch path with its cross-app vpid numbering, and the division of labor between the base orchestrator and the individual mappers. New contributors - human or AI - repeatedly have to rediscover these invariants before they can safely touch a mapper. Add an AGENTS.md to the framework directory describing the framework as a whole, and one to each component directory (round_robin, ppr, seq, rank_file, lsf) explaining that mapper's niche, priority, gate conditions, and algorithm in detail. These are orientation maps that point back at the authoritative docs, matching the repository's existing AGENTS.md convention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
rmaps_rank_file.c carried two static forward declarations - prte_rmaps_rf_lsf_convert_affinity_to_rankfile() and prte_rmaps_rf_process_lsf_affinity_hostfile() - that are never defined and never called. They are the remnant of an abandoned idea to convert an LSB_AFFINITY_HOSTFILE into rankfile form; LSF affinity is instead handled by the standalone rmaps/lsf component, which parses that file directly. Remove the dead declarations so the source reflects what the mapper actually does. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The repository exposes its orientation docs under both the AGENTS.md and CLAUDE.md names via a symlink, so tooling keyed to either name finds the same content - as the top-level CLAUDE.md -> AGENTS.md link already does. Add the matching CLAUDE.md -> AGENTS.md symlink alongside each rmaps guide (the framework directory and every component) so the new guides are discoverable under both conventions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The src/rml tree was once three MCA frameworks -- rml (the messaging API), routed (pluggable routing-tree algorithms), and oob (pluggable transports). Because PRRTE only ever had one RML, one routing algorithm (a radix tree), and one transport (TCP), the three were collapsed into a single directory -- but the multi-component abstractions were left in place. What remained was dead scaffolding that made the code harder to read and maintain: a "hand the failed message to another OOB component" failover chain that nothing drives, a relay macro nothing calls, a routed-module name carried in every wire header, and comments describing a world of many components, modules, and transports that no longer exists. This removes that scaffolding. Gone are prte_oob_ping and the hop_unknown / no_route / msg_error handlers and macros it fed -- one of which, PRTE_ACTIVATE_TCP_NO_ROUTE, could never have compiled, as it named a misspelled global -- along with the unused QUEUE_RELAY and POST_SEND macros and their now-orphaned caddy types, the unused MCA_OOB_TCP_PING type, an unused iovec send-callback typedef, a broken and unused radix static initializer, and several write-only base-struct fields. The vestigial routed[] field and PRTE_MAX_RTD_SIZE are removed from the on-wire TCP header, shrinking it; this is safe because every daemon in a DVM runs the same build, so the header is never exchanged across versions. The surviving multi-component comments are rewritten to describe the actual single path -- RML to radix routing to TCP -- with relay explained as re-entering the send path toward the next hop. While removing the dead fields this also fixes a latent bug: the prte_max_msg_size MCA parameter was registered against max_recon_attempts, so setting it silently corrupted the reconnect count while the real message-size limit stayed pinned at its default. To make the tree easier to modify going forward, this adds src/rml/README.md as an editing-oriented map and docs/how-things-work/rml as a narrative of how the revised RML works. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The repo root already follows the AGENTS.md convention: the real orientation file is AGENTS.md and CLAUDE.md is a symlink to it, so both agent-agnostic tooling and Claude Code load the same guidance. The RML editing map was the one per-directory orientation doc still named README.md, which no agent harness auto-loads when working under src/rml -- defeating the point of a document that describes itself as "an orientation map for anyone (human or agent)". Rename it to AGENTS.md, add the parallel CLAUDE.md symlink, fix the document's self-reference, and update the two live pointers in the how-things-work RML page. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
…paign) The shipped shrink path (shrink_campaign.rst) drains a campaign one daemon at a time: each targeted daemon self-exits, the HNP learns of each departure through the errmgr comm-failure path, and every departure drives its own routing-tree repair, so shrinking m daemons on one branch of the radix tree can run up to m sequential promotions. Issue #2492 proposes collapsing that to a single repair per campaign. This adds a plan document capturing the required revisions: hook the reliable xcast's completion so the HNP repairs the whole batch of targets in one prte_rml_repair_routing_tree() pass and emits one completion event; move the daemon side from self-exit to entering a leaving mode and waiting for its lifeline to drop; and retire the per-death completion logic and its idempotency stamping from the errmgr. It records the design decision that leaving mode must ride in the shrink command itself rather than a separate order, which makes the scheme race-free by construction because the broadcast reaches each doomed daemon through the very lifeline whose later failure triggers its termination. The document is explicit that the terminate-on-lifeline path does not exist today and must be built and gated so a daemon exits only when it has been ordered out, and it points validation at the ten-node Docker testbed. This is documentation only; no code behavior changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The shipped shrink path drained a campaign one daemon at a time: each targeted daemon self-exited, the HNP learned of every departure through the errmgr comm-failure path, and each departure drove its own routing-tree repair. A shrink of m daemons on one branch therefore ran up to m sequential promotion and descendant rewrites on the HNP, which review of the launch-fence work flagged as potentially expensive for a large shrink. This is the optimization tracked in issue #2492. Complete the campaign as a single collective event instead. The reliable xcast already knows when every daemon in the DVM has received a broadcast, so add an xcast_nb entry point that fires a completion callback on the master at that point; the shrink command registers one carrying its campaign. Because the initiating op is consumed while relaying to the master and the tracked op is rebuilt on receipt, the callback is carried in a FIFO the master pops when it relays its own broadcast back to itself. When the callback fires, the master tears the whole set of targets out of the DVM in one prte_rml_repair_routing_tree pass -- marking each not-alive, setting its state to TERMINATED, decrementing num_daemons, and detaching its node -- then drops the fence and emits the single PMIX_DVM_IS_READY completion event. A targeted daemon no longer self-exits on receipt: doing so before its subtree acknowledged the broadcast would keep the master's completion from ever firing. It instead records that it is leaving and departs on a bounded timer, or sooner when its lifeline drops (prte_rml_route_lost departs early once the daemon knows it is leaving, so a genuine unrelated fault still recovers). Because the master proactively marked every target not-alive and TERMINATED, the target's eventual real comm-failure is recognized as an already-departed daemon and ignored, rather than double-counting num_daemons or aborting the DVM; the per-death shrink-campaign bookkeeping in the errmgr is removed. The completion event and the fence release therefore fire once per campaign, and the routing tree is repaired once, regardless of how many daemons the shrink removes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The first cut of the collective shrink left three pieces running unconditionally: the xcast completion FIFO fired for every master broadcast, a daemon named in a PRTE_DAEMON_SHRINK_CMD recorded a leaving state and departed on a timer, and the errmgr ignored a comm-failure for an already-departed daemon. None of these is wrong outside elastic mode, but the collective completion they support only ever runs in elastic mode, so leaving them active needlessly perturbs the default launch and fault-handling paths that every non-elastic prterun and persistent DVM rely on. Gate all three on prte_elastic_mode. Outside elastic mode the xcast FIFO is never touched, so xcast behaves exactly as before; a shrink-command target does the legacy clean immediate exit rather than deferring to the departure timer; and the already-departed guard is skipped so daemon fault handling is unchanged. Inside elastic mode every path behaves as validated. Confirmed on the testbed that a non-elastic prterun across three nodes and a non-elastic persistent DVM plus prun still work, and that the elastic grow/shrink cycle still drains to a single completion event. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The master attaches a broadcast's completion callback to the op it builds when that broadcast is relayed back to itself, popping a FIFO of pending completions in the order the broadcasts were emitted. Enqueuing that FIFO entry in xcast_nb() left a gap: if begin_xcast() then failed to pack or send the broadcast, it returned without unwinding the entry, so the next master-originated broadcast would pop a stale head and every subsequent completion would be attached to the wrong op. Move the enqueue into begin_xcast(), immediately before the reliable send that emits the broadcast, and remove the entry if that send fails. The FIFO now tracks exactly the broadcasts that were actually put on the wire, in emission order, so a dropped send can no longer shift the alignment. Both the enqueue and the op-id stamping done on receipt run on the single progress thread with in-order delivery to self, so the ordering guarantee the FIFO relies on is preserved. The completion callback is stashed on the initiating op so begin_xcast can read it; it is never fired from that op, only from the op the master builds on receipt. The path remains gated on prte_elastic_mode. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The collective shrink-completion plan has now been implemented and exercised on the ten-node Docker testbed, so fold what was learned back into the document. Add an implementation-status section noting the change built warning-free under devel-check, that the general xcast_nb facility and the bounded-timer departure were the choices taken, and that survivor-side repair batching was deliberately left out because it would race the reliable xcast's ACK bookkeeping. Replace the speculative validation checklist with the results actually observed — single-branch and multi-branch shrinks each drain to one completion event with a single tree repair and the departures absorbed by the already-departed guard, a target crashed mid-shrink still drains once, and the fence does not wedge concurrent launches — and record why the in-flight-remap case could not be exercised in this harness. Note the two bugs found and fixed during validation (the completion callback lost across the master's relay-to-self, and the static-initialized FIFO that corrupted on append) so they are not reintroduced. Update the open questions to mark the terminate-on-lifeline, callback-shape, and comm-failure-fall-through items resolved and to carry survivor-side batching forward as the remaining follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
Record in the collective-shrink plan that the whole completion path — the xcast_nb enqueue on the master, the daemon leaving mode, and the already-departed guard in errmgr/dvm — is gated behind prte_elastic_mode, so a default prterun and a persistent DVM plus prun retain their prior launch and fault-handling behavior when elastic mode is off. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The collective-shrink completion callback cannot ride the op the master initiates, because that op is discarded once relayed; it rides a FIFO of pending completions that the master pops as it builds each tracked op on receipt. Document that mechanism in the plan, including that the FIFO entry is enqueued in begin_xcast immediately before the send and unwound on failure, so a dropped broadcast cannot misalign a completion onto the wrong op. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The title underline was one character shorter than the title (the em-dash in the heading pushed the title to 72 characters against a 71-character rule), which docutils flags as "Title underline too short." Extend the underline to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
When an elastic DVM shrinks a node out, the daemon's node object is left
carrying the launch state from its previous life in the DVM: the
PRTE_NODE_FLAG_DAEMON_LAUNCHED flag stays set and the node stays in the
daemon-job map. Every plm launcher consults PRTE_NODE_FLAG_DAEMON_LAUNCHED
to decide whether a node already has a daemon, so on a later grow the node
is skipped ("daemon already exists") and its prted is never relaunched;
the stale map entry additionally lets setup_vm add the node a second time,
duplicating it in the VM. Either way the re-grow silently misbehaves.
Introduce prte_plm_base_reset_dvm_node(), which returns such a node to a
pristine, never-launched state by clearing PRTE_NODE_FLAG_DAEMON_LAUNCHED
and PRTE_NODE_FLAG_LOC_VERIFIED and dropping the node from the daemon-job
map, and call it from both teardown paths that remove a node from the DVM:
the collective shrink completion and the grow-failure rollback. This is
launcher-agnostic and is a prerequisite for re-growing a previously shrunk
node.
Separately, route PRTE_PROC_STATE_FAILED_TO_CONNECT through the errmgr's
comm-failure handling (where a grow target's failure is rolled back and
the requester notified) rather than the fatal "UNSUPPORTED DAEMON ERROR
STATE" path, so a daemon that comes up but cannot complete its connect-back
during a grow no longer takes down the whole DVM.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Ralph Castain <rhc@pmix.org>
The collective-shrink design doc described the per-target HNP teardown but not the node launch-state reset a re-grow depends on. Document the new prte_plm_base_reset_dvm_node() step in the Step-2 teardown, note that the grow-failure rollback shares it and that FAILED_TO_CONNECT now routes into the grow rollback rather than aborting the DVM, and record that #2491 is only partially addressed here: the launcher-agnostic node reset is done, but the vpid-hole / routing-tree half still needs a launcher-agnostic fix because vpid reuse would break the sequential-vpid launchers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The positional radix routing tree spans [0, num_daemons) and decides who is reachable purely from vpid arithmetic, consulting failed_dmns to skip ranks that have left. prte_rml_compute_routing_tree() re-initializes failed_dmns every time it runs, which is fine at startup but wrong when a DVM grow re-runs it: a daemon that was shrunk out (or lost to a fault) leaves a permanent hole in the vpid space, because the DVM never reuses a daemon vpid. The recompute forgot that hole and rebuilt a tree that routed to the dead rank, so wireup to a newly grown daemon failed and the grow never completed (#2491). Add a persistent dead_dmns bitmap to prte_rml_base that, unlike failed_dmns, is initialized once and never re-initialized. prte_rml_repair_routing_tree() records every departing rank in it, and prte_rml_compute_routing_tree() restores those marks into the freshly initialized failed set and then routes the base tree around them with the same ancestor/child repair a fault would perform (minus the fault-handler notifications, since a recompute is not a new fault). The tree therefore tolerates a sparse vpid space for every launcher, which is the property the non-ssh launchers need since they assign daemons a sequential vpid range and cannot accept vpid reuse. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
A freshly grown daemon starts with an empty dead-daemon set and never witnessed the shrink events that removed its peers, so on its own it cannot know which ranks are permanent vpid holes. It also derived num_daemons from the count of live daemons in the nidmap, which undercounts the vpid space whenever a hole is present and leaves the new daemon disagreeing with the HNP about the size of the tree. Pack the true daemon vpid-space size (prte_process_info.num_daemons) into the nidmap alongside the live daemon list, and on decode set num_daemons from it and mark every rank in [0, num_daemons) that has no live daemon entry as permanently dead before computing the routing tree. Every daemon, newly grown or long-standing, then converges on the same vpid span and the same dead set as the HNP, so the tree is routed around the hole even in a deep tree where a new daemon's own computed ancestor is a departed rank. On an unshrunk DVM the packed span equals the live count, nothing is marked, and behavior is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
setup_vm computes map->num_new_daemons and map->daemon_vpid_start fresh on each call, but the reset that zeroes num_new_daemons sits after the construct: label while the grow path (PRTE_JOB_EXTEND_DVM) jumps straight to construct: and skips it. Across successive grows num_new_daemons therefore accumulates and daemon_vpid_start is never refreshed, which corrupts the grow campaign's target list and requester bookkeeping and suppresses its completion event. Reset both on entry to the grow branch so each grow accounts only for the daemons it actually launches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
Update the collective-shrink-completion plan's re-grow (#2491) section to match the implementation: the analysis of why a grow after a shrink failed is sharpened (the recompute wipes the shrink's failed-daemon marks), and the resolution is now described as implemented rather than deferred - the persistent dead-daemon set, the grow-path accounting reset, and the hole-aware nidmap that lets a brand-new daemon learn the vpid holes. The remaining VM-ready-gate follow-on stays tracked in #2491. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
…line A daemon that has entered leaving mode (prte_dvm_leaving) previously departed only when the specific route it lost was its lifeline. But a leaving daemon can observe a child connection drop before its lifeline does. Treating that earlier loss as an ordinary child failure makes the daemon emit an adoption notice for the rank promoted in the child's place; if that notice reaches the adoptee before word of this daemon's own departure, the adoptee marks its parent as failed and propagates a spurious fault up the tree. Depart on the first lost route while leaving, regardless of which route it was, so a doomed daemon's disconnects can never be misread as faults. The guard stays strictly gated on prte_dvm_leaving, so an unrelated fault in a daemon that is not leaving still takes the normal recovery path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The pending-completion FIFO that carries an xcast_nb callback across the master's relay-to-self was gated on prte_elastic_mode at both the enqueue (begin_xcast) and dequeue (xcast_recv) sites. That left the callback silently inert for any master-originated broadcast outside elastic mode: the caller passes a non-NULL cbfunc, but nothing is queued or re-attached, so finish_op never fires it. A facility that quietly drops the callback depending on a mode flag is a trap for future callers. Drop the elastic-mode gate so any master-originated broadcast enqueues an entry, keeping the FIFO aligned with exactly the broadcasts emitted, and let finish_op fire the callback only when one was actually registered. The cost is one list node per master xcast; correctness no longer depends on the mode flag. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
Fold the reviewer's corrections into the collective-shrink plan. The `global` argument to prte_rml_repair_routing_tree denotes the source of the failure information, not the intended response, so the note that survivors use global=false because "no global failure propagation is needed" is replaced with an accurate description. The claim that survivor-side repair batching is an open follow-up that would race the reliable xcast's ACK bookkeeping is corrected: per the xcast author, batching the repair at the root already drives a batched repair on the survivors, and the ACK bookkeeping is designed to absorb a mid-broadcast repair rather than race it. Finally, the terminate-on-lifeline description is updated to match the code now departing on the first lost route rather than only the lifeline. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
Check for definition of some newer PMIx attributes before using them Signed-off-by: Ralph Castain <rhc@pmix.org>
Serves as an example of how to initiate grow/shrink ops Signed-off-by: Ralph Castain <rhc@pmix.org>
The bootstrap method for standing up a persistent DVM -- an identical prted started on every node that self-assembles from a shared prte.conf, with the daemon on the controller host self-promoting to the HNP -- has only a partial draft in the tree (the config parser exists but derives no identity and forms no DVM). Before writing that code we need an agreed, authoritative description of the observable behavior and a concrete, reviewable implementation plan, so the eventual patches can be judged against a fixed contract rather than invented ad hoc. This adds a docs/plans/bootstrap directory holding a specification and an implementation plan. The specification defines the configuration-file contract, how each daemon derives its namespace and rank with no launcher, controller self-election, and wireup. The plan maps every launcher-supplied value onto the existing MCA-parameter plumbing so bootstrap reuses the normal daemon-startup path, and sequences the work: a shared config parser, identity derivation, controller election, contact-URI synthesis, address family selection (including IPv6-only support), ports, retry, the prted process-type branch with removal of the redundant prte --bootstrap entry point, the controller-side node pool, and the user-facing artifacts. The configuration documentation is updated to match the settled key set: a single DVMPort replacing the separate controller/daemon ports, and the new DVMNetworks, DVMNetmask, DVMIPVersion, and KeepFQDNHostnames keys, with the rule that the configuration file takes precedence over the MCA parameter file. A stray "PRRTED" typo in the log-path option is corrected. No runtime code changes; documentation only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The bootstrap plan initially left the daemon-to-controller connection retry entirely to the existing OOB parameters, seeding a fixed delay and a bounded attempt count. That is fragile: the daemons boot independently and the controller may arrive arbitrarily late, so a bounded retry can give up before the controller is ever ready, while a fixed short delay busy-spins against a controller that is down. Specify instead that a bootstrap daemon retries forever, with a capped exponential backoff: frequent attempts at first, then progressively longer delays that double up to a configured maximum, after which it keeps trying at that steady rate until the controller answers. The maximum delay is exposed as a new DVMRetryMaxDelay configuration key (default 5 seconds). The plan records the two supporting code changes this needs: a new prte_retry_max_delay OOB parameter that defaults to zero so the launched path keeps its current fixed-delay behavior, and a delay computation in the reconnect path that derives the backoff from the existing retry counter and clamps it to the cap. The specification, the configuration reference, the example prte.conf, and the configurator tool are updated to describe and carry the new key. No runtime code changes; documentation only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
Turn the partial bootstrap draft into a working launcher-less startup path, following the specification and plan under docs/plans/bootstrap. In the bootstrap method an identical "prted --bootstrap" is started on every node and each daemon assembles itself into the DVM by reading a shared prte.conf, with no interactive launcher reaching out over ssh or a resource manager. The duplicated Key=Value reader and DVMNodes regex expander that had been copied between the ess and ras paths are factored into a single shared parser in src/util/prte_bootstrap.c, which also owns the canonical rank-ordering rule (the controller is always rank 0; the remaining DVMNodes entries follow in listed order) and the host matching used for controller election. Both the daemon-side ess bootstrap and the controller-side ras component now call it, so the two can never diverge in how a configuration file is interpreted. The ess bootstrap now derives the local daemon's identity from the configuration and publishes it through the same MCA-parameter environment a launcher would have set: the DVM namespace and rank, the daemon count, the shared listening port, the address family, the inter-node networks, and -- for a non-controller daemon -- a controller contact URI synthesized entirely from the configuration so the daemon can phone home before any nidmap exists. The daemon on the controller host promotes itself to the HNP; prted branches on that result to initialize as master rather than as an ordinary daemon. Because the identity must be published before the global MCA parameters are registered, the bootstrap step is moved ahead of prte_register_params. To let a synthesized controller URI parse, the OOB now treats a missing or empty interface mask as universally reachable rather than rejecting the address. Connection retries gain a capped exponential backoff, governed by a new prte_retry_max_delay parameter (default zero, preserving the existing fixed-delay behavior), so a bootstrap daemon can retry a not-yet-present controller indefinitely without busy-spinning; the DVMRetryMaxDelay key sets the cap. To keep exactly one bootstrap story, --bootstrap is removed from the prte command and kept only on prted. The example prte.conf and the configurator tool are updated to the settled key set (a single DVMPort replacing the separate controller and daemon ports, plus DVMNetworks, DVMNetmask, DVMIPVersion, KeepFQDNHostnames, and DVMRetryMaxDelay), and the previously missing bootstrap diagnostic help entries are added. The daemon path is validated end to end: a bootstrapped prted derives the correct namespace and rank, synthesizes and accepts the controller URI, and begins retrying the connection. The controller self-promotion path compiles and initializes as master but its full persistent-DVM orchestration still needs multi-node validation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The launcher-less bootstrap method now lets an administrator control how the self-assembling daemons connect to one another. Two configuration keys are added: DVMRadix (default 64), which seeds the RML routing-tree radix so each daemon phones home to its parent rather than piling every connection onto the controller, and DVMConnectMaxTime (default 30), which bounds how long a daemon waits for a given parent during the startup race before healing up to the next ancestor. The healing reuses the existing lost-connection machinery. A connection attempt to any peer that is not our lifeline is now time-bounded, and in bootstrap mode a failed connection is routed through prte_rml_route_lost exactly as a lost live connection is, so a parent that never boots triggers the same grandparent promotion and the climb walks up the tree. The controller is always retried forever. Multi-node testing surfaced two latent defects that prevented the DVM from forming. First, none of bootstrap's global MCA parameters were taking effect: prte_register_params runs inside prte_init_util, before the bootstrap code had published its environment, and an MCA variable evaluates its environment only at first registration. A daemon that could not immediately reach the controller therefore gave up on the first attempt instead of retrying. Bootstrap is now split into a parameter phase, injected into prte_init_minimum just before the parameters are registered, and an identity phase that still runs once the hostname is known. Second, a daemon that promoted itself to controller stayed a daemon: prte_init_util had already stamped its proc_type, and the second init call skipped the assignment, so the ess framework selected the daemon module. The controller's proc_type is now upgraded to master before prte_init. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
In a launcher-less bootstrapped DVM the daemons are started independently on every node and wire themselves in; there is no process-launch manager to spawn or track them. Three gaps prevented the controller from ever completing DVM formation, and once fixed a further gap stranded application jobs launched into the running DVM. The self-promoted controller treats the configured DVMNodes list as its allocation: ras/bootstrap now marks the allocation managed and stamps each node UP with its slot count, so setup_virtual_machine takes the construct path over the authoritative pool instead of filtering an empty hostfile down to just the HNP. The controller then drives the ALLOCATE state and waits for the already-running daemons to report in rather than executing the daemon report-back sequence meant for a launched prted. A bootstrapped daemon must not recursively spawn children the way the ssh launcher does: with no plm selected its remote_spawn hook is NULL, so calling it unconditionally after wireup crashed every daemon. Guard the call on the hook being present. Finally, scope the bootstrap allocation-complete handling to the daemon job alone. It fired for every job, forcing application jobs into DAEMONS_LAUNCHED with nothing left to report in and hanging the launch. Application jobs now follow the normal path, where setup_virtual_machine finds every daemon already present and advances them to report. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
A bootstrapped daemon phones home to its parent in the radix tree, but until now it could only synthesize the controller's contact URI. That suffices only for a flat tree (default radix 64), where every daemon's parent is the controller. With a smaller DVMRadix the tree is several levels deep and an interior daemon's parent is another daemon whose address no nidmap has yet delivered, so its phone-home failed with PMIX_ERR_NOT_FOUND and the DVM never formed. Generalize the controller-URI synthesis to build the contact URI of any rank from the configuration alone: every daemon listens on the shared DVMPort, and the rank-to-host mapping is derivable from DVMNodes, so a peer's URI follows from its rank and host. prted synthesizes its parent's URI once prte_init has established the parent's rank through the routing tree, reusing the existing prte_parent_uri path the ssh launcher feeds; a flat tree still short-circuits because the parent is the controller. Resolving a parent's host name can return several addresses on a multi-homed node. Rather than dial an arbitrary one, filter the resolved addresses by the DVMNetworks CIDR and use the address on the DVM interconnect, taking the CIDR prefix as the URI's reachability mask when DVMNetmask is not given. When a host is multi-homed and no CIDR narrows the choice to exactly one address, fail to start with a diagnostic that names the host rather than baking in a wrong interface. This hardens the controller synthesis by the same path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
When a bootstrapped daemon loses its parent - during formation because the parent has not started yet, or in operation because it died - the routing tree heals by promoting the daemon to its next ancestor. The daemon must then phone home to that new parent, but its contact info was never distributed: only the original parent's URI was synthesized at boot, and the grandparent we now adopt is unknown, so the send failed with PMIX_ERR_NOT_FOUND and the heal stalled. Give the OOB the same escape hatch prted uses at boot. When it must route through a peer whose URI it does not know and we are in a bootstrapped DVM, synthesize that peer's contact URI from the configuration and connect to it, rather than declaring the message undeliverable. This is the general case of the boot-time parent synthesis, so rename the helper from _parent_uri to _peer_uri and let the OOB share it; the retained bootstrap configuration makes the lookup a pure computation with no file access on the fault path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The configurator and the bootstrap design docs still described DVMNetworks and DVMNetmask as they were first planned - a transport pin and an optional mask - and the plan's synthesis step and healing step predated two refinements that shipped: CIDR-based address disambiguation and on-demand synthesis of an adopted ancestor's URI during a heal. Bring them into agreement with the implementation. The configurator now explains that DVMNetworks CIDRs also disambiguate a multi-homed host's resolved address (and that an unresolvable ambiguity is a startup error), and that DVMNetmask defaults to the matched CIDR prefix. The plan's Step 4 records the disambiguation and the mask-from-prefix default and notes the synthesis is general over any peer rank; Step 5 cross-references that second use of DVMNetworks; Step 7b documents that the OOB synthesizes the adopted parent's URI on demand so the climb can re-home. The spec's heal description makes the same point in behavioral terms. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The src/rml editing map had drifted behind the code: the elastic-DVM and launcher-less bootstrap work added a permanent departed-daemon set, a leaving-daemon fast exit, on-demand peer-URI synthesis, and new connection-retry knobs, none of which were captured. The oob and relm subtrees, meanwhile, had no orientation map of their own, and the how-things-work narrative stopped at a single overview page that only gestured at the transport and said almost nothing about reliable messaging. Bring the documentation back in line. Update src/rml/AGENTS.md for the bootstrap and elastic revisions, add an AGENTS.md (with the CLAUDE.md symlink) to each of oob/, relm/, and relm/base/ so an agent editing those directories has a local map, and expand docs/how-things-work/rml/ with dedicated pages that walk through the TCP transport and the RELM protocol in detail. This is documentation only; no code changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
Five section titles had underlines one character shorter than the title text, which docutils rejects. Extend each to at least the title length. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The show_help messages are compiled into the binary from a generated source file that the build produces from the scattered help-*.txt files. Because the Make rule for that generated file depends only on the converter script and not on the help files themselves, an ordinary make silently leaves the old content in place after a help file is added, removed, or changed. Contributors then chase phantom failures where the .txt source looks correct but the running tool serves stale or missing messages. Document this as a golden rule in the agent orientation guide so both AI agents and human contributors know to remove the generated prte_show_help_content.* files and rebuild whenever they touch show_help content. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The RML bootstrap path heals around a lost daemon but has no way to re-insert one that returns: a departure is recorded permanently in dead_dmns, so a rebooted node that comes back with its original rank is ignored forever and its former children stay attached to the grandparent. Record a design for the reverse operation. The core observation is that the routing tree is a deterministic function of the live set, so revival is the inverse of repair: clear the rank and everyone recomputes. The plan splits a new clearable absent_dmns set out of the permanent dead_dmns, routes the return through an HNP-arbitrated global xcast that mirrors the death broadcast, adds a revival recompute with a demotion delta, and re-homes the orphaned children back down to the returned rank. It also resolves the incarnation-identity question in favor of a boot-timestamp epoch carried in the OOB wire header, which is safe because a DVM's daemons all run one build and the header is not an ABI. This is a design checkpoint only; no code changes accompany it. Whether the mechanism should extend beyond bootstrap to launched-DVM node reboots remains an open question in the document. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The RML records every departed daemon in dead_dmns, a set that is never cleared, so once a bootstrap node is lost its rank is treated as a permanent hole for the life of the DVM. That is correct for a launched or elastic DVM, where a vpid is retired on purpose and no launcher can re-place a daemon at a specific existing vpid, but it forecloses the bootstrap case a rebooted node can present: the node comes back and its daemon returns with the same rank. Introduce absent_dmns as a second persistent departure set to carry exactly those recoverable holes. Like dead_dmns it is initialized once and restored into the freshly-wiped failed_dmns on every routing-tree recompute, so while the daemon is gone the tree routes around its hole just as before; unlike dead_dmns it is meant to be cleared when the daemon returns. A fault in a bootstrapped DVM now records the rank as absent; a fault anywhere else still records it as dead, so launched and elastic behavior is byte-for-byte unchanged. Nothing clears absent_dmns yet, so this commit is behavior-preserving on its own; it only gives the forthcoming unheal path a set it is allowed to reverse. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The heal path removes a daemon from the routing tree incrementally: update_ancestors walks a dead ancestor forward to its next living inheritor, which can only ever shorten a daemon's ancestor list. That is fine for a departure but cannot express the reverse, where a returned daemon re-inserts itself above us and our ancestor list must grow. Rather than teach the incremental walk to grow a list, rebuild from the fault-free radix positions and then route around whatever remains failed. That is already what compute_routing_tree does for the holes it restores on a grow, so factor that base-rebuild into a shared helper and have revival reuse it: starting from the full-depth base list and dropping the still-failed ranks yields the correct tree whether a daemon left or returned, and the growing-ancestor-list problem simply does not arise. prte_rml_revive_routing_tree clears a rank's failure marks -- but only if it was recorded absent, so a permanently dead or never-failed rank is an idempotent no-op -- rebuilds the tree, and packages the prev-versus- current delta into a recovery status. A new demoted flag mirrors promoted for the direction revival moves us. Component notification is deliberately left as a marked TODO: the existing fault handlers would read the status as a fault and emit death/adoption notices, so the parallel revival notices come in the following stages. Nothing calls this routine yet, so the DVM's observable behavior is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The revival recompute exists but nothing invoked it. Add the global protocol that does, mirroring the death broadcast the heal path already uses. A bootstrap daemon announces itself to the HNP on startup with a new DAEMON_RETURNED message. The HNP treats it as authoritative: if the rank is still marked absent this is a genuine return, so it broadcasts DAEMON_REVIVED to the DVM; if the rank is already live -- a first boot, a duplicate, or an already-processed return -- it ignores the message, so the announcement is always safe to send and the whole exchange is idempotent. Every daemon converges on the broadcast by re-inserting the returned rank through prte_rml_revive_routing_tree, which is itself a no-op wherever the rank was not marked absent. The broadcast rides the existing reliable xcast, but on the opposite ordering from a death. A death is processed before it is forwarded, because it grows a survivor's child set and the repaired tree is needed to reach the promoted grandchildren. A revival shrinks the reshaping node's child set -- the returned rank reclaims its orphans -- so it must be forwarded before it is processed, or the very children about to re-home would be dropped from the forward set first. That is the default xcast path, so DAEMON_REVIVED stays off the process-first list; the reasoning is recorded there and at the HNP, which for the same reason does not revive its own tree inline but converges through its own relayed broadcast. Component notification from the revival recompute is still the deferred TODO, so this establishes tree convergence for a returned daemon that already holds current state; the RELM re-drive and re-home handshake follow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The return protocol had every bootstrap daemon announce itself directly to the HNP on startup. Even though the RML relays the announcement up the tree rather than opening a direct socket, it still funnels one message per daemon onto the root during formation, and the root's transport sustains that fan-in. An N-to-root pattern like this burdens the OS on the root node and degrades its responsiveness at the scales PRRTE runs at, which is exactly the shape we work to avoid. Announce one hop up to the parent instead. The parent is precisely the daemon that can tell a genuine return from a first boot: the global death broadcast marked the departed rank absent everywhere, so the parent either has the rank in its own absent set or it does not. On a first boot every parent simply drops the notice, so the root is never involved beyond its own handful of direct children; only a real return escalates a single relayed message to the HNP, which stays the sole arbiter that broadcasts the revival. The announcement rides the existing lifeline link, so no daemon opens a socket to the root, and the common path costs the root nothing. This resolves the trigger-source question the plan left open, in favor of parent-filtered escalation over root-direct announcement. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The revival recompute reshaped the routing tree but told no one, so a returned daemon rejoined the topology while grpcomm, filem, and relm kept stale in-flight state. Notify them, exactly as a fault does -- with one deliberate exclusion and two simplifications the xcast-driven design allows. The death handler prte_rml_fault_handler is not called: it purges the "failed" ranks and emits adoption and failure notices, all of which are wrong for a return. The remaining component handlers, however, key on the structural delta, and a revival is pure shrinkage from every reshaping daemon's view -- the returned rank's former parent swaps orphans for the rank in its child list, the orphans point their lifeline back at the rank, and daemons deeper down merely gain an ancestor. That trips only the parent-changed and children-changed paths, never the promotion-only ones (replay-pending, op-id-at-promotion) that exist for the growth direction, so the tested handlers are reused without a revival-specific branch. A separate re-home notice is likewise unnecessary: a revival is driven by the single broadcast rather than raced against it, so every daemon recomputes from the same signal and re-homes locally with no adoption-style ancestry reconciliation to perform. Partial returns need no special handling either. Once the returned rank's bit is cleared, the failed set is exactly what a fresh compute would hold, and revival derives the tree through the same build_tree_from_base helper, so it reproduces the tree a compute would build for that failed set -- a rank below a still-absent ancestor, or a subtree where only some absent daemons return, all fall out correctly. One watch item is recorded for harness validation: RELM link updates are depth stamped, revival changes depths, and the broadcast rides the xcast forward-first. Static analysis argues it is safe, because each daemon recomputes synchronously right after forwarding and settles its depth before any link update arrives, but the multi-hop update gating warrants confirmation on the multi-node harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
Exercised the implemented stages on the Docker swarm with a radix-2 bootstrap DVM. Killing an interior daemon healed its child up to the grandparent, and restarting it drove the return announcement, the HNP revival broadcast, and the child re-homing back to the returned rank -- the unheal topology works end-to-end. The test also pinned down the shape of the remaining state-resync work. Once the returned daemon was asked to join a reliable xcast it exited on an out-of-order op-id: it had rejoined with a fresh broadcast counter while the DVM's stream was already ahead. Note that in Stage 5 as the concrete meaning of the "already holds current state" precondition the earlier stages assume -- the returned daemon must be caught up to the current xcast op-id as well as the nidmap and job data. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
Reliable xcast enforces strict op-id ordering: finish_op raises PRTE_ERR_OUT_OF_ORDER_MSG unless an op's id is exactly one past the last completed one. A daemon starts at op-id zero, which is correct only for a daemon present from the first broadcast. A daemon that joins a running DVM -- grown in, or, on the bootstrap unheal path, returned after its node rebooted -- starts at zero while the DVM's broadcast stream is already well ahead, so the first op forwarded to it is far out of order and it force-exits. The harness reproduced exactly this: after an interior daemon rebooted and rejoined, the next job launch killed it on an out-of-order op-id. Recognize the late joiner and adopt the intervening ops as complete. A daemon that has never seen any xcast has op_id_inited still at zero; if it is then handed an op above one, it cannot possibly have the ops before it, so it sets its completed mark just below that op and treats the earlier ones as already done -- the same assume-complete the promotion path uses when a subtree grows. Ordering is then satisfied for this op and every one after. The master assigns op-ids and is never a late joiner, so it is excluded, and any daemon that has ever participated has a non-zero op_id_inited, so a genuine mid-stream ordering violation is still caught. This also removes a latent hazard for elastic grow, where a daemon added after enough broadcasts would have hit the same crash. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The op-id half of state resync is done and verified on the harness: the returned daemon no longer force-exits on an out-of-order op-id, and the elastic suite still passes, so the normal broadcast path is unaffected. With ordering fixed the returned daemon now reaches the launch itself and fails one layer deeper -- lacking the current nidmap and job data it recomputes an inconsistent tree mid-launch and sends to a node it wrongly believes is down. Note that as the concrete signature the remaining nidmap/job-data handoff must resolve. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
On a daemon comm failure the DVM controller decrements num_daemons and records the daemon COMM_FAILED. When a bootstrap node reboots, its daemon reports in again through the normal report-launch path, which resets the proc to RUNNING and ALIVE -- but nothing ever undoes the count the death removed. The controller is then left believing the DVM is one daemon smaller than it is. That stale count corrupts the launch nidmap, whose daemon vpid span is [0, num_daemons). With the span short by one, the returned daemon decodes its own rank -- or a peer past the truncated span -- as a permanent hole and marks it dead, then routes into that hole and force-exits on the first job. The harness reproduced exactly this: after the routing unheal, the next launch killed the returned daemon on a "node has gone down" send. Restore the count at report-in when the reporting daemon was COMM_FAILED, which only a returned daemon is -- a first launch or a grow target is not in that state here, so formation and grow are untouched, and the RUNNING transition that immediately follows keeps a duplicate report from double-counting. Gated on bootstrap, the only mode in which a daemon can return with its original vpid. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
When a bootstrap daemon's node powers down and later reboots, the DVM heals around the loss and then unheals when the daemon returns. During the return, before the daemon has formally reported its relaunch, the HNP still carries the departed daemon count in prte_process_info.num_daemons even though the departed node's node->daemon entry persists and is packed into the nidmap. Encoding num_daemons as the vpid span therefore declared a span one short of the highest vpid actually packed. The returning daemon decoded that short span, reset its own num_daemons to it, recomputed the routing tree over a rank space that excluded the top daemon, and then routed traffic for that live daemon to its parent -- eventually force-exiting with "node has gone down." The short span also under-sized the encode-side vpid buffer, writing one entry past its allocation whenever the pool held more daemons than the stale count. Derive the span from the pool instead: pre-scan for the highest daemon vpid and pack max(num_daemons, highest_vpid + 1). This covers every packed daemon while still preserving a top-of-range shrink hole (where num_daemons legitimately exceeds the live count), and it sizes the vpid buffer to match. Also record decode-side vpid holes as absent (clearable by the unheal path) rather than permanently dead when running a bootstrapped DVM, since a bootstrap hole may be a node that will reboot and return, unlike a launched/elastic DVM where a shrunk-out vpid is gone for good. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
The nidmap/job-data resync stage is complete: the returned-daemon force exit traced to a vpid-span bug in the handoff nidmap, not to missing job data. Update the plan's Stage 5 narrative to describe the root cause, the max(num_daemons, highest_packed_vpid + 1) span fix, the clearable-absent hole handling, and the harness verification, so the document reflects the shipped behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
A bootstrap daemon whose node powers off and later reboots returns as a new process wearing the same rank. Messages still in flight from the old incarnation, or late death/adoption notices, could be mis-delivered to the new one and corrupt its view of the tree. Nothing distinguished the two incarnations on the wire. Tag every process with a boot epoch -- a millisecond wall-clock timestamp captured once at RML startup -- and carry it in the OOB wire header as the origin's epoch. A message built here defaults to this process's epoch; a relayed message preserves the original sender's epoch from the received header. Each daemon keeps the highest epoch it has learned per rank and, in a bootstrapped DVM, drops daemon-namespace traffic stamped with a strictly older epoch for a rank -- the check runs after the full message has been read so the byte stream stays framed, and only for the daemon namespace since tool namespaces reuse rank numbers. A newer epoch passes but does not advance the table; the arbitrated revival does that authoritatively. The returning daemon announces its epoch in the DAEMON_RETURNED notice; the HNP accepts the return only if the epoch is strictly greater than the one last recorded for that rank -- rejecting a stale or degenerate same-timestamp reboot and forcing a retry -- then propagates the epoch in the DAEMON_REVIVED broadcast so every daemon records the new incarnation and drops any lingering traffic from the old one. Since the wire header is exchanged only among daemons of one DVM, all running the same build, adding the field carries no ABI concern. The drop path is bootstrap-gated, so launched and elastic DVMs are unaffected; the elastic suite (16/16) and the bootstrap unheal end-to-end both pass with the guard in place. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
Record that Stage 6 is implemented: the boot-epoch wire stamp, the per-rank epoch table and bootstrap-gated stale-traffic drop, and the epoch carried through the return/revival exchange with the HNP's strictly-greater acceptance test. Note the harness verification so the document reflects the shipped behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
A spawn request that fails while its job is being set up - most easily triggered by a map-by qualifier that passes prun's command-line sanity check but is rejected when the HNP parses the mapping policy, such as a non-numeric PE value - was able to take down an entire persistent DVM. A user's typo in a launch command should never kill a DVM that other jobs are relying on. The failure path in interim() reports the error to the requestor through the spawn-completion callback, which is the complete and correct notification. It then also activated PRTE_JOB_STATE_NEVER_LAUNCHED on the job. That job, however, is a bare object created at the top of interim(): it has no nspace, is absent from the job pool, and owns no procs or daemons. Driving such a phantom job into the job-termination state machine runs it through the errmgr, server, IOF, and session cleanup paths, all of which assume a real, registered job. On the phantom job that cleanup corrupts a live file descriptor - the HNP dies with an "Epoll MOD on fd NN failed: Bad file descriptor" - and every subsequent prun reports that no DVM is available. Split the failure path on prte_persistent. A persistent DVM simply discards the phantom job and keeps running; the requestor has already learned of the error through the callback, and there is nothing else to clean up. A one-shot launch (prterun is itself the requestor) retains the previous behavior: record the failure in our exit status and activate NEVER_LAUNCHED to tear the DVM down, preserving the deterministic non-zero exit established earlier. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Ralph Castain <rhc@pmix.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.