Node runtime split and role-specific rollout

[jmagar]

Summary

Implement docs/superpowers/plans/2026-04-24-node-runtime-split.md. Split lab serve into explicit controller and node runtime paths, stop deployed nodes from initializing controller-only surfaces, build role-specific controller/node artifacts, make nodes update verify real readiness and WebSocket reconnects, then prove the path with full verification and live rollout.

Beads

Epic: lab-686q

Plan

• docs/superpowers/plans/2026-04-24-node-runtime-split.md
• docs/NODE_RUNTIME_CONTRACT.md

Scope

• Runtime role resolution supports controller/node overrides and requires a controller host for node mode.
• Non-controller serve skips registry, gateway, MCP, Web UI, OAuth, marketplace, fs, full API, and controller logs startup.
• Node health is loopback-only and implemented with raw tokio TCP.
• nodes update builds/reuses distinct controller and node artifacts.
• Rollout ships the right artifact per target and reports artifact role/profile in JSON.
• Controller verification checks health, readiness, and fleet WebSocket listener.
• Node verification checks controller-observed connected state, not just inventory presence.
• Controller self-update failure reports recovery metadata and backup path.
• node-runtime feature can be checked without controller dependencies.

Acceptance Criteria

• just check passes.
• just build passes.
• just test passes.
• lint/no-run verification passes.
• live config audit passes.
• live nodes update --all passes with final JSON ok: true.

Execution Order

1. Runtime split and node health.
2. Role-specific nodes update artifacts and verification.
3. Live rollout until green.
4. Cargo feature slimming.
5. Generic deploy runner follow-up.

[traycerai]

Warning

Ticket Assist will be discontinued on May 1, 2026. After that date, the feature will no longer be available. We suggest installing the Traycer IDE Extension for continued AI-powered, spec-driven development. Thank you for choosing Traycer!

Plan

Observations

The binary is a single Rust workspace (crates/lab) with one main.rs entry point. lab serve is implemented in crates/lab/src/cli/serve.rs (45 KB) and starts every subsystem unconditionally. The device/fleet layer lives in crates/lab/src/api/device/fleet.rs (32 KB) and crates/lab/src/device/ws_client.rs. nodes update / deploy logic is in crates/lab/src/cli/deploy.rs. No RuntimeRole enum or node-runtime Cargo feature exists yet. tokio with net feature is already in workspace.dependencies.

Approach

Introduce a RuntimeRole enum in config.rs, thread it through serve.rs to gate controller-only subsystems, add a node-runtime Cargo feature, build role-specific artifacts in deploy.rs, and upgrade nodes update verification to check controller-observed WebSocket connected state. This follows the four execution-order phases from the issue exactly.

---

Phase 1 — Runtime Split & Node Health

1.1 — Add RuntimeRole to config.rs

In crates/lab/src/config.rs, add a RuntimeRole enum with variants Controller and Node. Add a role field to the top-level config struct (or a dedicated [serve] sub-table). Resolution priority: CLI flag --role → env var LAB_ROLE → config file → default Controller. For Node mode, require controller_host: Url to be set; fail fast at startup if absent.

1.2 — Expose --role flag on ServeArgs in cli.rs

In crates/lab/src/cli.rs, add --role <controller|node> and --controller-host <URL> to the Serve subcommand's argument struct. Pass these into the config resolution path.

1.3 — Gate controller-only subsystems in serve.rs

In crates/lab/src/cli/serve.rs, after resolving RuntimeRole, wrap each of the following startup blocks in if role == RuntimeRole::Controller { … }:

• MCP registry / gateway startup
• Web UI / OAuth server
• Marketplace
• Filesystem service
• Full REST API router (replace with a node-only minimal router)
• Controller log streaming

1.4 — Node-only loopback health endpoint

For Node mode, start a minimal raw tokio::net::TcpListener on 127.0.0.1:<port> that responds to a single GET /health with 200 OK and a JSON body {"role":"node","ok":true}. Do not use axum for this path — use raw TCP + hand-written HTTP/1.1 response to avoid pulling in controller dependencies. Place this in a new file crates/lab/src/cli/serve/node_health.rs.

---

Phase 2 — Role-Specific nodes update Artifacts & Verification

2.1 — Dual artifact build in deploy.rs

In crates/lab/src/cli/deploy.rs, update the nodes update build path to:

1. Build a controller artifact: cargo build --release (default features).
2. Build a node artifact: cargo build --release --features node-runtime --no-default-features (or a dedicated feature set — see Phase 4).
3. Cache each artifact by its content hash; reuse if unchanged.
4. Tag each artifact with role and profile fields in the output JSON.

2.2 — Add node-runtime Cargo feature to crates/lab/Cargo.toml

Add a [features] section with node-runtime = [] (and optionally default = []). This feature flag is the compile-time gate for Phase 4 slimming; for now it just needs to exist so the build succeeds.

2.3 — Rollout ships the right artifact per target

In deploy.rs, when iterating over fleet targets, inspect each node's configured role (from the fleet config or per-node override) and scp/rsync the matching artifact. Report role and profile in the per-node JSON result.

2.4 — Controller self-update failure recovery metadata

When the controller self-updates and the new binary fails to start, emit a JSON error object containing backup_path (path to the pre-update binary) and recovery_cmd (the command to restore it). This goes in the error branch of the self-update flow in deploy.rs.

---

Phase 3 — Verification Upgrades

3.1 — Controller verification in health.rs / deploy.rs

After deploying to a controller target, verify:

1. HTTP GET /health returns 200.
2. HTTP GET /readiness (or equivalent) returns 200.
3. The fleet WebSocket listener is accepting connections: attempt a raw tokio::net::TcpStream::connect to the controller's WS port and confirm the TCP handshake succeeds within a timeout.

3.2 — Node verification checks controller-observed connected state

After deploying to a node target, do not just check inventory presence. Instead, query the controller's fleet API (e.g., GET /api/device/fleet) and confirm the node appears with connected: true (or equivalent field in fleet.rs's response type). Poll with a configurable timeout and retry interval. Only report ok: true in the final JSON once this condition is met.

3.3 — WebSocket reconnect confirmation

After a node restarts with the new binary, the node's ws_client.rs will reconnect to the controller. The verification step in 3.2 implicitly confirms this because it checks the controller-observed connected state. No separate reconnect probe is needed.

---

Phase 4 — Cargo Feature Slimming

4.1 — Gate controller-only dependencies behind cfg(not(feature = "node-runtime"))

In crates/lab/src/lib.rs and relevant module files, wrap imports of axum, utoipa, utoipa-scalar, rmcp, rusqlite, jsonwebtoken, rsa, and rand with #[cfg(not(feature = "node-runtime"))] where they are only needed by controller surfaces. The node-runtime feature build must compile cleanly with only tokio, tracing, serde, serde_json, reqwest, and anyhow.

4.2 — Verify node-runtime feature compiles without controller deps

Add a just check-node recipe to Justfile:

check-node:
    cargo check -p lab --no-default-features --features node-runtime

---

Data Flow Summary

sequenceDiagram
    participant CLI as lab serve
    participant Config as config.rs
    participant Serve as serve.rs
    participant NodeH as node_health.rs
    participant Deploy as deploy.rs
    participant Fleet as fleet.rs (controller API)

    CLI->>Config: resolve RuntimeRole (flag → env → file → default)
    Config-->>Serve: RuntimeRole::Controller | Node

    alt Controller
        Serve->>Serve: start registry, gateway, MCP, Web UI, OAuth, full API
    else Node
        Serve->>NodeH: bind 127.0.0.1:<port> raw TCP health
        Serve->>Serve: start WS client → connect to controller_host
    end

    CLI->>Deploy: nodes update --all
    Deploy->>Deploy: build controller artifact (default features)
    Deploy->>Deploy: build node artifact (--features node-runtime)
    Deploy->>Deploy: scp role-matched artifact to each target
    Deploy->>Fleet: GET /api/device/fleet (poll connected state)
    Fleet-->>Deploy: node.connected == true
    Deploy-->>CLI: JSON { ok: true, role, profile }

Loading

---

File Change Summary

File	Change
crates/lab/src/config.rs	Add RuntimeRole enum + role / controller_host fields
crates/lab/src/cli.rs	Add --role and --controller-host to ServeArgs
crates/lab/src/cli/serve.rs	Gate controller-only subsystems on RuntimeRole::Controller
crates/lab/src/cli/serve/node_health.rs	New: raw tokio TCP loopback health listener
crates/lab/src/cli/deploy.rs	Dual artifact build, role-matched rollout, controller-observed node verification, self-update recovery metadata
crates/lab/Cargo.toml	Add [features] node-runtime = []
crates/lab/src/lib.rs + module files	#[cfg(not(feature = "node-runtime"))] guards on controller-only imports
Justfile	Add check-node recipe
docs/NODE_RUNTIME_CONTRACT.md	Create: document the controller/node contract (role resolution, health endpoint spec, WS reconnect behaviour)

Import In IDE

🤖 Prompt for AI Agents

## Observations

The binary is a single Rust workspace (`crates/lab`) with one `main.rs` entry point. `lab serve` is implemented in `crates/lab/src/cli/serve.rs` (45 KB) and starts every subsystem unconditionally. The device/fleet layer lives in `crates/lab/src/api/device/fleet.rs` (32 KB) and `crates/lab/src/device/ws_client.rs`. `nodes update` / deploy logic is in `crates/lab/src/cli/deploy.rs`. No `RuntimeRole` enum or `node-runtime` Cargo feature exists yet. `tokio` with `net` feature is already in `workspace.dependencies`.

## Approach

Introduce a `RuntimeRole` enum in `config.rs`, thread it through `serve.rs` to gate controller-only subsystems, add a `node-runtime` Cargo feature, build role-specific artifacts in `deploy.rs`, and upgrade `nodes update` verification to check controller-observed WebSocket connected state. This follows the four execution-order phases from the issue exactly.

---

## Phase 1 — Runtime Split & Node Health

### 1.1 — Add `RuntimeRole` to `config.rs`

In `crates/lab/src/config.rs`, add a `RuntimeRole` enum with variants `Controller` and `Node`. Add a `role` field to the top-level config struct (or a dedicated `[serve]` sub-table). Resolution priority: CLI flag `--role` → env var `LAB_ROLE` → config file → default `Controller`. For `Node` mode, require `controller_host: Url` to be set; fail fast at startup if absent.

### 1.2 — Expose `--role` flag on `ServeArgs` in `cli.rs`

In `crates/lab/src/cli.rs`, add `--role <controller|node>` and `--controller-host <URL>` to the `Serve` subcommand's argument struct. Pass these into the config resolution path.

### 1.3 — Gate controller-only subsystems in `serve.rs`

In `crates/lab/src/cli/serve.rs`, after resolving `RuntimeRole`, wrap each of the following startup blocks in `if role == RuntimeRole::Controller { … }`:
- MCP registry / gateway startup
- Web UI / OAuth server
- Marketplace
- Filesystem service
- Full REST API router (replace with a node-only minimal router)
- Controller log streaming

### 1.4 — Node-only loopback health endpoint

For `Node` mode, start a minimal raw `tokio::net::TcpListener` on `127.0.0.1:<port>` that responds to a single `GET /health` with `200 OK` and a JSON body `{"role":"node","ok":true}`. Do **not** use `axum` for this path — use raw TCP + hand-written HTTP/1.1 response to avoid pulling in controller dependencies. Place this in a new file `crates/lab/src/cli/serve/node_health.rs`.

---

## Phase 2 — Role-Specific `nodes update` Artifacts & Verification

### 2.1 — Dual artifact build in `deploy.rs`

In `crates/lab/src/cli/deploy.rs`, update the `nodes update` build path to:
1. Build a **controller** artifact: `cargo build --release` (default features).
2. Build a **node** artifact: `cargo build --release --features node-runtime --no-default-features` (or a dedicated feature set — see Phase 4).
3. Cache each artifact by its content hash; reuse if unchanged.
4. Tag each artifact with `role` and `profile` fields in the output JSON.

### 2.2 — Add `node-runtime` Cargo feature to `crates/lab/Cargo.toml`

Add a `[features]` section with `node-runtime = []` (and optionally `default = []`). This feature flag is the compile-time gate for Phase 4 slimming; for now it just needs to exist so the build succeeds.

### 2.3 — Rollout ships the right artifact per target

In `deploy.rs`, when iterating over fleet targets, inspect each node's configured `role` (from the fleet config or per-node override) and `scp`/`rsync` the matching artifact. Report `role` and `profile` in the per-node JSON result.

### 2.4 — Controller self-update failure recovery metadata

When the controller self-updates and the new binary fails to start, emit a JSON error object containing `backup_path` (path to the pre-update binary) and `recovery_cmd` (the command to restore it). This goes in the error branch of the self-update flow in `deploy.rs`.

---

## Phase 3 — Verification Upgrades

### 3.1 — Controller verification in `health.rs` / `deploy.rs`

After deploying to a controller target, verify:
1. HTTP `GET /health` returns `200`.
2. HTTP `GET /readiness` (or equivalent) returns `200`.
3. The fleet WebSocket listener is accepting connections: attempt a raw `tokio::net::TcpStream::connect` to the controller's WS port and confirm the TCP handshake succeeds within a timeout.

### 3.2 — Node verification checks controller-observed connected state

After deploying to a node target, do **not** just check inventory presence. Instead, query the controller's fleet API (e.g., `GET /api/device/fleet`) and confirm the node appears with `connected: true` (or equivalent field in `fleet.rs`'s response type). Poll with a configurable timeout and retry interval. Only report `ok: true` in the final JSON once this condition is met.

### 3.3 — WebSocket reconnect confirmation

After a node restarts with the new binary, the node's `ws_client.rs` will reconnect to the controller. The verification step in 3.2 implicitly confirms this because it checks the controller-observed connected state. No separate reconnect probe is needed.

---

## Phase 4 — Cargo Feature Slimming

### 4.1 — Gate controller-only dependencies behind `cfg(not(feature = "node-runtime"))`

In `crates/lab/src/lib.rs` and relevant module files, wrap imports of `axum`, `utoipa`, `utoipa-scalar`, `rmcp`, `rusqlite`, `jsonwebtoken`, `rsa`, and `rand` with `#[cfg(not(feature = "node-runtime"))]` where they are only needed by controller surfaces. The `node-runtime` feature build must compile cleanly with only `tokio`, `tracing`, `serde`, `serde_json`, `reqwest`, and `anyhow`.

### 4.2 — Verify `node-runtime` feature compiles without controller deps

Add a `just check-node` recipe to `Justfile`:
```
check-node:
    cargo check -p lab --no-default-features --features node-runtime
```

---

## Data Flow Summary

```mermaid
sequenceDiagram
    participant CLI as lab serve
    participant Config as config.rs
    participant Serve as serve.rs
    participant NodeH as node_health.rs
    participant Deploy as deploy.rs
    participant Fleet as fleet.rs (controller API)

    CLI->>Config: resolve RuntimeRole (flag → env → file → default)
    Config-->>Serve: RuntimeRole::Controller | Node

    alt Controller
        Serve->>Serve: start registry, gateway, MCP, Web UI, OAuth, full API
    else Node
        Serve->>NodeH: bind 127.0.0.1:<port> raw TCP health
        Serve->>Serve: start WS client → connect to controller_host
    end

    CLI->>Deploy: nodes update --all
    Deploy->>Deploy: build controller artifact (default features)
    Deploy->>Deploy: build node artifact (--features node-runtime)
    Deploy->>Deploy: scp role-matched artifact to each target
    Deploy->>Fleet: GET /api/device/fleet (poll connected state)
    Fleet-->>Deploy: node.connected == true
    Deploy-->>CLI: JSON { ok: true, role, profile }
```

---

## File Change Summary

| File | Change |
|---|---|
| `crates/lab/src/config.rs` | Add `RuntimeRole` enum + `role` / `controller_host` fields |
| `crates/lab/src/cli.rs` | Add `--role` and `--controller-host` to `ServeArgs` |
| `crates/lab/src/cli/serve.rs` | Gate controller-only subsystems on `RuntimeRole::Controller` |
| `crates/lab/src/cli/serve/node_health.rs` | New: raw tokio TCP loopback health listener |
| `crates/lab/src/cli/deploy.rs` | Dual artifact build, role-matched rollout, controller-observed node verification, self-update recovery metadata |
| `crates/lab/Cargo.toml` | Add `[features] node-runtime = []` |
| `crates/lab/src/lib.rs` + module files | `#[cfg(not(feature = "node-runtime"))]` guards on controller-only imports |
| `Justfile` | Add `check-node` recipe |
| `docs/NODE_RUNTIME_CONTRACT.md` | Create: document the controller/node contract (role resolution, health endpoint spec, WS reconnect behaviour) |

---

Execution Information

Branch: main
Commit: df6b50f

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.