feat(dev): bring run_inner to parity with dev-runner.sh and restore the log shipper in the live runner

[I-am-nothing]

1. Problem

The developer Logcat at /dev/module/<slug>/service is empty. The cause is two diverged dev runners, each missing what the other has:

• The live runner (shell). The compose runner service runs command: sh /modules/scripts/dev-runner.sh (ms-app-modules/docker-compose.yml). This shell script gives the full dev experience — air polling hot-reload, a Go dev-proxy multiplexer, esbuild + SSE livereload, and per-module platform-token files — but it never ships logs to dispatch. grep of both dev-runner.sh and dev-proxy.go returns zero matches for MS_INTERNAL_SECRET, /internal/modules, curl, or wget. Its only log handling is local terminal prefixing via sed ([<mod>] / [<mod>:web]).

• The runner with the shipper (Rust). mirrorstack dev --all → run_inner (mirrorstack-cli/src/commands/dev/mod.rs) does contain the feat(dev): ship runner module logs to dispatch (log pipeline) #36 log shipper (log_shipper.rs): it taps each child's stdout/stderr and POSTs batched lines to dispatch. But it spawns each module with a plain one-shot go run ./<dir> (mod.rs ~283) and has no hot-reload, no dev-proxy, no livereload, and writes no per-module token files. The advertised --watch flag is dead surface: run_inner(root, _args) discards its args, so args.watch is never read.

Net effect: the path that is actually executed (shell) lacks the shipper, and the path that has the shipper (Rust) lacks the dev features and so is never used. The compose comment block even describes a log shipper POSTing to /internal/modules/<id>/logs — but that describes the Rust runner's behavior, not the shell script that compose actually runs. The ingest endpoint POST /internal/modules/<id>/logs is live and auth-gated, the SDK now auto-emits access logs (app-module-sdk #128), and the login redirect is fixed (api-platform #276). The only missing link is getting the shipper into the live runner.

2. Current architecture & divergence

Compose runner container: golang:1.26-alpine + nodejs npm sed grep + go install air-verse/air@latest; repo bind-mounted at /modules, SDK mounted read-only; ports 9080:8080 (dev-proxy) and 9089:8089 (livereload base). command: sh /modules/scripts/dev-runner.sh.

Capability	dev-runner.sh (live, no shipper)	Rust run_inner (shipper, no dev features)
Module discovery	sed-parses use ( ... ) block of go.work; skips dir if no main.go or no ID: literal	workspace::discover_modules + module_meta::read_module_meta, keeps only non-empty meta.id (collapses both skip reasons to "no ID")
Per-module run	air -c /tmp/.air.<mod>.toml with poll=true poll_interval=2000, build go build -buildvcs=false -o /tmp/<mod> .	plain go run ./<dir> — one-shot compile+run, no rebuild watcher
--watch	n/a (always hot-reloads)	flag declared (mod.rs ~64) but unread (_args) — no-op
Internal ports	sequential from 18080 in go.work order	whatever the module binds (no assignment)
Reverse proxy	dev-proxy.go on :8080, routes /_m/<slug>/* → 127.0.0.1:<internal_port> (strips /_m/<slug> prefix, empty→/)	none — /_m/<slug> only exists conceptually in tunnel/dispatch
Web watcher / livereload	per-module node esbuild.config.mjs --watch with LR_PORT from 8089; SSE data: reload server lives in the module's esbuild config	none
Platform-token files	MS_PLATFORM_TOKEN_FILE=/modules/.ms-platform-token-<slug> per module (distinct token per tunnel; shared file would 401 all but the first)	does not set or forward MS_PLATFORM_TOKEN_FILE (not in env allowlist)
stdout/stderr	sed "s/^/[<mod>] /" (and [<mod>:web])	spawn_forwarder mirrors to terminal + ships to sink; colored <label> │ on TTY, [label] otherwise
Log shipping	absent	log_shipper::spawn → POST <MS_DISPATCH_URL>/internal/modules/<id>/logs, header X-MS-Service-Secret: <MS_INTERNAL_SECRET>, batched (1s flush / 200 max), best-effort. Disabled if MS_DISPATCH_URL OR MS_INTERNAL_SECRET is empty.
Env to children	container env inherited; PORT, MS_PLATFORM_TOKEN_FILE set per module	fixed allowlist: MS_LOCAL_DB_URL + MS_INTERNAL_SECRET, REDIS_URL, AWS_ENDPOINT_URL/REGION/ACCESS_KEY_ID/SECRET_ACCESS_KEY
Lifecycle	trap 'kill $pids; wait' INT TERM	busy-poll try_wait (200ms) + ctrl-c channel, kill all on signal

docker-compose.yml and scripts/ are untracked workspace-local files (git ls-files returns nothing for them); the compose comments mark them e2e-only. The SDK bind mount uses a hardcoded absolute host path matching the go.mod replace directives.

3. Goal

Bring Rust run_inner to feature parity with dev-runner.sh (polling hot-reload, dev-proxy multiplexer, livereload, per-module platform-token files, deterministic port assignment, prefixed stdout), then switch the compose command back to mirrorstack dev --all and retire dev-runner.sh + dev-proxy.go. This puts the existing Rust log shipper back on the live execution path so the Logcat populates end-to-end.

4. Scope — feature-parity checklist for run_inner

Each item maps to how dev-runner.sh / dev-proxy.go does it today. run_inner must gain all of these. Honor the --watch flag: stop taking _args; read args.watch (default the in-container path to watch=on).

• Module discovery (already present, keep). workspace::discover_modules + non-empty meta.id gate already mirrors the sed-parse of go.work + main.go ID: grep. Keep; this is the more robust path. Produce the module list in go.work order (needed for deterministic ports).

• Polling hot-reload per module (replaces plain go run). Mirror the air config exactly:

  • build cmd: go build -buildvcs=false -o /tmp/<mod> . run with current_dir = /modules/<mod>, then exec /tmp/<mod>.
  • watch only .go and .sql; exclude tmp, web, node_modules.
  • POLL the filesystem at 2000ms — do NOT use native fsnotify/inotify. OrbStack/Docker bind mounts do not deliver inotify events; native watching will silently miss every edit. On change: rebuild, and on success restart that module's child (kill old → spawn new binary). Two acceptable implementations: (a) shell out to the already-installed air per module using a generated /tmp/.air.<mod>.toml, or (b) implement a Rust poll loop (mtime scan every 2s over the included extensions) that rebuilds + restarts. Either way the shipper must re-tap the new child's stdout/stderr on restart.

• Deterministic internal port assignment. Assign PORT sequentially from 18080 in go.work order (oauth-core=18080, oauth-google=18081, …). Pass PORT=<n> into each child's env. These feed the proxy route table.

• In-process dev-proxy (replaces dev-proxy.go). Run an HTTP reverse proxy bound to :8080 inside run_inner. For each module register a route on prefix /_m/<slug>/ that strips /_m/<slug> from the request path (empty result → /) and forwards to http://127.0.0.1:<internal_port>. Reproduce the exact TrimPrefix("/_m/<slug>") + empty→/ semantics so e.g. GET /_m/oauth-core/foo → 127.0.0.1:18080/foo. Tolerate the startup race (module still compiling → 502 until first build completes), matching shell behavior; no readiness gate required for v1.

• Per-module esbuild web watcher + livereload. For any module with web/esbuild.config.mjs: cd <mod>/web, npm install --silent, then spawn node esbuild.config.mjs --watch with LR_PORT assigned sequentially from 8089 (incremented only per web-enabled module, so the LR index diverges from the internal-port index). The SSE data: reload server lives inside the module's esbuild config — the runner only needs to set LR_PORT and run it. Prefix this stream [<slug>:web].

• Per-module platform-token file. Set MS_PLATFORM_TOKEN_FILE=/modules/.ms-platform-token-<slug> on each module child (slug = dir basename). These files are written by the host run_outer after each tunnel's register_ack and reach the container via the .:/modules bind mount. Never a single shared file — dispatch mints a distinct service token per tunnel connection; a shared file authenticates only the first module and 401s the rest. Add MS_PLATFORM_TOKEN_FILE handling explicitly (it is currently absent from run_inner's env allowlist).

• Prefixed stdout/stderr (keep + align). spawn_forwarder already mirrors + ships per line. Ensure the terminal label reads [<slug>] (module) and [<slug>:web] (web watcher) so output matches the current shell prefixes and stays grep-friendly on non-TTY.

• Log shipper on the live path (the payoff). No shipper code change needed — log_shipper::spawn/run/flush are correct. Just ensure run_inner (now the live runner) reaches the existing tap: MS_DISPATCH_URL and MS_INTERNAL_SECRET both present → POST to <MS_DISPATCH_URL>/internal/modules/<id>/logs with X-MS-Service-Secret. When the sink is None (either var empty), print a one-line warning so a developer knows the Logcat is disabled by config rather than broken.

• Lifecycle. Keep the ctrl-c/try_wait supervisor, extended to also own the dev-proxy thread, the esbuild watchers, and the rebuild/restart loop; on shutdown tear down the whole tree. On Ctrl-C, drop the log sinks before exiting so each shipper thread's Disconnected branch flushes its final batch.

5. Out of scope

• Already merged, do not re-touch: login redirect fix (api-platform #276); SDK auto access-log emission (app-module-sdk #128). The ingest endpoint already exists and is auth-gated.
• Prod CloudWatch / prod log path — this PR is dev Logcat only.
• WSS tunnel internals, run_outer token minting/writing, the assertion/Google-OAuth env wiring — unchanged; run_inner continues to consume what the outer/host side produces.
• Shipper durability/retry, per-child backoff/health probes, structured-port configurability — best-effort/positional behavior matches the shell runner; not required to light up the Logcat.
• Refactoring Box::leak of the slug label.

6. Migration

1. Switch the compose command in ms-app-modules/docker-compose.yml from command: sh /modules/scripts/dev-runner.sh back to command: mirrorstack dev --all (the runner image must have the CLI binary available — build/copy it into the image, since the host CLI is no longer the in-container entrypoint).
2. Forward MS_INTERNAL_SECRET properly. The stopgap MS_INTERNAL_SECRET: ${MS_INTERNAL_SECRET:-} line already interpolates the host-minted secret into the container env. Keep it, but treat it as load-bearing now (the shipper auth depends on it) and document it: it is only minted by run_outer in --tunnel mode, sent on the register frame, and set on the docker compose up subprocess env. Without it the shipper sink is None and the Logcat stays empty. Verify MS_DISPATCH_URL: http://host.docker.internal:8083 remains forwarded (container→dispatch).
3. Correct/scope the misleading compose comment so it accurately describes the Rust run_inner shipper now actually running (not a behavior the retired shell script claimed).
4. Retire scripts/dev-runner.sh and scripts/dev-proxy.go — delete them once parity is verified. (Both are untracked workspace-local files; deletion is a local cleanup, not a tracked diff in ms-app-modules.)

7. Risks

• OrbStack/Docker has no inotify on bind mounts. Native fs-notify will silently miss every edit. The rebuild watcher must poll at ~2s (matching air's poll_interval=2000) or shell out to air (which polls). This is the single highest-risk item.
• Port wiring. Internal ports must start at 18080 in go.work order; the proxy must bind :8080 (host 9080); livereload LR_PORT from 8089 (host 9089). Off-by-one or wrong base breaks /_m/<slug> routing and the published port mapping.
• Hot-reload rebuild semantics in Rust. go run is one-shot; the new path must build to /tmp/<mod>, run the binary, and on change rebuild + restart the child AND re-tap its stdout/stderr into the shipper. A restart that forgets to re-attach the forwarder silently stops shipping for that module.
• Shipper secret/url wiring. If MS_INTERNAL_SECRET isn't forwarded into the container, or MS_DISPATCH_URL is wrong, the sink is None and the Logcat is empty with no obvious cause — hence the required warning line.
• dev-proxy routing regression. The in-process proxy must reproduce TrimPrefix("/_m/<slug>") + empty→/ exactly; any drift mis-routes module requests (and can break the login flow, which traverses /_m/oauth-core/*).
• Token file path. Must stay /modules/.ms-platform-token-<slug> per module; a single shared file or wrong path 401s all-but-first module.

8. Acceptance criteria

1. Logcat populates end-to-end. After bringing the stack up with mirrorstack dev --all as the runner command and making one request that hits a module (e.g. the oauth-core sign-in path through /_m/oauth-core/...), a log entry for that request is (a) present in dispatch's Redis ring for that module id, and (b) returned by the read endpoint backing /dev/module/<slug>/service.
2. Hot-reload works under OrbStack. Editing a .go file in a running module triggers a rebuild + restart within ~2s (polling), with no manual restart, and the module's logs keep shipping after the restart.
3. dev-proxy routes resolve. GET /_m/<slug>/<path> on host :9080 reaches 127.0.0.1:<internal_port>/<path> with the prefix stripped (empty→/), for every discovered module.
4. Per-module token auth holds. All modules (not just the first) authenticate platform-initiated calls — no 401 on the second+ module — proving distinct MS_PLATFORM_TOKEN_FILE per slug.
5. Login flow unaffected. The Google sign-in E2E completes (redirect resolves, no regression) with the new runner.
6. Shell artifacts retired. scripts/dev-runner.sh and scripts/dev-proxy.go are removed; compose command is mirrorstack dev --all.

9. Test / verify plan

Run from /Users/owo/Documents/MirrorStack-AI-V2/ms-app-modules unless noted. Host services (dispatch :8083, MinIO :9000, etc.) must be up first.

Bring up the live runner (tunnel mode mints + forwards the secret):

# host CLI (outer): mints MS_INTERNAL_SECRET, writes per-module token files, runs compose
cd /Users/owo/Documents/MirrorStack-AI-V2/mirrorstack-cli && cargo run -- dev --tunnel
# confirm the runner container is executing the CLI, not the shell script:
docker compose -f /Users/owo/Documents/MirrorStack-AI-V2/ms-app-modules/docker-compose.yml ps
docker compose -f /Users/owo/Documents/MirrorStack-AI-V2/ms-app-modules/docker-compose.yml logs runner | grep -E "starting [0-9]+ module|✓ (oauth-core|oauth-google)"

AC1 — Logcat populates:

# fire a request through the dev-proxy that exercises a module
curl -sS -i http://localhost:9080/_m/oauth-core/healthz   # or the real sign-in start path
# (a) entry landed in dispatch's Redis ring for the module id
docker exec -it $(docker ps -qf name=redis) valkey-cli --scan --pattern '*modules*log*'   # locate the ring key
docker exec -it $(docker ps -qf name=redis) valkey-cli LRANGE <ring-key> 0 -1 | head
# (b) read endpoint returns entries
curl -sS http://localhost:8083/internal/modules/<oauth-core-id>/logs -H "X-MS-Service-Secret: $MS_INTERNAL_SECRET" | jq '.entries | length'
# expect: >0, and the timestamp/msg of the request above appears

AC2 — hot-reload under OrbStack:

# note the module pid/start, edit a .go file, watch for rebuild+restart within ~2s
touch /Users/owo/Documents/MirrorStack-AI-V2/ms-app-modules/oauth-core/main.go
docker compose -f .../docker-compose.yml logs -f runner | grep -E "building|restarting|\[oauth-core\]"
# then re-fire AC1 curl and confirm a NEW log entry still ships (forwarder re-attached)

AC3 — proxy routing:

for slug in oauth-core oauth-google; do
  echo "== $slug =="; curl -sS -o /dev/null -w "%{http_code}\n" http://localhost:9080/_m/$slug/healthz
done
# expect each to reach its module (200/expected) — prefix stripped, empty→/

AC4 — per-module token auth (no all-but-first 401):

docker compose -f .../docker-compose.yml logs runner | grep -iE "401|unauthorized"   # expect none from token auth
ls -1 /Users/owo/Documents/MirrorStack-AI-V2/ms-app-modules/.ms-platform-token-*     # one file per module

AC5 — login flow: run the existing Google sign-in E2E against the new runner and confirm it completes (redirect resolves, session set).

AC6 — artifacts retired:

test ! -e /Users/owo/Documents/MirrorStack-AI-V2/ms-app-modules/scripts/dev-runner.sh && \
test ! -e /Users/owo/Documents/MirrorStack-AI-V2/ms-app-modules/scripts/dev-proxy.go && echo "retired"
grep -n "command:" /Users/owo/Documents/MirrorStack-AI-V2/ms-app-modules/docker-compose.yml   # expect: mirrorstack dev --all

Negative check — warning when shipping disabled:

# without --tunnel, MS_INTERNAL_SECRET is empty → sink None → expect a one-line warning, not silent
cd /Users/owo/Documents/MirrorStack-AI-V2/mirrorstack-cli && cargo run -- dev   # no --tunnel
# confirm runner logs warn that Logcat shipping is disabled (no secret), modules still run

---

🤖 Generated with Claude Code

Related (already merged — out of scope here): api-platform#276 (login redirect resolved), app-module-sdk#128 (SDK auto access-log so modules emit). Dispatch ingest POST /internal/modules/<id>/logs is live + auth-gated.

Local stopgap already applied (untracked e2e file, not a code change): MS_INTERNAL_SECRET: ${MS_INTERNAL_SECRET:-} was added to the runner service env in ms-app-modules/docker-compose.yml so the secret reaches the container. This PR should make that forward part of the supported wiring rather than a manual patch.

🤖 Generated with Claude Code