fix(backends): pass provider API keys through to cmux crewmate spawns

.gitignore

@@ -15,4 +15,5 @@ config/backlog-backend
 config/backend
 config/x-mode.env
 config/cmux-socket-password
+config/cmux-env-secrets
 config/wedge-alarm

AGENTS.md

@@ -159,7 +159,7 @@ Do not add model-specific versions of that policy.
 
 `secondmate-provisioning` owns secondmate harness pins and config inheritance, while `harness-adapters` owns the harness consequences.
 Dispatch only on a backend that `fm-spawn` validates as spawn-capable.
-A missing dependency, authentication failure, unsupported backend, or version refusal is a blocker; never silently retry on another backend.
+A missing dependency, authentication failure, unsupported backend, version refusal, or failed provider-credential fetch is a blocker; never silently retry on another backend.
 
 ## 5. Recovery
 

bin/backends/cmux.sh

@@ -341,24 +341,130 @@ fm_backend_cmux_surface_id_for_workspace() {  # <workspace_id>
     | jq -r '.panes[0] // {} | .selected_surface_id // (.surface_ids[0] // empty)' 2>/dev/null
 }
 
+# fm_backend_cmux_secret_specs_for_harness: zero or more "ENV_VAR secret-name
+# [region]" lines naming the provider credentials that must be explicitly
+# injected into a cmux workspace for <harness>, read fresh from the local,
+# gitignored config/cmux-env-secrets under the effective config dir (the same
+# read-fresh convention as fm_backend_cmux_password above). Each config line
+# is "<harness> <ENV_VAR> <secret-name> [<aws-region>]"; blank lines and
+# '#'-prefixed lines are ignored (docs/cmux-backend.md "Provider credential
+# passthrough (--env)"). Needed only for cmux: unlike tmux/herdr/zellij
+# (which fork a task inside the same host shell and so inherit firstmate's
+# own process environment), a cmux terminal surface's environment is heavily
+# scrubbed at spawn time - ambient provider API keys never reach it. An
+# absent file, or a harness with no matching line, means no injection at all;
+# a harness-matching line missing its secret name is skipped with a stderr
+# warning so a mistyped pairing surfaces at spawn instead of silently
+# reverting to no injection.
+fm_backend_cmux_secret_specs_for_harness() {  # <harness>
+  local harness=$1 config_dir="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" f h env_var secret_name region _rest
+  f="$config_dir/cmux-env-secrets"
+  [ -f "$f" ] || return 0
+  while read -r h env_var secret_name region _rest || [ -n "$h" ]; do
+    case "$h" in ''|'#'*) continue ;; esac
+    [ "$h" = "$harness" ] || continue
+    if [ -z "$secret_name" ]; then
+      echo "warning: cmux env passthrough: ignoring malformed line in $f (missing secret name): $h${env_var:+ $env_var}" >&2
+      continue
+    fi
+    if [ -n "$region" ]; then
+      printf '%s %s %s\n' "$env_var" "$secret_name" "$region"
+    else
+      printf '%s %s\n' "$env_var" "$secret_name"
+    fi
+  done < "$f"
+}
+
+# fm_backend_cmux_fetch_secret: print <secret-name>'s current value from AWS
+# Secrets Manager on stdout (in [region] when given, otherwise whatever
+# region is ambient in AWS config), using whatever AWS credentials/profile
+# are already ambient in this process's environment - never a hardcoded
+# profile. Fails loudly rather than continuing without the key: a missing
+# 'aws' CLI, a failed call (whose AWS stderr is included in the error - AWS
+# error text never contains the secret value), or an empty secret are all
+# refused.
+fm_backend_cmux_fetch_secret() {  # <secret-name> [region]
+  local secret_name=$1 region=${2:-} where value err_file aws_err status=0
+  where="AWS Secrets Manager${region:+ (region $region)}"
+  command -v aws >/dev/null 2>&1 || { echo "error: cmux env passthrough needs secret '$secret_name' but the 'aws' CLI is not installed" >&2; return 1; }
+  err_file=$(mktemp "${TMPDIR:-/tmp}/fm-cmux-aws-err.XXXXXX") || err_file=/dev/null
+  if [ -n "$region" ]; then
+    value=$(aws secretsmanager get-secret-value --secret-id "$secret_name" --region "$region" --query SecretString --output text 2>"$err_file") || status=$?
+  else
+    value=$(aws secretsmanager get-secret-value --secret-id "$secret_name" --query SecretString --output text 2>"$err_file") || status=$?
+  fi
+  aws_err=
+  if [ "$err_file" != /dev/null ]; then
+    aws_err=$(cat "$err_file" 2>/dev/null)
+    rm -f "$err_file"
+  fi
+  if [ "$status" -ne 0 ]; then
+    echo "error: failed to fetch secret '$secret_name' from $where - ${aws_err:-check that AWS credentials/profile are configured}" >&2
+    return 1
+  fi
+  if [ -z "$value" ] || [ "$value" = None ]; then
+    echo "error: secret '$secret_name' from $where returned an empty value" >&2
+    return 1
+  fi
+  printf '%s' "$value"
+}
+
+# fm_backend_cmux_env_args_for_harness: resolve <harness>'s secret specs (if
+# any, per fm_backend_cmux_secret_specs_for_harness) and set
+# FM_BACKEND_CMUX_ENV_ARGS to the "--env KEY=VALUE" pairs to pass to
+# `new-workspace`. A fetched value only ever lands in this array - never
+# printed, logged, or written to a file - mirroring how
+# fm_backend_cmux_password/CMUX_SOCKET_PASSWORD above handles the socket
+# password. Empty array (success) when the harness needs no injected secret.
+fm_backend_cmux_env_args_for_harness() {  # <harness>
+  FM_BACKEND_CMUX_ENV_ARGS=()
+  local harness=$1 env_var secret_name region value
+  [ -n "$harness" ] || return 0
+  while read -r env_var secret_name region; do
+    [ -n "$secret_name" ] || continue
+    value=$(fm_backend_cmux_fetch_secret "$secret_name" "$region") || return 1
+    FM_BACKEND_CMUX_ENV_ARGS+=(--env "$env_var=$value")
+  done < <(fm_backend_cmux_secret_specs_for_harness "$harness")
+  return 0
+}
+
 # fm_backend_cmux_create_task: create the task's workspace (one surface),
 # refusing an existing live <label> (finding #6: cmux enforces no uniqueness
 # itself). Resolves the fresh workspace's default surface via one list-panes
 # call (finding: a freshly created workspace already has exactly one surface,
 # so no separate new-surface call is needed). --focus false is passed for
 # defense in depth though verified to already be the default (finding:
 # workspace/surface/pane create all default focus to false) - no
-# focus-restore dance is needed, unlike zellij. Echoes "<workspace_id>
-# <surface_id>" on success.
-fm_backend_cmux_create_task() {  # <label> <cwd>
-  local label=$1 cwd=$2 title dup out wsid sfid
+# focus-restore dance is needed, unlike zellij. When <harness> is given and
+# config/cmux-env-secrets pairs it with provider credentials
+# (fm_backend_cmux_secret_specs_for_harness), they are fetched and passed as
+# `--env KEY=VALUE` (verified flag: cmux-control's own bin/cmuxctl,
+# cmuxNewWorkspace); with no pairing configured, nothing is fetched or
+# injected. Echoes "<workspace_id> <surface_id>" on success.
+fm_backend_cmux_create_task() {  # <label> <cwd> [harness]
+  local label=$1 cwd=$2 harness=${3:-} title dup out wsid sfid
   title=$(fm_backend_cmux_scoped_title "$label")
   dup=$(fm_backend_cmux_workspace_id_for_label "$title")
   if [ -n "$dup" ]; then
     echo "error: cmux workspace '$title' already exists" >&2
     return 1
   fi
-  out=$(fm_backend_cmux_cli new-workspace --name "$title" --cwd "$cwd" --focus false --id-format uuids 2>&1) || {
+  fm_backend_cmux_env_args_for_harness "$harness" || return 1
+  # Guarded append, never a bare "${arr[@]}" expansion of a possibly-empty
+  # array: under `set -u`, bash 3.2 (the macOS system bash) treats expanding a
+  # zero-element array as an unbound-variable error, and most callers spawn
+  # with no harness secret needed (FM_BACKEND_CMUX_ENV_ARGS stays empty).
+  local -a new_workspace_args=(new-workspace --name "$title" --cwd "$cwd")
+  [ "${#FM_BACKEND_CMUX_ENV_ARGS[@]}" -eq 0 ] || new_workspace_args+=("${FM_BACKEND_CMUX_ENV_ARGS[@]}")
+  new_workspace_args+=(--focus false --id-format uuids)
+  out=$(fm_backend_cmux_cli "${new_workspace_args[@]}" 2>&1) || {
+    if [ "${#FM_BACKEND_CMUX_ENV_ARGS[@]}" -gt 0 ]; then
+      local env_arg
+      for env_arg in "${FM_BACKEND_CMUX_ENV_ARGS[@]}"; do
+        [ "$env_arg" = --env ] && continue
+        out=${out//"${env_arg#*=}"/[redacted]}
+      done
+    fi
     echo "error: cmux new-workspace failed for '$title': $out" >&2
     return 1
   }

bin/fm-spawn.sh

@@ -25,8 +25,9 @@
 #   blocked backend contract. Default tmux spawns do not write backend= to meta;
 #   absent backend= means tmux. cmux does not support --secondmate spawns yet.
 #   A backend spawn refusal (missing dependency, version gate, unauthenticated
-#   socket, or unsupported secondmate mode) is terminal for that selected backend;
-#   callers must surface it instead of silently retrying another backend.
+#   socket, failed provider-credential fetch, or unsupported secondmate mode) is
+#   terminal for that selected backend; callers must surface it instead of
+#   silently retrying another backend.
 #   With no harness arg, a crewmate/scout spawn resolves the CREW harness only when
 #   config/crew-dispatch.json is absent. When that file exists, crewmate/scout
 #   spawns require an explicit harness so firstmate cannot silently skip dispatch
@@ -760,7 +761,7 @@ EOF
     ;;
   cmux)
     fm_backend_cmux_container_ensure || exit 1
-    CMUX_TASK_IDS=$(fm_backend_cmux_create_task "$W" "$PROJ_ABS") || exit 1
+    CMUX_TASK_IDS=$(fm_backend_cmux_create_task "$W" "$PROJ_ABS" "$HARNESS") || exit 1
     read -r CMUX_WORKSPACE_ID CMUX_SURFACE_ID <<EOF
 $CMUX_TASK_IDS
 EOF

docs/cmux-backend.md

@@ -20,6 +20,7 @@ Prerequisites:
 - `jq`, required to parse cmux's JSON output: `brew install jq` (or your platform's package manager).
 - The universal firstmate prerequisites - a verified crew harness plus the required toolchain, owned by [`docs/configuration.md`](configuration.md) ("Harness support", "Toolchain"); treehouse still provides the worktree, cmux only provides the session.
 - The cmux CLI binary is not guaranteed to be on `PATH` after a plain app install (see "CLI is not on PATH by default" below) - the adapter falls back to the well-known bundle path automatically, so this is not a blocker, just something to be aware of if you want to run `cmux` yourself from a shell.
+- The `aws` CLI, only if you configure provider-credential passthrough via `config/cmux-env-secrets` (see "Provider credential passthrough (`--env`)" below) - with a pairing configured, a spawn that cannot fetch the secret refuses loudly; with no pairing, `aws` is not needed at all.
 
 **One-time socket access setup (required, not optional):** cmux's control socket defaults to `automation.socketControlMode: "cmuxOnly"`, which rejects any CLI process not spawned inside cmux itself - firstmate always drives cmux from an external shell, so this must be changed before `backend=cmux` can work at all.
 Settings > Automation offers five Socket Control Mode values; their exact semantics were verified from cmux source (see "Socket control modes: the full matrix" below for the enforcement points and evidence):
@@ -171,7 +172,7 @@ No session field is needed - unlike herdr/zellij there is no session layer to re
 | Version gate | `cmux version` -> `"cmux 0.64.17 (97) [9ed29d81a]"` | Works with NO socket connection at all - a pure client-version check, verified even while the socket was still rejecting connections. |
 | Reachability/auth gate | `cmux ping` -> `"PONG"` or a typed error | Classified into `ok`\|`denied`\|`unauth`\|`down`\|`error` from the error text (`fm_backend_cmux_ping_state`); `fm_backend_cmux_ensure_running` launches the app (`open -a cmux`) only for `down`, and fails fast with an actionable message for `denied`/`unauth` since relaunching cannot fix a configuration problem. |
 | Duplicate task check | `cmux workspace list --json --id-format uuids`, match by home-scoped `.title` | cmux enforces NO title uniqueness for workspaces OR surfaces/tabs - verified live: two workspaces, and two surfaces within one workspace, all created successfully sharing one title. The adapter's own duplicate check is required, mirroring herdr/zellij, and it checks the scoped title such as `fm-firstmate-<8hex>-<id>`. |
-| Create task workspace | `cmux new-workspace --name <scoped-title> --cwd <dir> --focus false --id-format uuids` | Creates a workspace with exactly one default surface. `--focus` verified to already default to `false` for workspace/surface/pane creation - no focus-restore dance needed, unlike zellij. The caller passes `fm-<id>`, but the adapter creates `fm-<home-label>-<id>`. |
+| Create task workspace | `cmux new-workspace --name <scoped-title> --cwd <dir> [--env KEY=<value> ...] --focus false --id-format uuids` | Creates a workspace with exactly one default surface. `--focus` verified to already default to `false` for workspace/surface/pane creation - no focus-restore dance needed, unlike zellij. The caller passes `fm-<id>`, but the adapter creates `fm-<home-label>-<id>`. The `--env` pairs appear only when `config/cmux-env-secrets` pairs the spawning harness with provider credentials (see "Provider credential passthrough (`--env`)" below; flag verified from cmux-control's own `bin/cmuxctl` source). |
 | Workspace/surface id resolution | `cmux workspace list --json --id-format uuids` (find by home-scoped title), then `cmux list-panes --workspace <id> --json --id-format uuids` (`.panes[0].selected_surface_id`) | A freshly created workspace already has exactly one surface, so no separate `new-surface` call is needed. `--id-format uuids` (or `both`) is required to get a bare `id` field in JSON; the default JSON shape returns only short `ref` strings like `"workspace:2"`. |
 | Liveness / target readiness | `cmux list-panes --workspace <id> --json --id-format uuids`, checking the surface id appears in `.panes[].surface_ids` | Structural existence check, NOT a content read - see "read-screen fails on a genuinely fresh surface" below for why `read-screen` cannot be used here. Verified reliable on a completely untouched fresh surface, unlike `read-screen`. |
 | Send literal (unsubmitted) | `cmux send --workspace <id> --surface <id> -- <text>` | Verified live: does NOT auto-submit - text sits at the prompt, unexecuted, until a separate Enter. Matches every other backend's "literal-then-separate-Enter" contract. The `--` separator keeps option-shaped text such as `--help` literal. |
@@ -327,6 +328,21 @@ Same as herdr's tabs and zellij's tabs, unlike tmux's own window-name uniqueness
 Verified live: two workspaces created with the identical title `fm-test-dup` both succeeded and listed simultaneously with distinct ids; two surfaces within one workspace both renamed to the identical tab title also succeeded.
 `fm_backend_cmux_create_task`'s own title-based duplicate check is therefore required, mirroring both prior adapters' posture exactly.
 
+## Provider credential passthrough (`--env`)
+
+A cmux workspace's terminal surface starts with none of firstmate's own process environment worth relying on for provider credentials - unlike tmux/herdr/zellij, which fork a task inside the same host shell and so inherit it, a cmux surface is a fresh GUI-app-spawned process (see the wrapper-stripping finding above for a related, but distinct, case: that one is about the RUNTIME firstmate's own launch environment when firstmate itself runs inside a cmux tab; this one is about the environment of a workspace firstmate just CREATED for a task).
+So an ambient `FIREWORKS_API_KEY` set in firstmate's own shell never reaches a spawned opencode surface, and opencode's Fireworks-backed models (e.g. GLM 5.2) fail outright.
+
+`fm_backend_cmux_create_task` takes an optional third `<harness>` argument and, when the local, gitignored `config/cmux-env-secrets` pairs that harness with one or more provider credentials (`fm_backend_cmux_secret_specs_for_harness`, which reads the file fresh from the effective config dir the same way `config/cmux-socket-password` is read), fetches each from AWS Secrets Manager (using whatever AWS credentials/profile are already ambient - never a hardcoded profile) and passes it to `new-workspace` as `--env KEY=<value>`.
+Each `config/cmux-env-secrets` line is `<harness> <ENV_VAR> <secret-name> [<aws-region>]`, with blank lines and `#`-prefixed lines ignored; an omitted region defers to ambient AWS region config.
+For example, `opencode FIREWORKS_API_KEY opencode/fireworks-api-key us-east-1` injects the Fireworks key an opencode spawn needs for its Fireworks-backed models (e.g. GLM 5.2).
+An absent file, or a harness with no matching line, injects nothing and triggers no AWS call at all - the spawn proceeds exactly as it did before this knob existed.
+A harness-matching line missing its `<secret-name>` field is malformed: it is skipped with a one-line stderr warning naming the file and offending line, so a mistyped pairing surfaces at spawn instead of silently reverting to no injection.
+`--env KEY=VALUE` is cmux-control's own verified mechanism for this (`bin/cmuxctl`'s `cmuxNewWorkspace`, which builds one `--env` pair per entry of an `env` object): a real, working `new-workspace` flag firstmate's own adapter simply never used before.
+A failed fetch (missing `aws` CLI, a failed call - whose AWS stderr is included in the spawn-failure error, since AWS error text never contains the secret value - or an empty secret) fails the spawn loudly rather than silently launching the harness without a key it was configured to need.
+The fetched value only ever lands in a shell variable used to build the `--env` argument - never printed, logged, or written to a file, and redacted from any echoed-back `new-workspace` failure output - the same convention `fm_backend_cmux_password`/`CMUX_SOCKET_PASSWORD` above already follows for the socket password.
+`bin/fm-spawn.sh` passes the resolved `$HARNESS` through on every cmux spawn.
+
 ## Composer verification: structural border-row classification (adapted from herdr)
 
 cmux's `read-screen` gives plain-text capture with no cursor-row primitive and no ANSI style channel, unlike tmux's `#{cursor_y}` and herdr's `--format ansi` path for ANSI-aware ghost/placeholder classification.
@@ -370,5 +386,6 @@ All three tasks' cmux workspaces and worktrees were confirmed fully cleaned up a
   The one-time socket-access setup remains an unavoidable manual step regardless of how the backend was selected.
 - **`--secondmate` spawns are refused** (mirrors Orca's refusal) - no per-home container design (a herdr-style workspace-per-home split, or similar) has been designed or verified for cmux yet.
 - **The one-time socket-access setup is a real, undocumented-by-upstream onboarding step.** A captain who selects `backend=cmux` without first switching `automation.socketControlMode` away from its `cmuxOnly` default to a viable mode (Automation mode recommended; see "Setup") will see every spawn fail with an actionable error naming the viable modes and pointing back to this document, but there is no way for firstmate to complete that GUI-only setup step on the captain's behalf.
+- **`aws` is not bootstrap-detected** - `bin/fm-bootstrap.sh`'s backend-aware required-tool detection covers `cmux` and `jq` for a cmux backend selection, but does not add `aws` when `config/cmux-env-secrets` configures provider-credential passthrough; the credential fetch happens at spawn time instead and refuses loudly.
 - **A surface can still die in the brief window between `target_ready` succeeding and the operation's own call running.** That remaining race degrades to "the operation quietly did nothing" - the same class of gap firstmate already tolerates for an unverified send on any backend, caught downstream by `fm-spawn.sh`'s worktree-discovery poll timing out, `fm_backend_cmux_send_text_submit`'s retry loop (which reports `send-failed`/`pending`/`unknown` rather than a false "sent"), or the watcher's stale-pane detection.
 - **Windows cannot be closed over the control socket, and label lookup is current-window scoped** - both owned by "Closing the last workspace in a window" above. Teardown of a last-in-window task workspace therefore leaves that window a fresh default workspace rather than closing it, and `fm_backend_cmux_workspace_id_for_label`/`fm_backend_cmux_list_live` only see the current window, so a task workspace parked in a non-current window is a known blind spot for label-based recovery.