fix(backends): pass provider API keys through to cmux crewmate spawns

[Kadriya89]

Intent

Fix the cmux backend so provider API keys reach spawned crewmate processes: opencode spawns need FIREWORKS_API_KEY so opencode's Fireworks-backed models (e.g. GLM 5.2) work. fm_backend_cmux_create_task now builds --env FIREWORKS_API_KEY= for cmux new-workspace by fetching the key at spawn time from AWS Secrets Manager (secret opencode/fireworks-api-key, region us-east-1) using ambient AWS credentials - no hardcoded profile; a missing key fails the spawn loudly rather than silently continuing. Deliberate decisions already made and binding: the harness-to-secret pairing lives in local gitignored config/cmux-env-secrets so shipped code stays provider-neutral (captain decision); the fetched secret value is never printed, logged, or committed, and is redacted from cmux new-workspace failure output (captain decision); the SC2015 and-or chain in the empty-value check was rewritten as a proper if/then to satisfy shellcheck CI. Unit tests stub the aws CLI so no real AWS calls happen in tests. Push goes through the fork Kadriya89/firstmate per CONTRIBUTING.md since contributors lack upstream push access; PR targets kunchenguid/firstmate. An earlier hand-raised PR #360 exists and should be updated or superseded by the pipeline-raised PR.

What Changed

• bin/backends/cmux.sh gains a provider-credential passthrough for workspace creation: a new local, gitignored config/cmux-env-secrets file pairs a harness with ENV_VAR / secret-name / optional region lines, and fm_backend_cmux_create_task (now taking an optional harness argument, which bin/fm-spawn.sh passes on every cmux spawn) fetches each paired secret from AWS Secrets Manager at spawn time using ambient AWS credentials and injects it via cmux's new-workspace --env KEY=VALUE. This fixes opencode spawns on the cmux backend, whose scrubbed surface environment never received FIREWORKS_API_KEY, breaking Fireworks-backed models.
• The fetch fails the spawn loudly (missing aws CLI, failed call with AWS stderr surfaced, or empty secret) instead of silently launching without a configured key; the fetched value is never printed or logged and is redacted from echoed-back new-workspace failure output; malformed pairing lines are skipped with a stderr warning; an absent file or unpaired harness triggers no AWS call at all. An SC2015 and-or chain in the empty-value check was rewritten as if/then for shellcheck.
• Added tests/fm-backend-cmux.test.sh env-passthrough coverage (63/63 pass, including 7 new cases: spec parsing, malformed-line warning, --env injection, no-injection paths, loud fetch failure, redaction, missing-aws failure) with a stubbed aws CLI so no real AWS calls run; synced docs/cmux-backend.md, docs/configuration.md, docs/scripts.md, and AGENTS.md; and fixed unrelated test fixtures (composer-lib symlink, tasks-axi version skip, peer readiness) that the pipeline's test stage caught.

Risk Assessment

✅ Low: The feature is strictly opt-in via a local gitignored config (absent file preserves prior behavior exactly), fails loudly before workspace creation on any fetch problem, redacts the secret from error output, and is covered by eight targeted unit tests stubbing both cmux and aws.

Testing

Baseline full suite passed; re-ran the new cmux backend test file (63/63, covering all passthrough behaviors) and the four fixture-touched test files (all pass), then captured a manual end-to-end CLI transcript with stubbed aws and cmux showing the Fireworks key fetched from Secrets Manager with ambient credentials and injected via --env into the opencode workspace spawn, with no injection for unpaired harnesses, loud spawn failure on fetch errors, and secret redaction in error output.

Evidence: End-to-end demo transcript: cmux --env passthrough (4 scenarios)

== Scenario A: opencode spawn == aws secretsmanager get-secret-value --secret-id opencode/fireworks-api-key --region us-east-1 --query SecretString --output text cmux new-workspace --name fm-firstmate-960fe46d-glm52-demo --cwd /tmp/proj --env FIREWORKS_API_KEY=fw-DEMO-fake-fireworks-key-0000 --focus false --id-format uuids == Scenario B: claude spawn == no --env, aws never invoked == Scenario C: fetch failure == error: failed to fetch secret 'opencode/fireworks-api-key' from AWS Secrets Manager (region us-east-1) - AccessDeniedException...; new-workspace never called == Scenario D: redaction == ... --env FIREWORKS_API_KEY=[redacted] ... (fetched value absent from error output)

############################################################
## cmux provider-credential passthrough - end-to-end demo ##
## (real bin/backends/cmux.sh; stubbed aws + cmux CLIs)   ##
############################################################

== operator config: config/cmux-env-secrets (local, gitignored) ==
# provider credentials injected into cmux workspaces at spawn
opencode FIREWORKS_API_KEY opencode/fireworks-api-key us-east-1

== Scenario A: opencode spawn (the GLM 5.2 / Fireworks case) ==
$ fm_backend_cmux_create_task fm-glm52-demo /tmp/proj opencode   # exactly how fm-spawn.sh calls it
exit=0  stdout: bbbbbbbb-1111-1111-1111-111111111111 cccccccc-2222-2222-2222-222222222222

-- aws CLI invocation recorded by the stub (ambient credentials, no --profile):
aws secretsmanager get-secret-value --secret-id opencode/fireworks-api-key --region us-east-1 --query SecretString --output text
-- cmux new-workspace invocation recorded by the stub:
2:cmux new-workspace --name fm-firstmate-960fe46d-glm52-demo --cwd /tmp/proj --env FIREWORKS_API_KEY=fw-DEMO-fake-fireworks-key-0000 --focus false --id-format uuids

== Scenario B: claude spawn - no pairing, so no injection and aws is never called ==
exit=0  stdout: bbbbbbbb-1111-1111-1111-111111111111 cccccccc-2222-2222-2222-222222222222
-- new-workspace invocation (no --env anywhere):
2:cmux new-workspace --name fm-firstmate-960fe46d-claude-demo --cwd /tmp/proj --focus false --id-format uuids
-- aws.log is empty: aws never invoked

== Scenario C: secret fetch fails -> spawn fails loudly, workspace never created ==
exit=1
-- error surfaced to firstmate (includes AWS's own stderr):
error: failed to fetch secret 'opencode/fireworks-api-key' from AWS Secrets Manager (region us-east-1) - An error occurred (AccessDeniedException) when calling the GetSecretValue operation: not authorized
-- new-workspace never called

== Scenario D: new-workspace fails echoing its args -> secret value redacted from error ==
exit=1
-- error surfaced to firstmate:
error: cmux new-workspace failed for 'fm-firstmate-960fe46d-oc-redact': error: invalid arguments: new-workspace --name fm-firstmate-960fe46d-oc-redact --cwd /tmp/proj --env FIREWORKS_API_KEY=[redacted] --focus false --id-format uuids
-- fetched secret value absent from error output (redacted)

Evidence: fm-backend-cmux.test.sh full output (63/63 ok)

ok - fm_backend_cmux_version_check: accepts the verified minimum (0.64.17)
ok - fm_backend_cmux_version_check: accepts a newer version (0.70.0)
ok - fm_backend_cmux_version_check: refuses an old version loudly
ok - fm_backend_cmux_version_check: refuses loudly when cmux is not found on PATH or at the bundle path
ok - fm_backend_cmux_password: reads the first non-empty line of config/cmux-socket-password
ok - fm_backend_cmux_password: preserves spaces and tabs in config/cmux-socket-password
ok - fm_backend_cmux_password: respects FM_CONFIG_OVERRIDE
ok - fm_backend_cmux_password: empty when config/cmux-socket-password is absent
ok - fm_backend_cmux_cli: exports CMUX_SOCKET_PASSWORD when config/cmux-socket-password is set
ok - fm_backend_cmux_parse_target: splits '<workspace_uuid>:<surface_uuid>' on the first colon
ok - fm_backend_cmux_normalize_key: Enter/Escape/C-c map to cmux's verified enter/escape/ctrl-c
ok - fm_backend_cmux_scoped_title: scopes a primary task title with firstmate plus root hash
ok - fm_backend_cmux_scoped_title: scopes a secondmate task title with the home marker plus root hash
ok - fm_backend_cmux_scoped_title: includes the resolved FM_ROOT hash in the home label
ok - fm_backend_validate: cmux is a known backend
ok - fm_backend_busy_state: cmux (no native primitive) always reports unknown, same as tmux/zellij/orca
ok - fm_backend_composer_state: routes cmux to the cmux composer classifier
ok - fm_backend_cmux_ping_state: reports 'ok' on PONG
ok - fm_backend_cmux_ping_state: reports 'denied' when socketControlMode=cmuxOnly rejects the connection
ok - fm_backend_cmux_ping_state: reports 'unauth' when password mode rejects a missing/wrong password
ok - fm_backend_cmux_ping_state: reports 'unauth' when password mode rejects a wrong password (Invalid password)
ok - fm_backend_cmux_ping_state: reports 'down' when the app is not running yet
ok - fm_backend_cmux_ensure_running: returns immediately when cmux is already reachable
ok - fm_backend_cmux_ensure_running: fails fast on a denied socket without attempting to launch, naming every viable mode
ok - fm_backend_cmux_ensure_running: fails fast on an unauthenticated socket, naming the password config and the Automation mode alternative
ok - fm_backend_cmux_create_task: refuses a duplicate workspace title (cmux's own new-workspace has no uniqueness check)
ok - fm_backend_cmux_create_task: creates a workspace and parses workspace_id/surface_id from list responses
ok - fm_backend_cmux_secret_specs_for_harness: reads pairings (with optional region) from local config/cmux-env-secrets, skipping comments, blanks, unpaired harnesses, and an absent file
ok - fm_backend_cmux_secret_specs_for_harness: warns loudly (file + offending line) when a harness-matching pairing lacks a secret name, instead of silently skipping it
ok - fm_backend_cmux_create_task: passes --env FIREWORKS_API_KEY=<value> to new-workspace for a config-paired opencode spawn
ok - fm_backend_cmux_create_task: passes no --env flags, and never calls aws, for a harness config/cmux-env-secrets does not pair (e.g. claude)
ok - fm_backend_cmux_create_task: with no config/cmux-env-secrets, injects nothing and never calls aws - the spawn proceeds as before the knob existed
ok - fm_backend_cmux_create_task: fails before creating the workspace when the required secret cannot be fetched, surfacing the AWS CLI's stderr
ok - fm_backend_cmux_create_task: redacts the fetched secret from a failed new-workspace's echoed-back error output
ok - fm_backend_cmux_fetch_secret: fails loudly when the aws CLI is not installed, rather than continuing without the key
ok - fm_backend_cmux_target_ready: fails when the workspace/surface is not found (list-panes structural check)
ok - fm_backend_cmux_target_ready: verifies the workspace title against the expected label first
ok - fm_backend_cmux_target_ready: rejects a workspace id reused under a different title
ok - fm_backend_cmux_capture: fetches generously and trims to N lines locally
ok - fm_backend_cmux_capture: propagates a read-screen failure even when stdout is empty
ok - fm_backend_cmux_capture: fails when the target surface is absent
ok - fm_backend_cmux_send_key: normalizes the key (Escape -> escape) and targets the explicit workspace/surface
ok - fm_backend_cmux_send_key: recovers stale workspace/surface ids by expected label
ok - fm_backend_cmux_send_literal: calls send with an explicit workspace/surface and a -- separator
ok - fm_backend_cmux_current_path: actively probes with marked begin/end lines (zellij-shape frozen cwd)
ok - fm_backend_cmux_composer_state: a bare '❯' composer row reads empty
ok - fm_backend_cmux_composer_state: the ghost placeholder text reads empty, not pending
ok - fm_backend_cmux_composer_state: real composer text reads pending
ok - fm_backend_cmux_composer_state: a slash-command popup's argument-hint placeholder still reads pending (the incident fix)
ok - fm_backend_cmux_composer_state: reports unknown when the surface cannot be captured
ok - fm_backend_cmux_composer_state: reports unknown when no border-delimited composer row is found
ok - fm_backend_cmux_send_text_submit: reports 'empty' once the composer row reads empty after one Enter
ok - fm_backend_cmux_send_text_submit: reports 'pending' when the composer never clears after retried Enters (swallowed)
ok - fm_backend_cmux_send_text_submit: retries past a popup-placeholder-fill Enter and lands the real second Enter (the incident fix)
ok - fm_backend_cmux_send_text_submit: reports 'send-failed' when the target workspace/surface is absent
ok - fm_backend_cmux_window_of_workspace: walks windows and counts the membership-confirming workspace list
ok - fm_backend_cmux_window_of_workspace: echoes nothing when no window holds the workspace
ok - fm_backend_cmux_kill: closes the task workspace directly when it is not the last in its window
ok - fm_backend_cmux_kill: adds a throwaway sibling then closes the target when it is the last workspace in its window
ok - fm_backend_cmux_kill: never fails even when close-workspace fails
ok - fm_backend_cmux_kill: recovers stale workspace/surface ids by expected label
ok - fm_backend_cmux_list_live: lists only this home's scoped task workspaces using plain fm-<id> labels
ok - fm-spawn.sh: refuses backend=cmux for --secondmate spawns (mirrors Orca's refusal; no secondmate launch design exists yet)

- Outcome: 🔧 1 issue found → auto-fixed ✅ across 2 runs (51m1s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

🔧 **Rebase** - 3 issues found → auto-fixed ✅

• ⚠️ CONTRIBUTING.md - merge conflict rebasing onto origin/main
• ⚠️ docs/configuration.md - merge conflict rebasing onto origin/main
• ⚠️ docs/scripts.md - merge conflict rebasing onto origin/main

🔧 Fix applied.
✅ Re-checked - no issues remain.

⚠️ **Review** - 2 infos

• ℹ️ bin/backends/cmux.sh:426 - The fetched secret is passed as --env KEY=VALUE in the argv of the external cmux CLI process, making it briefly visible in the local process table (ps) while new-workspace runs. This is inherent to cmux-control's --env mechanism (no env-file/stdin alternative exists), but it is an exposure channel the "never printed, logged, or written to a file" documentation claim does not cover.
• ℹ️ bin/backends/cmux.sh:392 - The aws secretsmanager get-secret-value calls run inside a while read loop fed by process substitution, so the aws CLI inherits the spec stream as stdin. Adding < /dev/null to the aws invocations would guarantee a prompting or stdin-reading CLI can never consume subsequent pairing lines when multiple secrets are configured for one harness.

🔧 **Test** - 1 issue found → auto-fixed ✅

• 🚨 tests failed with exit code 1
• command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"

🔧 Fix: fix test fixtures: composer-lib symlink, tasks-axi skip, peer readiness
✅ Re-checked - no issues remain.

• command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
• Baseline configured suite (all tests/*.test.sh under tmux) — ran successfully before this round
• bash tests/fm-backend-cmux.test.sh — 63/63 pass, including the 7 new env-passthrough cases (spec parsing, malformed-line warning, --env injection for opencode, no injection for unpaired harness, no injection with absent config, loud fetch failure with AWS stderr, secret redaction, missing-aws-CLI failure)
• bash tests/fm-backlog-handoff.test.sh, bash tests/fm-gotmp.test.sh, bash tests/fm-watcher-lock.test.sh, bash tests/fm-secondmate-lifecycle-e2e.test.sh — the four files whose fixtures the branch's test-fix commit touched; all pass (tasks-axi cases skip cleanly on the older installed version, as intended)
• Manual end-to-end demo driving real bin/backends/cmux.sh via fm_backend_cmux_create_task with the harness argument fm-spawn.sh now passes, using stubbed aws + cmux CLIs: verified the recorded aws secretsmanager get-secret-value --secret-id opencode/fireworks-api-key --region us-east-1 call (no --profile), the --env FIREWORKS_API_KEY=<value> flag on new-workspace, no aws call / no --env for a claude spawn, spawn abort before workspace creation on AccessDenied, and [redacted] in echoed-back new-workspace failure output

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

[kunchenguid]

Automated reminder: thanks for the PR! This branch currently has a merge conflict with the base branch.

When you get a chance, please rebase onto (or merge) the latest base branch, resolve the conflict, and push. After that, checks will re-run and the PR will get looked at again.

_{Noted for firstmate#360 at ce2585f0.}

[kunchenguid]

Automated reminder: thanks for the PR! This branch currently has a merge conflict with the base branch.

When you get a chance, please rebase onto (or merge) the latest base branch, resolve the conflict, and push. After that, checks will re-run and the PR will get looked at again.

_{Noted for firstmate#360 at 94d64f2c.}