fix(api): persist stdout for automation-triggered script runs (#3162)

[ToddHebebrand]

Closes #3162

The bug

Automation run_script actions queued the device command with a synthetic
${runId}:${deviceId}:${actionIndex} executionId. handleScriptResult
(apps/api/src/routes/agentWs.ts) matches that against script_executions.id
— a uuid column — so Postgres raised 22P02: invalid input syntax for type uuid, the throw landed in a caught console.error, and the agent's real
stdout was thrown away. Automation script output was invisible everywhere:
no execution row, and automation_run_device_results.output only ever held
the automation's own log lines ("[info] Queued run_script action").

The fix

Persist. executeRunScriptAction now mints a genuine script_executions
row and passes its uuid as the payload's executionId, so the existing result
handler persists stdout/stderr/exitCode — and inherits the terminal-status
guard, the #2434 secret redaction, and the stale-command reaper for free.
org_id is the device's org (a partner-wide automation owns no org of its
own, #2133). trigger_type gets a new 'automation' value.

Correlate. New nullable bare script_executions.automation_run_id uuid

• partial index. Deliberately no FK / no Drizzle .references():
  schema/automations.ts already imports schema/scripts.ts, so the reverse
  reference would close an import cycle — mirroring the existing bare
  automation_runs.config_policy_id. A side benefit is that script_executions
  stays out of automation_runs' FK graph, so the children-before-parents
  org-cascade ordering contract is untouched.

Render. GET /automations/runs/:runId attaches per-device scriptResults,
and the run-history expanded panel gains a collapsible per-script
stdout/stderr block.

Races and volume, handled

• Terminal-status stomp. queueCommandForExecution delivers over the
  WebSocket synchronously, so a fast agent can drive the row completed
  (with its stdout) before we return. Both post-queue transitions are now
  guarded on the row still being pending; without that, an unguarded write
  would flip a completed row back to queued and the stale-command reaper
  would later mark it timeout.
• Delivery vs. enqueue. On actual delivery the row goes to running with
  a startedAt, mirroring services/scriptExecution.ts, so the UI can derive
  a duration. Otherwise it's merely queued.
• Offline devices. queueCommandForExecution refuses any non-online
  device, and automation runs do not filter by device status. Checking up front
  avoids minting one failed execution row per offline device per run — a
  daily automation over a fleet of asleep laptops would otherwise drown the
  device's 50-row script history and skew per-script success stats.
• Abuse signal. resource.volume_outlier's scripts_24h counted every
  execution per partner. One hourly automation across a fleet would now pin
  that signal for a perfectly ordinary MSP, so it excludes
  trigger_type = 'automation' — scheduled machine activity with no human
  behind it.

Contracts

• CORE_TENANT_EXPORT_POLICY updated — the one cascade list that fires on a
  column add to an already-registered table. script_executions was
  already in CORE_ORG_CASCADE_DELETE_ORDER and both device lists; no FK means
  no new ordering constraint.
• RLS unaffected: automation runs execute inside withSystemDbAccessContext,
  under which breeze_has_org_access short-circuits, so the INSERT WITH CHECK
  policy passes. The new read inherits the same org policy as the existing
  device-results query.
• ALTER TYPE ... ADD VALUE is the only statement in migration -a-, per the
  per-file-transaction rule. -b- adds the column + index. Both sort after
  2026-08-13-*.
• The execute-script request schema still accepts only the original four
  trigger types — 'automation' is provenance the runtime mints, never
  something an API caller can forge. The ScriptExecution response schema
  (openapi + packages/shared) was widened.

Verification

• apps/api: 422 passed / 2 skipped across the affected suites
  (automationRuntime.*, routes/automations, routes/agentWs,
  routes/scripts, routes/devices/scripts, abuseSignals/,
  db/autoMigrate), incl. 9 new automationRuntime.runScript.test.ts cases.
• apps/web: 107 passed (components/automations/ incl. 4 new script-output
  cases, no-silent-mutations) + 96 i18n gate tests.
• tsc --noEmit clean for apps/api, apps/web, packages/shared.
• Not run locally (no DB): the RLS/integration contract suites and
  db:check-drift.

Known follow-ups (deliberately out of scope)

• The execute_command action's output is still discarded — it runs ad-hoc
  content with no scripts row, so it can't have an execution row, and needs
  a different persistence target. The run-history panel will show output for
  run_script actions and nothing for execute_command ones.
• staleCommandReaper times automation rows out on a fixed 300s rather than
  the script's own timeoutSeconds, and doesn't reconcile
  automation_run_device_results.
• Per-script stats (aiToolsScripts.ts) now blend manual and automation runs
  with no way to split them.

🤖 Generated with Claude Code

[cloudflare-workers-and-pages]

Deploying breeze with    Cloudflare Pages

Latest commit:	e6d680b
Status:	✅  Deploy successful!
Preview URL:	https://9e37ca25.breeze-9te.pages.dev
Branch Preview URL:	https://fix-3162-automation-script-s.breeze-9te.pages.dev

View logs

[ToddHebebrand]

Review run: /pr-review-toolkit:review-pr — code-reviewer, silent-failure-hunter, pr-test-analyzer, comment-analyzer (plus one independent design review before the PR was opened).

Findings: 0 CRITICAL. 8 IMPORTANT + 6 factually-wrong comments raised → all addressed in ebdb0a7; 0 outstanding. The substantive ones:

• execute_command kept a synthetic executionId for a reason that turned out to be false. I had commented that the Go agent requires the field; it does not — handleScript keys the execution on cmd.ID, and only script_cancel reads executionId. So the field is now dropped entirely, removing the last producer of non-uuid ids and the per-device-per-run warn spam.
• The offline pre-check was wrong. It read the run's device snapshot (taken once at the top of a fleet run, so minutes stale), silently skipping devices that came online mid-run — and diverging from execute_command, which still got the live check. Replaced: the row is minted, and discarded if the command never reaches the queue. Same "no row for an offline device" outcome, no stale read.
• A queue throw stranded the row. queueCommandForExecution does several DB round-trips plus a JIT decrypt and can throw rather than return {error}, leaving the row at pending for the reaper to relabel timeout — "no response from agent", when no agent ever had it. Now wrapped.
• The run-detail response inlined uncapped stdout. It is re-polled every few seconds while a run is live and stdout is accepted up to 5MB per execution. Now truncated in SQL (16k/8k) with explicit stdoutTruncated/stderrTruncated flags.
• The UI would have shipped the same ambiguity the issue is about. A pending/queued/running execution rendered the empty-output placeholder — indistinguishable from "the script printed nothing". And because a run goes terminal as soon as its commands are queued, the parent's poll had already stopped by the time stdout arrived, so the first view froze until you collapsed and re-expanded the row. Both fixed.
• The catch that hid this bug is no longer blind. handleScriptResult's catch now calls captureException like its SNMP sibling.
• Six of my own comments were verifiably wrong (agent requirement, reaper joining device_commands, "LEFT JOIN" vs equality filter, queued vs running as the stompable state, "mirrors the manual path", and the abuse-signal rationale — the commands_24h axis is unfiltered, so excluding automations from the scripts axis does not de-noise the signal). All corrected.

Tests: apps/api 426 passed / 2 skipped across the affected suites (automationRuntime.*, routes/automations, routes/agentWs, routes/scripts, routes/devices/scripts, abuseSignals/, db/autoMigrate). apps/web 207 passed (components/automations/, i18n gates, no-silent-mutations). tsc --noEmit clean for apps/api, apps/web, packages/shared.

⚠️ CI has not run — GitHub Actions is in a major outage. The pull_request trigger dropped for this branch (no workflow run exists), gh workflow run returns HTTP 500, and other PRs' runs are queued 1h+. gh pr checks shows only Cloudflare Pages, which reads misleadingly like a short green list. Please confirm a real CI run before merging.

Not verified locally: the RLS/integration contract suites and db:check-drift — they need a live DB, and the Postgres containers on this machine belong to other worktrees. CORE_TENANT_EXPORT_POLICY gained automation_run_id (the one cascade contract that fires on a column add), and that suite is non-blocking on PRs but required on main, so it's worth running before merge.

Known follow-ups, deliberately out of scope:

• execute_command output is still discarded — ad-hoc content with no scripts row, so it can't have an execution row and needs a different target.
• A BullMQ execute-run retry re-executes actions and mints duplicate execution rows (executeAutomationRunInner has no run-status short-circuit). Pre-existing for device_commands, but these rows are now user-visible.
• staleCommandReaper times automation rows out on a fixed 300s rather than the script's own timeoutSeconds, and doesn't reconcile automation_run_device_results.

Status: review-clean, awaiting maintainer merge — gated on a real CI run once Actions recovers.