feat(orchestrations): give durable runs an identity and fail soat calls loudly

[arantespp]

Why

A soat tool node reaches the REST API over a loopback HTTP call, which carries an Authorization header only when the run is driven inline (wait: true). Every durable path — a queued start, a scheduler wake, a redrive after a crash, an awaiting_input resume — drove with no header at all, so the self-call came back 401.

Worse, executeSoatTool returned the response body whatever its status, so that 401 became the node's artifact and the run succeeded, as though the action had happened. A task-dispatched run is always durable (tasksDispatch.ts, deliberate per #855), so an orchestration dispatched by a workflow state could never move its own task — silently.

What changed

1. A non-2xx self-call fails the tool call. It throws HttpToolError, mapped at the call boundary to TOOL_HTTP_ERROR, exactly as an http tool's target rejecting a call already is. The upstream status survives in meta.tool_status_code — which isRetriableError already reads, so a terminal 4xx isn't retried. HttpToolError moves into its own module so both throw sites can import it without an import cycle.

2. A run carries an identity. It persists the principal that started it (principal_kind / principal_id — internal columns, not exposed in the API), and each background drive re-mints a short-lived run-as token from it, mirroring the existing triggerToken.ts. The token asserts identity only: authorization is still evaluated per call against current policies, so revoking access reaches a run already in flight — which a token minted once at run start and stored would not.

Keeping a run from out-reaching the credential that started it drove the rest:

• An API-key-started run carries a key claim, resolved into that key's policies as a boundary. A revoked key stops the run acting rather than falling back to its owner's broader access.
• Trigger- and OAuth-started runs record no principal. Their boundary lives in the token (the trigger's policy, the consented scope), not in the principal, so re-minting would silently drop it. They keep executing inline with the original token, exactly as today.
• Run tokens are marked with an orn claim. Without a marker the middleware can't tell one from an OAuth access token and builds a consent boundary from an absent scope claim — a policy that allows nothing. (Found by a test, not by reading.)

3. The identity survives a chain of states. Task automation threads the principal through: the task's creator, the principal that fired a transition, and for an automation-fired hop the principal of the run that routed it there. Without that last step a chain works once and then decays to no principal at the second state.

Testing

Tests drive the real loopback, binding the worker's own port the way mcp.test.ts does — without a listener the status under test never happens.

• a non-2xx action surfaces 502 TOOL_HTTP_ERROR with tool_status_code: 404 instead of returning the error body
• a background-driven run's soat node returns real, project-scoped data
• a run started by a read-only API key lists fine but is refused (403) on a write, and no resource is created — no escalation to the owning user
• a workflow-dispatched orchestration transitions its own task end to end, the loop this whole change exists for
• lib/orchestrationRunToken.test.ts covers the branches no entry point reaches: a deleted user, a revoked key, and trigger/OAuth/plain tokens that must not be mistaken for run tokens

Full server suite: 4916/4917 passing. The one failure (files.test.ts, upload_url returning an absolute URL) reproduces on a clean main in this environment and is unrelated. pnpm typecheck, pnpm eslint --fix, pnpm docs-lint, and the @soat/postgresdb suite (including schemaDrift) all pass.

Not run: smoke tests and tutorials — both need Docker, which isn't available in this environment. No smoke steps were added: this fixes existing behavior rather than adding a user-facing flow, and I'd rather not add assertions I couldn't execute.

Notes for review

• No API surface change — the principal columns are internal, so no OpenAPI / SDK / CLI regeneration, no new permission actions, no formation schema change.
• Attribution nuance: a key-started run acts as the key's owner for attribution purposes (principal_kind: user), while still bounded by the key's policies. Setting apiKeyPublicId on a JWT-derived auth context would name the key precisely but risks confusing code that assumes the scoped-key identity fields travel together.
• The composed cycle is still unbounded. A state whose orchestration transitions the task back into that same state loops forever: cycle detection is per-graph and workflow cycles are deliberate, so nothing rejects it. Documented in orchestrations.md; a run-depth or transition budget would be the fix if it ever bites.
• Per your call, no dedicated task_transition node — the docs now recommend a tool node bound to a soat tool for create-task / transition-task, and spell out that the edge should stay fire-and-forget (a graph waiting on a task inverts the two lifetimes).

Open questions, self-resolved

Q: run-scoped token vs. persisted principal re-evaluated at execution time?
A: persisted principal — resolved by long-term; checked: OrchestrationRun had no principal column, and authUser.resolveProjectIds is already an in-process IAM evaluation, so no new secret material is stored.

Q: how should a nested (loop / sub_orchestration) child record its identity?
A: read it back from the parent's run token — resolved by pareto; checked: children are handed the parent's authHeader and run `wait: true`, so the only path that loses identity is a crash-redrive, and threading a principal arg through every node executor would have been a far wider diff for the same guarantee.

Q: should trigger- and OAuth-started runs get run tokens too?
A: no — resolved by long-term; checked: resolveScopedBoundaryDocs derives their boundary from the `trg` / `scope` claim, so a re-minted plain run token would drop it and widen the run's access.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UTVN32qiBQxeS5o4PM8M4D

---

Generated by Claude Code

[github-actions]

Deploy Outputs

Package	Stack	Output Key	Output Value
@soat/website	SoatWebsite-claude-workflow-call-orchestration-m6agzw	BucketWebsiteURL	http://soatwebsite-claude-workflow-call-orch-staticbucket-1wbv6uabepj4.s3-website-us-east-1.amazonaws.com

[arantespp]

CI is green. Noting the one red job in the run history so it isn't mistaken for a real signal:

Tutorials Tests failed on the first attempt — client-tools step 17, where the generation came back status: "completed" / required_action: null instead of pausing on the forced get_order_status client tool. It passed unchanged on re-run.

Before re-running I checked it wasn't mine rather than assuming:

• Every path this PR changes is specific to soat tools, orchestration runs, task automation, or project-scoped token boundaries. That tutorial authenticates with a plain admin JWT, uses a client tool, and involves no orchestration — and the forcing itself is decided in ollama-tool-choice from the request's tool_choice field, which this diff never reaches.
• The base isn't the cause either: the PR that went green ~4 minutes earlier shares this one's base (both include fix(agents): fail a generation whose answer is a tool call written as text #870).
• Smoke Tests passed on the first attempt, which is the suite that actually exercises the changed behavior — agent-invoked soat tools now throw on a non-2xx self-call instead of returning the error body.

That leaves it as the residual nondeterminism in this step that .claude/rules/tests.md already documents for client-tools (#774). Flagging rather than silently re-running: if it recurs on an unrelated PR it's worth treating as a real gap in the tool_choice shim, not a flake to retry.

---

Generated by Claude Code