Skip to content

fix(orchestrations): pin a run to the graph it started on - #892

Open
arantespp wants to merge 1 commit into
mainfrom
claude/epic-next-steps-rt38qu
Open

fix(orchestrations): pin a run to the graph it started on#892
arantespp wants to merge 1 commit into
mainfrom
claude/epic-next-steps-rt38qu

Conversation

@arantespp

Copy link
Copy Markdown
Member

Closes #872. First step of the epic in #883.

Summary

Orchestration graphs had no versioning and runs were not pinned to the graph they started on, so all four execution entry points re-read the live Orchestration row:

Entry point Location
driveQueuedRun orchestrationEngine.ts
wakeRun orchestrationEngine.ts
resumeOrchestrationRunExecution orchestrationEngine.ts
redriveRun orchestrationEngine.ts

An update-orchestration therefore silently re-shaped runs already executing — including runs parked for days in sleeping or awaiting_input — with no error and no record of which topology actually ran. node_executions could reference node ids from a graph that no longer existed.

This adds OrchestrationVersion on the shared archive engine landed in #880 and resolves every execution through the run's pinned version.

Per #883, it skips the run-level JSONB snapshot that #877 listed as its step 1. That stopgap was justified by the shared engine not existing yet; #877's own step 3 already turned the snapshot into a version reference, so building it now would have been write-then-replace work.

Changes

File Change
postgresdb/models/OrchestrationVersion.ts New archive table, same column layout as AgentVersion / GuardrailVersion
postgresdb/models/Orchestration.ts version column (default 1) + HasMany to the archive
postgresdb/models/OrchestrationRun.ts orchestrationVersion — the pin
lib/orchestrationVersionSnapshot.ts Config projection + makeVersionStore (write side)
lib/orchestrationVersions.ts makeVersionArchive adapters — list / get / restore
lib/orchestrationRunGraph.ts resolveRunGraph — the single resolution seam
lib/orchestrations.ts Archive on create, on graph change, and delete with the parent
lib/orchestrationEngine.ts Four call sites resolve through the pin; start-orchestration-run stamps it
rest/v1/orchestrationVersions.ts Three routes, hanging off the orchestrations router the way agentVersions.ts hangs off agents

The single seam is the point. Each of the four sites previously did orch.nodes as OrchestrationNode[]; fixing three of them would have looked correct in review and left the bug in the path a run actually parks in.

The versioned surface is nodes, edges, state_schema, input_schema. Name and description stay metadata, for the same reason as a guardrail's: two version numbers must not denote the same topology, and the version number is exactly what a run cites.

API surface

  • Orchestration.version, OrchestrationRun.orchestration_version
  • version_label on create / update, matching agents and guardrails
  • GET /orchestrations/{id}/versions, GET .../versions/{version}, POST .../versions/{version}/restore
  • Three new IAM actions: orchestrations:ListOrchestrationVersions, GetOrchestrationVersion, RestoreOrchestrationVersion
  • MCP picks all three up automatically from the spec (verified against a live server)

Degradation, deliberately

Runs created before pinning existed carry a null orchestration_version and keep executing the live graph — the only graph they ever had. Refusing to drive them would strand every run in flight across the deploy. A pinned version whose archive row is missing degrades the same way rather than losing the run's work; it is unreachable through the API (a version is archived before any run can pin it, and versions are deleted only with their orchestration, which deletes its runs in the same transaction). Both paths log which graph they chose, and both are covered by tests.

Verification

Red/green throughout. The pinning tests were proven to fail before the fix by forcing resolveRunGraph to return the live graph — exactly the 5 pinning assertions went red while the 2 degradation tests stayed green.

  • pnpm typecheck — clean (server, sdk, cli)
  • pnpm eslint --fix — clean
  • Server suite: 181 suites / 4983 tests pass, coverage thresholds met
  • @soat/postgresdb (15), @soat/cli (57), @soat/app (279) — all pass
  • pnpm lint-openapi — no errors; docs-lint OK; @soat/website build — no broken links or anchors

New tests:

  • tests/unit/tests/rest/orchestrationVersions.test.ts (28) — archiving, no-op writes, JSON Logic round-trip, list/get/restore, 400/401/403/404
  • tests/unit/tests/lib/orchestrationRunPinning.test.ts (7) — one per entry point, each driven through its real entry point (wakeDueRuns, reapOrphanedRuns, POST /orchestration-runs/:id/resume, the worker's driveQueuedRun), with the graph edited after the run was created. Each graph's second node writes a marker into state, so the assertion names which topology ran rather than merely that the run finished.

Live QA against a running server reproduced #872's own scenario: an async run parked on a 20s delay, the graph rewired to v2 while it slept, and the woken run finished on v1{"status":"succeeded","orchestration_version":1,"answer":"v1"}. Restore-appends and metadata-only-archives-nothing were confirmed the same way.

Smoke tests were not run — Docker is unavailable in this environment. The steps are added to tests/smoke-tests.sh (version list/get/restore, plus asserting the synchronous run pins to the current version) and CI's smoke-test job is the verifier. The script passes sh -n.

Deployment note

orchestration_versions is a new table and orchestrations.version / orchestration_runs.orchestration_version are new columns, all created by sync --alter. No backfill is needed or wanted: existing orchestrations get version = 1 with no archived row, and existing runs get a null pin — together that is exactly the documented degradation path, so in-flight runs continue on the live graph and the first graph edit after deploy archives normally. No index was renamed or removed.

Open questions, resolved

Q: Snapshot nodes/edges onto the run (#877 step 1) or take #872 straight onto OrchestrationVersion?
A: OrchestrationVersion — resolved by long-term (durability ladder + debt containment).
   The stopgap's only justification was the absence of a shared engine, and #877
   step 3 already converts the snapshot into a version reference. Checked: #880
   shipped `resourceVersions.ts` with two working adapters, so the "small, ships
   fast" claim no longer holds.

Q: Store the pin as a version number or a FK to orchestration_versions?
A: Number — resolved by pareto. It is what the wire exposes and what an audit
   reader cites, it matches `Generation.agentVersion`, and the archive row is
   reachable from it with no join. Checked: no query in the codebase needs to
   join runs to version rows.

Q: Is `name` part of the versioned config?
A: No, metadata — resolved by long-term (pattern hygiene). Follows the guardrail
   precedent: bumping a version on a rename makes two version numbers denote the
   same topology, which is the one thing the number must not do. Checked:
   `GuardrailVersion` versions only `document`; agents version name because an
   agent's whole mutable surface is its behavior.

Q: Also guard `update-orchestration` against edits while runs are active, as #872
   suggests as a minimum?
A: No — resolved by pareto. Pinning is the complete fix, and a guard would remove
   a capability the design deliberately keeps: the live columns become a draft.
   Checked: no test or doc asserted that an edit during an active run should be
   refused.

Q: Should a `loop` / `sub_orchestration` child inherit its parent's pin?
A: No — forwarded as a documented boundary rather than silently chosen. A child is
   a new run of a *different* orchestration and pins that orchestration's current
   version. Freezing a whole nested pipeline would need a version map threaded
   through child dispatch; nothing has asked for it. Documented in
   `modules/orchestrations.md#versioning` so the semantics are not a surprise.

Not in this PR

Related: #877, #880, #882, #883

🤖 Generated with Claude Code

https://claude.ai/code/session_01PLPL6R4W6p4aLQ2rpiojSm


Generated by Claude Code

Orchestration graphs had no versioning and runs were not pinned, so all four
execution entry points re-read the live `Orchestration` row: `driveQueuedRun`,
`wakeRun`, `resumeOrchestrationRunExecution` and `redriveRun`. An
`update-orchestration` therefore re-shaped runs already executing — including
runs parked for days in `sleeping` or `awaiting_input` — with no error and no
record of which topology actually ran.

Adds `OrchestrationVersion` on the shared archive engine from #880 and resolves
every execution through the run's pinned version:

- `orchestrationVersionSnapshot.ts` projects the graph (`nodes`, `edges`,
  `state_schema`, `input_schema`) and owns the write side. Name and description
  stay metadata, as for a guardrail: two version numbers must not denote the
  same topology.
- `orchestrationVersions.ts` supplies the archive adapters — list, get, and a
  restore that appends rather than rewinding the counter.
- `orchestrationRunGraph.ts` is the single resolution seam. The four call sites
  each used to do `orch.nodes as OrchestrationNode[]`, so fixing three of them
  would have looked correct in review and left the bug in the path a run
  actually parks in.
- `OrchestrationRun.orchestrationVersion` records the pin, stamped at
  `start-orchestration-run` and exposed as `orchestration_version`.

Runs created before pinning existed carry a null version and keep executing the
live graph — the only graph they ever had. A pinned version whose archive row is
missing degrades the same way rather than losing the run's work.

Per #883, this skips the run-level JSONB snapshot #877 proposed as step 1: that
was justified by the shared engine not existing yet, and #877's own step 3
already turned the snapshot into a version reference.

No release/canary layer, per #883's third step. `releaseAssignment.ts` is
already extracted and pure for when something asks for it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PLPL6R4W6p4aLQ2rpiojSm
@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown

Deploy Outputs

Package Stack Output Key Output Value
@soat/website SoatWebsite-claude-epic-next-steps-rt38qu BucketWebsiteURL http://soatwebsite-claude-epic-next-steps-rt-staticbucket-rgutihd5sa1y.s3-website-us-east-1.amazonaws.com

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] Orchestration graphs are not versioned — editing an orchestration mutates in-flight runs (orchestrations)

2 participants