fix(orchestrations): pin a run to the graph it started on - #892
Open
arantespp wants to merge 1 commit into
Open
Conversation
Orchestration graphs had no versioning and runs were not pinned, so all four execution entry points re-read the live `Orchestration` row: `driveQueuedRun`, `wakeRun`, `resumeOrchestrationRunExecution` and `redriveRun`. An `update-orchestration` therefore re-shaped runs already executing — including runs parked for days in `sleeping` or `awaiting_input` — with no error and no record of which topology actually ran. Adds `OrchestrationVersion` on the shared archive engine from #880 and resolves every execution through the run's pinned version: - `orchestrationVersionSnapshot.ts` projects the graph (`nodes`, `edges`, `state_schema`, `input_schema`) and owns the write side. Name and description stay metadata, as for a guardrail: two version numbers must not denote the same topology. - `orchestrationVersions.ts` supplies the archive adapters — list, get, and a restore that appends rather than rewinding the counter. - `orchestrationRunGraph.ts` is the single resolution seam. The four call sites each used to do `orch.nodes as OrchestrationNode[]`, so fixing three of them would have looked correct in review and left the bug in the path a run actually parks in. - `OrchestrationRun.orchestrationVersion` records the pin, stamped at `start-orchestration-run` and exposed as `orchestration_version`. Runs created before pinning existed carry a null version and keep executing the live graph — the only graph they ever had. A pinned version whose archive row is missing degrades the same way rather than losing the run's work. Per #883, this skips the run-level JSONB snapshot #877 proposed as step 1: that was justified by the shared engine not existing yet, and #877's own step 3 already turned the snapshot into a version reference. No release/canary layer, per #883's third step. `releaseAssignment.ts` is already extracted and pure for when something asks for it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PLPL6R4W6p4aLQ2rpiojSm
Deploy Outputs
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #872. First step of the epic in #883.
Summary
Orchestration graphs had no versioning and runs were not pinned to the graph they started on, so all four execution entry points re-read the live
Orchestrationrow:driveQueuedRunorchestrationEngine.tswakeRunorchestrationEngine.tsresumeOrchestrationRunExecutionorchestrationEngine.tsredriveRunorchestrationEngine.tsAn
update-orchestrationtherefore silently re-shaped runs already executing — including runs parked for days insleepingorawaiting_input— with no error and no record of which topology actually ran.node_executionscould reference node ids from a graph that no longer existed.This adds
OrchestrationVersionon the shared archive engine landed in #880 and resolves every execution through the run's pinned version.Per #883, it skips the run-level JSONB snapshot that #877 listed as its step 1. That stopgap was justified by the shared engine not existing yet; #877's own step 3 already turned the snapshot into a version reference, so building it now would have been write-then-replace work.
Changes
postgresdb/models/OrchestrationVersion.tsAgentVersion/GuardrailVersionpostgresdb/models/Orchestration.tsversioncolumn (default 1) +HasManyto the archivepostgresdb/models/OrchestrationRun.tsorchestrationVersion— the pinlib/orchestrationVersionSnapshot.tsmakeVersionStore(write side)lib/orchestrationVersions.tsmakeVersionArchiveadapters — list / get / restorelib/orchestrationRunGraph.tsresolveRunGraph— the single resolution seamlib/orchestrations.tslib/orchestrationEngine.tsstart-orchestration-runstamps itrest/v1/orchestrationVersions.tsagentVersions.tshangs off agentsThe single seam is the point. Each of the four sites previously did
orch.nodes as OrchestrationNode[]; fixing three of them would have looked correct in review and left the bug in the path a run actually parks in.The versioned surface is
nodes,edges,state_schema,input_schema. Name and description stay metadata, for the same reason as a guardrail's: two version numbers must not denote the same topology, and the version number is exactly what a run cites.API surface
Orchestration.version,OrchestrationRun.orchestration_versionversion_labelon create / update, matching agents and guardrailsGET /orchestrations/{id}/versions,GET .../versions/{version},POST .../versions/{version}/restoreorchestrations:ListOrchestrationVersions,GetOrchestrationVersion,RestoreOrchestrationVersionDegradation, deliberately
Runs created before pinning existed carry a null
orchestration_versionand keep executing the live graph — the only graph they ever had. Refusing to drive them would strand every run in flight across the deploy. A pinned version whose archive row is missing degrades the same way rather than losing the run's work; it is unreachable through the API (a version is archived before any run can pin it, and versions are deleted only with their orchestration, which deletes its runs in the same transaction). Both paths log which graph they chose, and both are covered by tests.Verification
Red/green throughout. The pinning tests were proven to fail before the fix by forcing
resolveRunGraphto return the live graph — exactly the 5 pinning assertions went red while the 2 degradation tests stayed green.pnpm typecheck— clean (server, sdk, cli)pnpm eslint --fix— clean@soat/postgresdb(15),@soat/cli(57),@soat/app(279) — all passpnpm lint-openapi— no errors;docs-lintOK;@soat/website build— no broken links or anchorsNew tests:
tests/unit/tests/rest/orchestrationVersions.test.ts(28) — archiving, no-op writes, JSON Logic round-trip, list/get/restore, 400/401/403/404tests/unit/tests/lib/orchestrationRunPinning.test.ts(7) — one per entry point, each driven through its real entry point (wakeDueRuns,reapOrphanedRuns,POST /orchestration-runs/:id/resume, the worker'sdriveQueuedRun), with the graph edited after the run was created. Each graph's second node writes a marker into state, so the assertion names which topology ran rather than merely that the run finished.Live QA against a running server reproduced #872's own scenario: an async run parked on a 20s
delay, the graph rewired to v2 while it slept, and the woken run finished on v1 —{"status":"succeeded","orchestration_version":1,"answer":"v1"}. Restore-appends and metadata-only-archives-nothing were confirmed the same way.Smoke tests were not run — Docker is unavailable in this environment. The steps are added to
tests/smoke-tests.sh(version list/get/restore, plus asserting the synchronous run pins to the current version) and CI'ssmoke-testjob is the verifier. The script passessh -n.Deployment note
orchestration_versionsis a new table andorchestrations.version/orchestration_runs.orchestration_versionare new columns, all created bysync --alter. No backfill is needed or wanted: existing orchestrations getversion = 1with no archived row, and existing runs get a null pin — together that is exactly the documented degradation path, so in-flight runs continue on the live graph and the first graph edit after deploy archives normally. No index was renamed or removed.Open questions, resolved
Not in this PR
0.xbreaking-change bump withlerna version --dry-runbefore the next release) are per-deployment calls, not repo work.Related: #877, #880, #882, #883
🤖 Generated with Claude Code
https://claude.ai/code/session_01PLPL6R4W6p4aLQ2rpiojSm
Generated by Claude Code