Skip to content

fix(runtime): reconcile workflow timeouts fail closed - #1132

Merged
Brad-Edwards merged 7 commits into
devfrom
RUN-317-workflow-timeout-pr
Aug 13, 2026
Merged

fix(runtime): reconcile workflow timeouts fail closed#1132
Brad-Edwards merged 7 commits into
devfrom
RUN-317-workflow-timeout-pr

Conversation

@doublewhy

Copy link
Copy Markdown

Plain-language summary

  • Context: The control plane periodically checks running workflows and marks them timed out only after their configured wall-clock limit has actually elapsed. A timeout may also trigger compensation work.
  • Problem: Malformed, offset-free, future, or non-monotonic persisted timestamps could be swallowed or misclassified, and floating-point rounding over very large spans could declare a timeout slightly early. That can either silently disable reconciliation or create false timeout and compensation evidence.
  • Fix: Treat persisted clocks as untrusted input, require explicit offsets and monotonic ordering, fail without changing state when evidence is invalid, and compare exact integer elapsed seconds before producing a timeout.

Issue closed

Closes #1102

What changed

  • Parses the reconciliation clock once and rejects malformed or naive values.
  • Requires started_at <= updated_at <= reconciliation clock.
  • Validates persisted workflow state and timeout configuration strictly.
  • Leaves history, status, and compensation unchanged when time evidence is invalid.
  • Uses exact whole-second arithmetic instead of float total_seconds() or overflow-prone timestamp addition.
  • Keeps terminal reconciliation idempotent across restart.

Verification

  • Focused timeout suite: 35 passed.
  • Timeout plus control-plane API selection: 37 passed, 31 deselected.
  • Changed production code: 100% executable-line and branch coverage.
  • Ruff, repository policy, governance traceability, and diff hygiene pass.
  • Canonical default suite on this macOS host: 6,430 passed, 1 skipped; the remaining 18 failures are unchanged current-dev libvirt tests that require /usr/bin/busybox and are addressed separately by fix(libvirt): bind reproducible guest operations to fresh evidence #1128.
  • Rebuilt on current dev at 3d6e369b726607ff657a841c7f1dee0bec655b8f.

Scope

This PR covers only workflow timeout proof and reconciliation. Concurrent participant reservations/settlement (#1101), HTTP admission, and durable operation storage are separate successors.

doublewhy and others added 6 commits August 12, 2026 12:58
…conciliation

`_workflow_has_timed_out` wrapped both timestamp parses in
`except Exception: return False`, so any unparseable value reported
"not timed out". A RUNNING workflow whose recorded `started_at` could
not be parsed therefore had no derivable deadline and stayed RUNNING
for the lifetime of the control plane: reconciliation could never
reclaim it, even decades past a one-second timeout.

The two timestamps have different scope, so they are now handled
differently:

- `submitted_at` is the caller's reconciliation clock and governs every
  workflow in the pass, so an unusable value raises instead of quietly
  disabling all timeouts. The HTTP adapter already maps `ValueError` to
  409 for this route.
- A per-workflow `started_at` that cannot be parsed no longer blocks
  reclamation; the workflow is timed out under a distinct terminal
  reason so it stays diagnosable rather than looking like an ordinary
  timeout.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`timeout_seconds` carries no declared upper bound, so adding it to the
start instant overflowed for a very large value — as a float timestamp
and as a `timedelta`. The previous blanket `except Exception` hid that as
"not timed out"; with the exception handling narrowed, it would instead
abort the whole reconciliation pass and surface as a 500. Elapsed time is
now compared against the timeout, which Python evaluates exactly for an
arbitrarily large integer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants