Skip to content

fix(runtime): reconcile workflow timeouts fail closed #1102

Description

@doublewhy

Gap Claim

Workflow timeout reconciliation catches malformed persisted timestamps as if no timeout occurred, and compares the timeout against the wrong clock value. Corrupt state can therefore remain active indefinitely or be reconciled inconsistently.

Existing Surface Audit

Reviewed control_plane_timeouts.py, persisted operation timestamps, timeout state transitions, restart reconciliation, and the timeout fixes currently bundled into PR #1086.

Scope

OpenRAE runtime reliability only. Requirements: RUN-317 and RUN-318.

Chosen Architecture

Parse persisted timestamps through one strict boundary, reject malformed or future-inconsistent values with a stable fail-closed diagnostic, and compare monotonic elapsed duration against the configured workflow timeout without confusing wall-clock timestamps with durations.

Verification Plan

Cover malformed timestamps, timezone offsets, future timestamps, exact boundaries, elapsed-over-limit transitions, restart reconciliation, and stable public diagnostics. Keep unrelated PR #1086 changes outside this issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions