Skip to content

fix(runtime): make local operation state transactional and durable - #1136

Draft
doublewhy wants to merge 6 commits into
devfrom
API-404-durable-store-successor
Draft

fix(runtime): make local operation state transactional and durable#1136
doublewhy wants to merge 6 commits into
devfrom
API-404-durable-store-successor

Conversation

@doublewhy

@doublewhy doublewhy commented Aug 12, 2026

Copy link
Copy Markdown

Plain-language summary

  • Context: OpenRAE keeps local control-plane snapshots, operation status, idempotency claims, and audit history so an interrupted run can be inspected and retried safely.
  • Problem: The previous JSON store could lose concurrent writes, and a crash could leave backend effects, the snapshot, and operation status disagreeing. A second local process could also act from stale cached state.
  • Fix: Store local state in a transactional SQLite WAL database, commit terminal snapshots and records together, recover interrupted operations without replaying unknown backend effects, and admit exactly one local runtime owner at a time.

Closes #1092

Part of #8.

What changes

  • replaces whole-file JSON writes with indexed SQLite transactions and unique idempotency claims;
  • requires WAL and durable synchronization, validates SQLite files without retaining application-owned database descriptors, and synchronizes legacy backups before admission;
  • migrates legacy JSON under an exclusive lock, verifies the imported state, preserves the original files, and retains a timestamped backup;
  • persists the operation claim before backend execution, atomically commits the resulting snapshot and terminal record, and marks orphaned non-terminal records failed on restart without replaying the backend;
  • holds a process-bound lease for the lifetime of the runtime and drains active calls before release;
  • preserves the exact legacy custom-store protocol through one explicitly non-crash-atomic, deprecated compatibility seam;
  • publishes participant episode closure records in runtime-snapshot-v1 so the public contract matches the exhaustive durable snapshot codec.

Current-base integration

This revision is rebased on dev at 96b20ae8422fae9936bae928bbd8a1c14d2001da; its head is 1f7a01423dbcc6a8827ad62dbe1b31f8993597f6. It retains the merged timeout (#1132), OCI registry (#1130), authored OS realization (#1141), and solver-operation (#1142) behavior. The durability test provisioner now declares a coherent OS-capable envelope and binds guest-observed OS evidence through the production binding helper required by #1141; runtime admission was not weakened.

Scope boundary

This is a single-host, single-process local reference store. It does not claim distributed consensus, multi-worker cache coherence, durable job scheduling, backend-effect exactly-once guarantees, or request offload. HTTP admission and scheduler work remain outside this PR; already-merged timeout, OCI, realization, and solver behavior is inherited rather than implemented here.

Verification

  • focused crash consistency, process ownership, lifecycle, compatibility, API, and snapshot regressions: 204 passed on Python 3.14 free-threaded;
  • standard Python 3.14 free-threaded suite excluding only test_mcp_server.py: 6,910 passed, 1 skipped;
  • the unfiltered sweep reached 6,911 passed, 1 skipped; its 84 failures are confined to the current-base Python 3.14 free-threaded MCP event-loop incompatibility tracked by [BUG] MCP tests rely on removed implicit event-loop behavior on Python 3.14 #1117;
  • changed production coverage against 96b20ae: 1,088/1,088 executable lines and 278/278 branch exits;
  • full-tree Ruff format/check: passed;
  • repository policy against 96b20ae: passed; requirement governance used its documented Ground Control unavailable skip because DNS was unavailable;
  • schema publication, generated-schema drift, formal evidence replay, and the complete contracts graph: passed;
  • canonical hosted CI and Sonar passed on this exact head in run 31667112381.

@doublewhy

Copy link
Copy Markdown
Author

Fresh Sonar cleanup pushed in 8e95170d.

  • Replaced the over-broad BaseException catch with Exception, so process interrupts are not swallowed while ordinary reconciliation failures still poison the runtime.
  • Centralized the repeated database-file label without changing path validation.
  • Verification: 115 crash-consistency tests passed; both path modules reached 99% combined focused coverage (all changed lines covered and control_plane_store_paths.py at 100% line/branch); Ruff, repository policy, and diff hygiene passed.

The plain-language summary and Closes #1092 mapping are unchanged.

@doublewhy

Copy link
Copy Markdown
Author

Fresh Linux CI exposed and this update fixes one final WAL lifecycle race.

SQLite may unlink an ephemeral -wal/-shm path between pathname lookup and metadata return, yielding st_nlink == 0. The prior hard-link check rejected that benign disappearance and caused one of four concurrent writers to exit. Commit a6dcee0 now treats only zero-link sidecars as disappeared, still rejects multiple links, and keeps the main database pinned to exactly one link.

Verification: 115 crash-consistency tests pass; the affected path module is 100% line/branch covered; the original four-process write test passed 10 consecutive runs; Ruff, repository policy, and diff hygiene pass. Fresh CI has restarted on this exact SHA.

@doublewhy

Copy link
Copy Markdown
Author

Fresh Sonar analyzed the WAL-race head and reported one maintainability issue: the path-admission helper crossed the configured complexity threshold by one. Commit 5981ebe extracts link-count admission into a focused helper without changing behavior or suppressing the rule.

The full 115-test crash-consistency suite still passes, the path module remains 100% line/branch covered, and Ruff plus repository policy pass. Fresh CI/Sonar has restarted on this exact head.

@doublewhy
doublewhy marked this pull request as ready for review August 12, 2026 22:42
@doublewhy
doublewhy marked this pull request as draft August 13, 2026 04:10
@doublewhy
doublewhy force-pushed the API-404-durable-store-successor branch from 5981ebe to a8e9641 Compare August 13, 2026 04:19
@doublewhy
doublewhy force-pushed the API-404-durable-store-successor branch from a8e9641 to 1f7a014 Compare August 13, 2026 04:28
@doublewhy
doublewhy marked this pull request as ready for review August 13, 2026 04:41
@doublewhy
doublewhy marked this pull request as draft August 13, 2026 08:00
@doublewhy

Copy link
Copy Markdown
Author

Temporarily returned to draft under the breaking-change freeze. This PR changes persistence, worker/lifecycle behavior, and durable snapshot handling. Preserve the verified branch, but do not merge until the maintainer explicitly resumes runtime-semantic changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant