Skip to content

fix(runtime): settle concurrent participant batches reliably - #1135

Open
doublewhy wants to merge 3 commits into
devfrom
RUN-308-scheduler-successor
Open

fix(runtime): settle concurrent participant batches reliably#1135
doublewhy wants to merge 3 commits into
devfrom
RUN-308-scheduler-successor

Conversation

@doublewhy

@doublewhy doublewhy commented Aug 12, 2026

Copy link
Copy Markdown

Plain-language summary

Context: OpenRAE can ask several simulated participants to act at the same time. Its scheduler must still keep one trustworthy view of who is running, what changed, and how much execution capacity remains.

Problem: If one concurrent batch failed partway through, the scheduler could leave capacity stuck, erase accounting from earlier work, retry an action even though its external effect was uncertain, or let one backend result corrupt another participant's state. The old path also repeated expensive whole-snapshot work as a batch grew.

Fix: This change makes the safety boundary explicit. Before any action is sent, a failed batch can be rolled back exactly. After sending begins, every submitted action is settled without blind retry. Results are isolated and merged in a stable order, scheduler-owned state stays protected, and only this batch's capacity is released. The same path handles an 800-participant run without recursive or repeated full scans.

Closes #1101

What changes

  • Separates pre-dispatch rollback from post-dispatch indeterminate settlement. A backend exception, cancellation, or unpairable result count can no longer make an uncertain action due again.
  • Gives every worker an isolated predecessor and freezes returned envelopes before serialized commit, including repeated-object and mutation-after-return cases.
  • Assigns every RuntimeSnapshot field to either backend merge ownership or scheduler protection. New fields fail the ownership guard until their concurrent semantics are explicit.
  • Three-way merges backend-owned values in deterministic request order, rejects revision conflicts with stable redacted diagnostics, and still settles already-dispatched peers before applying failure_policy: stop.
  • Preserves pre-existing participant and execution-service work through delta accounting. In-flight participants are not selected twice, and exhausted capacity produces an explicit diagnostic instead of silent non-progress.
  • Stages intermediate snapshots and materializes the complete invariant once at the batch boundary. Due work is scanned once and processed with an iterative capacity-bounded loop.
  • Adds the architecture/non-claim decision record and RUN-308 issue, code, and test traceability.

Current-base evidence

Verified after rebasing onto dev at 96b20ae8422fae9936bae928bbd8a1c14d2001da:

  • Focused scheduler, concurrency, and participant-contract suite on Python 3.12.13: 154 passed, including the real 800-participant iterative reservation/settlement case.
  • Merged-boundary regression set for the 800-participant path, workflow-timeout reconciliation from fix(runtime): reconcile workflow timeouts fail closed #1132, and solver deadline enforcement from fix(processor): enforce solver deadline boundaries #1142: 68 passed.
  • Changed production-code coverage: 476/476 executable lines and 134/134 branch exits, with no changed-code exclusions.
  • Ruff 0.15.9 formatting and lint across the Python project and repository tooling: passed.
  • Repository policy and ADR governance: passed. Requirement governance reached the configured check but skipped locally because the Ground Control hostname did not resolve.
  • Rebase range-diff: all three commits are patch-identical to the reviewed pre-rebase series.

Hosted CI and Sonar passed for exact head 17dc65cd36e56fe2d873d68401325ee262c4b1a6 on base 96b20ae8422fae9936bae928bbd8a1c14d2001da.

Review boundary

This PR is deliberately limited to concurrent participant scheduling and its scale path. It does not change HTTP/API admission, durable workflow storage, the workflow-timeout implementation merged in #1132, the solver deadline implementation merged in #1142, dependency-cycle detection, or resolved-realization authority. It does not claim that native side effects can be rolled back after dispatch.

The rationale, alternatives, field-ownership boundary, and nonclaims are recorded in docs/decisions/issue-1101-concurrent-participant-rollback-preflight.md.

Base and provenance

  • Base: dev at 96b20ae8422fae9936bae928bbd8a1c14d2001da.
  • Head: 17dc65cd36e56fe2d873d68401325ee262c4b1a6.
  • Three focused commits reconstructed from the terminal scheduler lineage; no commits from the old integration branch are merged wholesale.

@doublewhy
doublewhy marked this pull request as ready for review August 12, 2026 21:14
@doublewhy
doublewhy marked this pull request as draft August 13, 2026 04:01
Treat pre-dispatch reservation failures as exact rollback and post-dispatch indeterminate work as non-retryable settlement. Protect scheduler-owned snapshot fields, merge backend state deterministically, preserve unrelated service counters, and process capacity-bounded chunks iteratively.

Stage shared snapshots once per batch so mutation isolation remains linear in participant count.

Requirement: RUN-308

Refs: #1101
Split concurrent commit and dispatch phases into focused modules, carry indeterminate settlement state as one cohesive value, and cover every entry branch without changing dispatch semantics.\n\nRequirement: RUN-308\n\nRefs: #1101
@doublewhy
doublewhy force-pushed the RUN-308-scheduler-successor branch from c26cc57 to 17dc65c Compare August 13, 2026 04:06
@doublewhy
doublewhy marked this pull request as ready for review August 13, 2026 04:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant