fix(runtime): close control-plane, timeout, and scheduler failure-handling gaps

[doublewhy]

Plain-language summary

• Context: The runtime API must authorize the right target, bound request memory, and settle scheduled work even when backends fail.
• Problem: Several independent failure paths could admit the wrong authorization scope, buffer too much input, leave reservations unsettled, or mishandle timeouts. Combining all seven defects in one PR also makes the safety boundaries difficult to review.
• Fix: Fail closed at each boundary and add regression coverage; focused successor PRs are replacing this bundled branch issue by issue.

Related issues

Closes #1090
Closes #1091

Related follow-up hardening remains tracked in #1101, #1102, and #1103.

---

Seven defects in the runtime control plane and the planner. Two are authentication flaws, one is a denial-of-service opening, three are reliability failures that leave state stuck, and one is a scalability limit.

#	What breaks	Severity
1	A credential issued for one experiment authenticates against another	auth bypass
2	A revoked credential keeps working, and the rejection is never audited	auth bypass
3	An upload with no declared size can exhaust memory	denial of service
4	A workflow can become permanently unstoppable	stuck state
5	A misbehaving backend permanently blocks its participants	stuck state
6	A long-enough timeout crashes timeout reconciliation	crash
7	A large scenario crashes planning instead of being planned	crash

Nothing under contracts/schemas/, specs/, _version.py or CHANGELOG.md is touched, so no published schema, canonical digest, signature, or evidence bytes change.

---

1. A credential issued for one experiment authenticates against another

Each control plane serves one target (one experiment apparatus). An identity can be scoped to a target so a credential for experiment A is useless against experiment B. That scoping was enforced for callers identified by proxy header, but skipped entirely for bearer tokens — the bearer branch returned its identity and never checked.

So a token minted for target A authenticated against target B's control plane. Run one control plane per experiment and a credential for one grants access to the others.

I enumerated the full decision space — all 72 combinations of (token, identity header, verified header, proxy-trust flag) — and ran it against both the old and new code:

• the flaw authenticated in 18 of 72 configurations
• including with trust_proxy_identity_headers=False, so it was not gated by the proxy-trust setting at all
• after the change, 0 configurations newly authenticate (19 stop, all of them these two flaws)

Both paths now share _require_target_binding.

2. A revoked credential keeps working, and the rejection is never audited

If a caller presented a bearer token that did not resolve, the code did not reject it. It fell through to the proxy-header path and tried to authenticate the request another way.

Two consequences. Where header identities are trusted, a revoked or rotated token keeps working — the revocation appears to have no effect. And because nothing was rejected, the failed credential never reached the audit log, so an operator reviewing the trail sees no sign of it. That matters for a system whose purpose is auditable evidence.

A presented token that does not resolve is now a 401.

Also here: tokens are compared in constant time over encoded bytes (a non-ASCII token previously raised instead of reporting unauthorized), and the security config's mappings are read-only so strict_defaults() cannot be granted principals or tokens after construction.

3. An upload with no declared size can exhaust memory

There is a request-size limit. It was enforced by reading the entire body into memory and then measuring it.

A request that declares content-length is caught by a separate header check. A request that does not declare one — Transfer-Encoding: chunked — is invisible to that check, so the only limit was the one that had already buffered everything. The response was still 413, but the memory was already gone.

The body is now measured chunk by chunk and refused as soon as the running total crosses the limit, before the offending chunk is copied. Starlette's body cache is seeded so route parsing is unchanged; _CachedRequest.wrapped_receive replays it to the inner app, which is the framework's intended hook for middleware that reads the body.

4. A workflow can become permanently unstoppable

Timeout reconciliation parsed two timestamps inside except Exception: return False. Any unparseable value therefore reported "not timed out".

A running workflow whose recorded started_at cannot be parsed has no derivable deadline, so it reported "not timed out" every time, forever. I verified one still RUNNING 30 years past a one-second timeout. No operator action reclaims it, because reconciliation is the mechanism that would.

The two timestamps have different scope, so they are now handled differently:

• submitted_at is the caller's clock for the whole pass. An unusable value raises, rather than silently disabling every timeout at once. The HTTP route already maps ValueError to 409.
• A per-workflow started_at that cannot be parsed no longer blocks reclamation. The workflow times out under a distinct terminal reason, so it stays diagnosable instead of looking like an ordinary timeout.

5. A misbehaving backend permanently blocks its participants

Before calling a backend to run a batch of participant actions, the scheduler reserves capacity for each participant. Only a committed result releases a reservation.

If the backend raised, or returned the wrong number of results, the exception escaped the scheduler. Every selected participant stayed marked in-flight with the service non-quiescent, and nothing existed to complete them — so no later action could be admitted for those participants. The escape also bypassed this layer's diagnostic channel, which every other backend fault uses.

Both cases are now reported as diagnostics, and the pre-batch snapshot is reinstated wholesale. Restoring the whole snapshot rather than adjusting counters matters: without per-action results there is no basis for the rest of a failed-action transition, and hand-adjusted counters have to stay consistent with in-flight work from an earlier batch that the scheduler does not exclude. Reinstating is exact for both. Only the exception type crosses the backend boundary, matching _backend_call_failed.

6. A large-enough timeout crashes timeout reconciliation

timeout_seconds has no declared upper bound. Adding a very large one to the start instant overflowed — as a float timestamp and as a timedelta. The old blanket except Exception hid this as "not timed out"; with the exception handling narrowed it would instead abort the whole reconciliation pass and surface as a 500.

Elapsed time is now compared against the timeout, which Python evaluates exactly for an arbitrarily large integer.

7. A large scenario crashes planning instead of being planned

Cycle detection walked the dependency graph recursively, so recursion depth tracked the longest dependency chain. A scenario with roughly a thousand chained resources raised RecursionError out of the planner: no plan, no diagnostic, just a crash. Measured to die between 900 and 2000 chained resources — while the iterative topological sort beside it handled the same graph.

This is a real ceiling for the project's own targets. BigRAE is described as instantiating 200 concurrent environments of 30+ assets each.

The walk is now driven from an explicit stack. A 5000-node chain and a 3000-node cycle both resolve. Detected cycles are unchanged: a differential property test compares against the recursive semantics over graphs containing self-loops and multi-node cycles, and I separately checked 20000 random graphs up to 22 nodes with zero divergence. Also removes _ordering_graph, which was unreferenced anywhere in the repository.

---

How I know

Every fix ships a regression test confirmed to fail before it and pass after.

Full nox -s verify green on Ubuntu 22.04 / Python 3.12 — all six lanes (unit, integration, contracts, static, participant-opacity-proof, docs-local) — at 91% total coverage.

🤖 Generated with Claude Code

[Brad-Edwards]

Please note in the comments or body which issue this PR closes. If no issue exists, please create one and link it.

Thank you!

[doublewhy]

Linked the issues in the PR body: this PR closes #1090 and #1091. I also noted the separately tracked scheduler, timeout, and planner follow-ups (#1101–#1103). Thank you.

[doublewhy]

Superseded by focused, issue-linked draft PRs: #1133 (HTTP authentication and request admission; closes #1090 and #1091), #1132 (workflow timeout reconciliation; closes #1102), #1135 (concurrent scheduler settlement; closes #1101), #1126 (bounded dependency-cycle detection; closes #1103), and #1134 (MCP event-loop and Python qualification; closes #1117). Together these retain and extend every independent fix from this bundled branch, while making each safety boundary reviewable on its own. Closing this overlapping PR; no unique implementation is being dropped.