The witnessed genesis: CORA records a TomoScan capture it only watched - #658
Merged
Conversation
Rung 1 of watching a TomoScan capture live over EPICS needs the same reach-tier vocabulary the Enclosure permit-probe trail already uses, but cora.run cannot depend on cora.enclosure.aggregates. Hoist ReachTier to cora.shared (zero dependencies, any BC may use it) and re-export it from its original home so every existing import keeps working unchanged. CaptureObserver mirrors EnclosureObserver at every layer: BC-local Protocol, a closed CapturePhase vocabulary with a first-class UNRECOGNIZED member for substrate vocabulary drift, and the same dual-clock and reach-tier shape. QuietCaptureObserver yields nothing rather than a synthesized reading, since there is no safe "always" value for a capture phase the way AlwaysPermittedEnclosureObserver has one for a permit status.
Rung 1 needs a place to declare which PVs to watch and what a deployment's tomoscan literal vocabulary means, without hardcoding 2-BM's status strings in the spine. capture_watch_pvs is code -> role -> PV (2-BM runs several tomoscan variants off one base class, so the outer key is the capture code); capture_status_phases maps a deployment's own ScanStatus literals onto the closed CapturePhase vocabulary, validated at boot so a typo fails loud rather than silently classifying everything as UNRECOGNIZED. CapturePhase itself gets the same hoist ReachTier already got: cora.infrastructure validates the deployment's declared table against it and cannot depend on cora.run.ports (tach), so the enum moves to cora.shared and is re-exported from its original home. Everything defaults empty/off, so a generic boot is unaffected.
The first slice with anything to see on the floor. RunWatcher drains a CaptureObserver and logs one event per observed phase (or the unreached/probe-only case), and does nothing else: no event store, no entries table, no Run command. This lets a deployment watch a real capture end to end, over a real substrate, before any write path exists to get wrong. Not yet wired into main.py: it takes any CaptureObserver, and the real ControlPort-backed one is the next slice. Tested here against a fake observer exercising every phase, the probe-only/unreached case, an observer that raises mid-stream, and the no-op-when-unconfigured lifespan shape.
ControlPortCaptureObserver bridges the Run BC's CaptureObserver port to the Operation BC's ControlPort at the composition root, mirroring ControlPortEnclosureObserver at every layer: sibling poll tasks feeding a shared queue, the same _PumpDone-counted drain, the same queue-drain race fix on teardown. One deliberate inversion from the Enclosure precedent, load-bearing enough to be its own module docstring section: the Enclosure adapter synthesizes an Unknown STATUS on disconnect so the run-start gate fails closed. There is no such gate here, so a disconnect carries no status claim and no phase at all, the same shape a probe-only poll tick already uses. Reading a disconnect as a real terminal would fabricate one the substrate never reported. Wired into main.py's lifespan behind run_watcher_enabled, a separate gate from capture_watch_pvs being configured, so a deployment can declare its PVs ahead of turning the watcher on. Still shadow only: nothing this slice adds can write anything.
docs/reference/modeling.md already named "Conducted vs recorded (who drives the act)" as an axis orthogonal to the Run/Procedure boundary, but nothing encoded it. Adds ConductMode (CONDUCTED / RECORDED) to Run's genesis, defaulting to CONDUCTED since every StartRun caller today is CORA's own Conductor. Nothing yet constructs a RECORDED Run: that is slice 8's not-yet-built promotion from RunWatcher's shadow observations into a real recorded genesis. This slice only gives that future work a field to import and pass RECORDED into, instead of inventing the vocabulary under gate-review pressure later. Pure additive threading: RunStarted event, Run state, the evolver's per-arm preservation, StartRun command, decider, and the proj_run_summary read model (new migration). No change to check_safety_envelope (its gates apply uniformly regardless of who drove the act) and no Procedure-side mirror (no second consumer yet). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ce 6) start_run needs a real plan_id (Plan -> Practice -> Method -> Capability); nothing in that chain had ever been registered for 2-BM, so a real Run would 404 with PlanNotFoundError on its first precondition. Checking what actually exists on arcturus also surfaced that the dark_field/flat_field recipes target StationShutter and the 2bmSP1: camera, neither of which was registered either (only the beamline root and the 31MP Camera_HighRes existed). Extends pilot_seed.py, the existing 2-BM ceremony pattern, rather than inventing a second mechanism: registers the two missing Assets, then the Capability/Method/Practice/Plan chain for dark_field and flat_field (both "conductible today" per recipes.md), bound to those Assets. Factored a shared seed_genesis helper for the four Recipe BC aggregates, since a fourth near-identical closure alongside Asset's and Supply's own crossed the rule-of-three the file's existing pattern otherwise avoids. energy_setting and hexapod_reboot stay unregistered: recipes.md marks both "design, pending executor", so a Plan for either would fail on its first conduct step. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…lice 7) located_in_enclosure_id is genesis-only (no update command exists), and a Run whose scoped Assets declare none is Permit-by-default per check_safety_envelope: leaving it unset on slice 6's StationShutter and Camera was a silent Enclosure-permit gate skip, not a cosmetic gap. Fixing it on already-registered Assets means decommission + re-register (the Lock A rebind path already used for controller_id), which means the two Plans binding them by id must be replaced too -- Capability, Method, and Practice reference no specific Asset id, so none of those three change. This is a one-time migration, permanent in pilot_seed.py from here on: the old StationShutter/Camera are decommissioned if present (no-op on a fresh deployment, which goes straight to the located _v2 registration), the old Plans are deprecated (hygiene; start_run already refuses a Plan bound to a Decommissioned asset regardless), and new _v2 Assets/Plans are registered with located_in_enclosure_id resolved by name via the same seed_enclosures() the real app's boot lifespan already uses -- never a hardcoded id, since Enclosure ids are minted at boot, not deterministic. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ce (slice 5 correction) Slice 5 shipped two gaps against its own roadmap spec, both latent only because nothing yet constructs a Recorded Run. StartRun carried a caller-settable conduct_mode that the driven decider copied verbatim, letting any caller of the driven path claim Recorded while taking the enforcing path: the exact laundering hole the axis exists to close. Separately, none of the three standing runtimes (RunSupervisor's hold FSM, its liveness/truncate/signal rules, RunInitiator's in-flight cap) distinguished conduct_mode at all, so enabling any of them against a mix of driven and watched Runs would try to hold, truncate, or otherwise act on a Run an external tool actually controls. StartRun now carries no conduct_mode field; the driven decider hardcodes CONDUCTED. list_runs surfaces conduct_mode so the runtimes have something to filter on. The RunSupervisor filters watched Runs out of its running/held population once, upstream of every mechanism, before the hold pass, liveness check, or Rule Q/R signal observation ever sees them. The RunInitiator's max_in_flight count is left unchanged and documented: it deliberately counts every Run regardless of conduct_mode, because the cap serializes access to shared single-stage hardware a watched capture is actively occupying. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
test_start_run_persists_event_with_full_upstream_chain_against_postgres asserts the persisted RunStarted payload field by field. Slice 5 added conduct_mode to that payload but never updated this test, so it has been silently broken since, caught now by running the integration suite (not part of slice 5's own unit+architecture verification sweep). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Slice 8's watched genesis needs a way to record the enclosure and beam readings instead of enforcing them, without risking the two paths ever disagreeing about what a gate actually checks. Refactor check_safety_envelope's four inline checks into standalone gate functions: check_clearance_gate and check_supply_gate still raise unconditionally (both are CORA's own data, not live facility signals, so they refuse on every path per the roadmap's rule), while enclosure_gate_refusal and beam_gate_refusal return the refusal they would raise instead of raising it. check_safety_envelope is unchanged in signature, behaviour, and raise order, now composed from the four gates; every existing caller needed zero changes, confirmed by the full pre-existing test suite passing unmodified. New witness_safety_envelope shares the exact same gate functions and converts only the enclosure/beam refusals into a SafetyEnvelopeVerdict, so the two entry points cannot silently drift: a change to a gate's logic changes what both paths see. SafetyEnvelopeVerdict carries plain bools only (the record exporter's generator drops bare str and Any) and lives beside ConductMode in state.py rather than in safety_envelope.py or events.py, so neither of those modules gains a new import edge to carry it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…es CORA only observes Slice 8's whole point: today only start_run can create a Run, and it always drives the act through CORA's own Conductor. A capture an external tool runs (2-BM's TomoScan) has nowhere honest to land except as a fabricated driven start, or not at all. RecordWatchedRun is that second path: same aggregate, same RunStarted event, but the decider hardcodes ConductMode.RECORDED and the command carries none of the operator-facing fields a driven start has, so a caller of the driven path can never claim to be recording, and a caller of this path can never claim to be driving. The governing rule, unchanged from the roadmap: refuse on what CORA can fix, witness what CORA cannot. Every CORA-side genesis invariant the driven decider enforces (deprecated Plan, decommissioned Asset, capability shortfall, bad wires, absent Clearance or Supply) still raises here. Only the two genuinely external live facility signals, enclosure permit and beam availability, are witnessed instead: recorded as a SafetyEnvelopeVerdict on the emitted event rather than gating the genesis. A trigger guard (RunMonitorTriggerNotPermittedError, mirroring the Enclosure BC's shipped lock) closes the one operator-reachable path in: only a literal "Monitor" trigger is accepted, and there is no REST route, no MCP tool, and no other construction site for ConductMode.RECORDED anywhere in the tree, each pinned by a dedicated architecture fitness test rather than left to convention. A naming-r3-reviewer pass on the staged diff caught five real R1/R2 issues before this landed: a property named `held` that collided with the Run BC's own RunStatus.HELD vocabulary, a field named `envelope_verdict` that collided with the "event envelope" infra term, two classes using a skeleton that didn't match their siblings (RunStartContext/RunStartEvents), and two gate-check functions whose word order didn't match their two siblings. All five renamed before commit; the rename also surfaced one more pre-existing exact-payload test gap of the same shape the previous commit fixed, caught by the same full-suite run and fixed alongside. Deferred to a follow-up commit: wiring this handler into _run_watcher.py's live promotion loop (the deployment settings, the seeded Agent principal, the edge-triggered dedup state machine, and the restart-rebuild logic), and seeding an actual fly-scan Plan for it to point at. Those are mechanical integration work once this slice exists; this commit is the safety-critical reviewable unit on its own. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
RunWatcher (cora.api._run_watcher) needs a seeded Agent + Actor at a pinned id to author RecordWatchedRun calls once it starts promoting BEGUN captures to real watched Runs, same shape as every other in-process runtime (RunSupervisor, RunInitiator, CalibrationWatcher). No behavior change to the watcher itself yet: this only gets the principal into the event store.
RunSummaryItem already carries conduct_mode (Conducted/Recorded), but the query had no way to filter on it. The RunWatcher's restart-rebuild query (a follow-up commit) needs exactly this: page through open Recorded Runs without also seeing Conducted ones. Pure filter-plumbing, mirroring the existing status/plan_id/campaign_id ScalarFilter shape; no projection or migration change.
Two new settings for the RunWatcher's promotion path (a follow-up commit): capture_watch_plan_id (which Plan a promoted watched Run references) and run_watcher_recording_enabled (a second kill switch above run_watcher_enabled, gating whether a BEGUN observation is actually promoted rather than just logged). Boot refuses to start with recording enabled unless both prerequisites are set, mirroring the existing _enforce_production_principal_policy shape: cheaper to catch a half-configured deployment at boot than the first time a real capture begins.
Wires the shadow-only capture-observation runtime to actually create a real watched (Recorded) Run via record_watched_run when it observes a BEGUN capture, behind the run_watcher_recording_enabled kill switch added in a prior commit. A per-capture-code dedup state machine (RunWatcherRecorder) ensures a single in-progress capture is promoted at most once; ENDED/ABORTED clears the local entry so a later capture on the same code can promote again. A boot-time restart-rebuild query (rebuild_open_captures) seeds that dedup state from every currently- Running Recorded Run's external_refs, so a capture still open across a process restart is never re-promoted. Shadow mode (recording disabled, the default) is byte-for-byte unchanged: observe_capture's body is untouched, and every existing lifespan/loop test that constructs no recorder keeps passing unmodified. Reviewed by naming-r3-reviewer before landing: renamed rebuild_open_capture_map -> rebuild_open_captures, capture_closed -> open_capture_cleared (it isn't a CapturePhase, so it doesn't belong to that log-event family), _CLOSING_PHASES -> _TERMINAL_PHASES (matching the codebase's terminal/closing vocabulary split), and the recorder's observe method -> observe_capture (so both dispatch branches in run_run_watcher read identically).
Seeds the actual 2-BM TomoScan workflow (fly_scan) as a third rung on the acquisition recipe ladder, alongside the existing dark_field / flat_field baseline captures. Watch-only: no operator REST/UI surface ever selects this Plan for start_run, matching record_watched_run's own stub route/tool, so capture_watch_plan_id has something real to point at once a deployment turns recording on.
Gate-review finding (security/authz lens): rebuild_open_captures ran unconditionally at boot with no exception handling, so a deployment that enabled run_watcher_enabled under enforced TrustAuthorize but forgot to grant the seeded RunWatcher agent ListRuns would crash the entire API's boot, not just this feature. Every sibling watcher probes its read grant via probe_read_grant and stays resilient in its own per-tick loop; this one-time boot read had neither. Adds the same probe_read_grant call (refuses boot with a clear message in strict mode) plus a fallback to an empty dedup map on UnauthorizedError in non-strict mode, so a missing grant degrades to "cold start, one possible re-promotion" rather than taking down every other bounded context's routes.
Gate-review finding (record/publishing lens): enclosure_permitted and beam_available exported unredacted (keep:number) purely because they are bool-typed, while the equivalent live PSS/interlock fact on EnclosurePermitObserved (from_status/to_status) is drop:text. Nobody made a deliberate export-sensitivity call for the new fields; the generator's type-driven default decided it. Adds _OVERRIDE_DISPOSITIONS to gen_record_dispositions.py (sibling to the existing _OVERRIDE_WIRE_KEYS per-field override table) so the two fields classify as drop:text like their EnclosurePermitObserved counterpart, while staying genuinely typed bool at the source (no str coercion just to game the classifier). Regenerated _dispositions.py; the diff is exactly the two flipped entries.
Gate-review finding (beamline domain lens): the seeded fly_scan recipe required only a Shutter + Camera, the same pair as the two static dark_field/flat_field baseline captures. A real fly-scan's defining feature is continuous sample rotation; omitting the Rotary stage meant an exported watched Run record would understate the equipment a real 2-BM TomoScan fly-scan actually uses. Seeds a RotaryStage-family Device Asset (RotaryStage is already a globally-bootstrapped family) and threads it through seed_acquisition_recipe via a new include_rotary flag, used only by the fly_scan call; dark_field/flat_field are unchanged.
cluster to witnessed (RunWatcher -> RunWitness)
Executes a naming decision locked in a separate session
(project_conduct_mode_witnessed_rename.md): RECORDED was not actually a
contrast pair with CONDUCTED, since every Conducted Run is also
recorded (in the event store, in proj_run_summary, in the export
bundle). WITNESSED is the only candidate mutually exclusive with
CONDUCTED, and is already the vocabulary the decider's own governing
rule uses ("refuse on what CORA can fix, witness what CORA cannot").
That memo's tripwire fires when a deployment is about to enable
run_watcher_enabled, which is exactly the next step after this
follow-up's five commits landed live promotion. Executing the sweep now
(worktree is clean, no in-flight edits to reconcile by hand) rather than
deferring further.
Cascades from the enum rename: RunWatcher -> RunWitness (runtime class,
module, seeded agent identity, settings, log namespace),
record_watched_run -> record_witnessed_run (feature slice, command,
context/events classes), and prose throughout ("watched genesis" ->
"witnessed genesis"). Also fixes a separate pre-existing bug the memo
called out: trigger_source referenced a class (CaptureWatcher) that
never existed in the codebase; it now names the real runtime
(RunWitness).
Naming-reviewed before landing (naming-r3-reviewer, two passes): first
pass found the RunWatcherRecorder.observe/RunWatcherRecorder naming
(now RunWitnessRecorder.observe_capture), rebuild_open_capture_map ->
rebuild_open_captures, and _CLOSING_PHASES -> _TERMINAL_PHASES from the
original commit. Second pass (this rename) found several persisted
strings and half-renamed prose (mixed CONDUCTED/witnessed vocabulary in
the same sentence), all fixed, plus a genuine architecture-fitness gap:
RunWitness doesn't end in any R5 doer-suffix
(-er/-or/-ist/-ant/-tor) the existing test/test_agent_kind_doer_form.py
checked, so a new _ZERO_CHANGE_DOER_WORDS allowlist was added for
whole-word doer nouns like Monitor/Coordinator's siblings.
Coverage report
Click to see where and how coverage changed
The report is truncated to 25 files out of 31. To see the full report, please visit the workflow summary page. This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rung 1 of the 2-BM watched-run ladder ("CORA watches a TomoScan scan happen"): a second, independent Run genesis for captures CORA observes but does not drive, plus the live promotion loop that turns a real capture into a real Run record.
ConductMode(who drove the act), the capture-observer port + shadow-only watcher runtime, and the 2-BM recipe ladder + Asset enclosure locations.27be6cbd2e+ corrections): the watched genesis itself —record_witnessed_run, a second Run command that witnesses (not enforces) the live enclosure-permit and beam-availability gates while still refusing on clearance/supply, with a full laundering-wall fitness test proving no operator path can reach it.list_runsconduct_mode filter, boot-time recording gate, restart-rebuild dedup, a realfly_scanrecipe).fly_scanrecipe missing its Rotary stage Asset.bdbe2d403e): executes a naming decision locked in a parallel session —ConductMode.RECORDEDwas not actually a contrast pair withCONDUCTED(every Conducted Run is also recorded), so it becomesWITNESSED, cascading toRunWatcher->RunWitnessand the whole cluster.Every commit is naming-reviewed and the full unit + architecture + integration suites are green throughout.
Deliberately NOT part of this PR:
RUN_WITNESS_RECORDING_ENABLEDstays off by default and must not be turned on for real until slice 9 (terminal recording) ships — a Witnessed Run currently has no automated path back out of Running once its capture ends.Test plan
RUN_WITNESS_ENABLED=true, recording left off) and verify the floor