[Architecture] Consolidate durable operation lifecycle into one typed state machine

[morluto]

Summary

Flameox currently has more than one durable-operation framework, and long-running features continue to implement lifecycle behavior directly. The code therefore duplicates idempotency, ownership, progress, cancellation, cleanup, terminalization, and restart recovery across operation families.

PR #79 introduced a generic durable operation framework to standardize these semantics, but the abstraction was adopted primarily by capability setup. Detached capture retained a parallel record/status/manager hierarchy, while inference replay/profiling and several publication/recovery paths still own bespoke terminal transitions.

The repeated lifecycle fixes in #75, #77, #87, #104, #118, and #127 are symptoms of this architectural split. The correct fix is not another feature-specific status field; it is one small durable state-machine kernel with typed operation adapters.

Evidence in the current architecture

OperationRunner is nominally generic but semantically capability-specific

src/flameox/application/operations.py defines useful common concepts:

• persisted operation records;
• request and idempotency digests;
• bounded progress and item outcomes;
• owner/heartbeat metadata;
• cancellation and cleanup state;
• terminal receipts and recovery actions.

However, the supposedly generic status projection contains branches such as record.operation == "capability.setup", and the runner is currently used by capability setup rather than being the lifecycle boundary for other long-running operations. Domain behavior has leaked back into the kernel.

Detached capture implements a parallel lifecycle system

src/flameox/application/detached.py separately defines:

• DetachedProgress;
• DetachedCaptureRecord;
• DetachedRecovery;
• DetachedCaptureStatus;
• DetachedCaptureManager;
• its own in-memory task ownership/restart logic;
• its own idempotency lookup;
• its own cancellation, cleanup, progress, and terminal transitions.

Idempotency lookup scans stored records instead of addressing an atomically unique durable key. That is both an efficiency smell and, more importantly, evidence that idempotency is being implemented as query convention rather than a storage invariant.

Other workflows still hand-code terminalization

Inference replay and profiling create run manifests, execute subprocesses, preserve partial artifacts, translate exceptions, append terminal revisions, publish evidence, and clean staging within large lifecycle methods. Artifact publication, repair, quarantine, and recovery have also required repeated fixes to ensure that interruption leaves a terminal or resumable state.

These services need domain-specific work, but they should not each invent the transport-level lifecycle around that work.

Root cause

The generic abstraction is drawn at the wrong boundary.

The repository currently mixes:

1. operation mechanics — idempotency, ownership, leases, progress, cancellation, cleanup, retries, terminal immutability, and crash recovery;
2. domain state — capture/run manifests, evidence publication, capability receipts, inference protocols, artifact registrations, and validation results;
3. presentation — MCP polling intervals, next-tool guidance, and operation-specific status fields.

When these concerns live in one feature manager, every new operation family creates another subtly different state machine. When they live in one generic model with hard-coded operation-name checks, the core abstraction becomes an untyped union implemented with strings.

Required invariant

Every long-running side-effecting operation uses one durable lifecycle kernel. Domain adapters may define work and typed receipts, but they may not redefine idempotency, ownership, cancellation, terminal-state, or recovery semantics.

For a stable (workspace, operation kind, idempotency key, request intent) tuple, all concurrent and retried starts must converge on one durable operation and at most one side-effecting execution.

Proposed design

1. Define one durable operation envelope

Keep the shared record small and stable. For example:

class OperationEnvelope(ContractModel):
    schema_version: Literal[1]
    operation_id: str
    workspace_id: str
    kind: OperationKind
    request_digest: str
    idempotency_digest: str
    state: OperationState
    phase: str
    revision: int

    owner_id: str | None
    owner_lease_expires_at: datetime | None
    heartbeat_at: datetime | None

    cancellation_requested_at: datetime | None
    cancellation_effective_at: datetime | None
    cleanup_state: CleanupState

    progress: OperationProgress
    payload: OperationPayload
    terminal_receipt: OperationReceipt | None
    recovery: RecoveryAction | None

OperationPayload and OperationReceipt should be discriminated typed unions owned by operation adapters. The lifecycle kernel should not inspect their operation-specific fields.

2. Use an explicit transition table

Do not update lifecycle fields through arbitrary model_copy() calls spread across services. Define and test legal transitions, for example:

pending -> starting -> running -> succeeded
                         |  |       failed
                         |  +-----> cancelling -> cancelled
                         +--------> recovering -> running/failed/cancelled

The exact states can differ, but the following rules should be encoded centrally:

• terminal states are immutable;
• cancellation request and cancellation completion are distinct;
• cleanup completion is independent from the primary process exit;
• only the current lease owner may advance active work;
• a revision/CAS check guards every transition;
• owner expiry enables a deterministic recovery decision;
• failure always records a bounded cause, phase, and recovery action;
• progress is monotonic where totals are known and phase-based otherwise.

3. Replace operation-name conditionals with typed adapters

Define an adapter protocol such as:

class OperationAdapter[RequestT, PayloadT, ReceiptT](Protocol):
    kind: OperationKind

    def canonical_request(self, request: RequestT) -> JsonValue: ...
    async def execute(self, context: OperationContext, request: RequestT) -> ReceiptT: ...
    async def cancel(self, context: OperationContext) -> None: ...
    async def recover(self, context: OperationContext) -> RecoveryDecision: ...
    def project_status(self, envelope: OperationEnvelope) -> OperationStatus: ...

The kernel controls transitions and persistence. The adapter performs domain work and returns typed events/receipts. Core code must not contain checks for "capability.setup", "capture", "inference", or other operation names.

4. Make idempotency a storage invariant

A start path must not scan records and then create a new record in a separate step.

Use one of these equivalent approaches:

• derive the durable operation key from (workspace_id, kind, idempotency_digest) and atomically create that exact record path;
• or store operations in a transactional database with a unique constraint on those fields.

The stored record must also contain request_digest. Reusing the same idempotency key with different intent must fail explicitly rather than reconnect to unrelated work.

If the existing JSON record store remains, its atomic create/CAS semantics can enforce this without requiring an immediate database migration. The architectural requirement is uniqueness at write time, not a particular database.

5. Standardize ownership, heartbeat, and recovery

All active operations should use the same lease model:

• owner identity;
• lease acquisition under CAS/transaction;
• periodic heartbeat;
• explicit expiry;
• restart reconciliation;
• proof that an old owner cannot continue advancing the record after lease loss.

Recovery is operation-specific, but the decision boundary is not. The adapter may choose resume, observe_external_work, finalize_from_domain_record, fail_with_recovery, or cannot_recover; the kernel persists and exposes that decision consistently.

6. Keep domain records authoritative

This issue must not replace RunManifest, artifact registrations, evidence generations, or inference protocol identities with one generic operation blob.

For capture and inference:

• the operation envelope owns request/idempotency/lease/cancellation/recovery semantics;
• the run manifest remains the authoritative scientific/execution evidence record;
• the operation payload links to the run ID and current domain revision;
• terminalization reconciles the operation receipt with the already persisted run state.

This avoids two competing copies of execution truth while still giving MCP/CLI one durable polling protocol.

7. Use one bounded progress/event model

Persist only a bounded recent progress history plus the current phase/aggregate. Events should be typed and sequence-numbered so callers can poll incrementally without unbounded records.

Progress reporting should have one definition across capabilities, capture, inference, reduction, and maintenance operations:

• known work: monotonic completed/total;
• unknown duration: phase and elapsed time, no invented percentage;
• cancellation: requested, observed, cleanup in progress, terminal;
• retry/recovery: explicit attempt and ownership metadata.

8. Apply the kernel incrementally

Recommended order:

1. Remove capability-specific branches from operations.py by introducing the adapter boundary.
2. Port DetachedCaptureManager onto the same kernel while preserving current MCP contracts.
3. Move inference replay/profiling to durable operations, reusing the authorized-plan work from the plan-integrity issue.
4. Move long artifact publication/recovery/repair operations where they benefit from polling, idempotency, and restart recovery.
5. Delete parallel record/status managers once migrations and compatibility windows are complete.

Non-goals

• Do not force bounded synchronous reads into durable operations.
• Do not erase domain-specific states or evidence models.
• Do not create a universal workflow language or scheduler.
• Do not serialize arbitrary Python callbacks as durable workflow definitions.
• Do not duplicate the typed-enum/discriminated-union work already underway in refactor: make invalid contract states unrepresentable #152; this issue is about transition authority and durable mechanics.

Storage and compatibility considerations

• Version the envelope and every typed payload/receipt.
• Add readers/migrators for existing operations/ and detached-capture records.
• Preserve existing operation/run IDs when they are externally referenced.
• Define whether old in-progress records can be resumed or must terminate with an explicit migration recovery receipt.
• Keep writes atomic and crash-safe; a process crash between domain mutation and envelope update must be reconcilable from the domain record or journal.
• Avoid holding a database transaction across external work. Persist a transition/lease, perform bounded work, then commit the next transition.

Acceptance criteria

• One lifecycle kernel owns idempotency, revision/CAS, leases, heartbeat, cancellation, cleanup, terminal immutability, progress, and recovery.
• The kernel contains no operation-name-specific branches.
• Capability setup and detached capture are adapters over the same kernel.
• Inference replay/profiling have a documented migration path to the same durable operation contract.
• Reusing an idempotency key with the same request reconnects; using it with a different request fails with a typed conflict.
• Two processes concurrently starting the same operation can produce at most one side effect.
• Idempotency lookup is keyed/atomic and does not require listing every record.
• Lease expiry and restart recovery are deterministic and covered by tests.
• Cancellation is tested before launch, during startup, during execution, during preservation/publication, and during cleanup.
• Every terminal result contains bounded phase/cause/cleanup/recovery information.
• Terminal states cannot be revised back to active states.
• Run manifests and evidence remain domain-authoritative; operation records link to them rather than duplicate them.
• Model/state-machine tests enumerate legal and illegal transitions.
• Crash-injection tests stop the process after each persisted transition and prove restart convergence.
• Existing operation/detached records have an explicit migration or terminal-recovery strategy.
• MCP and CLI expose one consistent start/status/cancel/recover vocabulary across migrated operations.

Best-practice references

• SQLite transactions provide atomic commit/rollback semantics, including crash behavior: https://www.sqlite.org/lang_transaction.html and https://www.sqlite.org/transactional.html
• SQLite uniqueness and ON CONFLICT can make an idempotency key a write-time invariant instead of a read-then-write convention: https://www.sqlite.org/lang_conflict.html
• Amazon's guidance on idempotent APIs covers stable request identity, retries after uncertain responses, and the dangerous same-key/different-intent case: https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/
• AWS durable-execution guidance distinguishes at-least-once/at-most-once behavior and recommends transactional/unique-key deduplication for side effects: https://docs.aws.amazon.com/durable-execution/patterns/best-practices/idempotency/

[morluto]

Related authority boundary: #154 defines the server-owned plan token/digest that side-effecting operations consume. The lifecycle kernel here should persist and reconnect to that consumed intent, but must never accept replacement argv/cwd/environment/output fields from a caller.

[morluto]

Triage refinement: keep the kernel small and migrate incrementally

This is P1 architectural consolidation. It should follow the immediate P0 containment work, but the first slice can land independently.

First PR

Refactor operations.py before adding more operation families:

• remove the record.operation == "capability.setup" branch from the generic status projection;
• define a typed operation adapter/event boundary;
• encode an explicit transition table;
• route every transition through validated construction per [Architecture] Ban non-validating mutation of ContractModel state #158;
• address operation records directly by the digest-derived durable key rather than scanning all records for idempotency.

Capability setup should remain the first adapter and preserve its current public contract.

Second PR

Port DetachedCaptureManager onto the same kernel while preserving the existing MCP start/get/cancel vocabulary. The run manifest remains authoritative; the operation envelope should link to run_id and reconcile from the run record after restart.

Later migrations

Then migrate inference replay/profiling and long projection/recovery work. Do not force ordinary bounded reads or small atomic record writes through the operation framework.

Additional closure requirements

• Lease ownership needs fencing: an expired/old owner must be unable to append active transitions after another owner takes over.
• Crash-injection tests should terminate after every persisted transition and prove restart convergence.
• Cancellation must be tested before launch, during startup, execution, artifact preservation/publication, and cleanup.
• Terminal states must be immutable, including under stale CAS retries.

Dependencies

• [Architecture] Ban non-validating mutation of ContractModel state #158 supplies validated state transitions.
• [Architecture] Encode workspace lock ordering in a ranked lock manager #161 supplies named lock intents.
• [Architecture] Replace stringly-typed agent workflow links with a validated tool/action graph #159 supplies typed recovery actions.
• [Architecture] Add an explicit migration contract for every persisted schema #157 versions existing and future operation envelopes.
• [Architecture] Eliminate run-manifest/evidence dual writes with a recoverable publication transaction #160 may use the operation kernel to run reconciliation, but projection intent remains a separate storage-consistency primitive.