Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
320 changes: 320 additions & 0 deletions CONTEXT.md

Large diffs are not rendered by default.

27 changes: 27 additions & 0 deletions docs/adr/0001-single-host-multi-process-topology.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Single-host, multi-process topology with fault isolation

OATH runs as several cooperating processes on a single host: one process per
**Adapter** (broker or data provider), one **Core** process (risk engine, order
execution, portfolio, and initially strategies), and one or more **Strategy
Node** processes. We chose this over a single-process monolith so that a flaky
external integration — the thing that actually hangs, leaks, or gets
rate-limited — cannot crash the decision-making centre, and over a
multi-machine distributed system because a retail, single-box deployment lets us
use shared-memory IPC for near-in-process latency.

## Consequences

- Fault isolation: an adapter crash takes down only that venue's connectivity.
- The **zero-copy fast path** (shared-memory IPC, e.g. iceoryx2) requires all
processes on one host. Network Bus backends (Kafka, RabbitMQ, …) relax this:
processes may live on different machines at a latency cost. Cross-machine
distribution is therefore supported via those backends, just not on the
optimized path. (See ADR-0002.)
- The dominant design concern becomes the inter-process contract (see ADR-0002),
not in-process trait dispatch.
- Strategies start co-located in Core and split into Strategy Nodes later; the
strategy↔Core seam is designed as a swappable call from day one.
- Adapters **and** Strategy Nodes are hot-pluggable: a new venue or strategy
process can join the Bus at runtime (subject to symbology resolution and a
registration handshake) without restarting Core — something a single-process
design cannot do.
39 changes: 39 additions & 0 deletions docs/adr/0002-backend-agnostic-bus-canonical-message-model.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Backend-agnostic Bus with one canonical message model

Processes communicate over a **Bus** abstracted behind a single trait, so the
transport backend is swappable (zero-copy shared memory such as iceoryx2 for the
fast path; Unix sockets or Kafka for other users). There is exactly **one
canonical message model** shared by every backend — backends differ only in how
they move bytes, never in the data they carry.

> _Refined by **ADR-0020**: the universal type bound is `Serialize` (POD is a
> backend-specific zero-copy discipline, not a trait requirement), and the
> per-topic delivery semantics below are realized as two classes (`LatestValue` /
> `Reliable`) × two access patterns (keyed store / stream)._

## Considered options

- *Per-backend data models* — rejected: it recreates, one layer up, the
representation swamp that ADR-0003 outlaws in the core.
- *One model, intersection of backend constraints* — chosen.

## Consequences

- Message payloads are `Serialize` (the shared trait bound); backends that
support zero-copy may additionally impose fixed-layout, `#[repr(C)]`
plain-old-data (POD) constraints as backend-specific discipline (ADR-0020).
- The trait must be designed to the **stricter** ownership contract: zero-copy
hands back a *loaned*, lifetime-bounded sample; network backends hand back an
*owned* value. Modelling the borrowed/lifecycle-bounded case keeps zero-copy.
- `#[repr(C)]` payloads are brittle under schema evolution; versioning is a
known open concern to be addressed separately.
- Delivery semantics are per-topic: market data is drop-to-latest (ring buffer);
orders and fills are reliably delivered (never dropped on the wire). Acting on
stale-but-delivered messages is a separate, consumer-side freshness concern.
- **Durability is a backend capability, not a Bus guarantee.** iceoryx2 is
ephemeral (zero-copy, fast, lost on crash); Kafka and Chronicle Queue are
durable replayable logs. A durable backend can double as Core's event log /
recovery journal (see ADR-0005).
- Pub/sub backends support **dynamic publishers/subscribers**, so adapters and
strategy nodes can be added or removed at runtime without restarting Core
(see ADR-0001).
20 changes: 20 additions & 0 deletions docs/adr/0003-canonical-model-adapter-translation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Canonical core model with adapter-side translation

The Core and the Bus speak exactly one canonical vocabulary (`Symbol`, `Price`,
`Quantity`, …), chosen at compile time and identical for every adapter. Each
**Adapter** owns the translation between its venue's representation and the
canonical model at its own boundary (an anti-corruption layer); venue-specific
representations never leak inward. A central symbology (e.g. perm_id / OpenFIGI)
gives the canonical identity, so the same instrument offered by different
brokers collapses to a single `Symbol` and is not double-counted by portfolio or
risk.

## Consequences

- Every adapter carries a translation layer (precision conversion, symbol
resolution, mapping tables) — accepted cost, paid to keep the core clean.
- `Price`/`Quantity` are newtypes over a swappable inner numeric type, so the
representation can change at compile time without touching call sites (later
settled on fixed-point `i128`/`u128` — ADR-0023).
- Translation lives in the adapter process, so a malformed-message bug in one
venue's translation cannot corrupt another's.
17 changes: 17 additions & 0 deletions docs/adr/0004-risk-as-continuous-control-loop.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Risk as a continuous, autonomous control loop

The Risk Engine is a continuous control loop with veto/cancel/amend authority
over every order, not a one-shot pre-trade gate. It surveils all live state and
may act autonomously — for example, cancelling a resting order already on a
venue's book because a correlated fill changed the exposure or capital is now
better deployed elsewhere. Strategies only *detect* and propose **Signals**;
Core *decides and acts*.

## Consequences

- The `risk-core` / `execution-core` traits are an event-in / command-out state
machine, not a `check(order) -> verdict` function.
- Risk is un-bypassable: a Strategy cannot reach a Broker directly; it emits a
Signal that Core adjudicates.
- Core must hold and continuously update the full state risk depends on, which
motivates the single-writer design under discussion in ADR-0005.
53 changes: 53 additions & 0 deletions docs/adr/0005-single-writer-event-sourced-core.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Single-writer, event-sourced, deterministic Core

Deterministic replay is a non-negotiable goal, and Core must enforce global
shared invariants (account buying power, global buying power, per-asset-class
risk) that are inherently cross-symbol. A global invariant plus deterministic
ordering forces a **single writer**: one logical thread owns all state — per
-symbol and global — and is the only thing that may mutate it. Inputs from all
processes are assigned a total order at a single ingress and appended to a
persisted, totally-ordered input log; Core's state is a pure fold over that log,
and **replay re-feeds the log through the identical fold with external side
effects suppressed**.

For the MVP this is implemented as a **single-threaded deterministic kernel**
with I/O and persistence offloaded to async worker threads that feed results
back as events — the model proven in production by NautilusTrader. The single
*writer* (the decision stage) is fundamental and stays; the single *thread* is
an MVP implementation detail we can later replace with a staged pipeline
(LMAX-Disruptor-shaped: parallel parse/journal/publish stages around one
decision stage) **if and only if** profiling shows the kernel is the bottleneck.

## Considered options

- *Shard by Symbol* — rejected: global cross-symbol invariants would force a
distributed two-phase reservation on the hot path, fighting both latency and
determinism.
- *Single-writer, single-threaded kernel + offloaded async I/O (MVP)* —
accepted. Matches NautilusTrader's shipping architecture.
- *LMAX-Disruptor staged pipeline (single decision stage)* — the measured
optimization path, not built until needed. Caveats: stall propagation,
busy-spin CPU burn, cache-line/`unsafe` tuning, ring sizing, debuggability.

## Consequences

- The persisted ordered input log is the `oath-persistence-core` event log; Core
is an event-sourced state machine and persistence is its journal.
- **Recovery = snapshot + replay-tail**: periodically snapshot Core state; on
restart, load the last snapshot then replay only the log tail after it. A
durable Bus backend (Kafka, Chronicle Queue) can serve as this log (see
ADR-0002).
- A hard line separates the *deterministic decision* (replayable) from the
*effectful action* (live-only, suppressed during replay). Reconciling
in-flight orders with the broker after a crash is a known open problem.
- The MVP kernel is bounded by one core; parallel work (ML, analytics) runs
off-loop in workers and feeds results back as events.
- `unsafe` is permitted where it demonstrably buys performance, via a justified
per-crate override of the workspace `unsafe_code = "deny"` lint.

## Future fault tolerance

A **hot-standby Core** that replays the same log via consensus (Aeron-Cluster
style replicated deterministic state machine) is the documented path to Core
failover. Out of scope for the MVP; recorded so the single-writer model is not
mistaken for a single point of failure with no exit.
35 changes: 35 additions & 0 deletions docs/adr/0006-broker-reconciliation-contract.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Crash recovery: broker-authoritative reconciliation

After a crash, Core recovers in two steps: **replay** the Event Log to rebuild
decision state (side effects suppressed), then **reconcile** against the broker,
which is the **single source of truth** for what actually happened to orders,
positions, and fills. Replay restores intent; reconciliation restores reality;
Core repairs any divergence into its recovered state.

## The reconciliation join key

Every order carries a **client order id** — the join key that lets Core ask the
broker a precise question ("what happened to `abc123`?") instead of matching by
ambiguous attributes (symbol/side/qty/price). Because the kernel is
deterministic, this id is regenerated identically on replay, so an order decided
before a crash is matched exactly to the broker's report. Without the id,
reconciliation degrades to fuzzy attribute matching and silently repairs
state incorrectly whenever two similar orders or partial fills exist.

## The ordering invariant (write-ahead)

The input(s) that trigger an order decision **must be durably appended to the
Event Log before the resulting order is transmitted to the broker.** Otherwise
replay cannot regenerate the order (or its client order id) and the broker is
left holding an unmatchable orphan order. A *separate* write-ahead "decision
record" is not needed — replaying the inputs regenerates the decision — but this
log-before-send ordering is mandatory.

## Mandatory adapter capability

Every **Broker** adapter MUST provide (a) **idempotent submit** keyed by a
client order id (deduping retransmissions), and (b) a **queryable** view of
order / position / fill state for reconciliation. Venues lacking native support
must emulate it inside the adapter (persisted id mapping, dedup table). Venues
where this cannot be made to hold are excluded — a deliberate scope boundary
favouring a clean, trustworthy core over maximal venue coverage.
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Binding time: runtime-pluggable iff cross-process

Every OATH binary — Core, each Adapter, each Strategy Node — is fully
monomorphized internally: **no `dyn` trait object sits on any decision hot
path.** "Swappable" is achieved two different ways depending on *what* is
swapped — **compile-time generics** for anything in-process, and **process-level
pluggability over the Bus** (spawn a process, run a registration handshake) for
anything that joins at runtime. The rule: *runtime-pluggable ⟺ a separate
process across the Bus; in-process ⟹ compile-time static.*

## Considered options

- *Trait objects (`dyn`) for swappable subsystems* — rejected: dynamic dispatch
on the per-event decision path defeats the latency goal (no inlining across the
call, vtable indirection) and is unnecessary — the only thing that must change
at runtime is *which processes are connected*, not which code a hot loop calls.
- *Compile-time generics in-process + process/Bus pluggability across* — chosen.

## Consequences

- Risk/execution Policies and I/O backends (net, persistence, Bus impl) are bound
at build time via generics; a user *builds their Core* with the rules and
backends they need. There is no runtime in-process plugin loader.
- The polymorphism that lets Core talk to "any adapter" is carried by the
canonical message model + Bus trait (ADR-0002/0003), **not** by an in-process
trait. Core holds a Bus handle, never a `Box<dyn Adapter>`, and never depends on
a `Broker` / `DataProvider` / `Strategy` trait — those traits live on the
adapter/strategy side and are called statically by their host harness.
- Adapters and Strategy Nodes are added at runtime by spawning a process that
connects to the Bus and completes a registration handshake (ADR-0001) —
pluggability without any in-process dynamic dispatch.
- Co-location is a *backend choice, not a code path*: an initially co-located
Strategy (ADR-0001) talks through an in-memory Bus backend; moving it to its own
process swaps the backend, not the strategy code. The strategy↔Core seam never
branches on local-vs-remote.
- Cost accepted: no pre-built generic binary with runtime-loadable in-process
plugins; each deployment is compiled for its chosen Policies/backends.
52 changes: 52 additions & 0 deletions docs/adr/0008-single-owner-kernel-stateless-policies.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Single-owner Kernel with stateless Policies over a StateView

Core's decision components — risk, execution, portfolio — hold **no state of
their own and do not depend on one another**. The single-writer **Kernel**
(ADR-0005) owns *all* canonical state; each component is a **stateless Policy**
the Kernel invokes as a decision function over a read-only `StateView` of that
state. This extends ADR-0005's single-owner argument one level down: the same
reason cross-symbol invariants forced a single writer *between* processes (rather
than sharding) forbids distributing state ownership *within* Core across
co-owning components.

## Considered options

- *Component co-owners* — portfolio owns positions, execution owns the open-order
table, risk owns limit state; the Kernel sequences calls but each mutates its
own slice. Rejected: a global invariant (e.g. buying power) spans all three
owners, so the check must either read across them (which re-creates the
`StateView` anyway) or be hoisted into the Kernel — and distributed mutation
widens the determinism-audit surface and complicates Snapshot. It re-litigates
ADR-0005 one level down.
- *Kernel sole owner; components are stateless Policies over a `StateView`* —
chosen.

## Consequences

- The old `risk → execution → portfolio` dependency edges vanish: a Policy
depends only on the `StateView` contract, never on a sibling. Portfolio is
modelled as *fold logic* (how a Fill updates positions) plus read accessors, not
as a state owner.
- "Swappable" here is a **Policy swap** (different risk *rules*, a different
execution *algo*), not a backend swap — bound at compile time per ADR-0007. A
swapped-in Policy holds a read-only view and therefore *cannot* corrupt
canonical state.
- A Policy that needs private state declares an associated `Private` type the
Kernel **custodies** and the Policy mutates **in place** (`&mut Self::Private`)
— keeping it inside the one Snapshot and one fold, and avoiding the per-event
allocation a functional `-> NewState` return would impose on the Disruptor
decision stage (ADR-0005).
- A Policy never *acts*; it emits a **Decision** — the set of intended actions for
one input (admit/shape/reject a Signal; cancel/amend/flatten resting orders).
The Kernel drains the Decision and performs every action — it is the sole actor,
with effects suppressed on replay. To avoid per-event allocation the Policy
pushes actions into a Kernel-provided, reused **sink** (`&mut ActionSink`)
rather than returning an owned collection — the same in-place discipline as
`Private`.
- `StateView` is a single read-only trait, **generic-dispatched** (monomorphized,
zero-cost, fixture-implementable for tests), exposing all canonical read-state.
It must expose **no mutation or interior mutability** and **no nondeterministic
iteration** (keyed/ordered accessors only), so replay cannot diverge.
- `StateView`, `Decision`, and the Policy traits are a *Core-subsystem* contract:
they live in a Core-internal hub crate, not in `oath-model` (reserved for
cross-subsystem / over-the-Bus types).
Loading