Backtest Forensics Engine

An event-driven, multi-asset backtesting engine built around one idea: a backtest result is not trustworthy until it has been forensically audited. Most backtesters will happily hand you a beautiful equity curve built on a subtle bug — a fill that used the signal bar's close instead of the next bar's open, an accounting identity that silently drifts, a strategy that (without anyone intending it) peeked at future data. This engine is designed to catch that class of bug automatically, on every run, before you trust the number.

Why this is different

Most open-source backtesters (backtrader, vectorbt, zipline) optimize for speed or expressiveness and treat correctness as the user's job — write a correct strategy, trust the framework's fills. This engine inverts that: the engine itself carries a forensic audit layer that runs whether or not you ask for it.

• Kahan-summation accounting. Every cumulative sum (equity curve, P&L, commission) uses compensated summation, not naive np.sum/np.cumsum. Floating-point drift over a 100,000-bar backtest is a real source of silently-wrong results; this engine doesn't have that failure mode.
• Conservation laws checked on every bar, not just at the end. The accounting identity (cash + market_value == equity), P&L decomposition, and time monotonicity are verified continuously. A violation surfaces at the bar it happened, not as a mysterious final-equity mismatch three weeks later.
• Canary strategies for look-ahead bias. Before you trust a strategy's result, run it through a harness of oracle strategies with known correct answers — including a perfect-foresight canary (trades with knowledge of the next bar's direction; must hit the exact theoretical-max P&L) and an anti-foresight canary (always trades the wrong way; must lose money). If anti-foresight is profitable, your fill timing has a look-ahead leak — full stop. Seven canaries ship out of the box.
• A 15-component Trust Score on every result — not optional. Statistical robustness (PSR, minimum backtest length), overfitting risk (DSR, walk-forward efficiency, trade count), mechanical integrity (conservation, determinism, data quality), risk profile (drawdown, tail risk, Sharpe stability), and trade quality (profit factor, win consistency) roll up into a single 0–100 score with an A–F grade. A shiny Sharpe ratio on a D-grade result is a red flag, not a strategy.
• Deterministic, reproducible runs. Order and fill IDs are monotonic counters, not UUIDs or wall-clock timestamps — the same inputs produce byte-identical outputs, so a result is SHA-256-reproducible.
• Zero external dependencies beyond numpy/scipy/pandas. The validation layer isn't a bolt-on library wrapping a "real" engine — it's built into the core, at the same dependency weight as the fill logic itself.

Quickstart

pip install -e ".[dev]"

from strategies.examples.ma_crossover import MACrossover
from engine.core.builder import BacktestBuilder
from engine.broker.commission import PerContractCommission
from engine.broker.slippage import FixedSlippage
from engine.data.csv_feed import CSVBarFeed

bars = list(CSVBarFeed("data/ES_1m.csv", instrument_id="ES"))

result = (BacktestBuilder()
    .set_strategy(MACrossover(fast_period=10, slow_period=30, instrument_id="ES"))
    .set_bars(bars)
    .set_capital(100_000)
    .set_commission(PerContractCommission(14.50))   # e.g. $29/round-turn, CME e-mini
    .set_slippage(FixedSlippage(ticks=1, tick_size=0.25))
    .set_multiplier("ES", 50.0)
    .build()
    .run())

print(f"P&L: ${result.total_pnl:,.2f}, Trades: {result.n_trades}")
print(f"Conservation clean: {result.conservation_report['is_clean']}")

Or from the command line:

python cli.py run --strategy ma_crossover --data data/ES_1m.csv --report tearsheet

Worked example: catching look-ahead bias with the canary harness

The engine ships a CanaryHarness that runs a battery of oracle strategies against known-answer synthetic data. Two of the seven are specifically designed to catch look-ahead bias in the fill-timing path:

from engine.validation.canary import CanaryHarness

verdict = CanaryHarness().run_all()

print(verdict.summary)          # e.g. "7/7 canaries passed"
for r in verdict.results:
    print(f"  [{'PASS' if r.passed else 'FAIL'}] {r.name}: {r.actual}")

assert verdict.passed, f"Canary failed: {verdict.failed_canary}"

What the two look-ahead canaries actually check:

• perfect_foresight — a synthetic strategy that "knows" the next bar's direction and trades accordingly. On the harness's deterministic bidirectional data, this must produce exactly the theoretical-maximum P&L, which the harness re-derives from the bar data itself. If the achieved P&L differs from that computed maximum, the engine is letting a strategy fill at a price it shouldn't be able to see — a look-ahead leak.
• anti_foresight — the mirror image: a strategy that always trades against the next bar's direction. On the same data it takes real trades and must lose money. If anti_foresight comes back profitable, something in the fill-timing path is letting the strategy act on information it shouldn't have yet — this is the single highest-signal check for look-ahead bias in an event-driven backtester, and it runs by default on every CanaryHarness().run_all() call.

The harness data is deliberately bidirectional (both up and down bars) with a gap between each bar's close and the next bar's open. That matters: on a monotone ramp (open[i+1] == close[i]), a fill that leaks to the signal bar's close is numerically identical to the correct next-open fill, so the leak is invisible. The gapped, two-directional series makes an off-by-one-bar fill show up as a P&L change — so the canaries genuinely catch the classic "used the signal bar's close instead of the next bar's open" bug rather than passing vacuously. (tests/engine/test_canary.py injects exactly that leak and asserts the harness verdict flips to fail.)

This is the check that a plain equity-curve eyeball review will never catch: a strategy that looks profitable because it is quietly cheating. Run the canary harness before you trust any new strategy or engine change.

Trust Score

Every backtest result can be scored end-to-end. Build a MetricsReport from the raw engine output (build_report does this for you, and is what cli.py uses), then hand it to compute_trust_score:

from engine.analytics.report import build_report
from engine.validation.trust_score import compute_trust_score

report = build_report(
    equity_curve=result.equity_curve,
    fills=result.fills,
    bars=bars,
    initial_capital=100_000,
    total_commission=result.total_commission,
    conservation_report=result.conservation_report,
)

score = compute_trust_score(
    metrics=report.metrics,
    conservation_report=result.conservation_report,
)
print(f"{score.grade} ({score.total_score:.1f}/100)")

Grade	Range	Meaning
A	90–100	Institutional quality, ready for allocation
B	75–89	Strong, minor concerns worth monitoring
C	60–74	Moderate, proceed with caution
D	40–59	Weak, significant concerns
F	0–39	Unreliable, do not allocate

Composed from 15 components across 5 weighted categories: statistical robustness (Probabilistic Sharpe Ratio, minimum backtest length, sample size), overfitting risk (Deflated Sharpe Ratio, walk-forward efficiency, trade count, PBO/CSCV), mechanical integrity (conservation-law compliance, determinism, data quality), risk profile (drawdown, tail risk, Sharpe stability), and trade quality (profit factor, win consistency, edge ratio).

Architecture

engine/                 Zero external deps beyond numpy/scipy/pandas
  analytics/            30+ metrics, streaming Welford, reports (text/JSON/HTML), tearsheets
  broker/               Order matching, commission, slippage, fill models
  core/                 BacktestEngine, Builder, Clock, EventBus, Journal
  data/                 ColumnarStore, DataValidator, Consolidators, CSVBarFeed
  instruments/          Instrument specs (ES, NQ, etc.), InstrumentRegistry
  microstructure/       Tick-level fill and microstructure modeling
  models/               Bar, Fill, Trade, Money, AccountState, enums, errors
  optimization/         Grid search + genetic optimizer, fitness functions
  portfolio/            Portfolio, Position, Accounting (Decimal cash)
  pricing/              Pricing utilities
  risk/                 RiskManager, CircuitBreaker (NORMAL/WARNING/HALTED/RECOVERY)
  strategy/             Strategy ABC, Context, Indicators (SMA/EMA/RSI/ATR/BB/MACD), scheduling
  tail_risk/            Tail-risk analytics
  validation/           Conservation laws, 7 canary strategies, Trust Score,
                         Walk-forward, Monte Carlo, DSR/PSR, Bayesian Sharpe,
                         Forensics, Assumption Registry
  vectorized/           Vectorized signal + backtest runner for fast screening

strategies/examples/    Example strategies (MA crossover, a QuantConnect-style demo)
tests/                  pytest suite covering the engine (see "Running tests")
docs/research/          QuantConnect/LEAN API + product reference notes used to
                         design a QC-portable strategy-authoring surface
cli.py                  CLI entry point (backtest-cli after `pip install -e .`)

Library core, no framework lock-in. engine/ is a standalone Python package with no knowledge of any particular trading desk, broker, or data vendor. Data feeds, dashboards, live-broker bridges, and reporting UIs are things you build on top (a minimal CSVBarFeed/CSVTickFeed is included in engine/data/csv_feed.py to get you started) — the engine's job is running backtests correctly and telling you when it doesn't trust its own answer.

Key design decisions

• Kahan summation from day one — compensated arithmetic in every cumulative computation. No naive np.sum/np.cumsum anywhere in the hot path.
• Conservation laws on every bar — accounting identity, P&L decomposition, and time monotonicity verified continuously, not just at the end of a run.
• 7 canary strategies, including perfect-foresight and anti-foresight, to catch look-ahead bias. Run them before trusting any new strategy.
• Trust Score on every result — 15-component, 0–100 score with an A–F grade. Not optional; it's part of BacktestResult, not a separate report you have to remember to run.
• Deterministic IDs — fill and order IDs are monotonic counters, not UUIDs or timestamps. Given the same inputs, results are byte-for-byte reproducible and SHA-256-checkable.
• Fill realism as a first-class citizen — orders fill at the next bar's open (not the signal bar's close), with configurable slippage and commission models, specifically so the canary harness has something meaningful to validate.

Related work

	Backtest Forensics Engine	backtrader	vectorbt	Zipline
Execution model	Event-driven	Event-driven	Vectorized	Event-driven
External deps	numpy/scipy/pandas only	none (pure Python)	numpy/numba	pandas/numpy/talib-adjacent stack
Built-in look-ahead-bias detection	Yes — canary harness (perfect/anti-foresight oracles)	No	No	No
Per-bar conservation/accounting checks	Yes	No	No (vectorized, no per-bar state)	No
Composite trust/quality score on every result	Yes (15-component, A–F)	No	No	No (pyfolio add-on gives metrics, not a pass/fail score)
Determinism / reproducibility as a design goal	Yes (monotonic IDs, Kahan summation)	Not a stated goal	Not a stated goal	Not a stated goal
Maturity / ecosystem	New, single maintainer	Mature, large community	Mature, active	Mature but maintenance has slowed

None of these projects are wrong to skip a forensic layer — it's a deliberate scope choice, and their vectorized/pure-Python approaches win on other axes (vectorbt's speed, backtrader's dependency-free portability, Zipline's Quantopian-era ecosystem). This engine's bet is narrower: for research where a false-positive backtest is expensive to discover after capital is committed, catching the bug mechanically beats catching it by eyeballing an equity curve.

API overview

# Strategy authoring — subclass Strategy, implement on_bar
from engine.strategy.base import Strategy
from engine.strategy.context import Context
from engine.core.events import BarEvent

class MyStrategy(Strategy):
    def on_bar(self, ctx: Context, bar: BarEvent) -> None:
        ctx.buy(bar.instrument_id, quantity=1)   # or ctx.sell(...)

# Building and running a backtest
from engine.core.builder import BacktestBuilder
result = (BacktestBuilder()
    .set_strategy(MyStrategy())
    .set_bars(bars)
    .set_capital(100_000)
    .build()
    .run())

# result.equity_curve, result.fills, result.n_trades, result.total_pnl,
# result.conservation_report, result.total_commission are all populated.

docs/research/2026-04-29-quantconnect-api.md maps this engine's strategy-authoring surface against QuantConnect/LEAN's QCAlgorithm API, for anyone porting strategies or evaluating how far this is from a QC-portable interface.

Running tests

python -m pytest tests/ -q

A handful of upstream tests that depend on broker-specific integrations (an IBKR data-feed adapter, a Streamlit reporting dashboard) or on a consumer's private data-persistence layer are intentionally not included in this public package — they test integration code, not the engine. Each omission is called out inline in the relevant test file. One real-data test (TestMACrossoverRealData in tests/engine/test_phase5.py) looks for a CSV under data/ and skips cleanly if you haven't supplied one — data/ is gitignored so you can drop your own OHLCV file there.

Requirements

• Python 3.11+
• numpy, scipy, pandas (core — the only required runtime dependencies)
• pytest, hypothesis (dev/test only)

License

MIT — see LICENSE.