backtest-guard

Most backtests that look great are overfit. backtest-guard tells you if yours is.

You backtested a strategy. The Sharpe ratio is 2.1. It's tempting to believe you found an edge. But if you (or your parameter sweep, or your feature-selection loop) quietly tried 200 variants before landing on this one, a Sharpe of 2.1 is close to what you'd expect from the best of 200 pure-noise strategies — no edge required. backtest-guard runs the statistical tests that tell these two situations apart, plus the leakage checks that catch the bugs that manufacture fake edges in the first place.

It implements four well-established, peer-reviewed methods from the quantitative finance literature (see References) as a small, dependency-light, well-tested library:

1. Deflated Sharpe Ratio (DSR) and Probabilistic Sharpe Ratio (PSR) — is your Sharpe ratio statistically distinguishable from noise, once you account for non-normal returns and how many strategies you tried?
2. Probability of Backtest Overfitting (PBO) via Combinatorially Symmetric Cross-Validation — across many train/test splits, does the in-sample winner actually tend to win out-of-sample, or is it a coin flip?
3. PurgedKFold — a scikit-learn-compatible cross-validation splitter that purges and embargoes overlapping label windows, the #1 source of leakage in financial ML pipelines.
4. Lookahead & leakage detectors — heuristic fingerprints that flag features suspiciously correlated with future returns, and duplicate rows leaking across a train/test split.

Install

pip install backtest-guard

Core dependencies are just numpy and scipy. Pandas is optional — every function accepts a pandas.Series / pandas.DataFrame directly, or install the extra to make sure it's available:

pip install "backtest-guard[pandas]"

Quickstart

import numpy as np
from backtest_guard import audit

returns = np.random.default_rng(0).normal(0.0008, 0.01, 500)  # your strategy's daily returns
report = audit(returns, n_trials=1)

print(report.summary())
print(report.overall)  # Verdict.PASS / WARN / FAIL

Worked example: catching a lucky noise strategy

This is the scenario backtest-guard exists for. Imagine you swept 200 parameter combinations, backtested each, and kept the one with the best Sharpe ratio. The winner's raw Sharpe ratio looks excellent — but it was selected because it was the best of 200 noisy trials, not because it has real skill. Run it:

import numpy as np
from backtest_guard import audit, deflated_sharpe_ratio, pbo

rng = np.random.default_rng(9)
n_trials, n_periods = 200, 250

# 200 strategies, every one of them pure noise (mean zero returns).
candidates = rng.normal(0.0, 0.01, size=(n_trials, n_periods))
sharpes = candidates.mean(axis=1) / candidates.std(axis=1, ddof=1)
best_idx = int(np.argmax(sharpes))
best_returns = candidates[best_idx]

print(f"Best of {n_trials} noise strategies -- naive Sharpe: {sharpes[best_idx]:.2f}")
# Best of 200 noise strategies -- naive Sharpe: 0.13  (looks like a plausible small edge!)

dsr = deflated_sharpe_ratio(best_returns, n_trials=n_trials)
print(f"Deflated Sharpe Ratio: {dsr:.1%} probability of a real edge")
# Deflated Sharpe Ratio: 23.3% probability of a real edge  (not significant)

result = pbo(candidates.T, n_splits=16)
print(f"Probability of Backtest Overfitting: {result.pbo:.1%}")
# Probability of Backtest Overfitting: 80.8%  (strong overfitting signal)

report = audit(best_returns, n_trials=n_trials, returns_matrix=candidates.T)
print(report.summary())

backtest-guard audit -- overall verdict: FAIL

[PASS] Sample Sharpe ratio (0.1310)
         Raw per-period Sharpe ratio over 250 observations (skew=-0.15, excess kurtosis=0.76).
         Informational only -- see PSR/DSR below for corrected assessments.
[PASS] Probabilistic Sharpe Ratio (PSR) (0.9791)
         P(true Sharpe > 0.0) = 97.9%, correcting for skew, kurtosis, and sample length. The
         observed Sharpe is statistically credible on its own (before considering how many
         strategies were tried).
[FAIL] Deflated Sharpe Ratio (DSR) (0.2326)
         P(true Sharpe > 0) = 23.3% after deflating for having selected the best of 200 tried
         strategies (n_trials=200). This performance is plausibly explained by picking the best
         of many noise strategies -- classic backtest overfitting.
[PASS] Minimum track record length (163.6230)
         Need >= 164 observations for 95% confidence the true Sharpe exceeds 0.0; you have 250.
         Track record is long enough.
[FAIL] Probability of Backtest Overfitting (PBO) (0.8076)
         PBO = 80.8% across 12870 combinatorial train/test splits. In-sample winners tend to be
         below-median out-of-sample -- strong overfitting signal.

The raw Sharpe ratio (0.13) and even the plain PSR (97.9%!) look like a perfectly reasonable strategy. DSR and PBO both see through it once they account for the fact that 200 variants were tried: this is exactly what pure noise looks like after you pick the best of 200 tries. See examples/lucky_noise.py for the full runnable script, including the contrast against a strategy with genuine, simulated skill.

API

Function / class	Purpose
probabilistic_sharpe_ratio(returns, sharpe_benchmark=0.0)	P(true Sharpe > benchmark), correcting for skew/kurtosis/sample length
deflated_sharpe_ratio(returns, n_trials)	PSR against the expected max Sharpe of n_trials independent noise strategies
minimum_track_record_length(returns, sharpe_benchmark=0.0, confidence=0.95)	Minimum observations needed to trust the Sharpe estimate
pbo(returns_matrix, n_splits=16)	Probability of Backtest Overfitting via combinatorial train/test splits
PurgedKFold(n_splits, label_end_times, embargo_fraction)	scikit-learn-compatible splitter with purging + embargo
check_lookahead(features, returns)	Flags features suspiciously correlated with same-period vs. next-period returns
check_train_test_leakage(train_features, test_features)	Flags near-duplicate rows shared across a split
audit(returns, n_trials=1, returns_matrix=None)	Runs the full battery, returns an AuditReport with .summary() and per-check verdicts

Full docstrings (with the exact formulas and paper citations) are in the source — every public function documents which paper and equation it implements.

Using PurgedKFold with scikit-learn

from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
from backtest_guard import PurgedKFold

# label_end_times[i] = the bar index at which the label for observation i
# stops depending on future data (e.g. i + holding_period_in_bars).
splitter = PurgedKFold(n_splits=5, label_end_times=label_end_times, embargo_fraction=0.01)
scores = cross_val_score(RandomForestClassifier(), X, y, cv=splitter)

backtest-guard does not depend on scikit-learn — PurgedKFold just follows its split() / get_n_splits() convention, so it works with any estimator that accepts a CV splitter.

References

• Bailey, D. H., & Lopez de Prado, M. (2012). The Sharpe Ratio Efficient Frontier. Journal of Risk, 15(2), 3-44. SSRN 1821643
• Bailey, D. H., & Lopez de Prado, M. (2014). The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality. Journal of Portfolio Management, 40(5), 94-107. SSRN 2460551
• Bailey, D. H., Borwein, J., Lopez de Prado, M., & Zhu, Q. J. (2015). The Probability of Backtest Overfitting. Journal of Computational Finance, 20(4), 39-69. SSRN 2326253
• Lopez de Prado, M. (2018). Advances in Financial Machine Learning. Wiley. Chapter 7: "Cross-Validation in Finance."

Related work

• mlfinlab (and its fork mlfinpy) implements a much broader slice of Advances in Financial Machine Learning, including PurgedKFold-style splitters, but as part of a large, heavier framework covering feature engineering, portfolio optimization, and more.
• pypbo implements CSCV/PBO specifically, and was a useful reference implementation while building this library's PBO module.

backtest-guard is intentionally narrower: it does one job — "is this backtest statistically sound?" — with zero required configuration, a two-dependency install (numpy + scipy), a permissive MIT license, and active maintenance. If you already have a heavier ML pipeline, mlfinlab/mlfinpy may serve you better; if you want a quick, sharp answer to "is my backtest overfit," this is built for that single question.

Development

git clone https://github.com/AgentJDrew/backtest-guard.git
cd backtest-guard
python -m venv .venv
.venv/Scripts/activate  # or `source .venv/bin/activate` on macOS/Linux
pip install -e ".[dev]"
pytest
ruff check .

License

MIT (c) 2026 Andrew Lazzeroni. See LICENSE.