trade-statistics — Trade History Analytics & Review Automation Pipeline

Purpose

A pipeline that takes discretionary-trading history CSVs and statistically visualizes which conditions lose money and when decision quality degrades, automating the human review loop.

The design philosophy is not to automate trading decisions, but to produce review material for examining decision quality. False alarms (flagging harmless trades) and misses (overlooking dangerous decision patterns) are treated as distinct costs — an evaluation framing shared with equipment diagnostics and anomaly detection.

Features

• 9-step analysis pipeline: performance report → feature enrichment (100+ indicators) → avoid-condition mining → feature importance → currency strength → deep-dive diagnostics → HTML report → Pine parameter optimization → Pine code generation
• Multi-method feature importance: combines Gini importance, Random Forest, permutation importance, SHAP, and statistical tests into a composite rank to avoid single-method bias
• Leakage-aware threshold evaluation: Pine parameter thresholds are selected on chronological training data and evaluated on a later test split
• Human-discretion diagnostics (human_discretion_diagnostics.py): extracts contexts where judgment quality degrades — re-entries right after a loss (revenge-trade candidates), additional trades after exceeding a daily stop line, losing positions held too long
• Monte Carlo simulation: estimates risk-of-ruin and drawdown distributions via resampling
• Multi-broker support: normalizes CSV formats from different brokers into a unified round-trip format (merge_brokers.py)

Quick Start (reproducible with synthetic data)

No real data, broker account, or MT5 required — three commands from synthetic data to diagnostics.

pip install -r requirements.txt

# 1. Generate synthetic trade history (fixed seed, reproducible)
python scripts/generate_synthetic_sample.py --out data/sample_trades.csv

# 2. Performance profile (loss clusters, time buckets, daily-stop candidates)
python scripts/sample_data_profile.py data/sample_trades.csv --out-dir output

# 3. Discretion-degradation diagnostics
python scripts/human_discretion_diagnostics.py data/sample_trades.csv --out-dir output

Outputs are written to output/ as CSV / JSON / Markdown review files.

For real broker CSVs (supported: OANDA MT5, GMO Click Securities, Matsui Securities), see docs/OPERATIONS.md (Japanese).

Input Format

Normalized round-trip CSV: entry_datetime, exit_datetime, symbol, side_buy (1=long / 0=short), qty, entry_price, exit_price, pnl.

Architecture

broker CSV (OANDA / GMO / Matsui)
        │  merge_brokers.py (format normalization)
        ▼
normalized round-trip CSV
        │
        ├── sample_data_profile.py           performance segmentation
        ├── human_discretion_diagnostics.py  decision-degradation events
        ├── trade_feature_analysis.py        OHLCV feature enrichment (MT5 required)
        │       ├── avoid_conditions.py      avoid-condition mining
        │       ├── feature_importance_analysis.py  SHAP/RF/permutation composite
        │       └── generate_html_report.py  12-tab Plotly dashboard
        └── monte_carlo_sim.py               risk simulation

Details: docs/ARCHITECTURE.md, docs/OPERATIONS.md, docs/PINE_INTEGRATION.md (Japanese).

Evaluation Design

• Avoid-conditions and thresholds are validated with chronological train/test splits (random splits would leak future information)
• Feature importance combines several methods to expose method-dependent rankings; the composite rank is not a causal estimate
• Permutation tests check whether performance differences are distinguishable from chance
• Warning events are evaluated by their impact on subsequent PnL, not by raw counts

Limitations & Untested Areas

• The full feature-enriched pipeline (Step 2 onward) requires MT5 on Windows; synthetic data reproduces the performance profile and discretion diagnostics only
• Synthetic data mimics statistical properties (negative edge, fat tails) but not real market microstructure
• Avoid-conditions can overfit on small segments; out-of-sample re-validation is assumed before operational use
• Discretion-event definitions (e.g., revenge trades) are heuristics, not direct measurements of psychological state

Background

Built to solve a personal pain point in FX/CFD trading — "verbalizing losing patterns depends on subjective memory" — by learning the required statistics and ML techniques and implementing the review loop end to end. An example of solving a field problem by acquiring and applying the necessary technology.

License

MIT