From 9b5a9c1e382d56a2b6677d8f24b99b21256d9255 Mon Sep 17 00:00:00 2001 From: JLaborda <15078416+JLaborda@users.noreply.github.com> Date: Wed, 1 Jul 2026 07:52:13 +0000 Subject: [PATCH 1/5] docs(mvp): migrate specs to spec-driven structure Move MVP documentation under spec/ with constitution, numbered features, ADRs, guides, and GitHub Issues workflow. Add feature spec template. Co-authored-by: Cursor --- AGENTS.md | 51 +- CHANGELOG.md | 4 +- CONTEXT.md | 18 +- Makefile | 2 +- README.md | 16 +- docs/README.md | 87 +--- spec/README.md | 82 ++++ spec/adr/0001-simfin-fundamentals-mvp.md | 15 + spec/adr/0002-june-demo-scope-cut.md | 15 + spec/archive/requirements.md | 41 ++ spec/backlog/backlog.md | 16 + spec/constitution/mission.md | 384 +++++++++++++++ spec/constitution/roadmap.md | 123 +++++ spec/constitution/tech-stack.md | 103 ++++ spec/features/001-backtesting/spec.md | 162 +++++++ spec/features/002-broker-execution/spec.md | 127 +++++ spec/features/003-cheap-stocks/spec.md | 113 +++++ .../004-corroborative-signals/spec.md | 86 ++++ spec/features/005-dashboard-reporting/spec.md | 135 ++++++ spec/features/006-etl-data-lake/spec.md | 451 ++++++++++++++++++ spec/features/007-high-quality-stocks/spec.md | 111 +++++ .../008-permanent-loss-filter/spec.md | 129 +++++ spec/features/010-sell-watch/spec.md | 132 +++++ .../011-universe-construction/spec.md | 166 +++++++ .../012-unstructured-financial-data/spec.md | 103 ++++ spec/guides/download-fundamentals.md | 178 +++++++ spec/guides/download-simfin.md | 80 ++++ spec/meta/feature-spec-template.md | 101 ++++ spec/meta/github-issues.md | 50 ++ spec/prds/ci-cd/ci-cd-prd.md | 231 +++++++++ spec/prds/devcontainer/prd.md | 131 +++++ spec/prds/phase2/prd.md | 322 +++++++++++++ 32 files changed, 3635 insertions(+), 130 deletions(-) create mode 100644 spec/README.md create mode 100644 spec/adr/0001-simfin-fundamentals-mvp.md create mode 100644 spec/adr/0002-june-demo-scope-cut.md create mode 100644 spec/archive/requirements.md create mode 100644 spec/backlog/backlog.md create mode 100644 spec/constitution/mission.md create mode 100644 spec/constitution/roadmap.md create mode 100644 spec/constitution/tech-stack.md create mode 100644 spec/features/001-backtesting/spec.md create mode 100644 spec/features/002-broker-execution/spec.md create mode 100644 spec/features/003-cheap-stocks/spec.md create mode 100644 spec/features/004-corroborative-signals/spec.md create mode 100644 spec/features/005-dashboard-reporting/spec.md create mode 100644 spec/features/006-etl-data-lake/spec.md create mode 100644 spec/features/007-high-quality-stocks/spec.md create mode 100644 spec/features/008-permanent-loss-filter/spec.md create mode 100644 spec/features/010-sell-watch/spec.md create mode 100644 spec/features/011-universe-construction/spec.md create mode 100644 spec/features/012-unstructured-financial-data/spec.md create mode 100644 spec/guides/download-fundamentals.md create mode 100644 spec/guides/download-simfin.md create mode 100644 spec/meta/feature-spec-template.md create mode 100644 spec/meta/github-issues.md create mode 100644 spec/prds/ci-cd/ci-cd-prd.md create mode 100644 spec/prds/devcontainer/prd.md create mode 100644 spec/prds/phase2/prd.md diff --git a/AGENTS.md b/AGENTS.md index 4b84e11..701b97b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,7 +4,7 @@ Instructions for AI agents working in the SmartWealthAI repository. ## Project phase -SmartWealthAI is in **MVP planning and spec refinement**, with a **June 30, 2026 demo slice** as the current delivery target. Architecture and features are defined in Markdown under `docs/mvp/`. Implementation work must align with a feature spec in `docs/mvp/features/` and respect cross-cutting rules in `docs/mvp/architecture/architecture.md`. +SmartWealthAI is in **MVP planning and spec refinement**, with a **June 30, 2026 demo slice** as the current delivery target. Specifications live under `spec/`. Implementation work must align with a feature `spec.md` under `spec/features/` and respect cross-cutting rules in `spec/constitution/mission.md`. Do not treat `src/` or `notebooks/` as canonical architecture; they are legacy exploration and out of scope for planning context. @@ -13,13 +13,15 @@ Do not treat `src/` or `notebooks/` as canonical architecture; they are legacy e | Path | Purpose | | --- | --- | | `CONTEXT.md` | Ubiquitous language (domain terms; extend via `/grill-with-docs`) | -| `docs/adr/*.md` | Architecture Decision Records (hard-to-reverse choices) | -| `docs/mvp/demo-slice.md` | June 30 delivery target (narrow vertical slice) | -| `docs/mvp/architecture/architecture.md` | MVP vision, principles, module table, closed decisions | -| `docs/mvp/features/*.md` | Per-module specs (scope, acceptance criteria, diagrams) | -| `docs/mvp/requirements/requirements.md` | Sprint 0 spike (superseded by MVP specs; historical reference) | -| `docs/mvp/backlog/backlog.md` | Informal product ideas mapped to MVP features | -| `docs/README.md` | Index of the docs tree | +| `spec/adr/*.md` | Architecture Decision Records (hard-to-reverse choices) | +| `spec/constitution/roadmap.md` | June 30 delivery target (narrow vertical slice) | +| `spec/constitution/mission.md` | MVP vision, principles, module table, closed decisions | +| `spec/features/00N-slug/spec.md` | Per-module specs (scope, acceptance criteria); optional `plan.md` / `tasks.md` | +| `spec/constitution/tech-stack.md` | Technologies, infrastructure, runtime conventions | +| `spec/meta/github-issues.md` | GitHub Issues workflow for feature implementation | +| `spec/archive/requirements.md` | Sprint 0 spike (superseded by MVP specs; historical reference) | +| `spec/backlog/backlog.md` | Informal product ideas mapped to MVP features | +| `spec/README.md` | Index of the docs tree | See also `.cursor/rules/*.mdc` for persistent agent guidance. @@ -29,8 +31,8 @@ Write all code, comments, docstrings, documentation, commits, and PR text in **E ## Spec-driven workflow -1. Identify the feature spec (e.g. `docs/mvp/features/cheap-stocks.md`). -2. Read `docs/mvp/architecture/architecture.md` for constraints (point-in-time data, exclusions, MLflow, paper trading, etc.). +1. Identify the feature spec (e.g. `spec/features/003-cheap-stocks/spec.md`). +2. Read `spec/constitution/mission.md` for constraints (point-in-time data, exclusions, MLflow, paper trading, etc.). 3. Resolve open questions in the spec; record decisions in the spec or architecture doc. 4. Implement only what the spec allows for the current phase. 5. After implementation, update the feature spec (implementation status, acceptance criteria, links to code when it exists). @@ -72,7 +74,7 @@ export SIMFIN_API_KEY="" poetry run download-simfin ``` -Spec: [`docs/mvp/features/etl-data-lake.md`](docs/mvp/features/etl-data-lake.md). Operator guide: [`docs/mvp/guides/download-simfin.md`](docs/mvp/guides/download-simfin.md). Delivery target: [`docs/mvp/demo-slice.md`](docs/mvp/demo-slice.md). +Spec: [`spec/features/006-etl-data-lake/spec.md`](spec/features/006-etl-data-lake/spec.md). Operator guide: [`spec/guides/download-simfin.md`](spec/guides/download-simfin.md). Delivery target: [`spec/constitution/roadmap.md`](spec/constitution/roadmap.md). ### Fundamentals — SEC spike (frozen, phase 2) @@ -81,7 +83,7 @@ export SEC_IDENTITY="Your Name your@email.com" poetry run download-fundamentals --universe dow30 ``` -Guide: [`docs/mvp/guides/download-fundamentals.md`](docs/mvp/guides/download-fundamentals.md). +Guide: [`spec/guides/download-fundamentals.md`](spec/guides/download-fundamentals.md). ## Gotchas @@ -91,23 +93,14 @@ Guide: [`docs/mvp/guides/download-fundamentals.md`](docs/mvp/guides/download-fun - `data/` is gitignored except `data/reference/**` (versioned universe and mapping CSVs). Never commit personal finance files or raw broker exports. -## Notion (task tracking) +## GitHub Issues (task tracking) -Git specs in `docs/mvp/` are canonical. Notion tracks execution tasks only. Setup: [`docs/mvp/NOTION_SETUP.md`](docs/mvp/NOTION_SETUP.md). +Git specs in `spec/` are canonical. **GitHub Issues** track feature implementation. See [`spec/meta/github-issues.md`](spec/meta/github-issues.md). -### Default task board - -| Setting | Value | -| --- | --- | -| **Board name** | `Cursor Agent Tasks` | -| **Location** | This project's Notion workspace (connected via Cursor Notion MCP) | -| **Board URL in repo** | Not stored — use MCP OAuth and search by board name | - -Agents must use the Notion MCP server (authenticated in **Cursor → Settings → MCP**) to find and update this board. Search the workspace for `Cursor Agent Tasks` before creating or editing tasks. Do not ask the user to commit a Notion URL to the repository. - -Each task should reference a spec path (e.g. `docs/mvp/features/universe-construction.md`) and copy acceptance criteria from that spec. When a task completes, update the Git spec first, then mark the Notion task done. - -**Skills (after MCP auth):** `spec-to-implementation`, `create-task`, `tasks-build`, `tasks-explain-diff`. For `tasks-build`, the user supplies a single task URL in chat (not in this file). +- **One issue per feature** when implementation starts. +- Issue body must link `spec/features/00N-slug/spec.md` (and `plan.md` / `tasks.md` when they exist). +- Label `ready-for-agent` when the spec is complete and work can start. +- On completion: update `spec.md` and `tasks.md`, then close the issue. **Commit workflow:** `/commit_split` — project skill [`.cursor/skills/commit-split/SKILL.md`](.cursor/skills/commit-split/SKILL.md); splits into conventional commits; out-of-scope paths follow [`.gitignore`](.gitignore). @@ -117,7 +110,7 @@ Matt Pocock engineering skills ([`mattpocock/skills`](https://github.com/mattpoc ### Issue tracker -GitHub Issues on `JLaborda/SmartWealthAI` via the `gh` CLI. MVP specs in `docs/mvp/` remain canonical; Notion is optional for execution only. See [`.cursor/rules/issue-tracker.md`](.cursor/rules/issue-tracker.md). +GitHub Issues on `JLaborda/SmartWealthAI` via the `gh` CLI. MVP specs in `spec/` remain canonical. See [`.cursor/rules/issue-tracker.md`](.cursor/rules/issue-tracker.md) and [`spec/meta/github-issues.md`](spec/meta/github-issues.md). ### Triage labels @@ -125,4 +118,4 @@ Default vocabulary: `needs-triage`, `needs-info`, `ready-for-agent`, `ready-for- ### Domain docs -Single-context: `CONTEXT.md` + `docs/adr/` at repo root; MVP specs in `docs/mvp/` during planning. See [`.cursor/rules/domain.md`](.cursor/rules/domain.md). +Single-context: `CONTEXT.md` + `spec/adr/`; MVP specs in `spec/`. See [`.cursor/rules/domain.md`](.cursor/rules/domain.md). diff --git a/CHANGELOG.md b/CHANGELOG.md index 943c42a..3bd2247 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,12 +19,12 @@ First public release: the **June 30 demo slice** — a runnable Greenblatt-style - **Streamlit dashboard** — Overview, Ranking, and Portfolio pages (`run-dashboard`). - **MLflow run logging** — params, metrics, portfolio parquet artifact, and git commit SHA tag per pipeline run. - **Hermetic CI** — pytest fixtures; no live SimFin or yfinance calls in PR workflows. -- **MVP specs and ADRs** under `docs/mvp/` and `docs/adr/`. +- **MVP specs and ADRs** under `spec/` and `spec/adr/`. ### Requirements - Python 3.11+, Poetry. -- `SIMFIN_API_KEY` for live data download (see [download-simfin guide](docs/mvp/guides/download-simfin.md)). +- `SIMFIN_API_KEY` for live data download (see [download-simfin guide](spec/guides/download-simfin.md)). ### Quickstart diff --git a/CONTEXT.md b/CONTEXT.md index abc9705..41fc2d1 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -1,6 +1,6 @@ # SmartWealthAI -Ubiquitous language for the quantitative value-investing MVP. Canonical formulas and acceptance criteria live in `docs/mvp/`; this file is the concise vocabulary agents and humans share. Extend via `/grill-with-docs` when terms are resolved. +Ubiquitous language for the quantitative value-investing MVP. Canonical formulas and acceptance criteria live in `spec/`; this file is the concise vocabulary agents and humans share. Extend via `/grill-with-docs` when terms are resolved. ## Language @@ -29,7 +29,7 @@ Backtesting only companies that still exist today, overstating returns. Mitigate _Avoid_: living-universe backtest **Universe**: -Investable tickers for a run date. **June 30 demo:** all SimFin US companies minus banks/insurers/utilities (`IndustryId` exclusions + bank/insurance sanity check). **Phase 2:** historical S&P 500 constituents including delisted names. Spec: `docs/mvp/features/universe-construction.md`. +Investable tickers for a run date. **June 30 demo:** all SimFin US companies minus banks/insurers/utilities (`IndustryId` exclusions + bank/insurance sanity check). **Phase 2:** historical S&P 500 constituents including delisted names. Spec: `spec/features/011-universe-construction/spec.md`. _Avoid_: watchlist, portfolio, benchmark index today **Industry classification**: @@ -45,7 +45,7 @@ A company removed entirely from ranking (not a score penalty). Permanent loss an _Avoid_: penalize, down-rank, soft filter **Permanent loss filter**: -Hard exclusion for fraud or bankruptcy/distress risk before any score. Spec: `docs/mvp/features/permanent-loss-filter.md`. +Hard exclusion for fraud or bankruptcy/distress risk before any score. Spec: `spec/features/008-permanent-loss-filter/spec.md`. _Avoid_: risk score, stop-loss, drawdown rule **Review queue**: @@ -61,7 +61,7 @@ Income and cash-flow statements use SimFin **TTM**; balance sheet uses the lates _Avoid_: mixing balance-sheet TTM into ROC denominators, using annual income for ranking between rebalance dates **Return on capital (ROC)**: -`EBIT / (Net Working Capital + Net Fixed Assets)`. Quality factor; higher is better; cross-sectional **ROC rank** (1 = best). Spec: `docs/mvp/features/high-quality-stocks.md`. +`EBIT / (Net Working Capital + Net Fixed Assets)`. Quality factor; higher is better; cross-sectional **ROC rank** (1 = best). Spec: `spec/features/007-high-quality-stocks/spec.md`. _Avoid_: ROE, ROIC (unless explicitly that metric) **Net working capital (NWC)**: @@ -69,7 +69,7 @@ _Avoid_: ROE, ROIC (unless explicitly that metric) _Avoid_: total working capital without the excess-cash adjustment **Earnings yield (EY)**: -`EBIT / Enterprise Value`. Cheapness factor; higher is cheaper; cross-sectional **EY rank** (1 = cheapest). Spec: `docs/mvp/features/cheap-stocks.md`. +`EBIT / Enterprise Value`. Cheapness factor; higher is cheaper; cross-sectional **EY rank** (1 = cheapest). Spec: `spec/features/003-cheap-stocks/spec.md`. _Avoid_: dividend yield, earnings/price without EV **Enterprise value (EV)**: @@ -105,7 +105,7 @@ Simulated orders only; no real capital in the MVP. _Avoid_: live trading, shadow trading with real broker **Sell-watch**: -Daily monitor of model holdings for quality drop, fraud/bankruptcy, overvaluation, or opportunity cost; emits signals, no auto-execution. Spec: `docs/mvp/features/sell-watch.md`. +Daily monitor of model holdings for quality drop, fraud/bankruptcy, overvaluation, or opportunity cost; emits signals, no auto-execution. Spec: `spec/features/010-sell-watch/spec.md`. _Avoid_: stop-loss, trailing stop (deferred) **Sell signal**: @@ -113,7 +113,7 @@ Recommendation to exit a holding; requires explicit user confirmation before ord _Avoid_: auto-sell, trim (deferred state) **Walk-forward backtest**: -Rolling train/validation windows (3–5 years) over 20+ years of PIT data; annual rebalance. Spec: `docs/mvp/features/backtesting.md`. **Deferred to phase 2** for the June 30 demo MVP; demo slice stops at ranked model portfolio + dashboard. +Rolling train/validation windows (3–5 years) over 20+ years of PIT data; annual rebalance. Spec: `spec/features/001-backtesting/spec.md`. **Deferred to phase 2** for the June 30 demo MVP; demo slice stops at ranked model portfolio + dashboard. _Avoid_: single in-sample fit, peeking at hold-out (when backtest ships) **Block bootstrap**: @@ -138,7 +138,7 @@ _Avoid_: ad-hoc snapshot without run id ## Flagged ambiguities -Resolved scope cuts (see ADRs and [`docs/mvp/demo-slice.md`](docs/mvp/demo-slice.md)): +Resolved scope cuts (see ADRs and [`spec/constitution/roadmap.md`](spec/constitution/roadmap.md)): - **June 30 demo MVP:** SimFin bulk US → raw → normalizer → **universe (US market)** → ROC/EY → combined rank → top-30 EW model portfolio → Streamlit dashboard. No permanent loss filter, backtest, sell-watch, or paper trading in this slice. - SEC ETL spike (`sec_client`, `edgartools_client`, `download-fundamentals`) is **frozen** in repo for phase 2; demo pipeline uses SimFin bulk for fundamentals and run-date prices (`shareprices/latest`). @@ -149,4 +149,4 @@ Terminology reminders: - “Cheap” means high **EY**, not low P/E—use **EY rank** in issues and code names. - “Quality” means high **ROC**, not ESG or subjective moat—use **ROC rank**. - “Value trap” in specs means negative EBIT routed to **review queue**, not a separate score. -- MVP specs in `docs/mvp/` remain canonical until an ADR or architecture decision supersedes them; update `CONTEXT.md` when `/grill-with-docs` resolves a term conflict. +- MVP specs in `spec/` remain canonical until an ADR or architecture decision supersedes them; update `CONTEXT.md` when `/grill-with-docs` resolves a term conflict. diff --git a/Makefile b/Makefile index 665216f..6479df9 100644 --- a/Makefile +++ b/Makefile @@ -10,7 +10,7 @@ lint: test: poetry run pytest --cov=smartwealthai --cov-report=term-missing -# Requires SEC_IDENTITY in the environment. See docs/mvp/guides/download-fundamentals.md +# Requires SEC_IDENTITY in the environment. See spec/guides/download-fundamentals.md download-fundamentals: poetry run download-fundamentals --universe dow30 diff --git a/README.md b/README.md index d69378d..8fdc9a4 100644 --- a/README.md +++ b/README.md @@ -5,13 +5,13 @@ **A quantitative value-investing MVP: Greenblatt-style ranking on US equities.** -*Status: **v0.1.0** — [June 30 demo slice](docs/mvp/demo-slice.md) runnable from CLI (Poetry + SimFin API key).* +*Status: **v0.1.0** — [June 30 demo slice](spec/constitution/roadmap.md) runnable from CLI (Poetry + SimFin API key).* ## Project vision SmartWealthAI is a modular quantitative value investing system: SimFin fundamentals, point-in-time correctness, explainable ROC/EY ranking, and a Streamlit dashboard. The full architecture (backtest, sell-watch, paper trading) is the north star; the demo slice ships a narrower vertical first. -Canonical specs: [`docs/mvp/`](docs/mvp/) · Ubiquitous language: [`CONTEXT.md`](CONTEXT.md) · ADRs: [`docs/adr/`](docs/adr/) +Canonical specs: [`spec/`](spec/) · Ubiquitous language: [`CONTEXT.md`](CONTEXT.md) · ADRs: [`spec/adr/`](spec/adr/) ## Tech stack @@ -20,7 +20,7 @@ Canonical specs: [`docs/mvp/`](docs/mvp/) · Ubiquitous language: [`CONTEXT.md`] * **Data (demo):** SimFin bulk fundamentals and `shareprices/latest` (`simfin`) * **Data (phase 2):** `yfinance` and free vendor fallbacks for prices / personal NAV * **Core libraries:** `pandas`, `simfin`, `yfinance`, `requests` (SEC spike: `edgartools` — frozen) -* **MVP specs:** `docs/mvp/` (architecture + per-module features) +* **MVP specs:** `spec/` (architecture + per-module features) ## 🚀 Quickstart @@ -35,7 +35,7 @@ Canonical specs: [`docs/mvp/`](docs/mvp/) · Ubiquitous language: [`CONTEXT.md`] ``` 3. **Demo slice docs** — start here before coding: - [`docs/mvp/demo-slice.md`](docs/mvp/demo-slice.md) + [`spec/constitution/roadmap.md`](spec/constitution/roadmap.md) 4. **Run the full demo pipeline** (one command for a `run_date`): @@ -44,7 +44,7 @@ Canonical specs: [`docs/mvp/`](docs/mvp/) · Ubiquitous language: [`CONTEXT.md`] poetry run run-demo-pipeline --run-date 2026-06-18 ``` - Individual stages (`download-simfin`, `normalize-simfin`, `build-universe`, `score-universe`, …) are also available. Guide: [`docs/mvp/guides/download-simfin.md`](docs/mvp/guides/download-simfin.md). + Individual stages (`download-simfin`, `normalize-simfin`, `build-universe`, `score-universe`, …) are also available. Guide: [`spec/guides/download-simfin.md`](spec/guides/download-simfin.md). 5. **Demo dashboard** (after the pipeline for the same `run_date`): @@ -52,7 +52,7 @@ Canonical specs: [`docs/mvp/`](docs/mvp/) · Ubiquitous language: [`CONTEXT.md`] poetry run run-dashboard --data-dir data --run-date 2026-06-18 ``` - Spec: [`docs/mvp/features/dashboard-reporting.md`](docs/mvp/features/dashboard-reporting.md). + Spec: [`spec/features/005-dashboard-reporting/spec.md`](spec/features/005-dashboard-reporting/spec.md). 6. **SEC fundamentals spike (frozen, phase 2):** ```bash @@ -60,11 +60,11 @@ Canonical specs: [`docs/mvp/`](docs/mvp/) · Ubiquitous language: [`CONTEXT.md`] poetry run download-fundamentals --universe dow30 ``` - Guide: [`docs/mvp/guides/download-fundamentals.md`](docs/mvp/guides/download-fundamentals.md). + Guide: [`spec/guides/download-fundamentals.md`](spec/guides/download-fundamentals.md). ## Roadmap -See [`docs/mvp/demo-slice.md`](docs/mvp/demo-slice.md) for the **June 30, 2026** delivery target and [`docs/mvp/architecture/architecture.md`](docs/mvp/architecture/architecture.md) for the full MVP north star. +See [`spec/constitution/roadmap.md`](spec/constitution/roadmap.md) for the **June 30, 2026** delivery target and [`spec/constitution/mission.md`](spec/constitution/mission.md) for the full MVP north star. ### Demo slice (v0.1.0) diff --git a/docs/README.md b/docs/README.md index 6eae17a..f93d3ee 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,84 +1,9 @@ -# SmartWealthAI documentation +# Documentation moved -All MVP architecture and feature specifications live here. This tree is the **versioned source of truth** for the project. Notion is used only for task tracking (see [AGENTS.md](../AGENTS.md)). +All specifications moved to [`spec/`](../spec/README.md). -## Ubiquitous language +- Constitution: [`spec/constitution/`](../spec/constitution/) +- Feature specs: [`spec/features/`](../spec/features/) +- ADRs: [`spec/adr/`](../spec/adr/) -Domain vocabulary lives in [`CONTEXT.md`](../CONTEXT.md) at the repo root (Matt Pocock [single-context](https://github.com/mattpocock/skills) pattern). Extend it with `/grill-with-docs`; agent consumption rules are in [`.cursor/rules/domain.md`](../.cursor/rules/domain.md). - -## Layout - -```text -docs/ - adr/ # Architecture Decision Records (why, not how) - mvp/ - demo-slice.md # June 30 delivery target (narrow vertical) - architecture/architecture.md # MVP vision, principles, module table, decisions - features/ # One spec per module - guides/ # Operator guides for implemented slices - prds/ # Implementation PRDs (tooling and cross-cutting) - requirements/requirements.md # Sprint 0 spike (historical; superseded by MVP specs) - backlog/backlog.md # Informal ideas mapped to features -``` - -## Architecture Decision Records (ADRs) - -Short notes on hard-to-reverse choices. See [`docs/adr/`](../adr/). - -| ADR | Decision | -| --- | --- | -| [0001](../adr/0001-simfin-fundamentals-mvp.md) | SimFin fundamentals for MVP; SEC ETL phase 2 | -| [0002](../adr/0002-june-demo-scope-cut.md) | June 30 demo slice scope cut | - -## Demo slice - -**Current delivery target:** [`demo-slice.md`](mvp/demo-slice.md) — SimFin → US universe → ROC/EY → top-30 portfolio → dashboard. - -## Operator guides - -| Guide | When to use | -| --- | --- | -| [download-simfin.md](mvp/guides/download-simfin.md) | **Active demo path:** download SimFin bulk fundamentals and `shareprices/latest`, then build universe and run-date price snapshots. | -| [download-fundamentals.md](mvp/guides/download-fundamentals.md) | **Frozen SEC spike** (phase 2): download `companyfacts` + `edgartools` for a universe. Active demo path uses SimFin — see [`demo-slice.md`](mvp/demo-slice.md) and [`etl-data-lake.md`](mvp/features/etl-data-lake.md). | - -## PRDs - -| PRD | Scope | -| --- | --- | -| [devcontainer/prd.md](mvp/prds/devcontainer/prd.md) | Reproducible dev environment (Cursor / EC2) | -| [ci-cd/ci-cd-prd.md](mvp/prds/ci-cd/ci-cd-prd.md) | Phase 0 CI/CD, AWS integration, ECS deploy path | -| [phase2/prd.md](mvp/prds/phase2/prd.md) | Phase 2: Quantitative Value funnel, cloud deploy, light/full backtest, sell-watch | - -## Feature specs - -| Spec | Module | -| --- | --- | -| [etl-data-lake.md](mvp/features/etl-data-lake.md) | ETL and data lake | -| [universe-construction.md](mvp/features/universe-construction.md) | Universe construction | -| [permanent-loss-filter.md](mvp/features/permanent-loss-filter.md) | Permanent loss filter | -| [high-quality-stocks.md](mvp/features/high-quality-stocks.md) | Quality scoring | -| [cheap-stocks.md](mvp/features/cheap-stocks.md) | Cheapness scoring | -| [corroborative-signals.md](mvp/features/corroborative-signals.md) | Corroborative signals | -| [unstructured-financial-data.md](mvp/features/unstructured-financial-data.md) | Unstructured data | -| [backtesting.md](mvp/features/backtesting.md) | Backtesting | -| [sell-watch.md](mvp/features/sell-watch.md) | Sell watch | -| [portfolio-evolution.md](mvp/features/portfolio-evolution.md) | Portfolio evolution (model vs personal vs benchmarks) | -| [dashboard-reporting.md](mvp/features/dashboard-reporting.md) | Dashboard and reporting | -| [broker-execution.md](mvp/features/broker-execution.md) | Broker execution (paper) | - -Start with [architecture.md](mvp/architecture/architecture.md) for global constraints, then open the feature spec for the area you are changing. - -## Spec-driven workflow - -1. Read or update the relevant feature spec. -2. Align with architecture decisions (point-in-time data, exclusions, paper trading, etc.). -3. Implement only what the spec allows for the current phase. -4. Update the spec after implementation (status, acceptance criteria, code links). - -## Notion tasks - -- Duplicate the [Code with Notion board template](https://notion.notion.site/code-with-notion-board). -- Link each task to a spec path under `docs/mvp/features/`. -- Task board: **`Cursor Agent Tasks`** in Notion (MCP OAuth; see [NOTION_SETUP.md](mvp/NOTION_SETUP.md) and [AGENTS.md](../AGENTS.md)). - -Cursor agent rules: `.cursor/rules/*.mdc`. +Ubiquitous language remains in [`CONTEXT.md`](../CONTEXT.md) at the repo root. diff --git a/spec/README.md b/spec/README.md new file mode 100644 index 0000000..0c8b1bc --- /dev/null +++ b/spec/README.md @@ -0,0 +1,82 @@ +# SmartWealthAI specifications + +All MVP architecture and feature specifications live here. This tree is the **versioned source of truth** for the project. GitHub Issues track execution (see [`meta/github-issues.md`](meta/github-issues.md)). + +## Ubiquitous language + +Domain vocabulary lives in [`CONTEXT.md`](../CONTEXT.md) at the repo root. Extend it with `/grill-with-docs`; agent consumption rules are in [`.cursor/rules/domain.md`](../.cursor/rules/domain.md). + +## Layout + +```text +spec/ + README.md + constitution/ + mission.md # Vision, principles, module map, closed domain decisions + tech-stack.md # Technologies, infrastructure, runtime conventions + roadmap.md # Delivery phases, demo slice, feature priority + features/ + 00N-slug/ + spec.md # Scope and acceptance criteria (always) + plan.md # Implementation plan (when in progress) + tasks.md # Versioned checklist (when in progress) + adr/ # Architecture Decision Records + guides/ # Operator guides for implemented slices + prds/ # Tooling and cross-cutting PRDs + meta/ # Agent workflow conventions + backlog/ # Informal product ideas + archive/ # Superseded historical specs +``` + +## Constitution + +| Document | Purpose | +| --- | --- | +| [`mission.md`](constitution/mission.md) | What we build, for whom, principles, module map | +| [`tech-stack.md`](constitution/tech-stack.md) | Python, AWS, SimFin, MLflow, CI/CD | +| [`roadmap.md`](constitution/roadmap.md) | June 30 demo slice, phase 2 order, feature registry | + +## Architecture Decision Records + +| ADR | Decision | +| --- | --- | +| [0001](adr/0001-simfin-fundamentals-mvp.md) | SimFin fundamentals for MVP; SEC ETL phase 2 | +| [0002](adr/0002-june-demo-scope-cut.md) | June 30 demo slice scope cut | + +## Operator guides + +| Guide | When to use | +| --- | --- | +| [download-simfin.md](guides/download-simfin.md) | **Active demo path:** SimFin bulk fundamentals and prices | +| [download-fundamentals.md](guides/download-fundamentals.md) | **Frozen SEC spike** (phase 2) | + +## Feature specs + +| ID | Module | Spec | +| --- | --- | --- | +| 001 | Backtesting | [spec.md](features/001-backtesting/spec.md) | +| 002 | Broker execution | [spec.md](features/002-broker-execution/spec.md) | +| 003 | Cheap stocks | [spec.md](features/003-cheap-stocks/spec.md) | +| 004 | Corroborative signals | [spec.md](features/004-corroborative-signals/spec.md) | +| 005 | Dashboard reporting | [spec.md](features/005-dashboard-reporting/spec.md) | +| 006 | ETL + data lake | [spec.md](features/006-etl-data-lake/spec.md) | +| 007 | High-quality stocks | [spec.md](features/007-high-quality-stocks/spec.md) | +| 008 | Permanent loss filter | [spec.md](features/008-permanent-loss-filter/spec.md) | +| 009 | Portfolio evolution | [spec.md](features/009-portfolio-evolution/spec.md) | +| 010 | Sell-watch | [spec.md](features/010-sell-watch/spec.md) | +| 011 | Universe construction | [spec.md](features/011-universe-construction/spec.md) | +| 012 | Unstructured financial data | [spec.md](features/012-unstructured-financial-data/spec.md) | + +Start with [`constitution/mission.md`](constitution/mission.md) for global constraints, then open the feature spec for the area you are changing. + +## Spec-driven workflow + +1. Read or update the relevant feature `spec.md`. +2. Align with [`constitution/mission.md`](constitution/mission.md) (point-in-time data, exclusions, paper trading, etc.). +3. When implementation starts, add `plan.md` and `tasks.md` under the feature folder. +4. Open **one GitHub issue per feature** (see [`meta/github-issues.md`](meta/github-issues.md)). +5. Update the spec after implementation (status, acceptance criteria, code links). + +## New feature specs + +Copy [`meta/feature-spec-template.md`](meta/feature-spec-template.md) into `spec/features/00N-slug/spec.md`. Register the ID in [`constitution/roadmap.md`](constitution/roadmap.md). diff --git a/spec/adr/0001-simfin-fundamentals-mvp.md b/spec/adr/0001-simfin-fundamentals-mvp.md new file mode 100644 index 0000000..287e94c --- /dev/null +++ b/spec/adr/0001-simfin-fundamentals-mvp.md @@ -0,0 +1,15 @@ +--- +status: accepted +--- + +# SimFin as MVP fundamentals source (SEC ETL deferred) + +For the June 30 demo and the near-term MVP pipeline, US fundamentals and **run-date share prices** come from **SimFin** (free tier, bulk download via the `simfin` Python package), not SEC EDGAR. Demo prices use SimFin bulk `shareprices/latest` joined to the universe by ticker. The existing SEC spike (`sec_client`, `edgartools_client`, `download-fundamentals`) stays in the repo **frozen** for phase 2; `yfinance` remains a possible fallback for phase 2 backtests and personal NAV, not the demo pipeline. + +**Why:** SEC ETL complexity and rate limits were blocking progress on the scoring pipeline. SimFin provides standardized income, balance, and cash-flow statements with `Publish Date` / `Restated Date` for point-in-time queries, ~20 years of US history on the free tier, and a separate industry taxonomy—enough to ship a Magic Formula demo by end of June. + +**Trade-offs:** `as_of_date` uses SimFin `Publish Date` (not EDGAR acceptance). Phase 2 SEC ingestion may require a reconciliation or re-backtest. SimFin free-tier datasets refresh roughly weekly, which is acceptable for annual-rebalance logic but not for intraday freshness. + +**Considered:** Continue with SEC `companyfacts` only (rejected for June deadline); paid vendors Bloomberg/FactSet (out of budget). + +**Consequences:** Update `etl-data-lake.md` and `universe-construction.md`; add SimFin connector + normalizer; keep `curated/fundamentals` schema provider-agnostic so scoring modules do not change. diff --git a/spec/adr/0002-june-demo-scope-cut.md b/spec/adr/0002-june-demo-scope-cut.md new file mode 100644 index 0000000..9231a0d --- /dev/null +++ b/spec/adr/0002-june-demo-scope-cut.md @@ -0,0 +1,15 @@ +--- +status: accepted +--- + +# June 30 demo slice — simplest Magic Formula vertical + +The **June 30 deliverable** is a reduced vertical slice, not the full architecture vision. Pipeline: **SimFin ETL → universe (US market) → ROC/EY → combined rank → top-30 equal-weight model portfolio → Streamlit dashboard**. Permanent loss filter, backtesting, sell-watch, paper trading, watchlist, and corroborative/unstructured modules are **deferred to phase 2**. + +**Why:** The project is behind schedule. The portfolio demo must show an explainable, end-to-end Magic Formula run on real data—not every MLOps and risk gate in the full spec. + +**Trade-offs:** No backtest gate before orders (orders are out of scope anyway). Universe is all SimFin US companies minus banks/insurers/utilities—not historical S&P 500 with delisted names (survivorship bias mitigation waits for backtest phase). No permanent-loss filter in the demo path. + +**Considered:** Full MVP including 20-year walk-forward backtest (rejected for June); demo + minimal backtest (rejected—user chose fastest path); Zipline for backtests (rejected—incompatible with Python 3.11, unmaintained, poor fit for fundamental annual rebalance). + +**Consequences:** Documented in [`spec/constitution/roadmap.md`](../constitution/roadmap.md). Full feature specs remain the north star; modules marked deferred are unchanged in intent. MLflow logs **pipeline runs** in the demo; the `backtesting` experiment starts in phase 2. diff --git a/spec/archive/requirements.md b/spec/archive/requirements.md new file mode 100644 index 0000000..7d23520 --- /dev/null +++ b/spec/archive/requirements.md @@ -0,0 +1,41 @@ +> **Superseded.** Historical Sprint 0 spike. Canonical specs live under `spec/`. + +# Sprint 0 requirements: Magic Formula screener spike + +> **Status:** Historical reference only. The full MVP is defined in [mission.md](../constitution/mission.md) and [features/](../features/). Do not treat this document as the current MVP scope. + +## Primary goal + +Build a minimal Python pipeline that downloads financial data for a very small set of tickers, computes a simple ranking, and prints the result to the console. + +## Functional requirements + +### Input + +- Start from a **static hardcoded list** of 5–10 known tickers (e.g. `["AAPL", "MSFT", "GOOGL", "JNJ", "KO"]`). +- Do not download the full S&P 500 in this spike (API rate limits and runtime). + +### Processing + +- Connect to a free API (recommended: `yfinance`). +- Fetch proxy metrics for the Magic Formula: + - **Return on Capital (ROC)**, or fallback **ROE** / **ROA** + - **Earnings Yield**, or fallback inverse **P/E** +- Rank each metric from 1 to N across the universe and **sum ranks** for a final Magic Rank. + +### Output + +- Print the final ranking to the console (plain `print` or a small pandas table), best to worst. + +## Technical requirements + +- **Language:** Python 3.x (project now standardizes on 3.11+ via Poetry) +- **Libraries:** `yfinance`, `pandas` +- **Version control:** Git with a few local commits + +## Explicitly out of scope for Sprint 0 + +- Databases, GUI, Docker, machine learning +- Downloading thousands of tickers + +Those belong to later MVP modules documented under `spec/features/`. diff --git a/spec/backlog/backlog.md b/spec/backlog/backlog.md new file mode 100644 index 0000000..f6d6b40 --- /dev/null +++ b/spec/backlog/backlog.md @@ -0,0 +1,16 @@ +# Product backlog (informal) + +Informal ideas from the product owner. Each item should eventually map to one or more [feature specs](features/) and [mission.md](../constitution/mission.md). GitHub issues should reference the relevant spec path. + +| # | Idea | Likely MVP feature(s) | +| --- | --- | --- | +| 1 | Detect financial problems with holdings in my portfolio | [sell-watch.md](../010-sell-watch/spec.md), [permanent-loss-filter.md](../008-permanent-loss-filter/spec.md), [dashboard-reporting.md](../005-dashboard-reporting/spec.md) | +| 2 | Track portfolio evolution over time and compare to benchmarks (e.g. S&P 500) | [dashboard-reporting.md](../005-dashboard-reporting/spec.md), [backtesting.md](../001-backtesting/spec.md) | +| 3 | AI-assisted detector for undervalued stocks (Peter Lynch filters, Magic Formula) | [cheap-stocks.md](../003-cheap-stocks/spec.md), [high-quality-stocks.md](../007-high-quality-stocks/spec.md), [universe-construction.md](../011-universe-construction/spec.md) | +| 4 | Diversification analysis beyond sector labels (clustering in growth vs contraction regimes) | [corroborative-signals.md](../004-corroborative-signals/spec.md), [dashboard-reporting.md](../005-dashboard-reporting/spec.md) — may need a future spec | +| 5 | Detect overvalued positions where selling or trimming may make sense | [sell-watch.md](../010-sell-watch/spec.md), [cheap-stocks.md](../003-cheap-stocks/spec.md) | +| 6 | Rebalancing guidance (owner questions whether rebalance fits buy-cheap / sell-dear philosophy) | [mission.md](../constitution/mission.md) (annual rebalance decision), [broker-execution.md](../002-broker-execution/spec.md) | + +## Priority + +Ordering is not fixed here. During MVP planning, promote items into feature specs with acceptance criteria before implementation. diff --git a/spec/constitution/mission.md b/spec/constitution/mission.md new file mode 100644 index 0000000..ffbd4fe --- /dev/null +++ b/spec/constitution/mission.md @@ -0,0 +1,384 @@ +# Mission + +This document captures product vision, architecture principles, module map, and closed domain decisions for the SmartWealthAI MVP. + +This is a portfolio project intended to showcase MLOps practices applied to a quantitative value investing system. The end user is a particular investor, but the system itself behaves as an automated agent that runs end-to-end without manual intervention. + +## June 30 demo slice (current delivery target) + +The **first shippable vertical** is narrower than the full vision below. See [`roadmap.md`](roadmap.md) and [ADR-0002](../adr/0002-june-demo-scope-cut.md): SimFin ETL → US-market universe → ROC/EY → top-30 equal-weight portfolio → Streamlit dashboard. Backtest, permanent loss filter, sell-watch, and paper trading are **phase 2**. The full architecture in this document remains the north star. + +## MVP vision + +Build a modular quantitative value investing system that: + +1. Retrieves financial data from **SimFin** (fundamentals, demo) and free price providers; SEC EDGAR deferred to phase 2 ([ADR-0001](../adr/0001-simfin-fundamentals-mvp.md)). +2. Stores raw and curated data in an AWS-based, incrementally refreshed data lake. +3. Filters out companies with high risk of permanent capital loss (fraud and bankruptcy). +4. Identifies high-quality companies. +5. Identifies cheap companies. +6. Uses corroborative signals to strengthen or weaken investment theses. +7. Analyzes unstructured financial data (filings, transcripts, news). +8. Builds a ranked watchlist and a long-only model portfolio of 15 to 30 US stocks. +9. Backtests the strategy over 20+ years against benchmarks and historical crises. +10. Continuously monitors the model portfolio to detect sell signals. +11. Visualizes the evolution of the model portfolio and of the user's personal portfolio. +12. Prepares broker orders in paper trading mode only. +13. Surfaces every decision in a dashboard with auditable explanations. + +The MVP prioritizes traceability, reproducibility, point-in-time correctness, low operational cost, and a clean separation between data ingestion, rules, scoring, portfolio construction, monitoring, and execution. + +## Architecture principles + +- **Modularity**: each module evolves independently and can be replaced without rewriting downstream code. +- **Traceability and explainability**: every score, exclusion, buy, or sell decision is reconstructible from its input data and rule version. +- **Reproducibility**: a run over a given universe and date can be replayed bit-for-bit from versioned data and code. +- **Point-in-time correctness**: no module is allowed to use data that was not yet publicly available at the decision date. Look-ahead bias is treated as a critical defect. +- **Raw before transformed**: provider responses are stored verbatim in a raw zone before any normalization, so any bug downstream can be replayed from source. +- **Incremental data lake**: new filings or prices update only what changed; we never reprocess the full history unless we explicitly request it. +- **Provider abstraction**: SimFin, SEC EDGAR (phase 2), free price providers, and future sources are wrapped behind interchangeable connectors; `curated/fundamentals` schema is provider-agnostic. +- **AWS-first, cheapest-first**: data lake, compute, secrets, and dashboard all live in AWS, choosing the cheapest viable option at MVP scale. Heavier infrastructure (Kubernetes, paid data) is a growth path, not an MVP requirement. +- **MLOps and CI/CD by design**: every pipeline component is built, tested, packaged, deployed, scheduled, and observed. +- **Paper trading first**: the broker module never touches real money in the MVP, even by accident. +- **Backtesting before broker**: the strategy must pass a backtest before any order, even a paper one, is generated. **Does not apply to the June demo slice** (no broker in demo). +- **Specs before code**: this document and the feature specs are refined before implementation begins. + +## Architecture diagram + +```mermaid +flowchart LR + subgraph Sources["Data sources"] + SimFin["SimFin (fundamentals, MVP)"] + SEC["SEC EDGAR (phase 2)"] + Prices["Prices: SimFin shareprices/latest (demo) + yfinance / vendor fallback (phase 2)"] + UserPort["User portfolio CSV (data/clean/personal_finance/...)"] + News["News and transcripts (later)"] + end + + subgraph ETL["ETL + Data Lake (S3 + DuckDB, incremental)"] + Ingest["Ingestion connectors"] + Raw["Raw zone (immutable)"] + Normalized["Normalized / curated zone (versioned schema)"] + PITStore["Point-in-time store (as_of_date = SimFin Publish Date)"] + QualityChecks["Data quality + review queue"] + end + + subgraph Universe["Universe construction"] + SP500Hist["Universe: SimFin US (demo) / S&P 500 historical (phase 2)"] + UniFilters["Filters: IndustryId exclusion banks / insurers / utilities"] + end + + subgraph Analysis["Analysis engine"] + PermanentLoss["Permanent loss filter (fraud + bankruptcy)"] + Quality["Quality score (ROC)"] + Cheapness["Cheapness score (Earnings Yield)"] + Signals["Corroborative signals"] + Unstructured["Unstructured data analysis"] + end + + subgraph Decision["Investment decision"] + RiskGate["Risk gate (max 10% per name)"] + Ranking["Greenblatt-style ranking + market-cap tie-break"] + Sizing["Portfolio construction (15-30 names, EW/SW/RP)"] + Watchlist["Watchlist + model portfolio"] + end + + subgraph Live["Live monitoring (daily)"] + SellWatch["Sell-watch: quality drop + fraud/bankruptcy + overvaluation + opportunity cost"] + PortfolioEvo["Portfolio evolution (model vs personal vs benchmarks)"] + EmailAlerts["Dashboard + AWS SES email"] + end + + subgraph Validation["Validation"] + Backtest["Backtest (>= 20 years, annual rebalance, walk-forward 3-5y) + Monte Carlo"] + CrisisReport["Crisis drawdown report (informational)"] + Benchmarks["Benchmarks: S&P 500 CW + S&P 500 EW + Russell 3000 + Magic Formula"] + end + + subgraph Execution["Execution"] + OrderBuilder["Order builder (manual confirmation)"] + Paper["Paper trading"] + end + + subgraph Ops["MLOps + Reporting"] + Snapshots["MLflow runs + S3 artifacts (immutable snapshots)"] + Dashboard["Streamlit dashboard"] + CICD["GitHub Actions CI/CD + Prefect orchestration"] + Runtime["ECS Fargate Spot (or AWS Batch) tasks"] + Secrets["GitHub Secrets (build) + AWS Secrets Manager (runtime)"] + end + + SimFin --> Ingest + SEC -. "phase 2" .-> Ingest + Prices --> Ingest + UserPort --> Ingest + News --> Ingest + + Ingest --> Raw + Raw --> Normalized + Normalized --> PITStore + PITStore --> QualityChecks + + SP500Hist --> UniFilters + UniFilters --> PermanentLoss + QualityChecks --> PermanentLoss + QualityChecks --> Quality + QualityChecks --> Cheapness + QualityChecks --> Signals + QualityChecks --> Unstructured + + PermanentLoss --> RiskGate + Quality --> Ranking + Cheapness --> Ranking + Signals --> Ranking + Unstructured --> Ranking + RiskGate --> Ranking + Ranking --> Sizing + Sizing --> Watchlist + + PITStore --> Backtest + Ranking --> Backtest + Sizing --> Backtest + Benchmarks --> Backtest + Backtest --> CrisisReport + Backtest -. "Sharpe > all benchmarks" .-> OrderBuilder + + Watchlist --> OrderBuilder + OrderBuilder --> Paper + + Watchlist --> SellWatch + QualityChecks --> SellWatch + Ranking --> SellWatch + SellWatch --> EmailAlerts + SellWatch -. "after user confirmation" .-> OrderBuilder + + Paper --> PortfolioEvo + UserPort --> PortfolioEvo + Benchmarks --> PortfolioEvo + + Ranking --> Snapshots + Sizing --> Snapshots + SellWatch --> Snapshots + Backtest --> Snapshots + Snapshots --> Dashboard + + Secrets --> Ingest + Secrets --> Paper + CICD --> Dashboard +``` + +## MVP modules + +| Module | Spec | Main responsibility | +| --- | --- | --- | +| ETL + Data Lake | [../features/006-etl-data-lake/spec.md](../features/006-etl-data-lake/spec.md) | Download, version, validate, and store financial data with point-in-time guarantees and incremental refresh. | +| Universe construction | [../features/011-universe-construction/spec.md](../features/011-universe-construction/spec.md) | **Demo:** SimFin US minus banks / insurers / utilities. **Phase 2:** historical S&P 500 (incl. delisted), common-stock filters, share-class dedup. | +| Permanent loss filter | [../features/008-permanent-loss-filter/spec.md](../features/008-permanent-loss-filter/spec.md) | Hard-exclude companies with fraud or bankruptcy risk; include the Enron / Lehman / WorldCom regression test. | +| High-quality stocks | [../features/007-high-quality-stocks/spec.md](../features/007-high-quality-stocks/spec.md) | Score quality starting from Greenblatt's ROC. | +| Cheap stocks | [../features/003-cheap-stocks/spec.md](../features/003-cheap-stocks/spec.md) | Score valuation starting from Earnings Yield. | +| Corroborative signals | [../features/004-corroborative-signals/spec.md](../features/004-corroborative-signals/spec.md) | Buybacks, insider activity, and other confirming signals. | +| Unstructured financial data | [../features/012-unstructured-financial-data/spec.md](../features/012-unstructured-financial-data/spec.md) | Extract useful information from filings, transcripts, and news. | +| Backtesting + crisis report | [../features/001-backtesting/spec.md](../features/001-backtesting/spec.md) | Walk-forward backtest (3-5y windows) over 20+ years, Monte Carlo, benchmarks (S&P 500 CW/EW, Russell 3000, Magic Formula), crisis drawdown report. | +| Sell-watch / vigilance | [../features/010-sell-watch/spec.md](../features/010-sell-watch/spec.md) | Daily monitor of model portfolio for quality drop, fraud/bankruptcy, overvaluation, and opportunity cost. Emits signals (no auto-execution). | +| Portfolio evolution | [../features/009-portfolio-evolution/spec.md](../features/009-portfolio-evolution/spec.md) | Track the model portfolio and the user's personal portfolio over time and compare against configurable benchmarks. | +| Broker execution | [../features/002-broker-execution/spec.md](../features/002-broker-execution/spec.md) | Convert confirmed decisions into paper trading orders only. | +| Dashboard + reporting | [../features/005-dashboard-reporting/spec.md](../features/005-dashboard-reporting/spec.md) | Surface every input, score, decision, and explanation. Functional-first for the MVP. | + +## Functional flow — June 30 demo slice + +See [`roadmap.md`](roadmap.md). Steps not listed here are **phase 2**. + +1. Pipeline run for a `run_date`; secrets from env / AWS Secrets Manager (`SIMFIN_API_KEY`, etc.). +2. Bulk-download SimFin US datasets if older than `refresh_days`; store verbatim under `raw/simfin/`. +3. Build the demo universe: SimFin US companies minus banks / insurers / utilities (`IndustryId` CSV + bank/insurance sanity check). +4. Build run-date prices from SimFin bulk `shareprices/latest`: join universe tickers, take the latest `Date <= run_date`, and store `curated/prices`. +5. Run the SimFin normalizer → `curated/fundamentals` with PIT `as_of_date` from SimFin `Publish Date`. +6. Calculate ROC and Earnings Yield; combined rank with market-cap tie-break. +7. Select top **30** names, equal-weight model portfolio. +8. Log an MLflow run (params, metrics, portfolio artifact, git SHA). +9. Publish the Streamlit dashboard: ranking table, portfolio, per-name ROC/EY explainability. + +## Functional flow — full MVP (phase 2) + +North-star end-to-end flow after the demo slice ships: + +1. CI/CD pipeline triggers a daily run (cron via Prefect / EventBridge) and pulls secrets. +2. Build the run-date universe from historical S&P 500 constituents, apply universe filters, deduplicate share classes. +3. Ingest fundamentals from SimFin and/or SEC EDGAR and prices from free providers, storing raw responses immutably. +4. Incrementally normalize new or restated data and write to the point-in-time store (`as_of_date` = provider publish or EDGAR acceptance). +5. Run data quality checks; failing rows go to the review queue and are excluded if not resolved. +6. Apply the permanent loss filter (fraud + bankruptcy) as a hard exclusion with stored reasons. CI runs the Enron / Lehman / WorldCom regression check. +7. Calculate ROC (quality) and Earnings Yield (cheapness). +8. Apply corroborative and unstructured signals. +9. Build a Greenblatt-style combined ranking; break ties by ascending market cap. +10. Select 15 to 30 long-only names, max 10% per name; portfolio weighting (EW, SW, RP) is a backtest hyperparameter. +11. Backtest the configuration on 20+ years of point-in-time data, walk-forward 3-5 year windows. If Sharpe does not beat all benchmarks (S&P 500 CW, S&P 500 EW, Russell 3000, Magic Formula), do not auto-promote any new configuration; the configuration that runs in production is the last one that passed. +12. The sell-watch module re-scores current holdings daily; any sell trigger creates a signal that goes to dashboard + email; only proceeds to the order builder after explicit user confirmation. +13. Generate paper trading orders only. +14. Track the model portfolio (as if it were traded) and the user's personal portfolio (from the cleaned CSV) and compare against configurable benchmarks. +15. Log every run as an MLflow run with artifacts in S3 (immutable snapshot). +16. Publish the dashboard with inputs, scores, decisions, and explanations. + +## Decisions made so far + +These items are now closed for the MVP. They can be reopened in later iterations. + +### Universe and data + +| Area | Decision | +| --- | --- | +| **June demo** | See [`roadmap.md`](roadmap.md). SimFin → US universe → ROC/EY → top 30 EW → dashboard. | +| Markets | US only. Other markets deferred. | +| Universe (demo) | All SimFin US companies minus banks/insurers/utilities. | +| Universe (full MVP) | S&P 500 historical constituents (incl. delisted). Phase 2. | +| Sector classification (demo) | SimFin `IndustryId` + `load_industries()`; exclusions in `data/reference/simfin_industry_exclusions.csv`. | +| Sector classification (full MVP) | SIC from SEC EDGAR when SEC ETL ships. | +| Sectors excluded | Banks, insurers, and utilities (incomparable accounting for ROC/EY). | +| Sector limits | No sector / country / industry quotas. Out of MVP scope. | +| Share classes | Treat as the same company; keep the class with the highest average trading liquidity and drop the rest. Phase 2 for demo. | +| Market cap floor | Optional parameter. Off in demo. | +| Trading volume floor | Optional; off in demo. | +| Primary fundamentals source | **SimFin** (free tier, bulk download). [ADR-0001](../adr/0001-simfin-fundamentals-mvp.md). | +| SEC ETL | Frozen spike in repo; phase 2 normalizer. | +| Primary price source | **Demo:** SimFin bulk `shareprices/latest`. **Phase 2:** `yfinance`; free tiers of FMP, Alpha Vantage, and EODHD as redundancy / fallback. | +| Data lake | S3 (raw + curated zones) + DuckDB as the analytical engine (`duckdb` reads parquet directly from S3, no Athena bill). | +| Data lake refresh | Bulk re-download on schedule (`refresh_days=7` on free tier); incremental normalize by `Publish Date` watermark. | +| Schema versioning | Normalized schemas are versioned with explicit migrations. | +| Raw data policy | Store provider responses verbatim in the raw zone (SimFin bulk files for variants the pipeline downloads). | +| Retention policy | Keep curated data long-term; purge raw data only once curated data has been validated. | +| Point-in-time | Required. SimFin `Publish Date` is `as_of_date`; `Restated Date` for new versions; `Report Date + lag` fallback → review queue. | +| Fundamentals periodicity | Income/cashflow TTM; balance sheet quarterly (latest PIT snapshot). | +| Missing data | Flag for review for the MVP. If review backlog grows, fall back to exclusion. | + +### Scoring and portfolio construction + +| Area | Decision | +| --- | --- | +| Ranking style | Greenblatt-style: ROC for quality + Earnings Yield for cheapness. Treated as a placeholder until replaced by a more practical model. | +| Tie-break | Sort ties by ascending market cap; smaller names have priority (more room to grow). | +| Permanent loss | Hard exclusion. Scope: fraud + bankruptcy only. | +| Portfolio size | **Demo:** top 30. **Full MVP:** 15 to 30 long-only positions. | +| Short positions | Not allowed. | +| Per-name cap | 10% of portfolio (full MVP; irrelevant for demo EW top 30). | +| Weighting | **Demo:** equal-weight only. **Full MVP:** EW / SW / RP as backtest hyperparameters. | +| Rebalancing | Annual fixed for the full MVP. Demo is single `run_date` snapshot. | +| Outputs | **Demo:** model portfolio + full ranking in dashboard. **Full MVP:** watchlist + model portfolio + evolution. | + +### Risk, explainability, and operations + +| Area | Decision | +| --- | --- | +| Explainability | Dashboard shows raw inputs + scores + explanations + the rules that fired. Functional-first; visual polish later. | +| Run snapshots | MLflow is enabled from day one. Every pipeline run and every backtest is an MLflow run with parameters, metrics, and artifacts in S3 (see "MLflow as the snapshot store" below). | +| Risk checks that block orders | Permanent loss filter must have flagged `pass`; no duplicate orders; data freshness within threshold; full scoring pipeline completed; cash availability within configured limit; backtest must beat all benchmark Sharpes. | +| FP/FN review | Backtest builds a confusion matrix per rule and a curated regression test forces the bankruptcy filter to flag Enron, Lehman, and WorldCom. | +| Overfitting controls | Walk-forward backtesting + hold-out years never used for tuning + an in-sample vs out-of-sample Sharpe divergence flag treated as a red signal. | +| Run frequency | Daily. Cheap by design. | +| Secrets management | GitHub Secrets at build / deploy time, AWS Secrets Manager at runtime. | +| Broker mode | Paper trading only. | +| Primary user | Particular investor consuming a dashboard; the system itself runs as an automated agent. | + +### Infrastructure (AWS, cheapest-first) + +| Area | Decision | +| --- | --- | +| Storage | S3 (raw + curated parquet) + DuckDB as the local query engine. No Athena bill. | +| Experiment tracking | MLflow from day one. Tracking server on a small EC2 (SQLite backend) with `s3://` as the artifact root. | +| Dashboard | Streamlit. | +| Email alerts | AWS SES (sporadic emails, very cheap). | +| Compute / runtime | ECS Fargate Spot tasks (or AWS Batch on Fargate Spot), whichever is cheaper for the daily run. Triggered by Prefect. | +| Orchestration | Prefect Core (self-hosted on the same EC2 as MLflow, or via Prefect Cloud free tier). | +| Scheduling | EventBridge cron triggers the Prefect deployment once per day. | +| CI/CD | GitHub Actions builds and pushes Docker images to ECR. | +| Observability | CloudWatch Logs + Prefect UI for the MVP. Per-module metrics added if/when needed. | + +### Backtesting + +| Area | Decision | +| --- | --- | +| Historical depth | At least 20 years (value-investing horizon). | +| Crisis scenarios | All major historical crises included (dotcom, GFC, COVID, 2022 rate shock). Drawdown per crisis is reported but the MVP does not require crisis pass / fail. | +| Monte Carlo | Yes, in addition to historical replay. See "Monte Carlo simulation" below. | +| Transaction costs | Not modeled in the MVP. | +| Taxes | Not modeled in the MVP. | +| Survivorship bias | Avoided by including delisted historical S&P 500 constituents. Bankruptcies remain in the universe as evidence. | +| Pass criterion | Strategy's Sharpe must beat all of: S&P 500 cap-weighted, S&P 500 equal-weighted, Russell 3000, and Greenblatt Magic Formula. | +| Walk-forward windows | 3 to 5 year train / validation splits, given the long horizon. | +| Backtest rebalancing | Annual (matches production for the MVP). | +| Benchmarks | S&P 500 CW, S&P 500 EW, Russell 3000, Greenblatt Magic Formula portfolio. | + +### Sell-watch + +| Area | Decision | +| --- | --- | +| Scope | Model portfolio only. The user's personal portfolio is not actively monitored (those are personal orders). | +| Signals | Quality deterioration, fraud / bankruptcy flag turning on after entry, overvaluation, and opportunity cost (a better candidate exists in the watchlist). | +| Overvaluation trigger | Earnings Yield below the cross-sectional 10th percentile **or** Earnings Yield below 5% absolute. Both thresholds are starting points and treated as hyperparameters. | +| Quality deterioration trigger | ROC YoY drop greater than 30% **or** the name dropping out of the ROC top decile. Both thresholds are starting points and treated as hyperparameters. | +| Opportunity cost trigger | A watchlist candidate must outrank the held name by more than 5 positions in the combined Greenblatt ranking before the holding is flagged. | +| Frequency | Daily. | +| States | Hard `sell` only for the MVP. `trim` / `hold-with-warning` deferred. | +| Auto-execution | None. Signals require manual user confirmation before any order is built. | +| Alerts | Dashboard badge + AWS SES email. | +| Rules | Fundamentals-based + opportunity cost. Price-based stops (trailing / drawdown) deferred. | + +### Portfolio evolution + +| Area | Decision | +| --- | --- | +| User portfolio source | `data/clean/personal_finance/operations/my_operations_eur.csv` (already in EUR, derived from two broker exports). | +| Views | Cumulative return, drawdown, rolling Sharpe, holdings over time, contribution / attribution, vs S&P 500, vs the model portfolio. | +| Paper-traded model | The model portfolio is simulated as if it were actually traded, so the user can see what they would have earned or lost by following it daily. | +| Benchmarks | Configurable (S&P 500, MSCI World, others). | +| Update frequency | Daily. | +| Diversification / clustering | Deferred to a later milestone (kept in the long-term wishlist). | + +## Clarifications captured from this iteration + +### Point-in-time data and look-ahead bias + +A backtest (or any historical scoring) must only use information that was publicly available at the decision date. If on `2019-03-31` the system uses Q4 2018 earnings to rank a stock, but those earnings were not filed until `2019-04-25`, the backtest is leaking future information into the past. The same applies to: + +- Restated financials. The "as-known-in-2018" version is what 2018 decisions must use, not today's restated version. +- Index reconstitution. Using today's S&P 500 constituents to backtest 2010 introduces survivorship bias. +- Corporate actions (splits, dividends, delistings) and ticker changes. + +The MVP's point-in-time store records, for every fundamental value, the `as_of_date` (SimFin `Publish Date` in the demo; EDGAR acceptance in phase 2) and a `version_id`. Any historical query is forced to filter by `as_of_date <= decision_date`. When the publish date is missing or unreliable, a conservative lag (period end + 45 days for 10-Q, + 90 days for 10-K) is used and the row is flagged for review. This is conservative enough for a long-term value strategy. + +### MLflow as the snapshot store + +User question: "are immutable snapshots like artifacts? What if we used MLflow?" + +Yes. MLflow is a natural fit here, and it covers three needs at once: + +- **Runs**: each pipeline execution (daily, plus every backtest) is logged as an MLflow run. Parameters (universe definition, rebalance frequency, weighting scheme, thresholds, git commit SHA) are logged via `mlflow.log_param`. Metrics (Sharpe, drawdown, CAGR, hit rate, alpha, number of exclusions, count of sell signals) are logged via `mlflow.log_metric`. +- **Artifacts**: the curated input slice (or its hash), the watchlist, the model portfolio, the backtest report, and the markdown / HTML dashboard snapshot are logged as artifacts. The artifact store points at S3 with versioning and / or object lock so a past run is byte-for-byte recoverable. +- **Model registry (optional, future)**: when the scoring model evolves beyond the Greenblatt placeholder, MLflow's model registry can promote a candidate from `staging` to `production` and tie that decision back to a backtest run. + +Practical MVP setup: + +- MLflow tracking server: a tiny EC2 (or AWS Fargate task on demand) with SQLite or RDS Postgres as the backend store. For the absolute cheapest setup, an MLflow tracking server is not strictly required: `mlflow.start_run(...)` with `file://` or `s3://` as the artifact root works for a single user. +- Artifact root: `s3://smartwealthai-mlflow-artifacts/`. +- Each run is tagged with the commit SHA and pipeline name, which makes the "immutable snapshot" effectively the MLflow run id. + +Treating this as nice-to-have for the MVP is fine: we can start by writing snapshots straight to S3 with predictable paths, and slot in MLflow once the pipeline stabilizes. + +### Monte Carlo simulation of fundamentals and prices + +User question: "we cannot simulate company results coherently with prices, right?" + +Three options, in increasing complexity: + +1. **Block bootstrap of historical paths (recommended for the MVP).** Resample contiguous blocks (e.g., 6 or 12 months) from the real historical dataset across all companies simultaneously. This preserves the joint distribution of prices and fundamentals because both come from the same period of real data. It generates new "alternate histories" without inventing relationships that did not exist. It is the standard technique in academic backtests. +2. **Factor-based simulation.** Estimate a small number of factor returns (market, value, quality, size) plus idiosyncratic noise, and re-simulate company returns from those factors. Fundamentals are then assumed to evolve along their historical AR(1) / random-walk paths conditional on the factor regime. More flexible than bootstrap, but requires estimating a factor model. +3. **Generative joint model (out of MVP).** A VAR / copula / GAN / diffusion model trained on (prices, fundamentals) per company. Very powerful but easy to misuse, and effectively impossible to validate at MVP scale. + +The MVP uses block bootstrap. The synthetic-data approach is captured as a stretch goal. + + +## Related documents + +- [`roadmap.md`](roadmap.md) — current delivery target and feature priority +- [`tech-stack.md`](tech-stack.md) — technologies, infrastructure, and runtime conventions +- [`../features/`](../features/) — per-module specs (`spec.md`, optional `plan.md` / `tasks.md`) +- [`../adr/`](../adr/) — architecture decision records diff --git a/spec/constitution/roadmap.md b/spec/constitution/roadmap.md new file mode 100644 index 0000000..a578190 --- /dev/null +++ b/spec/constitution/roadmap.md @@ -0,0 +1,123 @@ +# Roadmap + +## June 30 demo slice — simplest Magic Formula + +**Status:** accepted (see [ADR-0002](../adr/0002-june-demo-scope-cut.md)) +**Target date:** 2026-06-30 +**North star:** Full MVP in [`mission.md`](mission.md) — this document defines only what ships first. + +## Objective + +Deliver a working, explainable Greenblatt-style Magic Formula pipeline on real US data: ingest fundamentals, rank the market, build a model portfolio, show results in Streamlit. No historical validation or execution in this slice. + +## In scope + +| Step | Module / spec | Notes | +| --- | --- | --- | +| 1 | SimFin ETL | [`etl-data-lake.md`](features/006-etl-data-lake/spec.md) — bulk US download, raw zone, SimFin normalizer | +| 2 | Universe | [`universe-construction.md`](features/011-universe-construction/spec.md) — demo mode: SimFin US minus sector exclusions | +| 3 | Quality | [`high-quality-stocks.md`](features/007-high-quality-stocks/spec.md) — ROC | +| 4 | Cheapness | [`cheap-stocks.md`](features/003-cheap-stocks/spec.md) — Earnings Yield | +| 5 | Ranking | Combined rank = ROC rank + EY rank (lower is better); tie-break ascending market cap | +| 6 | Model portfolio | Top **30** names, **equal-weight** only | +| 7 | Dashboard | [`dashboard-reporting.md`](features/005-dashboard-reporting/spec.md) — ranking table, portfolio, per-name explainability | +| 8 | MLflow | Log each pipeline run (params, scoring metrics, portfolio artifact) — `src/smartwealthai/mlflow_run_logging.py`, wired in `score-universe` ([#61](https://github.com/JLaborda/SmartWealthAI/issues/61)) | + +## Out of scope (phase 2) + +- Permanent loss filter ([`permanent-loss-filter.md`](features/008-permanent-loss-filter/spec.md)) +- Backtesting and crisis report ([`backtesting.md`](features/001-backtesting/spec.md)) +- Sell-watch ([`sell-watch.md`](features/010-sell-watch/spec.md)) +- Paper trading / broker ([`broker-execution.md`](features/002-broker-execution/spec.md)) +- Watchlist (ranking table in dashboard is enough) +- Corroborative signals, unstructured data, portfolio evolution (personal CSV) +- SEC EDGAR ETL (frozen spike remains in repo — [ADR-0001](../adr/0001-simfin-fundamentals-mvp.md)) +- Historical S&P 500 universe with delisted names +- Score-weighted and risk-parity weighting + +## Data sources + +| Data | Source | Tier | +| --- | --- | --- | +| Fundamentals | SimFin bulk (`income` TTM, `balance` quarterly, `cashflow` TTM, `companies`, `industries`) | Free | +| Prices (demo) | SimFin bulk `shareprices/latest` | Free; same ticker namespace as universe | +| Prices (phase 2) | SimFin `shareprices/daily` or vendor fallback | Backtest and personal NAV | +| Industry exclusions | `data/reference/simfin_industry_exclusions.csv` + bank/insurance dataset sanity check | Versioned CSV | + +## Pipeline diagram + +```mermaid +flowchart LR + SimFin["SimFin bulk US"] --> Raw["raw/simfin/"] + Raw --> Norm["SimFin normalizer"] + Raw --> SharePx["shareprices/latest"] + Norm --> Fund["curated/fundamentals"] + SharePx --> Prices["curated/prices"] + Companies["SimFin companies + industries"] --> Uni["Universe (US − exclusions)"] + Fund --> Uni + Uni --> ROC["ROC rank"] + Uni --> EY["EY rank"] + Prices --> ROC + Prices --> EY + Fund --> ROC + Fund --> EY + ROC --> Rank["Combined rank"] + EY --> Rank + Rank --> Port["Top 30 EW portfolio"] + Port --> Dash["Streamlit dashboard"] + Port --> MLflow["MLflow run"] +``` + +## Key decisions (closed for demo) + +| Topic | Decision | +| --- | --- | +| Backfill | Full SimFin US bulk per dataset variant; normalizer filters to universe tickers | +| `as_of_date` | SimFin `Publish Date`; restatements → new `version_id` with `Restated Date`; missing publish → `Report Date + lag` → review queue | +| Fundamentals periodicity | Income/cashflow **TTM**; balance sheet **quarterly** (latest PIT row) | +| Raw vs curated | Raw verbatim for downloaded variants; curated minimal (provider-agnostic schema) | +| Universe | All SimFin `market=us` companies minus banks/insurers/utilities (`IndustryId` CSV + bank/insurance sanity check) | +| Share prices | SimFin `shareprices/latest`; `price_date` may lag `run_date` by ~30 days (free tier) — OK for demo | +| Portfolio | Top 30, equal-weight, market-cap tie-break on ranks | +| SEC code | Frozen, not called by demo pipeline | + +## Acceptance criteria + +- [x] One command (or Prefect flow) runs the full demo pipeline for a `run_date`. +- [x] Dashboard shows combined rank, ROC/EY inputs, and top-30 portfolio with explanations. +- [ ] Every curated fundamental row has `as_of_date <= run_date` when queried PIT. +- [ ] No bank/insurer/utility from the exclusion list appears in the ranked universe. +- [x] MLflow run exists with portfolio parquet artifact and git commit SHA tag. +- [x] Hermetic CI tests do not call SimFin or yfinance live. + +## After the demo (phase 2 order) + +1. Historical S&P 500 universe + permanent loss filter +2. Minimal backtest (annual rebalance, 20 years) — custom pandas/DuckDB loop, not Zipline +3. Walk-forward, Monte Carlo, benchmark gate +4. Sell-watch + paper trading +5. SEC EDGAR normalizer (optional PIT upgrade) +6. Quantitative Value metrics (multi-period fundamentals from raw SimFin archives) + +## Feature registry (stable IDs) + +Chronological spec IDs (creation order). **Priority** is defined by this roadmap and [`../backlog/backlog.md`](../backlog/backlog.md), not by the number. + +| ID | Feature | Spec | +| --- | --- | --- | +| 001 | Backtesting | [`spec/features/001-backtesting/spec.md`](../features/001-backtesting/spec.md) | +| 002 | Broker Execution | [`spec/features/002-broker-execution/spec.md`](../features/002-broker-execution/spec.md) | +| 003 | Cheap Stocks | [`spec/features/003-cheap-stocks/spec.md`](../features/003-cheap-stocks/spec.md) | +| 004 | Corroborative Signals | [`spec/features/004-corroborative-signals/spec.md`](../features/004-corroborative-signals/spec.md) | +| 005 | Dashboard Reporting | [`spec/features/005-dashboard-reporting/spec.md`](../features/005-dashboard-reporting/spec.md) | +| 006 | Etl Data Lake | [`spec/features/006-etl-data-lake/spec.md`](../features/006-etl-data-lake/spec.md) | +| 007 | High Quality Stocks | [`spec/features/007-high-quality-stocks/spec.md`](../features/007-high-quality-stocks/spec.md) | +| 008 | Permanent Loss Filter | [`spec/features/008-permanent-loss-filter/spec.md`](../features/008-permanent-loss-filter/spec.md) | +| 009 | Portfolio Evolution | [`spec/features/009-portfolio-evolution/spec.md`](../features/009-portfolio-evolution/spec.md) | +| 010 | Sell Watch | [`spec/features/010-sell-watch/spec.md`](../features/010-sell-watch/spec.md) | +| 011 | Universe Construction | [`spec/features/011-universe-construction/spec.md`](../features/011-universe-construction/spec.md) | +| 012 | Unstructured Financial Data | [`spec/features/012-unstructured-financial-data/spec.md`](../features/012-unstructured-financial-data/spec.md) | + +## Informal backlog + +See [`../backlog/backlog.md`](../backlog/backlog.md). diff --git a/spec/constitution/tech-stack.md b/spec/constitution/tech-stack.md new file mode 100644 index 0000000..670a1e4 --- /dev/null +++ b/spec/constitution/tech-stack.md @@ -0,0 +1,103 @@ +# Tech stack + +Technologies, infrastructure, and runtime conventions for SmartWealthAI. +## Infrastructure (AWS, cheapest-first) + + +| Area | Decision | +| --- | --- | +| Storage | S3 (raw + curated parquet) + DuckDB as the local query engine. No Athena bill. | +| Experiment tracking | MLflow from day one. Tracking server on a small EC2 (SQLite backend) with `s3://` as the artifact root. | +| Dashboard | Streamlit. | +| Email alerts | AWS SES (sporadic emails, very cheap). | +| Compute / runtime | ECS Fargate Spot tasks (or AWS Batch on Fargate Spot), whichever is cheaper for the daily run. Triggered by Prefect. | +| Orchestration | Prefect Core (self-hosted on the same EC2 as MLflow, or via Prefect Cloud free tier). | +| Scheduling | EventBridge cron triggers the Prefect deployment once per day. | +| CI/CD | GitHub Actions builds and pushes Docker images to ECR. | +| Observability | CloudWatch Logs + Prefect UI for the MVP. Per-module metrics added if/when needed. | + + +## MLOps stack (cost-aware) + +User proposed: GitHub Actions + MLflow + Prefect + Kubernetes. + +For a daily run over the S&P 500 historical universe, Kubernetes is overkill and expensive. Recommended cost-aware mapping: + +| User goal | MVP-cheap option | Growth path | +| --- | --- | --- | +| Orchestration | Prefect Core running on a small EC2 (or Prefect Cloud free tier) | Prefect on EKS | +| Execution | AWS Batch or ECS Fargate spot tasks triggered by Prefect, or a small EC2 with cron + Docker | EKS with autoscaling | +| Experiment tracking | MLflow with SQLite + S3 artifact root, on the same EC2 | MLflow on RDS + EC2 / Fargate | +| CI/CD | GitHub Actions building Docker images and pushing to ECR | Same | +| Scheduling | EventBridge cron triggering a Prefect deployment | Same | +| Secrets | GitHub Secrets in CI; AWS Secrets Manager at runtime | Same | +| Observability | CloudWatch Logs + Prefect UI | CloudWatch Logs + Prefect Cloud + Grafana | + +This still showcases MLOps competence (CI/CD, container build, orchestrator, experiment tracking, secrets, observability) without paying for EKS in the MVP. + + +## MLflow + +User question: "are immutable snapshots like artifacts? What if we used MLflow?" + +Yes. MLflow is a natural fit here, and it covers three needs at once: + +- **Runs**: each pipeline execution (daily, plus every backtest) is logged as an MLflow run. Parameters (universe definition, rebalance frequency, weighting scheme, thresholds, git commit SHA) are logged via `mlflow.log_param`. Metrics (Sharpe, drawdown, CAGR, hit rate, alpha, number of exclusions, count of sell signals) are logged via `mlflow.log_metric`. +- **Artifacts**: the curated input slice (or its hash), the watchlist, the model portfolio, the backtest report, and the markdown / HTML dashboard snapshot are logged as artifacts. The artifact store points at S3 with versioning and / or object lock so a past run is byte-for-byte recoverable. +- **Model registry (optional, future)**: when the scoring model evolves beyond the Greenblatt placeholder, MLflow's model registry can promote a candidate from `staging` to `production` and tie that decision back to a backtest run. + +Practical MVP setup: + +- MLflow tracking server: a tiny EC2 (or AWS Fargate task on demand) with SQLite or RDS Postgres as the backend store. For the absolute cheapest setup, an MLflow tracking server is not strictly required: `mlflow.start_run(...)` with `file://` or `s3://` as the artifact root works for a single user. +- Artifact root: `s3://smartwealthai-mlflow-artifacts/`. +- Each run is tagged with the commit SHA and pipeline name, which makes the "immutable snapshot" effectively the MLflow run id. + +Treating this as nice-to-have for the MVP is fine: we can start by writing snapshots straight to S3 with predictable paths, and slot in MLflow once the pipeline stabilizes. + + +## Runtime (local development) + +- **Python:** 3.11+ (`requires-python = "^3.11"` in `pyproject.toml`) +- **Dependencies:** Poetry + +```bash +poetry env use python3.11 +poetry install +poetry run pytest +``` + +### MLflow (demo pipeline runs) + +`mlflow-skinny` is a Poetry dependency. No local MLflow UI server is required for pipeline logging. + +```bash +export MLFLOW_TRACKING_URI="file://$(pwd)/mlruns" +export MLFLOW_ALLOW_FILE_STORE=true +``` + +### Secrets + +- **Local dev:** `.env` (gitignored) for `SIMFIN_API_KEY`; devcontainer loads via `.devcontainer/install-env-hook.sh` +- **CI:** GitHub Actions secrets +- **Cloud runtime:** AWS Secrets Manager + +### Data lake (technical) + +- S3 raw + curated parquet zones; DuckDB as analytical engine +- Provider connectors: SimFin (demo), SEC EDGAR (phase 2 spike frozen) +- Cache: `yfinance` responses on S3 under `s3://smartwealthai-cache/yfinance/` (phase 2) + +## CI/CD and tooling PRDs + +| PRD | Path | +| --- | --- | +| Devcontainer | [`spec/prds/devcontainer/prd.md`](../prds/devcontainer/prd.md) | +| CI/CD | [`spec/prds/ci-cd/ci-cd-prd.md`](../prds/ci-cd/ci-cd-prd.md) | +| Phase 2 coordination | [`spec/prds/phase2/prd.md`](../prds/phase2/prd.md) | + +## Operator guides + +| Guide | Path | +| --- | --- | +| SimFin download (demo) | [`spec/guides/download-simfin.md`](../guides/download-simfin.md) | +| SEC fundamentals (frozen spike) | [`spec/guides/download-fundamentals.md`](../guides/download-fundamentals.md) | diff --git a/spec/features/001-backtesting/spec.md b/spec/features/001-backtesting/spec.md new file mode 100644 index 0000000..b56f379 --- /dev/null +++ b/spec/features/001-backtesting/spec.md @@ -0,0 +1,162 @@ +# Feature: Backtesting and Crisis Report + +## Implementation status + +**Deferred** for the June 30 demo slice ([ADR-0002](../../adr/0002-june-demo-scope-cut.md)). Spec remains the target for phase 2. + +## Objective + +Validate the strategy on at least 20 years of point-in-time data before any paper trading order is generated. Backtests must be reproducible, point-in-time correct, free of survivorship bias, and tracked end-to-end via MLflow. A backtest run that does not beat all benchmarks on Sharpe blocks order generation for that configuration. + +## MVP scope + +- Walk-forward backtest with 3 to 5 year train / validation windows. +- Annual rebalancing (matches production). +- Long-only, 15 to 30 names, max 10% per name (matches production). +- Weighting (equal-weight, score-weighted, risk-parity) as a hyperparameter; the winning weighting on validation Sharpe is used in production. +- Block-bootstrap Monte Carlo to generate alternate histories from the same data. +- Benchmark suite: S&P 500 cap-weighted, S&P 500 equal-weighted, Russell 3000, Greenblatt Magic Formula canonical replica. +- Crisis drawdown report (dotcom 2000-2002, GFC 2008-2009, COVID 2020, 2022 rate shock). Drawdowns reported, not gated. +- MLflow run per backtest with parameters, metrics, and artifacts. + +## Out of MVP scope + +- Transaction costs, slippage, bid-ask spreads. +- Capital-gains taxes. +- Currency hedging. +- Synthetic data via generative models (GAN / diffusion / VAR). +- Live optimization of the rebalance frequency (fixed annual for the MVP). +- Short positions, leverage. + +## Inputs + +| Input | Source | +| --- | --- | +| PIT fundamentals | `curated/fundamentals` | +| Adjusted prices | `curated/prices` | +| Historical universe per run date | `curated/universe` (rebuilt for each rebalance date) | +| Permanent loss exclusions | `curated/permanent_loss` | +| Quality and cheapness scores | `curated/scores/quality` + `curated/scores/cheap` (recomputed inside the backtest with PIT inputs) | +| Benchmarks reference | `data/reference/benchmarks/` with constituents of S&P 500 CW/EW and Russell 3000 over time; Magic Formula portfolio is rebuilt at every rebalance from the same universe and PIT data | +| Backtest config | `config/backtest/.yaml` (parameters: start date, end date, rebalance, weighting, MC settings, walk-forward windows) | + +## Outputs + +All outputs live as MLflow run artifacts under the `backtesting` experiment. + +| Output | Description | +| --- | --- | +| Equity curves parquet | Daily NAV per backtest variant, per benchmark | +| Trade ledger parquet | Every simulated trade at each rebalance | +| Holdings parquet | Holdings per name per date | +| Metrics dictionary | Sharpe, CAGR, max drawdown, turnover, hit rate, alpha vs each benchmark, in-sample vs out-of-sample Sharpe gap | +| Crisis report HTML | Drawdown per named crisis vs each benchmark | +| Monte Carlo distribution parquet | Per-trial returns, Sharpe distribution, drawdown distribution | +| Pass / fail flag | `True` only if strategy Sharpe strictly beats every benchmark | +| MLflow tags | `git_sha`, `config_hash`, `pit_data_hash` | + +## Pass criterion + +The backtest passes (and therefore allows production paper orders) only if all of the following hold on the out-of-sample window: + +- `Sharpe(strategy) > Sharpe(S&P 500 CW)` +- `Sharpe(strategy) > Sharpe(S&P 500 EW)` +- `Sharpe(strategy) > Sharpe(Russell 3000)` +- `Sharpe(strategy) > Sharpe(Magic Formula canonical replica)` + +Additionally, the **in-sample vs out-of-sample Sharpe gap** is logged. A gap larger than a configurable threshold (default: in-sample Sharpe more than 50% above out-of-sample Sharpe) flips a `overfit_risk` flag in the MLflow metrics. The flag does not block by itself, but the run is highlighted in the dashboard. + +Crisis windows have drawdowns reported but do not gate the run for the MVP. + +## Walk-forward design + +- Start date: backtest config (default: 20 years before the run date). +- End date: most recent year with full data, leaving the last 1 to 2 years as a frozen hold-out. +- Window length: 3 to 5 years (parameter). Hyperparameters (weighting, score thresholds) are tuned on the first part of each window, validated on the second. +- Rolling step: 1 year forward. +- Hold-out: an explicit final window the strategy never sees during tuning. Reported separately. + +## Monte Carlo (block bootstrap) + +- Resample contiguous blocks of `block_length` months (default 12) from the historical multi-asset return panel. +- Trial count `n_trials` (default 500). +- Same strategy logic runs on each synthetic path. +- Logged outputs: Sharpe distribution, max drawdown distribution, 5th / 50th / 95th percentile equity curves. +- Block bootstrap preserves the joint distribution of prices and fundamentals because both come from the same sampled blocks. + +## Mermaid diagram + +```mermaid +flowchart TD + Config["config/backtest/.yaml"] --> Engine["Backtest engine"] + PIT["curated/fundamentals (PIT)"] --> Engine + Prices["curated/prices"] --> Engine + UniverseHist["curated/universe (per rebalance date)"] --> Engine + PLoss["curated/permanent_loss (per rebalance date)"] --> Engine + + Engine --> Rebalance["For each rebalance date"] + Rebalance --> Scoring["Recompute ROC + EY (PIT)"] + Scoring --> RankBuild["Combined rank + portfolio construction"] + RankBuild --> Holdings["Holdings parquet"] + Holdings --> NAV["Daily NAV"] + + NAV --> Metrics["Sharpe / CAGR / drawdown / turnover / alpha"] + Metrics --> Benchmarks["vs S&P 500 CW + EW + Russell 3000 + Magic Formula"] + Benchmarks --> Pass{"Sharpe > all benchmarks?"} + Pass -->|Yes| PassFlag["pass = True"] + Pass -->|No| FailFlag["pass = False"] + + Engine --> MC["Block-bootstrap Monte Carlo"] + MC --> MCDist["Sharpe and drawdown distributions"] + + Metrics --> CrisisReport["Crisis drawdown report"] + + PassFlag --> MLflow["MLflow run"] + FailFlag --> MLflow + MCDist --> MLflow + CrisisReport --> MLflow +``` + +## Expected flow + +1. The pipeline triggers a backtest when the configuration changes, when a new release is built, or on a manual request from the dashboard. +2. The engine reads the config and resolves the start / end dates and walk-forward windows. +3. For each rebalance date (annually, starting at `start_date`): + 1. Build the universe via `universe-construction` at that date. + 2. Apply the permanent loss filter (PIT). + 3. Recompute ROC and EY on PIT fundamentals. + 4. Combine ranks, apply tie-break, select top 15 to 30 names with 10% cap. + 5. Compose the holding using the weighting variant under test. +4. Simulate daily NAV from rebalance to rebalance using adjusted prices. +5. Compute metrics overall, per walk-forward window, and per crisis window. +6. Run the same logic against the benchmark constructions to produce comparable Sharpe / drawdown. +7. Run the Monte Carlo block-bootstrap loop. +8. Decide pass / fail. Log everything to MLflow. + +## Acceptance criteria + +- A backtest run with the same config and the same PIT data hash produces metrics within numerical tolerance across two executions. +- Every backtest is logged as an MLflow run under the `backtesting` experiment with the `git_sha`, `config_hash`, and `pit_data_hash` tags. +- The engine never reads data more recent than the rebalance date during simulation; an automated check verifies this (e.g., max `as_of_date` per slice equals the rebalance date). +- The Magic Formula benchmark is built from the same universe and rebalanced annually like the strategy. +- The hold-out window is reported separately and is never touched by hyperparameter tuning. +- The crisis report includes at minimum dotcom 2000-2002, GFC 2008-2009, COVID 2020, 2022 rate shock. +- The pass flag is `True` only if the strategy Sharpe strictly beats every benchmark Sharpe. +- The Monte Carlo distribution has at least 500 trials by default. +- Production order generation reads the pass flag of the most recent backtest before emitting any order. + +## Open questions + +- Should we use the full S&P 500 CW total-return series as one benchmark and an equal-weight backtest of the same survivors as another? Recommendation: use a published total-return index (e.g., SPX TR via `^SP500TR` or a stitched series) for CW; build the EW from the historical constituents we already have. +- For Russell 3000 historical constituents we do not have a free reliable source. Recommendation: pin the Russell 3000 total-return index as a price series only (no constituent rebuild) for the MVP and revisit if it becomes the bottleneck. +- For Magic Formula benchmark, do we constrain it to the same exclusions (no banks / insurers / utilities) as the strategy, or run it on the full common-stock universe? Recommendation: same exclusions, so the comparison is fair to the strategy's universe. +- What is the right `block_length` for Monte Carlo: 6 or 12 months? Recommendation: 12 to capture annual cyclicality; expose as a parameter. +- Should we emit a pass / fail per weighting variant separately, or only for the winning variant? Recommendation: log every variant; promote only the winner. + +## Risks + +- Backtests on free data can have subtle PIT errors (missing restatements, late filings). The `as_of_date` check above is the main defense. +- Survivorship bias is removed by the historical universe but can creep back through benchmarks that only include current survivors. The benchmark sources matter. +- Monte Carlo block bootstrap underestimates extreme tails because it cannot generate scenarios outside historical experience. Documented as a known limitation. +- A strategy that beats all benchmarks on Sharpe can still be unstable in a single bad year. The crisis drawdown report is the user-facing warning. +- Without modeling transaction costs, the strategy looks better than it would in real life. Documented; revisit before any live trading. diff --git a/spec/features/002-broker-execution/spec.md b/spec/features/002-broker-execution/spec.md new file mode 100644 index 0000000..94260ad --- /dev/null +++ b/spec/features/002-broker-execution/spec.md @@ -0,0 +1,127 @@ +# Feature: Broker Execution + +## Implementation status + +**Deferred** for the June 30 demo slice ([ADR-0002](../../adr/0002-june-demo-scope-cut.md)). Spec remains the target for phase 2. + +## Objective + +Convert the model portfolio's target positions, plus confirmed sell-watch signals, into broker-compatible orders. For the MVP, every order is **paper-traded** in a simulator. Real broker connectivity is out of scope until the user explicitly opts in. + +## MVP scope + +- Translate target weights (from `portfolio-construction`) into target share counts using today's closing price. +- Compare target positions against the current paper book to compute required trades. +- Accept confirmed sell-watch signals (status `confirmed` in `curated/sell_watch/confirmations.parquet`) as forced sells. +- Run pre-trade risk checks (see "Pre-trade risk checks" below) and reject orders that violate them. +- Submit accepted orders to the internal paper trading simulator. +- Track order lifecycle (`proposed`, `accepted`, `rejected`, `submitted`, `filled`, `partially_filled`, `cancelled`). +- Reconcile the paper book against the expected target positions after fills. +- Idempotent: a given `(run_date, ticker, side, quantity, source)` tuple cannot produce two orders. +- Runs on ECS Fargate Spot once per trading day, after the daily pipeline finishes. + +## Out of MVP scope + +- Real broker connectivity (Alpaca, Interactive Brokers, etc.). +- Real money. +- Smart order routing. +- Limit orders, stop orders, options, futures, margin. +- Fractional-share semantics beyond what the simulator supports (the simulator accepts fractional volume to mirror the personal CSV). +- Tax-aware lot selection (FIFO at aggregate level). + +## Inputs + +| Input | Source | +| --- | --- | +| Target model portfolio | `curated/portfolio/model/holdings.parquet` (today's run) | +| Current paper book | `curated/broker/paper_book.parquet` | +| Confirmed sell signals | `curated/sell_watch/confirmations.parquet` (only `status = confirmed`) | +| Backtest pass flag | Latest MLflow run in `backtesting` experiment must have `pass = True` | +| Cash balance | `curated/broker/cash.parquet` | +| Today's adjusted close | `curated/prices` | +| Run date | Pipeline parameter | +| Broker config | `config/broker.yaml` (mode = `paper`, cash floor, per-order limit, daily turnover cap) | + +## Outputs + +| Output | Path / target | +| --- | --- | +| Proposed orders | `curated/broker/proposed_orders.parquet` | +| Risk-check log | `curated/broker/risk_checks.parquet` | +| Submitted orders + fills | `curated/broker/orders.parquet` (lifecycle) | +| Updated paper book | `curated/broker/paper_book.parquet` | +| Updated cash | `curated/broker/cash.parquet` | +| Reconciliation report | `curated/broker/reconciliation.parquet` | +| MLflow run | Parameters (config), metrics (orders proposed / submitted / rejected, turnover, cash before / after), artifacts (per-run report) | + +## Pre-trade risk checks (blocks order generation when any fails) + +| Check | Block? | +| --- | --- | +| Latest backtest `pass = True` | Yes | +| Permanent loss filter has flagged `pass` for every target ticker (not `exclude` or `unknown`) | Yes | +| Full scoring pipeline completed today (quality, cheapness, ranking parquets present) | Yes | +| Data freshness within threshold (every input price has a row dated today or the previous trading day) | Yes | +| No duplicate order in `curated/broker/orders.parquet` for the same `(run_date, ticker, side, quantity, source)` | Yes | +| Cash balance is above the configured floor after the trade | Yes | +| Per-order notional is below the configured per-order limit | Yes | +| Daily turnover (sum of trade notional) is below the configured cap | Yes | +| Position concentration after the trade keeps every name <= 10% of NAV | Yes | + +Any failing check moves the corresponding order to `rejected` with the failure reason. The pipeline continues with the orders that passed. + +## Mermaid diagram + +```mermaid +flowchart TD + Targets["curated/portfolio/model/holdings.parquet"] --> OrderBuilder["Compute required trades"] + PaperBook["curated/broker/paper_book.parquet"] --> OrderBuilder + Confirmed["confirmed sell-watch signals"] --> OrderBuilder + + OrderBuilder --> Proposed["proposed_orders.parquet"] + Proposed --> Checks["Pre-trade risk checks"] + + Checks --> Pass{"All checks pass?"} + Pass -->|No| Rejected["risk_checks.parquet (rejected)"] + Pass -->|Yes| Paper["Paper trading simulator"] + Paper --> Orders["orders.parquet (lifecycle)"] + Orders --> NewBook["Updated paper_book.parquet"] + Orders --> NewCash["Updated cash.parquet"] + NewBook --> Reconcile["reconciliation.parquet"] +``` + +## Expected flow + +1. Load target positions and the current paper book. +2. Compute required trades (target shares - current shares) using today's closing price. +3. Add forced sells for every confirmed (and not yet submitted) sell-watch signal. +4. Run pre-trade risk checks; mark each order `accepted` or `rejected`. +5. For accepted orders, submit them to the paper trading simulator at today's close. The simulator immediately marks them `filled` with the close price as the fill price. +6. Update the paper book and cash balance. +7. Run reconciliation: assert that `paper_book` equals `target_positions` for the accepted set; log any mismatch. +8. Persist all parquet outputs and log MLflow. + +## Acceptance criteria + +- Live execution code paths do not exist in the MVP. The simulator is the only execution backend. +- The pipeline cannot produce two identical orders. The deduplication key is documented and tested. +- Every rejection is logged with the failing check id; rejections never silently drop. +- The reconciliation report flags discrepancies between the paper book and the target positions. +- If the latest backtest `pass = False`, no order is submitted, even for confirmed sell-watch signals. (Sell signals remain confirmed and pending until a passing backtest exists.) +- The MLflow run logs at minimum: orders proposed, orders submitted, orders rejected (per check), and turnover. +- The module runs on ECS Fargate Spot triggered by Prefect, after the daily pipeline. + +## Open questions + +- Should confirmed sells be allowed to execute even when the latest backtest fails? Argument for yes: protecting capital is more important than waiting for a fresh backtest. Recommendation for the MVP: no, to keep the safety boundary clean; mark as open for review once we have data. +- Do we model intraday vs end-of-day fills? Recommendation: end-of-day close only for the MVP. +- Do we charge a simulated commission per trade? Recommendation: no for the MVP (matches backtest assumptions); revisit before any real trading. +- Where does the paper book start? Recommendation: configurable initial cash in `config/broker.yaml`, default 100,000 EUR. +- Does the simulator support fractional shares like the personal CSV? Recommendation: yes; matches the real broker behavior the user already experienced. + +## Risks + +- Once real-broker code exists, it can be enabled by accident. The MVP module should not import any live broker SDK; a separate module behind an explicit feature flag will be added later. +- A bug in deduplication can flood the paper book with phantom positions. The unit tests must cover replays and idempotency. +- Stale prices on a market holiday could produce incorrect fills. The freshness check is the primary defense. +- The user can ignore reconciliation failures. The dashboard surfaces them prominently. diff --git a/spec/features/003-cheap-stocks/spec.md b/spec/features/003-cheap-stocks/spec.md new file mode 100644 index 0000000..25fe902 --- /dev/null +++ b/spec/features/003-cheap-stocks/spec.md @@ -0,0 +1,113 @@ +# Feature: Cheap Stocks + +## Implementation status + +done (demo cross-sectional slice) — EY scoring and ranks: `src/smartwealthai/magic_formula_ranking.py`, CLI `score-universe` ([#60](https://github.com/JLaborda/SmartWealthAI/issues/60)). Single-ticker tracer: `magic_formula_metrics.py`, `pit_fundamentals.py`, `compute-metrics` ([#44](https://github.com/JLaborda/SmartWealthAI/issues/44)). + +## Objective + +Score the cheapness of every company that survives the universe filter, the permanent loss filter, and the quality scoring step. For the MVP, cheapness is a strict Greenblatt-style **Earnings Yield (EY) = EBIT / Enterprise Value**. Future iterations can plug additional valuation signals (FCF yield, EV/EBITDA, shareholder yield) through the same interface. + +## MVP scope + +- Compute `EY = EBIT / EV` per the canonical Greenblatt definition. +- Use the most recent point-in-time fundamentals available on the decision date and the run-date market data for EV. +- Produce a cross-sectional cheapness rank (lower rank = cheaper) for every passing company. +- Validate denominators: rows with `EV <= 0` or `EBIT` missing are flagged for review and excluded from the ranking. +- Avoid blindly ranking value traps as attractive: rows with negative EBIT are routed to the review queue rather than being inverted into "expensive". +- Log MLflow metrics (count valid, count invalid, EY quantiles). +- Expose the inputs alongside the score so the dashboard can explain "why is this company cheap?". + +## Out of MVP scope + +- Multi-metric cheapness score (FCF yield, EV/EBITDA, P/B, P/S, shareholder yield). +- Sector-relative valuation. +- Value-trap defense beyond the permanent loss filter (no quality threshold required to enter the cheapness rank; quality is a separate, parallel rank that combines later). +- Cyclically adjusted earnings. +- Forward-looking estimates. + +## Inputs + +| Input | Source | Notes | +| --- | --- | --- | +| Passing universe + permanent loss filter pass list | `curated/universe` + `curated/permanent_loss` | Only `pass` rows are scored. | +| PIT fundamentals (income statement, balance sheet) | `curated/fundamentals` | Filtered by `as_of_date <= run_date`. | +| Run-date market cap | `curated/prices/run_date=/prices.parquet` join `curated/fundamentals` | `shares_outstanding * adj_close`; `price_date` is the latest trading day ≤ `run_date`. | +| Enterprise value components | `curated/fundamentals` | Total debt, preferred equity, minority interest, cash. | +| Run date | Pipeline parameter | | +| EY formula version | `config/cheap/ey.yaml` | Versioned. | + +## EY definition (canonical Greenblatt) + +``` +EY = EBIT / Enterprise Value +EV = Market Cap + Total Debt + Preferred Equity + Minority Interest - Cash and Equivalents +``` + +with: + +- `EBIT` = Operating income before interest and taxes. Trailing twelve months. Same definition as in `high-quality-stocks.md` so both scores share the same `EBIT`. +- `Market Cap` = `shares_outstanding * close` on `run_date`. +- All other components from the latest filing whose `as_of_date <= run_date`. + +The formula and its variants are versioned in `config/cheap/ey.yaml`. Any change requires a new version id so backtests on prior versions remain reproducible. + +## Outputs + +| Output | Path / target | +| --- | --- | +| Cheapness scores parquet | `curated/scores/cheap/run_date=/scores.parquet` with `cik, ticker, ebit, market_cap, total_debt, preferred_equity, minority_interest, cash, ev, ey, ey_rank, formula_version, as_of_date` | +| Review queue rows | `curated/issues/run_date=/cheap.parquet` for invalid denominators, negative EBIT, missing inputs | +| MLflow metrics | `cheap_n_valid`, `cheap_n_invalid`, EY quantiles | + +## Mermaid diagram + +```mermaid +flowchart TD + Passing["Universe pass + Permanent loss pass"] --> Loader["Load PIT fundamentals + market cap"] + Loader --> Components["EBIT, Market Cap, Total Debt, Preferred Equity, Minority Interest, Cash"] + Components --> EV["Compute EV"] + EV --> Validate{"EV > 0 and EBIT present?"} + + Validate -->|No| Review["Review queue (cheap.parquet)"] + Validate -->|Yes| EBITSign{"EBIT > 0?"} + + EBITSign -->|No| Review + EBITSign -->|Yes| EY["EY = EBIT / EV"] + EY --> Rank["Cross-sectional rank (descending EY)"] + Rank --> Output["cheap/scores.parquet"] + Output --> MLflow["MLflow metrics"] +``` + +## Expected flow + +1. Load the passing universe and join with PIT fundamentals + market cap. +2. Compute Enterprise Value from `market_cap + total_debt + preferred_equity + minority_interest - cash`. +3. Validate inputs: missing component rows are dropped; `EV <= 0` and `EBIT <= 0` rows are flagged for review. +4. Compute `EY`. +5. Produce a cross-sectional rank from highest EY (rank 1) to lowest. +6. Persist parquet and log MLflow metrics. +7. The combined Greenblatt rank (`roc_rank + ey_rank`) is built downstream by `portfolio-construction` (inside the same pipeline). Tie-break by market cap from `high-quality-stocks` carries over. + +## Acceptance criteria + +- Same `(universe, run_date, formula_version)` produces byte-identical output (hash-verifiable). +- The score is purely a function of curated PIT data; no network calls. +- Every row has the EY value and all components that produced it. +- Rows with invalid EV or negative EBIT are visible in the review queue and not silently flipped to "expensive". +- The formula version travels with each scored row. +- The MLflow run logs at minimum count of valid rows, count of invalid rows, EY median, and EY quantiles. + +## Open questions + +- For Enterprise Value, do we use `Long Term Debt + Short Term Debt + Capital Lease Obligations` for `Total Debt`, or a narrower definition? Recommendation: include capital leases under `Total Debt` and document as `formula_version = v1`. +- ~~Cash definition: `CashAndCashEquivalents` only, or `CashAndCashEquivalents + ShortTermInvestments`?~~ **Closed (v1):** include short-term investments — SimFin column `Cash, Cash Equivalents & Short Term Investments` maps to curated `cash`. +- Preferred equity: use book value or market value? Recommendation: book value (market is rarely available for free). +- Should the EY rank skip companies that fail to score on quality (i.e., invalid ROC denominator)? Recommendation: no; keep the two ranks independent so the combined score only excludes a name when both fail. + +## Risks + +- Negative-EBIT companies are silent value traps that easy EY models can mislabel as attractive. The MVP routes them to the review queue, which avoids the trap but may exclude legitimate turnarounds. +- One-off items in EBIT distort EY. Same caveat as in `high-quality-stocks`; accepted as a Greenblatt placeholder. +- `Total Debt` reported by EDGAR has multiple equally defensible definitions. Version locking is the only durable mitigation. +- Restated balance sheet items shift EV across versions. PIT versioning handles it; the test suite must cover restatements explicitly. diff --git a/spec/features/004-corroborative-signals/spec.md b/spec/features/004-corroborative-signals/spec.md new file mode 100644 index 0000000..6c2e483 --- /dev/null +++ b/spec/features/004-corroborative-signals/spec.md @@ -0,0 +1,86 @@ +# Feature: Corroborative Signals + +> **Status for the MVP: deferred.** No corroborative signal is computed or used in the first version of the pipeline. The architecture leaves a slot for this module in the ranking diagram, but the MVP combined rank is exactly `quality_rank + cheapness_rank` (Greenblatt). This spec captures the design we will revisit once the MVP is validated. + +## Objective + +Add signals that corroborate or challenge the core quality and cheapness ranking. These signals are auxiliary; they never overrule the permanent loss filter and they should not dominate the value framework. + +## Out of MVP scope (entire module) + +The whole module is parked. The first iteration of the system runs without any corroborative input. The reasons: + +- The Greenblatt placeholder is intentionally minimal until the user finishes reading *Quantitative Value* and chooses the next factors. +- Insider transaction data, short interest data, and institutional ownership data all require either paid feeds or fragile scraping. The MVP's "free data only" constraint makes this hard to deliver reliably. +- A noisy corroborative score on top of a placeholder Greenblatt rank is more likely to hurt than help. + +## Candidate signals (future iterations) + +Listed for memory. None of them is implemented now. + +### Shareholder return + +- Net buyback yield. +- Share count reduction over time. +- Dividend yield. +- Dividend growth. +- Total shareholder yield. + +### Insider activity + +- Insider buying by executives or directors. +- Cluster buying. +- Insider selling after large price appreciation. +- Insider ownership level. + +### Market and ownership context + +- Short interest. +- Institutional ownership changes. +- Activist involvement. + +### Corporate events + +- Spin-offs. +- Tender offers. +- Debt refinancing. +- Management changes. + +## Mermaid diagram (future state) + +```mermaid +flowchart TD + CuratedData["Curated financial data"] --> Buybacks["Buyback signals"] + InsiderData["Insider transactions"] --> Insiders["Insider signals"] + MarketData["Market / ownership data"] --> Ownership["Ownership and short interest"] + Events["Corporate actions"] --> EventSignals["Event signals"] + + Buybacks --> SignalAggregator["Signal aggregator"] + Insiders --> SignalAggregator + Ownership --> SignalAggregator + EventSignals --> SignalAggregator + + SignalAggregator --> Adjustment["Adjustment to combined rank"] + SignalAggregator --> Flags["Review flags"] +``` + +## Open questions (for the iteration when this module is reactivated) + +- Which corroborative signal should be included first? Likely candidates: shareholder yield (gettable from EDGAR + prices for free) and SEC Form 4 insider transactions (also free). +- Should corroborative signals adjust the combined rank, or only appear as flags? +- How large must a buyback be to matter, and how do we strip out stock-based-compensation noise? +- How do we avoid double-counting dividends in cheapness and in shareholder yield? +- Should corroborative signals ever override a permanent loss exclusion? Recommendation locked: **no**. + +## Acceptance criteria for the future module + +- The module is opt-in via configuration; the MVP default keeps it off. +- It can be added without changing the ETL or the scoring modules: it consumes curated parquet and writes its own parquet. +- Its contribution to the combined rank is explicit, bounded, and documented. +- Missing signal data does not silently penalize a company. + +## Risks (future) + +- Insider data quality varies. Form 4 filings are reliable; aggregator interpretations are not. +- Short interest is reported with significant delay; pretending it is real-time invites bias. +- Buybacks can be value-destructive at high prices. The signal must condition on cheapness, not just on the existence of a buyback. diff --git a/spec/features/005-dashboard-reporting/spec.md b/spec/features/005-dashboard-reporting/spec.md new file mode 100644 index 0000000..7fbb43e --- /dev/null +++ b/spec/features/005-dashboard-reporting/spec.md @@ -0,0 +1,135 @@ +# Feature: Dashboard and Reporting + +## Implementation status + +**done** — demo slice three-page app ([#62](https://github.com/JLaborda/SmartWealthAI/issues/62)); full MVP pages in phase 2. + +### Demo run + +```bash +# After score-universe (or full demo pipeline) for a run_date: +poetry run run-dashboard --data-dir data --run-date 2026-06-18 +``` + +Environment variables: `SMARTWEALTHAI_DATA_DIR` (lake root, default `data`), `SMARTWEALTHAI_RUN_DATE` (optional override). + +Code: `apps/dashboard/` (Streamlit UI), `src/smartwealthai/dashboard_data.py` (parquet readers), `tests/test_dashboard_data.py`. + +## Objective + +Surface every input, score, decision, and audit trail produced by the pipeline in a single Streamlit dashboard. The dashboard is the primary product surface for the user. Visual polish is explicitly deferred; functional completeness comes first. + +## MVP scope + +### Demo slice (June 30) + +- Streamlit app (local or lightweight AWS deploy). +- Pages: **Overview**, **ETL & data quality**, **Universe**, **Quality (ROC)**, **Cheapness (EY)**, **Ranking + model portfolio** (top 30 EW). +- Per-name explainability: ROC/EY inputs and combined rank. +- MLflow run link per pipeline execution. +- Read-only views (no sell-watch confirmation in demo). + +### Full MVP (phase 2) + +- Streamlit app deployed on AWS (likely Fargate Spot behind an ALB, or App Runner if cheaper at MVP scale). +- Additional pages: permanent loss filter, sell-watch, backtests, portfolio evolution. +- Sell-watch confirmation writes back to `curated/sell_watch/confirmations.parquet`. +- Authentication: simple username + password from AWS Secrets Manager for the MVP (or Cognito if the cheapest path is similar). +- Reports rendered as HTML inside Streamlit and persisted as static HTML snapshots in S3 per run date. + +## Out of MVP scope + +- Custom domain + TLS beyond what App Runner / ALB provides by default. +- Multi-user workspaces, roles, or permissions. +- Real-time websocket updates (page refreshes are enough for a daily cadence). +- LLM-generated narratives. +- Mobile-optimized layouts. +- Editable model parameters from the UI (those live in YAML, version-controlled). + +## Inputs + +| Input | Source | +| --- | --- | +| Curated parquet from every module | `s3://smartwealthai-data-lake/curated/...` | +| MLflow tracking server | `http://:5000` | +| Latest run metadata | MLflow `latest_versions` per experiment | +| Sell-watch state | `curated/sell_watch/` | +| Personal and model NAV | `curated/portfolio_evolution/` | +| User credentials | AWS Secrets Manager | + +## Pages + +### Demo slice + +| Page | What it shows | +| --- | --- | +| **Overview** | Latest pipeline run timestamp, model portfolio headline, link to MLflow run. | +| **ETL & data quality** | Last successful SimFin / yfinance ingestion, freshness, review-queue count. | +| **Universe** | Today's universe and exclusion log (sector + bank/insurance sanity). | +| **Quality** | ROC distribution + top / bottom names + per-name component breakdown. | +| **Cheapness** | EY distribution + top / bottom names + per-name component breakdown. | +| **Ranking + model portfolio** | Combined Greenblatt rank with tie-break; top **30** equal-weight holdings. | +| **MLflow links** | Direct links to runs by date and `git_sha` tag. | + +### Full MVP (phase 2) + +| Page | What it shows | +| --- | --- | +| **Overview** | Headline KPIs (latest backtest Sharpe pass/fail, model NAV, personal NAV, open sell signals), latest pipeline run timestamp, links to MLflow runs. | +| **Permanent loss** | Today's exclusions with rule and value; trend line of count of exclusions over time; CI status of the Enron / Lehman / WorldCom regression. | +| **Sell-watch** | Open signals (proposed), confirmed history, dismissed history; each signal has a confirm and dismiss button. | +| **Backtests** | Equity curves, Sharpe table, crisis drawdown, Monte Carlo distribution, overfit flag, pass/fail. | +| **Portfolio evolution** | Personal NAV, model paper NAV, benchmark overlays, drawdown, rolling Sharpe, attribution. | + +All pages (demo and full MVP): every score row shows input components and rules that fired; every page links to the MLflow run id that produced the displayed data. + +## Mermaid diagram + +```mermaid +flowchart LR + User["User browser"] --> Streamlit["Streamlit app (Fargate Spot)"] + Streamlit --> S3["S3 (curated parquet + reports)"] + Streamlit --> MLflow["MLflow tracking server"] + Streamlit --> SES["AWS SES (confirm action triggers email follow-up)"] + Streamlit --> SellWatch["curated/sell_watch/confirmations.parquet"] + SellWatch --> Orders["Order builder (paper)"] +``` + +## Expected flow + +1. The user signs in. +2. The Streamlit app loads parquet directly from S3 via DuckDB for speed. +3. Each page queries the curated zone and renders the relevant tables and plots. +4. The sell-watch page lists `proposed` signals; clicking confirm or dismiss writes back to `confirmations.parquet` and the broker module picks up confirmed signals on its next run. +5. Each page footer shows the MLflow run id and a link. + +## Acceptance criteria + +### Demo slice + +- [x] Dashboard shows combined rank, ROC/EY inputs, and top-30 equal-weight portfolio with explanations. +- [x] Dashboard is readable from cached parquet; no live SimFin or yfinance calls for display. +- [x] Every numeric score traces to a curated parquet row. +- [x] Dashboard renders correctly when curated parquet for a module is missing (clear empty state). + +### Full MVP (phase 2) + +- The dashboard is fully readable from cached parquet; no network calls to the live pipeline are made for display. +- Every numeric score on the dashboard can be traced to a row in a curated parquet file. +- Sell-watch confirmation is the only write operation triggered by the dashboard. +- The static HTML snapshot per run date is stored in `s3://smartwealthai-reports/run_date=/index.html` and is browsable. +- Authentication blocks unauthenticated access. +- The dashboard build is published from GitHub Actions to ECR and deployed to Fargate Spot. + +## Open questions + +- Hosting choice: Fargate Spot behind ALB, App Runner, or even a small EC2 with Caddy. Recommendation: App Runner if the price difference at the MVP scale is small; otherwise Fargate Spot. +- Custom domain in the MVP, or just the default AWS URL? Recommendation: default URL for the MVP; custom domain is a follow-up. +- Persisting the HTML snapshot per run date: do we generate it from Streamlit (which is not natively static) or from a small jinja template fed by the same parquet? Recommendation: jinja template; Streamlit is the live interactive surface, the snapshot is the immutable record. + +## Risks + +- Streamlit reloads on each interaction; large parquet datasets must be cached aggressively in memory. +- App Runner / Fargate Spot can be killed mid-session by AWS; the user just refreshes. Documented as acceptable for the MVP. +- Authentication via Streamlit secrets is weak; the deployment must sit behind an AWS auth layer (Cognito, ALB auth, or IAM) before any real user lands on it. +- Sell-watch confirmations from the dashboard must be idempotent; double-clicking confirm should not create two orders. diff --git a/spec/features/006-etl-data-lake/spec.md b/spec/features/006-etl-data-lake/spec.md new file mode 100644 index 0000000..ba93021 --- /dev/null +++ b/spec/features/006-etl-data-lake/spec.md @@ -0,0 +1,451 @@ +# Feature: ETL and Data Lake + +## Implementation status + +**done** (demo slice) — SimFin connector + normalizer + shareprices snapshot + end-to-end orchestrator `run-demo-pipeline` ([#63](https://github.com/JLaborda/SmartWealthAI/issues/63)). SEC spike frozen ([ADR-0001](../../adr/0001-simfin-fundamentals-mvp.md)). + +## Objective + +Build the module that downloads, validates, normalizes, and stores financial data so that every downstream module (universe construction, scoring, backtesting, sell-watch, portfolio evolution) can rely on a single trustworthy source. The data lake lives on AWS S3 and is queried with DuckDB. Point-in-time correctness and incremental refresh are mandatory. + +## MVP scope + +### Demo slice (June 30 — primary) + +- Ingest US fundamentals from **SimFin** bulk download (`simfin` Python package, free tier). +- Datasets: `companies`, `industries`, `income` (TTM), `balance` (quarterly), `cashflow` (TTM), `shareprices` (`latest`) for `market=us`. +- Store SimFin bulk responses verbatim under `raw/simfin/`. +- Normalize into provider-agnostic `curated/fundamentals` (same schema scoring modules expect). +- Point-in-time: `as_of_date` = SimFin `Publish Date`; restatements via `Restated Date` + new `version_id`. +- Build run-date prices from SimFin bulk `shareprices/latest` joined to the universe (one row per ticker). +- Run data quality checks; failing rows → review queue. +- Weekly bulk refresh on free tier (`refresh_days=7`); incremental normalize by publish-date watermark. +- DuckDB views on curated parquet. + +### Full MVP (phase 2 additions) + +- SEC EDGAR ETL (frozen spike: `sec_client`, `download-fundamentals`). +- Incremental per-CIK filing ingest when SEC normalizer ships. +- S&P 500–scoped backfill policies. + +## Out of MVP scope + +- Real-time streaming ingestion. +- Paid data providers (Bloomberg, FactSet, CRSP). +- Cross-currency data (only USD-denominated US issuers). +- Dividend history for the personal portfolio (tracked separately; see `portfolio-evolution.md`). +- Cross-region replication or HA setups for S3. + +## Inputs + +| Input | Source | Notes | +| --- | --- | --- | +| Income statement (TTM) | SimFin bulk `income` variant `ttm` | EBIT, interest, revenue, net income. | +| Balance sheet (quarterly) | SimFin bulk `balance` variant `quarterly` | NWC, PP&E, debt, cash, shares. | +| Cash flow (TTM) | SimFin bulk `cashflow` variant `ttm` | Phase 2 permanent-loss filter; ingest in demo for raw archive. | +| Publish / report / restated dates | SimFin statement rows | `Publish Date` → `as_of_date`. | +| Company metadata | SimFin `companies` | `Ticker`, `CIK`, `IndustryId`, `SimFinId`. | +| Industry labels | SimFin `industries` | Sector/industry names for exclusions CSV. | +| Share prices (demo) | SimFin bulk `shareprices` variant `latest` | Run-date close for market cap; same ticker namespace as universe. | +| Share prices (phase 2) | SimFin `shareprices/daily` or vendor fallback | Backtest and personal NAV. | +| Industry exclusions | `data/reference/simfin_industry_exclusions.csv` | Banks, insurers, utilities. | +| Reference ticker map | `data/reference/ticker_mapping.csv` | Broker symbol → yfinance symbol. | +| SEC `companyfacts` (frozen) | SEC EDGAR | Phase 2 only; spike under `raw/sec_edgar/`. | + +## Outputs + +All outputs live under `s3://smartwealthai-data-lake/` and are queryable from DuckDB. + +| Dataset | Path (S3) | Partitioning | Notes | +| --- | --- | --- | --- | +| Raw SimFin bulk | `raw/simfin/dataset=/variant=/market=us/as_of_date=/` | by dataset, variant, download date | Verbatim CSV/ZIP from SimFin bulk API. | +| Raw share prices | `raw/simfin/dataset=shareprices/variant=latest/market=us/as_of_date=/` | by download date | Verbatim SimFin bulk CSV. | +| Raw prices (phase 2) | `raw/yfinance/...` or `shareprices/daily` | by ticker / date | Vendor fallback for backtest. | +| Raw SEC (frozen) | `raw/sec_edgar/cik=/endpoint=companyfacts/...` | by CIK | Phase 2; existing spike layout. | +| Curated fundamentals (PIT) | `curated/fundamentals/cik=/period=/` | by CIK and fiscal period | Provider-agnostic schema; `as_of_date`, `version_id`, `fiscal_period_end`. | +| Curated prices (demo) | `curated/prices/run_date=/prices.parquet` | by run date | One row per universe ticker: `run_date`, `ticker`, `price_date`, `close`, `adj_close`, `volume`. | +| Curated prices (phase 2) | `curated/prices/ticker=/year=/` | by ticker and year | Full daily history for backtest and NAV. | +| Universe history | `curated/universe/run_date=/` | by run date | Built by `universe-construction`. | +| Issue registry | `curated/issues/run_date=/` | by run date | Rows that failed quality checks. | +| yfinance cache | `cache/yfinance///.parquet` | by ticker, endpoint, date | TTL per endpoint. | + +## Mermaid diagram + +```mermaid +flowchart TD + Scheduler["Pipeline run"] --> SimFinConn["SimFin bulk connector"] + + SimFinConn --> RawSF["raw/simfin/ (immutable)"] + RawSF --> SFNorm["SimFin fundamentals normalizer"] + RawSF --> SharePx["shareprices/latest"] + SharePx --> PriceNorm["Demo price snapshot builder"] + Scheduler -. "phase 2" .-> YFConn["yfinance / vendor fallback (cached)"] + YFConn --> Cache["yfinance cache (S3, TTL)"] + Cache --> RawYF["raw/yfinance/"] + RawYF --> PriceHist["Phase 2 price normalizer"] + + SFNorm --> Curated["Curated parquet (S3)"] + PriceNorm --> Curated + PriceHist --> Curated + Curated --> PITStore["PIT store (curated/fundamentals)"] + PITStore --> QC["Data quality checks"] + Curated --> QC + + QC --> ReviewQueue["Review queue (curated/issues)"] + QC --> DuckDB["DuckDB views"] + DuckDB --> Downstream["Downstream modules"] +``` + +## Expected flow (demo) + +1. Download SimFin bulk US datasets (`companies`, `industries`, `income-ttm`, `balance-quarterly`, `cashflow-ttm`, `shareprices-latest`) if older than `refresh_days`. Store verbatim under `raw/simfin/...`. +2. Build universe for `run_date` (see `universe-construction.md`). +3. Join universe tickers to `shareprices/latest`; for each ticker take the latest `Date <= run_date`; write `curated/prices/run_date=/prices.parquet`. Missing tickers → error summary, excluded from scoring join. +4. Run the **SimFin normalizer** on fundamentals bulk snapshots (see *SimFin normalizer* below). +5. Quality checks; failures → `curated/issues/`. +6. Publish DuckDB views. Downstream reads curated only. + +## Expected flow (SEC — phase 2, frozen spike) + +Existing `download-fundamentals` CLI and `sec_client` remain in repo for reference. Not invoked by the demo pipeline. When resumed: per-CIK `companyfacts` download, EDGAR `acceptance-datetime` as `as_of_date`, separate SEC normalizer path documented below. + +## Data quality checks (initial set) + +| Check | Severity | +| --- | --- | +| Mandatory fields present (revenue, EBIT, net income, total assets, total liabilities, shares outstanding) | Block | +| `as_of_date` exists and is not in the future relative to `run_date` | Block | +| `fiscal_period_end <= as_of_date` | Block | +| Reported currency is USD | Block (non-USD goes to review queue) | +| Restated values produce a new `version_id` for the same `(cik, fiscal_period_end)` | Warn | +| Price gap larger than configurable threshold without a corresponding corporate action | Warn | +| Volume zero across multiple consecutive trading days | Warn | +| Schema migration mismatch | Block | + +## Point-in-time semantics + +- Every curated fundamentals row has `(cik, fiscal_period_end, as_of_date, version_id)` as the natural key. +- A query "fundamentals as of decision date D" returns, per `(cik, fiscal_period_end)`, the row with the highest `as_of_date <= D` and, on tie, the highest `version_id`. +- The same logic applies when re-running historical backtests: the backtest engine pins `D = decision_date` for each rebalance and never sees a row with `as_of_date > D`. +- Restated financials are kept as new versions; the prior version is preserved for replay of past decisions. +- **Demo share prices:** curated `price_date` comes from SimFin `shareprices/latest` (free tier refreshes ~weekly). **`price_date` may trail `run_date` by up to ~30 days**; no block or review queue for staleness in the demo slice. Phase 2 uses `shareprices/daily` or vendor fallback when same-day accuracy matters. + +Phase 2 yfinance cache semantics: + +- Cache key: `(ticker, endpoint, as_of_date)`. +- TTL per endpoint: + - Daily prices (`history`): 1 day after market close. + - Corporate actions (`actions`): 7 days. + - Static company info (`info`): 30 days. + - Income statement / balance sheet / cash flow: 90 days (yfinance fundamentals are sanity cross-check only; SimFin is canonical). +- Cache miss triggers a live call and writes the response to both the cache and the raw zone. +- Cache hits never trigger network calls. + +## Incremental refresh strategy + +- **SimFin (demo):** Re-download bulk US files when on-disk age exceeds `refresh_days` (default `7` on free tier). Normalizer processes only rows with `Publish Date` newer than the last successful watermark per dataset. +- **Initial backfill:** One manual bulk download of all demo datasets; normalizer filters to universe tickers. +- **yfinance / vendor fallback (phase 2):** Per-ticker watermark as before. +- Curated zones are append-only. Restatements create new versions; we never overwrite a prior version. + +## Acceptance criteria + +- Raw and curated zones are clearly separated; raw is never read by scoring modules. +- Every fundamentals value can be traced to a source file under `raw/simfin/` and its SimFin publish metadata. +- A query for "fundamentals available on date D" never returns rows with `as_of_date > D`. +- The same ingest run can fail for one ticker without aborting the rest. +- Schema versions and migrations are explicit; downstream views do not break silently. +- `yfinance` is not called when a valid cache entry exists. +- A full daily incremental run for the demo universe completes inside the Fargate Spot task budget (target: under 30 minutes; to validate during implementation). Phase 2 S&P 500 historical universe may need a separate budget check. +- A backtest run never triggers fresh `yfinance` calls; it only reads curated parquet. +- Schema, partitioning, and DuckDB view names are documented in the spec, not only in code. + +### Progress notes + +- SimFin bulk connector implemented in `src/smartwealthai/download_simfin.py`, + `simfin_client.py`, and `lake_paths.simfin_bulk_path`. Operator guide: + [`spec/guides/download-simfin.md`](../../guides/download-simfin.md). +- SimFin fundamentals normalizer in `src/smartwealthai/simfin_normalizer.py` and + `normalize_simfin.py` CLI; mapping at `config/fundamentals/simfin_mapping_v1.yaml`. + Hermetic tests in `tests/test_simfin_normalizer.py` with fixtures under + `tests/fixtures/lake/raw/simfin/`. +- Hermetic tests in `tests/test_download_simfin.py` (path layout, skip/force, + mocked download, per-dataset failure handling). +- A hermetic fixture lake contract is implemented for CI in + `tests/fixtures/lake/README.md` with raw SEC, raw SimFin, and raw yfinance + snapshots, curated derived fundamentals, and a provenance manifest with + checksums. +- Point-in-time selection and raw fixture loading behavior are covered by tests in + `tests/test_ci_baseline.py` via `smartwealthai.fixture_lake`. +- Fundamentals download spike modules are implemented under `src/smartwealthai/` + (`sec_client`, `edgartools_client`, `download_fundamentals`) — **frozen** for phase 2. + SimFin connector + normalizer are the active demo path ([ADR-0001](../../adr/0001-simfin-fundamentals-mvp.md)). +- SimFin shareprices snapshot in `price_ingest.py` and `download_prices.py` CLI + (`poetry run download-prices`). Requires `shareprices/latest` from `download-simfin`. + Writes `curated/prices/run_date=/prices.parquet`. Hermetic tests in + `tests/test_download_prices.py`. + +**Operator sequence (demo pipeline):** + +```bash +poetry run run-demo-pipeline --run-date 2026-06-19 +poetry run run-dashboard --data-dir data --run-date 2026-06-19 +``` + +Equivalent manual steps: + +```bash +poetry run download-simfin --as-of-date 2026-06-19 +poetry run build-universe --run-date 2026-06-19 +poetry run normalize-simfin --snapshot-date 2026-06-19 --universe-run-date 2026-06-19 +poetry run download-prices --run-date 2026-06-19 --snapshot-date 2026-06-19 +poetry run score-universe --run-date 2026-06-19 +``` + +## Decisions made (fundamentals) + +| Area | Decision | +| --- | --- | +| **Demo normalizer input** | SimFin bulk parquets from `raw/simfin/` (income TTM + balance quarterly + cashflow TTM). | +| **`as_of_date` (demo)** | SimFin `Publish Date`; `Restated Date` → new `version_id`. | +| **SEC normalizer (phase 2)** | `companyfacts` JSON from `raw/sec_edgar/...`; EDGAR acceptance as `as_of_date`. | +| **Mapping** | `config/fundamentals/simfin_mapping_v1.yaml` (demo); `mapping_v1.yaml` (SEC phase 2). | +| Canonical fields | Same ~12 curated columns for ROC, EY, and QC regardless of provider. | +| Provenance | Per-field source column + `mapping_version`. | +| Downstream contract | Scoring reads `curated/fundamentals` only — provider-agnostic schema. | +| SEC spike | Frozen in repo; not deleted. | + +## SimFin bulk connector (demo) + +Downloads US fundamentals via the `simfin` Python package into `raw/simfin/`. +Operator guide: [`spec/guides/download-simfin.md`](../../guides/download-simfin.md). + +### CLI + +```bash +export SIMFIN_API_KEY="" +poetry run download-simfin +poetry run download-simfin --refresh-days 7 --force +``` + +### Module map + +| Module | Role | +| --- | --- | +| `smartwealthai.simfin_client` | API key config, safe bulk download (zip-slip guarded), cache CSV path. | +| `smartwealthai.download_simfin` | CLI orchestration, skip/force by `refresh_days`, run summary. | +| `smartwealthai.lake_paths` | `simfin_bulk_path`, `simfin_errors_path`. | + +### Acceptance criteria (SimFin connector) + +- [x] `simfin` dependency in `pyproject.toml`; API key from `SIMFIN_API_KEY`. +- [x] CLI downloads all six demo datasets into stable `raw/simfin/` partitions. +- [x] Re-run without `--force` skips datasets fresher than `refresh_days`; `--force` overwrites. +- [x] Per-dataset failures recorded in run summary; batch continues when possible. +- [x] Hermetic tests cover path building, skip/force logic, and mocked download. +- [x] Bulk ZIP extraction validates member paths (zip-slip guard); does not use simfin `load_*` extractall path. +- [x] Operator steps in [`download-simfin.md`](../../guides/download-simfin.md). + +## SimFin normalizer (demo) + +Transforms SimFin bulk statements into curated canonical parquet. Joins income TTM with the latest quarterly balance row per ticker subject to PIT filters. Curated rows are accumulated in memory and written in bulk (one `to_parquet` per `cik`/`period` partition; deduped `mkdir`; parallel thread pool for I/O). Interactive runs show two Click progress bars: tickers during transform, partitions during write (`--quiet` to suppress; `--progress` to force on non-TTY). + +### Configuration + +| Setting | Source | Notes | +| --- | --- | --- | +| API key | `SIMFIN_API_KEY` env var | Required; AWS Secrets Manager at runtime. Never commit to repo. | +| Data root | `--data-dir` or S3 lake root | Local dev default `data/`. | +| Refresh | `refresh_days` | Default `7` for free tier. | +| Column mapping | `config/fundamentals/simfin_mapping_v1.yaml` | SimFin column → canonical field. | + +### Local raw layout (demo) + +| Dataset | Path | +| --- | --- | +| SimFin bulk snapshot | `raw/simfin/dataset=/variant=/market=us/as_of_date=/` | + +### Acceptance criteria (SimFin normalizer) + +- [x] Reads bulk files from `raw/simfin/` only. +- [x] Emits same curated schema as SEC path would (see canonical fields below). +- [x] PIT natural key `(cik, fiscal_period_end, as_of_date, version_id)`. +- [x] Hermetic tests with fixture SimFin CSV snippets. +- [x] `simfin_mapping_v1.yaml` drives column resolution. + +### Module map (SimFin normalizer) + +| Module | Role | +| --- | --- | +| `smartwealthai.simfin_normalizer` | Raw SimFin CSV join + PIT stamping + curated parquet writer. | +| `smartwealthai.normalize_simfin` | CLI entry point (`poetry run normalize-simfin`). | +| `config/fundamentals/simfin_mapping_v1.yaml` | SimFin column → canonical field mapping. | +| `smartwealthai.lake_paths` | `curated_fundamentals_path`, `curated_issues_path`, `fiscal_period_label`. | + +```bash +poetry run normalize-simfin --snapshot-date 2026-06-18 --universe-run-date 2026-06-18 +poetry run normalize-simfin --snapshot-date 2026-06-18 --ticker AAPL --ticker MSFT +poetry run normalize-simfin --snapshot-date 2026-06-18 --universe-run-date 2026-06-18 --quiet +``` + +## SEC fundamentals normalizer (phase 2) + +**GitHub issue:** [#52](https://github.com/JLaborda/SmartWealthAI/issues/52) — blocks [#44](https://github.com/JLaborda/SmartWealthAI/issues/44) (ROC/EY) and feeds [#50](https://github.com/JLaborda/SmartWealthAI/issues/50) (PIT selection). + +Transforms immutable `companyfacts` JSON into curated canonical parquet. Discovery logic from `notebooks/poc_metrics.ipynb` (JNJ EBIT walk-up, debt summation, NWC components) is productized here — not in scoring code. + +### Pipeline stages + +```mermaid +flowchart LR + Raw["raw/companyfacts JSON"] --> Long["Long facts table"] + Long --> Resolve["Resolve canonical fields (mapping_v1)"] + Resolve --> Curated["curated/fundamentals parquet"] + Resolve --> Issues["curated/issues (review queue)"] +``` + +1. **Ingest:** Parse `companyfacts` into a long table: `(cik, concept, fiscal_period_end, value_usd, as_of_date, form, accession)`. +2. **Resolve:** For each canonical field, apply `config/fundamentals/mapping_v1.yaml` rules: + - **Direct:** read a single XBRL concept when populated (e.g. `OperatingIncomeLoss` for EBIT). + - **Fallback chain:** try ordered alternative concepts (e.g. `InterestExpense`, then `InterestExpenseNonoperating`). + - **Derived:** compute from other resolved components (e.g. EBIT walk-up from net income + taxes + interest; `total_debt` as sum of components in scoring, not necessarily stored). + - **Review queue:** if resolution fails or QC blocks, write to `curated/issues/` — do not silently impute. +3. **Stamp:** Attach `as_of_date`, `fiscal_period_end`, `version_id`, `mapping_version`, and per-field provenance. +4. **Append:** Write append-only parquet under `curated/fundamentals/cik=/period=/`. + +### Canonical output fields (mapping v1) + +Fields required by `high-quality-stocks.md`, `cheap-stocks.md`, and ETL QC: + +| Canonical column | Used for | +| --- | --- | +| `ebit` | ROC, EY | +| `current_assets`, `current_liabilities`, `cash`, `short_term_debt` | Net working capital | +| `ppe_net` | ROC denominator | +| `long_term_debt`, `preferred_equity`, `minority_interest` | Enterprise value | +| `shares_outstanding` | Market cap join | +| `revenue`, `net_income`, `total_assets`, `total_liabilities` | Data quality checks | + +`total_debt`, `nwc`, `ev`, `roc`, and `ey` are computed in scoring modules from curated inputs plus prices — not stored in curated fundamentals unless a future spec revision says otherwise. + +### Configuration + +| Setting | Path | Notes | +| --- | --- | --- | +| XBRL → canonical mapping | `config/fundamentals/mapping_v1.yaml` | Versioned; new file for breaking mapping changes. | +| ROC formula | `config/quality/roc.yaml` | Scoring layer (downstream). | +| EY formula | `config/cheap/ey.yaml` | Scoring layer (downstream). | + +### Acceptance criteria (normalizer) + +- [ ] Reads only `companyfacts` from raw; no dependency on edgartools parquets. +- [ ] `mapping_v1.yaml` drives resolution; provenance columns on every output row. +- [ ] PIT natural key `(cik, fiscal_period_end, as_of_date, version_id)` on curated output. +- [ ] JNJ resolves EBIT via walk-up when `OperatingIncomeLoss` is blank (POC-validated). +- [ ] Hermetic tests with fixture `companyfacts` JSON; golden checks for JNJ + at least one direct-tag issuer. +- [ ] Per-field coverage summary for Dow 30 (`% direct` / `% fallback` / `% review queue`). +- [ ] CLI entry point documented in operator guide when implemented. + +### Out of normalizer scope (MVP) + +- Full US-GAAP taxonomy materialization. +- Per-ticker special cases (`if ticker == "JNJ"`). +- edgartools as a second normalization path. +- S3 upload (local `--data-dir` first; S3 follows CI/CD lake work). + +## Fundamentals download spike (local) + +First vertical slice: download and persist raw SEC `companyfacts` for a parameterized +universe. Optionally also download standardized annual statements from `edgartools` for +notebook exploration and cross-checks — **not** for the production normalizer. + +No `submissions` ingest, no curated parquet normalizer in this slice (normalizer: [#52](https://github.com/JLaborda/SmartWealthAI/issues/52)), and no S3 upload. + +**Operator guide:** [`spec/guides/download-fundamentals.md`](../../guides/download-fundamentals.md) + +### Scope + +- Universe presets backed by versioned CSV files under `data/reference/universes/`. + Initial preset: `dow30` (30 tickers with fixed CIKs). Expand later to S&P 500, + Russell 3000, or Nasdaq as additional presets. +- SEC REST: verbatim `companyfacts` JSON per CIK (**required** for normalizer). +- `edgartools` (optional): `Company(ticker).get_facts()` → income, balance, and cash-flow + statements via `.income_statement()`, `.balance_sheet()`, and + `.cashflow_statement()` with `period="annual"` and configurable `periods` + (default 16). Retained for dev/QC; may be dropped from the CLI once [#52](https://github.com/JLaborda/SmartWealthAI/issues/52) is stable. +- Local raw zone only (`--data-dir`, default `data/`). Paths mirror the production + lake layout so the module can move to S3 later without renaming. + +### Configuration + +| Setting | Source | Notes | +| --- | --- | --- | +| SEC identity | `SEC_IDENTITY` env var (required) | Used for SEC REST `User-Agent` and `edgartools.set_identity()`. | +| Data root | `--data-dir` CLI flag | Default `data/`. | +| Universe | `--universe` preset or `--universe-file` | Preset `dow30` reads `data/reference/universes/dow30.csv`. | +| History depth | `--periods` | Default `16` annual columns from `edgartools`. | +| Snapshot date | `--as-of-date` | Default: UTC today. Partition key for immutable daily snapshots. | +| Re-download | `--force` | Ignore existing files for the chosen `as_of_date`. | + +### Local raw layout + +| Dataset | Path | +| --- | --- | +| SEC companyfacts | `raw/sec_edgar/cik=/endpoint=companyfacts/as_of_date=/response.json` | +| edgartools income | `raw/edgartools/cik=/as_of_date=/income_statement_annual.parquet` | +| edgartools balance | `raw/edgartools/cik=/as_of_date=/balance_sheet_annual.parquet` | +| edgartools cash flow | `raw/edgartools/cik=/as_of_date=/cashflow_statement_annual.parquet` | +| Run errors | `raw/download_runs/as_of_date=/errors.json` (written only when failures occur) | + +### Cache and refresh + +- Partition by `as_of_date`. If a target file for today already exists, skip the + network call unless `--force` is set. +- SEC requests are throttled (max ~8 req/s) and retried up to three times with + exponential backoff on transient errors (429, 5xx, timeouts). +- A failure for one CIK does not abort the run. Permanent errors (e.g. 404) are + not retried. Exit code is `1` when any CIK fails, `0` otherwise. + +### CLI + +```bash +export SEC_IDENTITY="Your Name your@email.com" +python -m smartwealthai.download_fundamentals --universe dow30 +python -m smartwealthai.download_fundamentals --universe dow30 --periods 16 --force +``` + +### Module map + +| Module | Role | +| --- | --- | +| `smartwealthai.sec_client` | SEC REST client (throttle, retry, `companyfacts` download). | +| `smartwealthai.edgartools_client` | Optional `get_facts()` statement extraction to parquet (dev/QC). | +| `smartwealthai.download_fundamentals` | CLI orchestration, universe loading, run summary. | +| `smartwealthai.normalize_fundamentals` (planned) | `companyfacts` → curated canonical parquet ([#52](https://github.com/JLaborda/SmartWealthAI/issues/52)). | + +### Acceptance criteria (spike) + +- [x] `dow30` preset loads 30 `(ticker, cik)` rows from a git-versioned CSV. +- [ ] Each successful CIK produces the four raw artifacts above for the run date. +- [x] Re-running without `--force` on the same day skips existing files. +- [ ] `--force` re-downloads and overwrites today's partition. +- [ ] One failing CIK does not stop the rest; failures are listed in `errors.json`. +- [x] Hermetic unit tests cover universe loading, path building, and skip/force logic. + +## Open questions + +- For SEC EDGAR, do we use `sec-edgar-downloader` (filings as files), the `sec_api` (paid), or the official EDGAR REST APIs (`/submissions`, `/companyfacts`)? **Closed:** official REST APIs for fundamentals (`/companyfacts`); `sec-edgar-downloader` only when full 10-K / 10-Q text is needed by `unstructured-financial-data`. +- ~~Should `edgartools` be a normalizer input alongside `companyfacts`?~~ **Closed:** `companyfacts` only; edgartools optional dev/QC ([#52](https://github.com/JLaborda/SmartWealthAI/issues/52)). +- Do we keep daily prices only, or also intraday OHLC? Recommendation: daily-only for the MVP. +- Do we need a separate metadata table tracking ingestion provenance (URL, response code, byte size, hash), or is the S3 path enough? +- What is the policy when the same field disagrees between EDGAR and yfinance? Recommendation: EDGAR wins for fundamentals; yfinance wins for prices and corporate actions; disagreements are logged. +- Should the DuckDB views materialize parquet artifacts or always read directly from S3? Recommendation: read directly from S3 for the MVP; materialize only if query latency becomes a bottleneck. +- Do we add a hash of the raw payload to detect silent provider changes? + +## Risks + +- yfinance is a community wrapper around an undocumented Yahoo endpoint. It can break with little warning. Mitigations: cache aggressively, treat fallbacks as first-class, log every miss. +- SEC EDGAR rate limits requests (10 req/sec, with a required User-Agent). Mitigations: respect headers, throttle, retry with backoff. +- Free-tier providers have monthly quotas. The cache and provider abstraction must make it easy to skip a provider when its quota is exhausted. +- Restated fundamentals are easy to miss if the normalizer overwrites rows instead of creating new versions. The unit tests must explicitly cover this case. +- Mishandled timezones can shift `as_of_date` by a day and create silent look-ahead bias. All timestamps are stored in UTC. diff --git a/spec/features/007-high-quality-stocks/spec.md b/spec/features/007-high-quality-stocks/spec.md new file mode 100644 index 0000000..aeaf903 --- /dev/null +++ b/spec/features/007-high-quality-stocks/spec.md @@ -0,0 +1,111 @@ +# Feature: High-Quality Stocks + +## Implementation status + +done (demo cross-sectional slice) — ROC scoring and ranks: `src/smartwealthai/magic_formula_ranking.py`, CLI `score-universe` ([#60](https://github.com/JLaborda/SmartWealthAI/issues/60)). Single-ticker tracer: `magic_formula_metrics.py`, `pit_fundamentals.py`, `compute-metrics` ([#44](https://github.com/JLaborda/SmartWealthAI/issues/44)). + +## Objective + +Score the economic quality of every company that survives the universe filter and the permanent loss filter. For the MVP, the quality factor is a strict Greenblatt-style **Return on Capital (ROC)** computed from point-in-time fundamentals. Future iterations can plug additional quality signals into the same interface. + +## MVP scope + +- Compute `ROC = EBIT / (Net Working Capital + Net Fixed Assets)` per the canonical Greenblatt definition. +- Use the most recent point-in-time fundamentals available on the decision date. +- Produce a cross-sectional quality rank (lower rank = higher quality) for every passing company. +- Apply a market-cap tie-break: when ROC ties, the smaller market cap wins. +- Validate denominator: rows with `Net Working Capital + Net Fixed Assets <= 0` are flagged for review and excluded from the ranking. +- Log MLflow metrics: distribution of ROC, count of valid vs invalid rows, percentile statistics. +- Expose the score, the input components, and the explanation downstream so the dashboard can show "why is this company high quality?". + +## Out of MVP scope + +- Multi-metric quality scores (ROIC, ROE, ROA, FCF margin, accruals, balance sheet sub-score). Captured as candidates for the next iteration. +- Sector-relative quality (sectors with unusual accounting are already excluded upstream). +- Earnings quality / accruals scoring. +- Capital allocation scoring. +- Forward-looking estimates. +- Machine learning quality prediction. + +## Inputs + +| Input | Source | Notes | +| --- | --- | --- | +| Passing universe + permanent loss filter pass list | `curated/universe` + `curated/permanent_loss` | Only `pass` rows are scored. | +| PIT fundamentals (income statement, balance sheet) | `curated/fundamentals` | Filtered by `as_of_date <= run_date`. | +| Market cap | `curated/prices/run_date=/prices.parquet` join `curated/fundamentals` | Tie-break uses `shares_outstanding * adj_close` on `run_date`. | +| Run date | Pipeline parameter | | +| ROC formula version | `config/quality/roc.yaml` | Versioned to allow future variants. | + +## ROC definition (canonical Greenblatt) + +``` +ROC = EBIT / (Net Working Capital + Net Fixed Assets) +``` + +with: + +- `EBIT` = Operating income before interest and taxes. Trailing twelve months. +- `Net Working Capital` = `max(Current Assets - Excess Cash - Current Liabilities + Short-Term Debt, 0)` (Greenblatt uses non-interest-bearing current liabilities; we use this approximation and version it). +- `Net Fixed Assets` = Total fixed assets (PP&E net of depreciation). +- All values from the latest filing whose `as_of_date <= run_date`. + +The formula and its variants are versioned in `config/quality/roc.yaml`. Any change requires a new version id so backtests on prior versions remain reproducible. + +## Outputs + +| Output | Path / target | +| --- | --- | +| Quality scores parquet | `curated/scores/quality/run_date=/scores.parquet` with `cik, ticker, ebit, nwc, net_fixed_assets, roc, roc_rank, market_cap, tiebreak_rank, formula_version, as_of_date` | +| Review queue rows | `curated/issues/run_date=/quality.parquet` for invalid denominators and other warnings | +| MLflow metrics | `quality_n_valid`, `quality_n_invalid`, ROC quantiles | + +## Mermaid diagram + +```mermaid +flowchart TD + Passing["Universe pass + Permanent loss pass"] --> Loader["Load PIT fundamentals + market cap"] + Loader --> Compute["Compute EBIT, NWC, Net Fixed Assets"] + Compute --> Validate{"Denominator > 0?"} + + Validate -->|No| Review["Review queue (quality.parquet)"] + Validate -->|Yes| ROC["ROC = EBIT / (NWC + Net Fixed Assets)"] + ROC --> Rank["Cross-sectional rank (descending ROC)"] + Rank --> TieBreak["Tie-break by ascending market cap"] + TieBreak --> Output["quality/scores.parquet"] + Output --> MLflow["MLflow metrics"] +``` + +## Expected flow + +1. Load the passing universe and join with PIT fundamentals. +2. Compute `EBIT`, `Net Working Capital`, and `Net Fixed Assets` using the formula version configured for the run. +3. Validate inputs: drop rows with missing components; flag rows with `denominator <= 0` and route them to the review queue. +4. Compute `ROC`. +5. Produce a cross-sectional rank from highest ROC (rank 1) to lowest. +6. Resolve ties by ascending market cap. +7. Persist the parquet output and log MLflow metrics. + +## Acceptance criteria + +- Same `(universe, run_date, formula_version)` produces byte-identical output (hash-verifiable). +- The score is purely a function of curated PIT data; no network calls. +- Every row has both the ROC value and the components that produced it. +- Rows with invalid denominators are visible in the review queue and not silently dropped or auto-scored. +- The ranking is stable: changing only the market cap of a non-tied row never changes the rank order. +- The MLflow run logs at minimum count of valid rows, count of invalid rows, ROC median, and ROC quantiles. +- The formula version travels with each scored row, so a backtest using a past `formula_version` is reproducible. + +## Open questions + +- Greenblatt himself uses Pre-Tax Operating Earnings; do we use `EBIT` straight from EDGAR (`OperatingIncomeLoss + InterestAndDebtExpense`) or compute Pre-Tax Operating Earnings explicitly? Recommendation: use `OperatingIncomeLoss` from EDGAR and document the choice as `formula_version = v1`. +- ~~Excess cash definition for `Net Working Capital`~~ **Closed (v1):** curated `cash` uses SimFin `Cash, Cash Equivalents & Short Term Investments` for both NWC and EV (known approximation — NWC excess-cash adjustment is slightly aggressive vs cash-equivalents-only). +- Should very small ROC differences (e.g., < 0.1 percentage point) be treated as ties for the market-cap tie-break? Recommendation: no in the MVP; revisit if rank stability becomes a problem. +- For companies with negative EBIT but positive denominator, ROC is negative. Do we exclude them, or rank them at the bottom? Recommendation: rank them at the bottom; they will likely never enter the top 30 anyway. + +## Risks + +- `Net Working Capital` and `Net Fixed Assets` definitions vary across textbooks and providers. Locking the formula version is the only way to keep backtests reproducible. +- Single-metric quality leaves the strategy exposed to capital-light tech businesses whose balance sheets distort ROC. We accept this in the MVP and document the limitation. +- Restatements can move ROC sharply between versions. The PIT store keeps both versions; backtests must pick the version available at `as_of_date`. +- One-off items in EBIT can produce false positives. The MVP does not adjust for them; this is a known weakness of the Greenblatt placeholder. diff --git a/spec/features/008-permanent-loss-filter/spec.md b/spec/features/008-permanent-loss-filter/spec.md new file mode 100644 index 0000000..821b89f --- /dev/null +++ b/spec/features/008-permanent-loss-filter/spec.md @@ -0,0 +1,129 @@ +# Feature: Permanent Loss Filter + +## Implementation status + +**Deferred** for the June 30 demo slice ([ADR-0002](../../adr/0002-june-demo-scope-cut.md)). Spec remains the target for phase 2. + +## Objective + +Identify companies in the investable universe with elevated risk of permanent capital loss and remove them from the ranking before any score is computed. For the MVP, "permanent loss" is defined narrowly as **fraud or bankruptcy / financial distress**. Companies flagged by either subfilter are hard-excluded. + +## MVP scope + +- Hard exclusion (not a score penalty). A flagged company never enters the ranking. +- Two subfilters: fraud signals and bankruptcy / distress signals. +- Inputs come from the curated point-in-time store; no live network calls. +- Every exclusion records the triggered rule, the inputs that fired it, and the `as_of_date`. +- Regression test in CI: the bankruptcy subfilter must flag Enron, Lehman, and WorldCom on the dates each company was already in clear distress (e.g., Enron Q3 2001 10-Q, Lehman Q2 2008 10-Q, WorldCom Q1 2002 10-Q). If any of these stops being flagged, the CI build fails. + +## Out of MVP scope + +- Manipulation / accruals models (Beneish M-score, accruals-based scores). Deferred. +- Machine learning fraud detection. +- LLM-driven qualitative analysis of filings (handled later by `unstructured-financial-data`). +- Sector-specific distress models (banks, insurers, REITs and utilities are already excluded upstream by `universe-construction`). +- `Penalize` and `unknown` states. Only `pass` and `exclude` for the MVP. + +## Inputs + +| Input | Source | +| --- | --- | +| Curated PIT fundamentals (income statement, balance sheet, cash flow) | `curated/fundamentals` | +| Adjusted prices and corporate actions | `curated/prices` | +| SEC filing index (form type, accession, acceptance datetime) | `curated/sec_edgar/submissions` | +| Auditor information (when extracted) | `curated/sec_edgar/auditor` (future) | +| Regression test fixtures | `tests/fixtures/permanent_loss/` (CIK + as_of_date + expected `exclude`) | +| Run date | Pipeline parameter | + +## Outputs + +| Output | Path / target | +| --- | --- | +| Exclusion table | `curated/permanent_loss/run_date=/exclusions.parquet` with columns `cik, ticker, subfilter, rule_id, rule_version, triggered_value, threshold, as_of_date, explanation` | +| Filter status | `pass` or `exclude` per `(cik, as_of_date)` | +| Logged metrics (MLflow) | Number of evaluations, number of exclusions per subfilter, list of newly excluded companies | + +## Bankruptcy / distress subfilter (MVP rules) + +The MVP implements a small but well-known set of distress indicators. Each rule has a versioned id so historical decisions can be replayed. + +| Rule id | Definition | Threshold (initial) | Source | +| --- | --- | --- | --- | +| `BK_ALTMAN_Z` | Altman Z-score for non-financials | Z < 1.81 | `curated/fundamentals` | +| `BK_INT_COVERAGE` | Interest coverage (EBIT / Interest Expense), TTM | < 1.0 | `curated/fundamentals` | +| `BK_NETDEBT_EBITDA` | Net debt to EBITDA, TTM | > 7.0 with negative FCF | `curated/fundamentals` | +| `BK_NEGATIVE_EQUITY` | Stockholders' equity | < 0 | `curated/fundamentals` | +| `BK_GOING_CONCERN` | "Going concern" language flag from latest 10-K (provided by `unstructured-financial-data` once available) | flag present | `curated/text_flags` (future) | +| `BK_DELISTED` | Listing status | `delisted` and `delisting_reason in {bankruptcy, regulatory}` | `curated/prices` | + +A company is excluded if **any** of the above rules fire. + +## Fraud subfilter (MVP rules) + +The MVP fraud signals are intentionally narrow. They detect structural / accounting events, not subjective judgments. + +| Rule id | Definition | Threshold (initial) | +| --- | --- | --- | +| `FRD_RESTATEMENT_RECENT` | Material restatement of prior reported figures in the last 12 months (e.g., 10-K/A or 10-Q/A filings) | `>= 1` filing | +| `FRD_AUDITOR_CHANGE_REPEATED` | Auditor change in 2 of the last 3 fiscal years | `>= 2` changes | +| `FRD_REGULATORY_ACTION` | Open SEC enforcement action against the issuer | `True` | +| `FRD_LATE_FILER` | Filed `NT 10-K` or `NT 10-Q` (late filing notification) in last 12 months | `>= 1` filing | + +A company is excluded if **any** rule fires. Rules that depend on data not yet available in the MVP (`FRD_REGULATORY_ACTION`, derived from EDGAR enforcement feeds) are coded but tolerated as `unavailable` until the data is wired. Their absence is logged. + +## Mermaid diagram + +```mermaid +flowchart TD + Universe["v_universe (today)"] --> Loader["Load PIT fundamentals + prices + filings"] + Loader --> Bankruptcy["Bankruptcy / distress rules"] + Loader --> Fraud["Fraud rules"] + + Bankruptcy --> Decision{"Any rule fired?"} + Fraud --> Decision + + Decision -->|Yes| Exclude["Exclude (hard)"] + Decision -->|No| Pass["Pass to scoring modules"] + + Exclude --> Output["curated/permanent_loss exclusions.parquet"] + Output --> MLflow["MLflow metrics + artifact"] + + subgraph Tests["CI regression"] + Enron["Enron Q3 2001"] --> RegTest["Must be excluded"] + Lehman["Lehman Q2 2008"] --> RegTest + WorldCom["WorldCom Q1 2002"] --> RegTest + end +``` + +## Expected flow + +1. Read the universe for the run date from `v_universe`. +2. Join with the latest PIT fundamentals (using `as_of_date <= run_date`). +3. Compute each bankruptcy and fraud rule. Rules with missing required inputs are recorded as `unavailable` and the row is sent to the review queue (not auto-excluded). +4. If any rule fires, mark the company as `exclude` with the rule id, threshold, and the values that triggered it. +5. Write the exclusion parquet and log MLflow metrics. +6. Hand the passing set of `(cik, ticker)` to the scoring modules. + +## Acceptance criteria + +- The module is a pure function of curated parquet + reference rules: same inputs produce byte-identical output (verifiable by hash). +- The regression CI test for Enron, Lehman, and WorldCom blocks the build if any of the three stops being flagged. +- Every excluded row carries the `rule_id`, `rule_version`, `triggered_value`, and `threshold`. +- Rule definitions live in code, but thresholds live in a YAML config under `config/permanent_loss/` so they can be tuned by backtests without code changes. +- The filter never queries network resources. +- Companies with `unavailable` rule outputs do not pass silently: they enter the review queue. +- Each MLflow run for the permanent loss filter logs the count of exclusions per rule. + +## Open questions + +- Should `BK_NETDEBT_EBITDA` be sector-relative even though banks / insurers / utilities are excluded? Recommendation: keep it absolute for the MVP; revisit when those sectors are reintroduced. +- Threshold for `BK_NETDEBT_EBITDA` should probably be revisited per backtest; the initial 7.0 is a placeholder. +- Where does the auditor-change history come from? Recommendation: parse `acceptedAccountingFirm` from EDGAR if available; otherwise wait for `unstructured-financial-data` to provide it. +- Do we want a "watchlist" state (`watch`) between `pass` and `exclude`? Recommendation: no for the MVP; that role is fulfilled by `sell-watch` once a company is held. + +## Risks + +- Excluding bankrupt companies after the fact is easy; excluding them *before* is the hard part. The Altman Z-score has known weaknesses for tech / asset-light companies. The MVP accepts this in exchange for simplicity. +- Restatements happen for innocent reasons (acquisitions, IFRS-to-GAAP changes). The MVP rule will produce some false positives; they are tracked in the FP/FN review process. +- Removing companies for "late filer" status can be aggressive. We log every `NT 10-K` and `NT 10-Q` so the FP review can tune the rule. +- `BK_DELISTED` is only useful historically; it cannot prevent a loss in real time. Its purpose is to make the historical backtest realistic (a delisted-for-bankruptcy company never enters the post-delisting universe). diff --git a/spec/features/010-sell-watch/spec.md b/spec/features/010-sell-watch/spec.md new file mode 100644 index 0000000..a9b68d5 --- /dev/null +++ b/spec/features/010-sell-watch/spec.md @@ -0,0 +1,132 @@ +# Feature: Sell-Watch / Vigilance + +## Implementation status + +**Deferred** for the June 30 demo slice ([ADR-0002](../../adr/0002-june-demo-scope-cut.md)). Spec remains the target for phase 2. + +## Objective + +Monitor every name held in the **model portfolio** every day and emit a hard `sell` signal when the thesis breaks. Signals never auto-execute: they appear in the dashboard and trigger an AWS SES email so the user can review and confirm. Sell-watch does not monitor the user's personal portfolio (those are personal decisions). + +## MVP scope + +- Daily run against the live model portfolio holdings. +- Four trigger families: quality deterioration, fraud / bankruptcy flag turning on after entry, overvaluation, opportunity cost. +- Hard `sell` only. No `trim` or `hold-with-warning` states. +- Manual confirmation required: a sell signal must be confirmed in the dashboard before the broker module builds an order. +- Alerts: dashboard badge + AWS SES email per signal. +- Logs every evaluation (signal or no signal) for audit and FP/FN review. +- MLflow run per daily evaluation with parameters, metrics, and artifacts. + +## Out of MVP scope + +- Monitoring of the user's personal portfolio. +- Price-based stops (trailing stop, drawdown stop). +- Time-based stops. +- Automatic execution. +- Multi-state output (`trim`, `hold-with-warning`). +- LLM-driven narrative explanation (deferred). + +## Inputs + +| Input | Source | +| --- | --- | +| Current model portfolio holdings | `curated/portfolio/model/holdings.parquet` (produced by portfolio-construction) | +| Latest PIT fundamentals | `curated/fundamentals` | +| Latest prices | `curated/prices` | +| Latest quality scores | `curated/scores/quality` | +| Latest cheapness scores | `curated/scores/cheap` | +| Latest watchlist (top-ranked names not yet held) | `curated/portfolio/watchlist.parquet` | +| Permanent loss filter output | `curated/permanent_loss` | +| Confirmed signals history | `curated/sell_watch/confirmations.parquet` (so we do not re-alert on the same signal day after day) | +| Sell-watch config | `config/sell_watch.yaml` (thresholds, opportunity-cost margin, lookback for ROC YoY) | + +## Trigger definitions (MVP) + +All thresholds are starting points and live in `config/sell_watch.yaml`. Each is hyperparameter-able by the backtest engine. + +| Trigger id | Rule | Default threshold | +| --- | --- | --- | +| `SW_PERMANENT_LOSS` | The permanent loss filter, evaluated on the holding today, flags `exclude` | n/a | +| `SW_QUALITY_DROP_YOY` | ROC YoY drop greater than `quality_yoy_drop` | 30% | +| `SW_QUALITY_DECILE_DROP` | The holding is no longer in the top decile of cross-sectional ROC | top 10% | +| `SW_OVERVALUATION_PCT` | EY below the cross-sectional `overvaluation_percentile` of the current universe | 10th percentile | +| `SW_OVERVALUATION_ABS` | EY below the absolute `overvaluation_floor` | 5.0% | +| `SW_OPPORTUNITY_COST` | A watchlist candidate outranks the holding by more than `opportunity_cost_margin` positions on the combined Greenblatt rank | 5 positions | + +A holding is flagged `sell` if `SW_PERMANENT_LOSS` fires, **or** any quality trigger fires (`SW_QUALITY_DROP_YOY` or `SW_QUALITY_DECILE_DROP`), **or** any overvaluation trigger fires (`SW_OVERVALUATION_PCT` or `SW_OVERVALUATION_ABS`), **or** `SW_OPPORTUNITY_COST` fires. + +## Outputs + +| Output | Path / target | +| --- | --- | +| Signals parquet | `curated/sell_watch/run_date=/signals.parquet` with `ticker, triggers, fired_thresholds, values, message_id, status` | +| Audit parquet | `curated/sell_watch/run_date=/evaluations.parquet` (every holding evaluated, signal or no signal) | +| Email payload (per signal) | Subject + body + dashboard deep link | +| MLflow metrics | Count of signals per trigger, daily count of evaluations, count of confirmed vs ignored signals | + +The `status` field starts as `proposed` and moves to `confirmed` or `dismissed` when the user acts on it from the dashboard. + +## Mermaid diagram + +```mermaid +flowchart TD + Holdings["Model portfolio holdings (today)"] --> Eval["Evaluate triggers"] + PLoss["Permanent loss filter today"] --> Eval + ROC["Quality score (today and 1y ago)"] --> Eval + EY["Cheapness score (today, cross-section)"] --> Eval + Watchlist["Watchlist (top-ranked non-holders)"] --> Eval + + Eval --> AnyTrigger{"Any trigger fired?"} + AnyTrigger -->|No| Audit["audit parquet only"] + AnyTrigger -->|Yes| Dedup["Dedup against confirmations history"] + Dedup --> SignalsOut["sell_watch/signals.parquet"] + SignalsOut --> Dashboard["Dashboard sell-watch panel"] + SignalsOut --> SES["AWS SES email"] + Dashboard --> User["User confirms or dismisses"] + User -->|Confirm| Orders["Order builder (paper)"] + User -->|Dismiss| Audit2["confirmations.parquet (dismissed)"] +``` + +## Expected flow + +1. Pull today's holdings from the model portfolio table. +2. For each holding: + 1. Look up its current permanent loss status. If `exclude`, fire `SW_PERMANENT_LOSS`. + 2. Compute ROC today and ROC 1 year ago from PIT data. Fire `SW_QUALITY_DROP_YOY` if drop > threshold. + 3. Locate the holding's ROC rank among the current universe. Fire `SW_QUALITY_DECILE_DROP` if it left the top decile. + 4. Locate the holding's EY in today's universe percentile. Fire `SW_OVERVALUATION_PCT` if below the configured percentile. + 5. Read the holding's absolute EY. Fire `SW_OVERVALUATION_ABS` if below the absolute floor. + 6. Compare the holding's combined Greenblatt rank against the best non-held watchlist candidate. Fire `SW_OPPORTUNITY_COST` if margin exceeds threshold. +3. If any trigger fired, check the confirmations history to avoid re-alerting on an already-active signal. If it is new, write a signal row and send an SES email. +4. Write the audit parquet covering every evaluation. +5. The dashboard exposes the open signals. The user clicks confirm or dismiss, which writes `confirmations.parquet`. +6. Confirmed signals flow into the broker module as sell orders. Dismissed signals are remembered so we do not re-fire the same signal until the underlying input changes materially. +7. Log MLflow metrics for the daily run. + +## Acceptance criteria + +- The module never fires an order autonomously. The broker module requires a confirmed signal. +- A signal is deduplicated against the confirmations history so the same trigger does not email the user every day. +- Every signal row contains the trigger ids, the input values, the thresholds in effect, and the `as_of_date`. +- The audit parquet contains a row for every holding evaluated, signal or no signal. +- Thresholds live in YAML and travel with the run; backtests can sweep them. +- Emails sent through AWS SES include a dashboard deep link to the signal. +- The pipeline is idempotent for a given run date: re-running produces the same signal set without duplicate emails (idempotent by `message_id`). +- The MLflow run logs at minimum: number of evaluations, number of signals, number per trigger. + +## Open questions + +- For the "no longer in top decile" rule, do we use the decile of the same universe used to enter the position, or today's universe? Recommendation: today's universe; matches the spirit of opportunity cost. +- For `SW_QUALITY_DROP_YOY`, how do we handle restatements that change ROC retroactively? Recommendation: compare today's PIT ROC against the ROC value used at entry (snapshot at purchase), not against today's "1 year ago" PIT slice. +- The opportunity-cost trigger requires the watchlist to be sorted by the same combined rank used to enter. Should the watchlist be recomputed daily, or only at rebalance? Recommendation: daily, cheap. +- Should we dampen the email frequency with a rate limit (e.g., max 5 signals per day)? Recommendation: yes, with an MLflow metric reporting the suppression count. +- For the dismissed-signal memory: how long do we wait before re-firing a dismissed signal? Recommendation: until either the trigger value changes by more than 10% from the dismissal value, or 90 days have passed, whichever comes first. + +## Risks + +- Daily signals can desensitize the user. The dedup + dismissal memory exists to prevent this. +- The opportunity-cost trigger is the most rank-sensitive: small ranking noise can produce churn. The 5-position margin is the dampener; the backtest must validate that it is not too aggressive. +- Restated fundamentals can produce false sells if we rely on today's PIT slice for "ROC 1 year ago". The snapshot-at-entry approach above mitigates this. +- AWS SES may rate-limit or land in spam if the sender domain is not verified. The infrastructure spec must include verifying the SES sender identity. +- The user can dismiss legitimate signals out of bias. The FP/FN review of dismissed signals is a follow-up improvement. diff --git a/spec/features/011-universe-construction/spec.md b/spec/features/011-universe-construction/spec.md new file mode 100644 index 0000000..c40eff3 --- /dev/null +++ b/spec/features/011-universe-construction/spec.md @@ -0,0 +1,166 @@ +# Feature: Universe Construction + +## Implementation status + +**done** (demo slice) — universe builder ([#58](https://github.com/JLaborda/SmartWealthAI/issues/58)); industry exclusions reference CSV ([#56](https://github.com/JLaborda/SmartWealthAI/issues/56)). Full S&P 500 historical mode in phase 2. + +## Objective + +Produce the investable universe of US common stocks for each decision date. This module is the single entry point for "which tickers does the strategy consider today?" and is the upstream dependency of every downstream module. It must be point-in-time correct and survivorship-bias-free. + +## MVP scope + +### Demo slice (June 30) + +- Seed universe: all SimFin US companies (`load_companies(market='us')`). +- Exclude banks, insurers, and utilities via `data/reference/simfin_industry_exclusions.csv` (`IndustryId` list built from `load_industries()`). +- **Regeneration rules** (applied by `build_exclusions` in `src/smartwealthai/simfin_industry_exclusions.py`): + - `bank`: SimFin industry name exactly `Banks` + - `insurer`: industry name contains `Insurance` + - `utility`: SimFin sector exactly `Utilities` +- Regenerate after SimFin industry label changes: `poetry run generate-simfin-industry-exclusions --industries ` +- Sanity check: exclude tickers present in SimFin `income_banks` or `income_insurance` bulk datasets even if `IndustryId` is missing from the CSV. +- No S&P 500 historical file required for demo. +- No market-cap or ADV floors in demo (optional parameters disabled). +- Produce daily snapshot under `curated/universe/run_date=/`. + +### Full MVP (phase 2) + +- S&P 500 historical constituents (incl. delisted) from `data/reference/sp500_constituents.csv`. +- Common-stock filters (exclude ADRs, REITs, BDCs, ETFs, preferred-only). +- SIC-based sector exclusions when SEC ETL is available. +- Share-class deduplication by ADV. +- Optional market-cap and volume floors. + +## Out of MVP scope + +- Non-US universes. +- Index families other than S&P 500 (Russell 3000, MSCI USA, etc.) as the seed. +- Liquidity rules beyond a static daily-volume threshold. +- Sector exposure limits (out of MVP scope per architecture decision). +- Automatic re-classification of issuers as they change SIC code. + +## Inputs + +| Input | Source | Notes | +| --- | --- | --- | +| US company list | SimFin `companies` (demo) | `Ticker`, `CIK`, `IndustryId`. | +| Industry metadata | SimFin `industries` + `simfin_industry_exclusions.csv` | Sector/industry names for audit. | +| Bank/insurance sanity | SimFin `income_banks` / `income_insurance` ticker index | Secondary exclusion signal. | +| Historical S&P 500 constituents | `data/reference/sp500_constituents.csv` | Phase 2 only. | +| SIC codes | SEC EDGAR submissions | Phase 2 only. | +| Daily prices and volume | `curated/prices` | For ADV dedup and floors (phase 2). | +| Market cap | `curated/fundamentals` join `curated/prices` | Tie-break and optional floors. | +| Run date | Pipeline parameter | PIT universe slice. | + +## Outputs + +| Dataset | Path | Schema | +| --- | --- | --- | +| Daily universe | `s3://smartwealthai-data-lake/curated/universe/run_date=/universe.parquet` | `run_date, ticker, cik, industry_id, sector, market_cap_usd, exclusion_reasons` (demo schema; `sic_code` added in phase 2) | +| Exclusion log | `s3://smartwealthai-data-lake/curated/universe/run_date=/exclusions.parquet` | One row per excluded ticker with the triggered rule(s). | +| DuckDB view | `v_universe` | Latest universe view, partitioned on `run_date`. | + +## Mermaid diagram (demo) + +```mermaid +flowchart TD + Companies["SimFin companies (market=us)"] --> Seed["Seed universe at run_date"] + ExclCSV["simfin_industry_exclusions.csv"] --> SectorFilter{"IndustryId excluded?"} + BankSanity["Bank / insurance statement indices"] --> SanityFilter{"Bank or insurer ticker?"} + Seed --> SectorFilter + SectorFilter -->|Yes| Excluded["exclusions.parquet"] + SectorFilter -->|No| SanityFilter + SanityFilter -->|Yes| Excluded + SanityFilter -->|No| Universe["universe.parquet"] + Universe --> DuckDBView["v_universe"] + Excluded --> ExclusionLog["exclusions.parquet"] +``` + +## Mermaid diagram (full MVP — phase 2) + +```mermaid +flowchart TD + SP500["data/reference/sp500_constituents.csv"] --> Seed["Build seed universe at run_date"] + Curated["curated/fundamentals + curated/prices"] --> Enrich["Enrich with SIC, market cap, ADV"] + Seed --> Enrich + + Enrich --> CommonOnly{"Common stock?"} + CommonOnly -->|No| Excluded["Excluded: non-common-stock"] + CommonOnly -->|Yes| SectorFilter{"SIC in banks / insurers / utilities?"} + + SectorFilter -->|Yes| Excluded2["Excluded: sector"] + SectorFilter -->|No| Dedup["Deduplicate share classes by ADV"] + Dedup --> Floors{"Market cap and volume floors"} + Floors -->|Below| Excluded3["Excluded: too small / illiquid"] + Floors -->|Above| Universe["Daily universe (curated/universe)"] + + Excluded --> ExclusionLog["exclusions.parquet"] + Excluded2 --> ExclusionLog + Excluded3 --> ExclusionLog + + Universe --> DuckDBView["v_universe"] +``` + +## Expected flow (demo) + +1. Load SimFin `companies` for `market=us` from curated or raw snapshot. +2. Join `IndustryId` to `simfin_industry_exclusions.csv`; excluded rows → `exclusions.parquet` with reason `sector`. +3. Drop tickers found in bank/insurance SimFin statement indices (sanity check). +4. Persist `universe.parquet` for `run_date`. + +## Expected flow (full MVP — phase 2) + +1. Read `data/reference/sp500_constituents.csv` and compute historical membership through `run_date`. +2. Map each ticker to its CIK and `sic_code` via the curated EDGAR submissions table. +3. Filter to common stocks. The MVP keeps only issuers whose SEC form types include `10-K` and `10-Q` filed on a standard schedule, and excludes: + - ETFs and ETN issuers (form `N-CSR`, `N-Q`, fund-specific filings). + - REITs (`SIC 6798`). + - BDCs (`SIC 6770` and explicit BDC registrants). + - Preferred-only listings. + - Foreign private issuers filing `20-F` instead of `10-K` (ADRs). +4. Exclude sectors by SIC code range: + - Banks: `6020-6199`. + - Insurers: `6311-6411`. + - Utilities: `4900-4999`. + The full SIC-to-bucket mapping table is materialized in the spec for review. +5. For each issuer with multiple share classes, compute the trailing-90-day average daily volume per class and keep the class with the highest figure. All other classes go to `exclusions.parquet` with reason `share_class_lower_liquidity`. +6. Apply optional floors: + - `market_cap_usd >= market_cap_floor` (parameter, default off in the MVP). + - `avg_daily_volume_usd_90d >= adv_floor` (parameter, default `1_000_000` USD). +7. Persist the universe and exclusion log for `run_date`, alongside the parameter values used. + +## Acceptance criteria + +### Demo + +- [x] `data/reference/simfin_industry_exclusions.csv` versioned with banks, insurers, utilities (`industry_id`, `industry_name`, `sector`, `exclusion_reason`). +- [x] Same `run_date` → byte-identical `universe.parquet`. +- [x] No excluded `IndustryId` appears in the universe. +- [x] No bank/insurance sanity-check ticker appears in the universe. +- [x] Module consumes only curated/raw SimFin snapshots (no network). + +**Code:** `src/smartwealthai/universe_builder.py`, CLI `poetry run build-universe`. + +### Full MVP (phase 2) + +- Bankrupt companies that were once in the index appear in past universe snapshots up to their delisting date and are excluded only after that date with reason `delisted`. +- No company whose SIC code is in the excluded sector ranges appears in any universe snapshot. +- Share class deduplication is reversible from the exclusion log. +- The schema of `universe.parquet` is versioned and documented. + +## Open questions + +- Source of the historical constituents file. Proposed: `github.com/fja05680/sp500` snapshot pinned in `data/reference/sp500_constituents.csv`. Need user confirmation. +- How do we handle additions / removals on the same day a ticker is also evaluated for inclusion? Recommendation: include the ticker if it was in the index at the close of the prior trading day. +- Do we want a manual override list (`data/reference/universe_overrides.csv`) so the user can pin or blacklist tickers for testing? Recommendation: yes, but only honored when an explicit flag is set on the run. +- For dual-class issuers, do we collapse activity from both classes for the personal portfolio module, or do we keep them separate? Recommendation: keep separate in `portfolio-evolution`, deduplicate only in the investment universe. +- Do we want to record, in the universe snapshot, the SIC code reclassifications that happen mid-history? Recommendation: yes, store both the current and the as-of-date SIC code. + +## Risks + +- The community S&P 500 constituents dataset can have errors (missing additions, wrong dates). Mitigation: pin a snapshot and add a smoke test that asserts a known set of historical events (e.g., Lehman removal 2008, Tesla addition 2020). +- SIC codes are not a perfect sector classifier. Some banks file under non-bank SIC codes and vice versa. Mitigation: keep an explicit override list per CIK and review it during exclusions analysis. +- Survivorship bias still creeps in if the constituents file is built from "currently listed" companies. Mitigation: verify a sample of known-bankrupt companies (Lehman, Enron, WorldCom) are present in the historical file. +- Share class deduplication based on liquidity can flip the kept class across days for low-liquidity issuers. Mitigation: smooth the volume metric over 90 days and require a margin before flipping. +- Excluding banks, insurers, and utilities removes a sizable chunk of the index. Documented as an MVP trade-off. diff --git a/spec/features/012-unstructured-financial-data/spec.md b/spec/features/012-unstructured-financial-data/spec.md new file mode 100644 index 0000000..92533f9 --- /dev/null +++ b/spec/features/012-unstructured-financial-data/spec.md @@ -0,0 +1,103 @@ +# Feature: Unstructured Financial Data + +> **Status for the MVP: minimal.** The MVP does not run LLM analyses, summaries, or embeddings on filings. It only stores raw filing references and exposes one targeted text-flag pipeline that the permanent loss filter can consume: a **"going concern" detector** on the latest 10-K. Everything else (transcripts, news, sentiment, RAG) is parked until after the MVP is validated. + +## Objective + +Provide a thin text-processing layer that complements the structured pipeline. For the MVP, the only consumer is the permanent loss filter, which benefits from a high-precision "going concern" flag pulled from the latest 10-K. + +## MVP scope + +- Persist the raw 10-K text (or filing reference) in the data lake under `raw/sec_edgar/...` (already produced by `etl-data-lake`). +- Run a rule-based scanner that looks for "going concern" language patterns in the latest 10-K per company. +- Emit a boolean flag plus the matched passage(s) and the filing url. +- Expose results in a curated parquet (`curated/text_flags/going_concern.parquet`) consumed by `permanent-loss-filter`. +- Run weekly (filings do not change daily); incremental. +- No LLM, no embeddings, no summarization in the MVP. + +## Out of MVP scope + +- Summarization or LLM-driven narratives. +- Sentiment analysis. +- Year-over-year risk-factor diffs. +- Earnings call transcripts. +- News scraping. +- Multilingual filings. +- RAG / vector search infrastructure. +- Auditor-change extraction (handled by a different rule once the data is available). + +## Inputs + +| Input | Source | +| --- | --- | +| Latest 10-K filing per CIK | `raw/sec_edgar/.../form=10-K/...` | +| List of patterns to match | `config/text_flags/going_concern_patterns.yaml` (versioned) | + +## Outputs + +| Output | Path | +| --- | --- | +| Going concern flag | `curated/text_flags/going_concern.parquet` with `cik, accession, as_of_date, flag_bool, matched_phrases, source_url, pattern_version` | +| MLflow metrics | Count of CIKs scanned, count of flags raised | + +## Patterns (initial set) + +Stored in YAML, versioned. Match is case-insensitive, regex-based, restricted to the "Notes to Consolidated Financial Statements" and "Management's Discussion" sections when section markers can be found; otherwise applied to the full text. + +```yaml +patterns: + - "substantial doubt about (its|the company.s) ability to continue as a going concern" + - "substantial doubt regarding the company.s ability to continue as a going concern" + - "raise substantial doubt about (our|the company.s) ability to continue as a going concern" +``` + +A single match is enough to flag. + +## Mermaid diagram + +```mermaid +flowchart TD + Raw["raw/sec_edgar/.../form=10-K"] --> Reader["Filing reader (text extract)"] + Patterns["config/text_flags/going_concern_patterns.yaml"] --> Scanner["Regex scanner"] + Reader --> Scanner + Scanner --> Flag{"Match found?"} + Flag -->|Yes| Out["going_concern.parquet (flag = True)"] + Flag -->|No| OutNo["going_concern.parquet (flag = False)"] + Out --> PLF["permanent-loss-filter (BK_GOING_CONCERN)"] + OutNo --> PLF +``` + +## Expected flow + +1. Locate the latest 10-K per CIK whose `acceptance-datetime <= run_date`. +2. Extract plain text from the filing (HTML to text, no OCR; 10-Ks are HTML on EDGAR). +3. Run the regex scanner. +4. Persist a row per CIK with the flag, the matched phrase(s), the filing URL, the accession, and the pattern version. +5. The permanent loss filter joins this table on its `BK_GOING_CONCERN` rule. + +## Acceptance criteria + +- The going concern flag is reproducible for the same accession and the same pattern version (deterministic). +- A CIK without a recent 10-K is not silently flagged as `False`; it is marked `unknown` and routed to the review queue. +- The matched passage is stored alongside the flag for human review. +- Adding a new pattern requires a new `pattern_version`; old runs do not re-flag retroactively unless the user triggers a backfill. + +## Open questions + +- Do we want to also scan 10-Qs, or 10-Ks only? Recommendation: 10-Ks only for the MVP; going concern is mostly disclosed in the annual report. +- Section extraction: do we attempt to limit the search to specific 10-K items (Item 7, Item 8 notes), or scan the full filing? Recommendation: full filing for the MVP; precision is high enough. +- Should the flag have a TTL (e.g., expires 13 months after the filing date)? Recommendation: yes, default 400 days. + +## Risks + +- Some 10-Ks contain "going concern" language in a hypothetical or risk-factor context. The MVP rule will produce some false positives. Logged for FP/FN review. +- HTML parsing of EDGAR documents can fail on edge cases. The pipeline must capture and log parse errors instead of crashing. +- A pattern list in YAML is easy to break if patterns conflict. The `pattern_version` discipline is the only mitigation. + +## Future iterations (parked) + +- Auditor-change extraction (`FRD_AUDITOR_CHANGE_REPEATED` in permanent-loss-filter). +- Risk-factor year-over-year diff. +- Earnings call transcript ingestion + topic flags. +- LLM-driven summary with citation enforcement. +- Embeddings + RAG search inside the dashboard. diff --git a/spec/guides/download-fundamentals.md b/spec/guides/download-fundamentals.md new file mode 100644 index 0000000..76f44ae --- /dev/null +++ b/spec/guides/download-fundamentals.md @@ -0,0 +1,178 @@ +# Guide: Download fundamentals (local spike — frozen) + +> **Status:** This guide documents the **frozen SEC ETL spike** ([ADR-0001](../../adr/0001-simfin-fundamentals-mvp.md)). The June 30 demo pipeline uses **SimFin** instead — see [`roadmap.md`](../roadmap.md) and [`etl-data-lake.md`](../../006-etl-data-lake/spec.md). Do not delete this spike; it resumes in phase 2. + +Operator guide for the first ETL vertical slice: download raw SEC `companyfacts` and +standardized annual statements from `edgartools` for a parameterized universe. + +**Canonical spec:** [`../../006-etl-data-lake/spec.md`](../../006-etl-data-lake/spec.md) (section +*Fundamentals download spike*). + +## Prerequisites + +1. Python 3.11+ and Poetry installed. +2. Project dependencies installed: + + ```bash + poetry install + ``` + +3. **SEC identity** (required by SEC EDGAR and `edgartools`): + + ```bash + export SEC_IDENTITY="Your Name your@email.com" + ``` + + Use a real contact address. SEC may block requests with generic or invalid identities. + +4. Network access to `data.sec.gov` and SEC endpoints used by `edgartools`. + +## Quick start (Dow 30) + +```bash +export SEC_IDENTITY="Your Name your@email.com" +poetry run download-fundamentals --universe dow30 +``` + +Equivalent module invocation: + +```bash +poetry run python -m smartwealthai.download_fundamentals --universe dow30 +``` + +A full Dow 30 run downloads **120 artifacts** (4 per CIK: 1 JSON + 3 parquet files) and +typically takes several minutes because of SEC rate limits and `edgartools` parsing. + +## CLI reference + +Built with [Click](https://click.palletsprojects.com/). Run `download-fundamentals --help` for +auto-generated option docs. + +| Flag | Default | Description | +| --- | --- | --- | +| `--universe` | — | Preset name. Currently: `dow30`. | +| `--universe-file` | — | Path to a custom CSV (`ticker,cik`). Overrides preset when both are set. | +| `--data-dir` | `data` | Local data lake root. | +| `--periods` | `16` | Annual fiscal columns requested from `edgartools`. | +| `--as-of-date` | UTC today | Partition date (`YYYY-MM-DD`) for immutable daily snapshots. | +| `--force` | off | Re-download even when today's partition already exists. | + +### Examples + +```bash +# Custom universe CSV +poetry run download-fundamentals --universe-file data/reference/universes/dow30.csv + +# Pin snapshot date (reproducible backfill slice) +poetry run download-fundamentals --universe dow30 --as-of-date 2026-06-07 + +# Force refresh after a failed partial run +poetry run download-fundamentals --universe dow30 --force + +# Write to a temp lake (CI / experiments) +poetry run download-fundamentals --universe dow30 --data-dir /tmp/swai-lake +``` + +## Universe files + +Presets map to versioned CSV files under `data/reference/universes/`. See +[`../../../data/reference/universes/README.md`](../../../data/reference/universes/README.md). + +Format: + +```csv +ticker,cik +AAPL,0000320193 +MSFT,0000789019 +``` + +- `ticker` — trading symbol passed to `edgartools.Company(ticker)`. +- `cik` — 10-digit zero-padded SEC CIK used for `companyfacts` URLs. + +Fixed CIKs avoid ambiguity across share classes and ticker renames. + +## Output layout + +Under `{data-dir}/raw/`: + +```text +sec_edgar/cik=/endpoint=companyfacts/as_of_date=/response.json +edgartools/cik=/as_of_date=/income_statement_annual.parquet +edgartools/cik=/as_of_date=/balance_sheet_annual.parquet +edgartools/cik=/as_of_date=/cashflow_statement_annual.parquet +download_runs/as_of_date=/errors.json # only when failures occur +``` + +### Artifact summary + +| Artifact | Source | Contents | +| --- | --- | --- | +| `response.json` | SEC REST `/api/xbrl/companyfacts/CIK*.json` | Verbatim XBRL facts (raw zone). | +| `*_annual.parquet` | `edgartools` `get_facts()` | Parsed annual statements with `concept`, `label`, `section`, `FY 20xx` columns. | + +Raw downloads are **never** consumed directly by scoring modules in the MVP; a future +normalizer will produce curated parquet with point-in-time semantics. + +## Cache and re-runs + +```mermaid +flowchart TD + Start["Run download_fundamentals"] --> Load["Load universe CSV"] + Load --> Loop["For each ticker/CIK"] + Loop --> Check{"Today's file exists?"} + Check -->|yes, no --force| Skip["Skip network call"] + Check -->|no or --force| Fetch["Download from SEC / edgartools"] + Fetch --> Write["Write under raw/.../as_of_date=today/"] + Skip --> Next["Next CIK"] + Write --> Next + Next --> Loop + Loop --> Summary["Print summary; exit 1 if any failures"] +``` + +- Re-running on the **same day** without `--force` skips existing files. +- `--force` overwrites today's partition only. +- Prior dates remain immutable (append-only by `as_of_date`). + +## Error handling + +- One failing CIK does **not** abort the run. +- Transient SEC errors (429, 5xx, timeouts) retry up to 3 times with exponential backoff. +- Permanent errors (404, invalid CIK) fail immediately for that issuer. +- Failures are written to `raw/download_runs/as_of_date=/errors.json`. +- Exit code: `0` if all issuers succeed, `1` if any fail. + +## Module map + +| Module | Responsibility | +| --- | --- | +| `smartwealthai.download_fundamentals` | CLI orchestration and run summary. | +| `smartwealthai.universe` | Preset resolution and CSV loading. | +| `smartwealthai.lake_paths` | Path builders for the local raw zone. | +| `smartwealthai.sec_client` | SEC REST client (throttle, retry, `companyfacts`). | +| `smartwealthai.edgartools_client` | `get_facts()` statement extraction to parquet. | + +## Tests + +Hermetic unit tests (no network): + +```bash +poetry run pytest tests/test_download_fundamentals.py -q +``` + +Integration smoke test (requires `SEC_IDENTITY` and network) is intentionally **not** part +of PR CI. Run locally on a small CSV when validating credentials. + +## Expanding universes + +1. Add `data/reference/universes/.csv` with `ticker,cik` rows. +2. Register the preset in `UNIVERSE_PRESETS` inside `src/smartwealthai/universe.py`. +3. Run: `poetry run download-fundamentals --universe `. + +Planned expansions: S&P 500, Russell 3000, Nasdaq — same CSV + preset pattern. + +## Out of scope (this spike) + +- `submissions` ingest (SIC, filing index). +- Curated parquet / point-in-time normalizer. +- S3 upload and DuckDB views. +- Quarterly statements (`period="quarterly"`). diff --git a/spec/guides/download-simfin.md b/spec/guides/download-simfin.md new file mode 100644 index 0000000..d3c5362 --- /dev/null +++ b/spec/guides/download-simfin.md @@ -0,0 +1,80 @@ +# Download SimFin bulk fundamentals and prices (demo) + +Operator guide for the **SimFin bulk connector** on the June 30 demo path. Canonical spec: [`etl-data-lake.md`](../../006-etl-data-lake/spec.md). + +## Prerequisites + +- Poetry environment installed (`poetry install`) +- `SIMFIN_API_KEY` in the environment (free tier from [simfin.com](https://simfin.com); never commit) + +```bash +export SIMFIN_API_KEY="" +``` + +## Download US bulk datasets + +Downloads six demo datasets into the raw lake under `data/raw/simfin/`: + +| Dataset | Variant | Lake partition | +| --- | --- | --- | +| `companies` | `default` | `dataset=companies/variant=default/market=us/` | +| `industries` | `default` | `dataset=industries/variant=default/market=us/` | +| `income` | `ttm` | `dataset=income/variant=ttm/market=us/` | +| `balance` | `quarterly` | `dataset=balance/variant=quarterly/market=us/` | +| `cashflow` | `ttm` | `dataset=cashflow/variant=ttm/market=us/` | +| `shareprices` | `latest` | `dataset=shareprices/variant=latest/market=us/` | + +Each partition also includes `as_of_date=/` and the verbatim SimFin CSV filename (e.g. `us-income-ttm.csv`). + +```bash +poetry run download-simfin +poetry run download-simfin --data-dir data --refresh-days 7 +poetry run download-simfin --as-of-date 2026-06-18 --force +``` + +This command writes raw snapshots only. Run the full demo pipeline in order: + +```bash +poetry run download-simfin --as-of-date 2026-06-18 +poetry run build-universe --run-date 2026-06-18 +poetry run normalize-simfin --snapshot-date 2026-06-18 --universe-run-date 2026-06-18 +poetry run download-prices --run-date 2026-06-18 --snapshot-date 2026-06-18 +poetry run compute-metrics --ticker AAPL --as-of-date 2026-06-18 +``` + +Or run ingest → score in one command (then launch the dashboard): + +```bash +poetry run run-demo-pipeline --run-date 2026-06-18 +poetry run run-demo-pipeline --run-date 2026-06-18 --skip-download +poetry run run-dashboard --data-dir data --run-date 2026-06-18 +``` + +Pass `--ticker` to limit the normalize step to specific names (intersect universe). Use `compute-metrics` for single-ticker ROC/EY smoke tests without a full scoring run. + +`normalize-simfin` requires `--universe-run-date` (after `build-universe`) or `--ticker` for smoke tests. It does not process the full SimFin US table by default. + +## Refresh and cache behaviour + +- **Skip:** Re-run without `--force` when the on-disk lake copy is younger than `--refresh-days` (default `7`). +- **Force:** `--force` re-downloads from SimFin and overwrites today's partition regardless of age. +- **SimFin package cache:** Intermediate downloads land in `data/cache/simfin/` before being copied into `raw/simfin/`. + +## Failures + +- A failure for one dataset does not stop the rest. +- Non-critical dataset (`cashflow`) failure still exits `0` when critical datasets succeed. +- Exit code `1` when all critical datasets (`companies`, `industries`, `income`, `balance`, `shareprices`) fail, or when `SIMFIN_API_KEY` is missing. +- Per-run errors are written to `raw/simfin/download_runs/as_of_date=/errors.json` when any dataset fails. + +## Module map + +| Module | Role | +| --- | --- | +| `smartwealthai.simfin_client` | Configure API key, safe bulk download (zip-slip guarded), resolve cache CSV path. | +| `smartwealthai.download_simfin` | CLI orchestration, skip/force logic, run summary. | +| `smartwealthai.lake_paths` | Raw lake path builders for `raw/simfin/`. | + +## Tests + +Hermetic tests live in `tests/test_download_simfin.py`. They mock SimFin network calls; PR CI does not require a live API key. diff --git a/spec/meta/feature-spec-template.md b/spec/meta/feature-spec-template.md new file mode 100644 index 0000000..beccb85 --- /dev/null +++ b/spec/meta/feature-spec-template.md @@ -0,0 +1,101 @@ +# Feature spec template + +Copy this file when creating a new feature. Do **not** edit this template in place. + +## Folder layout + +```text +spec/features/00N-slug/ + spec.md ← required (this template) + plan.md ← add when implementation starts (in_progress) + tasks.md ← add when implementation starts (checklist) +``` + +### Stable ID (`00N`) + +- Assign the **next unused three-digit prefix** when the spec file is first created (chronological ID, not priority). +- Check [`../constitution/roadmap.md`](../constitution/roadmap.md) for existing IDs. +- **Never renumber** an existing folder. Priority changes go only in `roadmap.md`. +- Use a lowercase **slug** (`kebab-case`) describing the module. + +### Workflow + +1. Copy sections below into `spec/features/00N-slug/spec.md`. +2. Align with [`../constitution/mission.md`](../constitution/mission.md) and related feature specs. +3. Add the feature to [`../constitution/roadmap.md`](../constitution/roadmap.md) (priority / phase). +4. Update [`../../CONTEXT.md`](../../CONTEXT.md) if new domain terms are introduced (`/grill-with-docs`). +5. Open **one GitHub issue** per feature when moving to implementation — see [`github-issues.md`](github-issues.md). + +--- + +# Feature: + +## Implementation status + + + + +## Objective + + + +## MVP scope + + + +## Out of MVP scope + + + +## Inputs + +| Input | Source | Notes | +| --- | --- | --- | +| | | | + +## Outputs + +| Output | Path / target | +| --- | --- | +| | | + +## Mermaid diagram + + + +```mermaid +flowchart TD + A["Input"] --> B["Step"] + B --> C["Output"] +``` + +## Expected flow + + + +1. … +2. … + +## Acceptance criteria + + + +- … + +## Open questions + + + +- … + +## Risks + + + +- … + +## Related specs + + + +- [`../NNN-other-feature/spec.md`](../NNN-other-feature/spec.md) — … diff --git a/spec/meta/github-issues.md b/spec/meta/github-issues.md new file mode 100644 index 0000000..29799a0 --- /dev/null +++ b/spec/meta/github-issues.md @@ -0,0 +1,50 @@ +# GitHub Issues workflow + +Git specs under `spec/` are canonical. **GitHub Issues** track execution. + +## One issue per feature + +When a feature moves to implementation: + +1. Open **one GitHub issue** for the feature (e.g. `Feature: Cheap stocks — EY rank`). +2. Add label `ready-for-agent` when the spec is complete and work can start. +3. Create `plan.md` and `tasks.md` under `spec/features/00N-slug/` when implementation begins. + +## Issue body template + +```markdown +## Spec +spec/features/003-cheap-stocks/spec.md + +## Plan +spec/features/003-cheap-stocks/plan.md (when exists) + +## Tasks +See spec/features/003-cheap-stocks/tasks.md +``` + +Copy acceptance criteria from `spec.md` into the issue description or link to the spec path. + +## Sync discipline + +1. Change MVP decisions in the Git spec first (`spec.md`, or `constitution/` for cross-cutting rules). +2. Create or update the GitHub issue. +3. On completion: update `spec.md` (implementation status, acceptance criteria), check off `tasks.md`, then close the issue. + +## `tasks.md` format + +Markdown checklist in the feature folder. Example: + +```markdown +# Tasks: Cheap stocks + +- [ ] Implement EY from curated fundamentals +- [ ] Cross-sectional EY rank +- [ ] Wire into combined rank +``` + +`tasks.md` is the versioned checklist; the GitHub issue is the tracking unit on the project board. + +## Issue tracker + +Repository: `JLaborda/SmartWealthAI` via `gh` CLI. See [`.cursor/rules/issue-tracker.md`](../../.cursor/rules/issue-tracker.md). diff --git a/spec/prds/ci-cd/ci-cd-prd.md b/spec/prds/ci-cd/ci-cd-prd.md new file mode 100644 index 0000000..210f82b --- /dev/null +++ b/spec/prds/ci-cd/ci-cd-prd.md @@ -0,0 +1,231 @@ +# PRD: CI/CD and MLOps Infrastructure (Phase 0) + +**Status:** Ready for implementation +**Canonical architecture:** `spec/constitution/mission.md` +**Related specs:** ETL + data lake, permanent loss filter, backtesting, sell-watch (pipeline vertical slice) + +--- + +## Problem Statement + +SmartWealthAI is a portfolio-grade quantitative value-investing MVP that must demonstrate MLOps competence on AWS: reproducible builds, automated quality gates, containerized pipeline execution, and a clear path from development to production. Today the repository has Poetry dependencies and a development container, but no Makefile, no production Dockerfile, no GitHub Actions workflows, and no defined contract for how local development, CI tests, AWS data lake access, and deployment relate to each other. + +The developer also needs a phased approach that does not over-build infrastructure before the core investment pipeline exists (data ingestion → permanent loss filter → quality and cheapness scoring → ranking → model portfolio → backtest → sell-watch). Without an explicit CI/CD plan, work on the data lake and AWS risks becoming confusing: it is unclear what runs locally, what runs in CI, what touches S3, and when full continuous deployment should begin. + +## Solution + +Establish a **Phase 0 CI/CD foundation** that separates three concerns: + +1. **Fast, deterministic PR CI** — lint and unit/smoke tests against pinned fixtures; no network, no AWS, no live SEC or price provider calls. +2. **Independent AWS integration tier** — a manual and scheduled workflow that proves ingestion can write to the dev S3 data lake using GitHub OIDC (no long-lived AWS keys). +3. **Deferred full CD (Phase 1–2)** — after the pipeline container and business logic exist, deploy to ECS Fargate on merge to `develop` (dev) and `main` (prod with approval), using multiple Docker images over time but shipping only the **pipeline image** first. + +The data lake uses the **same layout everywhere** (raw, curated, point-in-time zones) with a configurable lake root URI: local file mirror for optional offline work, S3 dev bucket as the canonical store for real ingestion, S3 prod bucket for promoted runs. DuckDB reads Parquet from either backend. + +GitFlow maps environments: pull requests run CI on all branches; merge to `develop` eventually deploys dev; merge to `main` eventually deploys prod behind a GitHub Environment approval gate. + +## User Stories + +1. As a developer, I want a single Makefile with standard targets for install, lint, test, and local ingestion, so that dev, CI, and documentation all reference the same commands. +2. As a developer, I want Poetry to manage Python 3.11 dependencies and an in-project virtualenv, so that the environment matches the devcontainer and CI runners. +3. As a developer, I want PR CI to run automatically on every pull request, so that broken changes are caught before merge. +4. As a developer, I want PR CI to complete quickly without external network calls, so that feedback is reliable and merges are not blocked by SEC or yfinance outages. +5. As a developer, I want unit and smoke tests to use pinned fixtures representing universe, fundamentals, and prices, so that scoring and filtering logic is testable without a live data lake. +6. As a developer, I want Ruff to enforce lint and format checks in CI, so that code quality is consistent from day one. +7. As a developer, I want a smoke test that exercises the core pipeline path on fixtures (permanent loss → ROC → EY → combined rank → portfolio selection), so that regressions in the vertical slice are caught early. +8. As a developer, I want real data ingestion to target AWS S3 in a dev bucket, so that the project demonstrates cloud-native data lake practice rather than local-only storage. +9. As a developer, I want ingestion tested independently from PR CI via a separate workflow, so that AWS integration is proven without making every PR flaky or slow. +10. As a developer, I want the ingest integration workflow to be triggerable manually and on a weekly schedule, so that I can validate connectors after changes without waiting for a release. +11. As a developer, I want separate dev and prod S3 buckets from the start, so that test artifacts never mix with production data and IAM can be scoped correctly. +12. As a developer, I want GitHub Actions to authenticate to AWS via OIDC role assumption, so that no long-lived AWS access keys are stored in GitHub Secrets. +13. As a developer, I want distinct IAM roles for dev and prod GitHub environments, so that production permissions are tighter and auditable. +14. As a developer, I want a pipeline Docker image that is separate from future dashboard and control-plane images, so that batch jobs stay minimal and deploy boundaries are clear. +15. As a developer, I want only the pipeline image built and deployed in Phase 0–1, so that infrastructure work stays proportional to existing application code. +16. As a developer, I want the pipeline container to run on ECS Fargate (Spot where viable), so that daily batch execution is cost-effective and aligned with the architecture doc. +17. As a developer, I want merge to `develop` to eventually deploy to the dev environment (ECR tag + ECS task revision), so that integrated changes are runnable in AWS before production. +18. As a developer, I want merge to `main` to eventually deploy to prod with a required GitHub Environment approval, so that production promotion is deliberate. +19. As a developer, I want runtime secrets (API keys, broker credentials) sourced from AWS Secrets Manager at task runtime, so that secrets never live in the repository or container image. +20. As a developer, I want build-time configuration limited to non-secret environment identifiers (bucket names, regions, cluster names), so that the security model matches architecture decisions. +21. As a developer, I want a configurable lake root URI so the same ingestion and query code works against local mirrors and S3, so that I understand the data lake as layout plus Parquet, not as “local vs cloud” code forks. +22. As a developer, I want an optional gitignored local lake mirror for speed or offline work, so that I am not blocked when AWS is unavailable, without making local storage the source of truth. +23. As a developer, I want the permanent loss filter regression cases (Enron, Lehman, WorldCom) to run in CI once that module exists, so that bankruptcy/fraud exclusions remain auditable. +24. As a developer, I want full 20-year walk-forward backtests to run outside PR CI (manual or scheduled), so that long-running validation does not block every commit. +25. As a developer, I want CI to respect point-in-time correctness in fixture design, so that tests reinforce the project’s core constraint against look-ahead bias. +26. As a portfolio reviewer, I want the README and docs to explain the three-tier testing model (fixtures / AWS ingest-smoke / deploy), so that the MLOps story is interview-ready. +27. As a developer, I want Prefect orchestration deferred until the ECS task path works, so that scheduling complexity does not block the first end-to-end AWS run. +28. As a developer, I want the Streamlit dashboard image and CD deferred until dashboard code exists, so that deploy pipelines are not empty scaffolding. +29. As a developer, I want MLflow experiment tracking integrated after the pipeline stabilizes, so that run snapshots do not slow initial delivery. +30. As a developer, I want infrastructure definitions (buckets, OIDC provider, IAM roles, ECR repository, ECS cluster skeleton) versioned alongside the application, so that AWS setup is reproducible. +31. As a developer, I want conventional commit and GitFlow branch conventions documented and followed, so that `develop` and `main` map cleanly to dev and prod deploy workflows. +32. As a developer, I want the devcontainer to remain development-only and not used as the CI runner image, so that dev ergonomics and production slim images stay separate concerns. +33. As a developer, I want ECR images tagged with git commit SHA and environment labels, so that any ECS run is traceable to source control. +34. As a developer, I want CloudWatch Logs as the initial observability sink for ECS tasks, so that pipeline failures are debuggable without heavier tooling. +35. As a developer, I want a clear milestone sequence from M0 (Makefile + CI) through M4 (Fargate runs pipeline in dev) before prod CD, so that implementation order is unambiguous. + +## Implementation Decisions + +### Scope and phasing + +- **Phase 0 (now):** Makefile, Poetry groups, PR CI workflow, fixture layout, pipeline package skeleton, ingest-smoke workflow to dev S3, OIDC + bucket provisioning. No full deploy on every merge yet. +- **Phase 1:** Pipeline Dockerfile, ECR push on merge to `develop`, ECS Fargate task definition update for dev. +- **Phase 2:** Prod deploy on merge to `main` with GitHub Environment approval; prod OIDC role and prod bucket writes restricted to promoted tasks. +- **Phase 3+:** Prefect orchestration, dashboard image, MLflow server, EventBridge daily schedule — explicitly later. + +### Deployable units (multi-image, pipeline first) + +- Target architecture uses **multiple Docker images** over the MVP lifetime: pipeline worker (batch), dashboard (Streamlit), and optionally separate control-plane services. +- **Phase 0–1 ships only the pipeline image.** Dashboard and Prefect worker images are out of scope until their application modules exist. +- The pipeline image entrypoint runs batch stages (ingest, normalize, score, rank, backtest slice, sell-watch) driven by CLI subcommands or a single orchestrated command — exact CLI shape to be defined during M1 implementation. + +### Data lake and environment configuration + +- Lake layout follows architecture zones: **raw** (immutable provider payloads), **curated** (normalized Parquet), **pit** (point-in-time store keyed by as-of date), plus **curated/issues** for the review queue. +- A single configuration value, **lake root URI**, selects the storage backend: + - Local optional mirror: file-backed root under a gitignored directory for developer convenience. + - Dev canonical store: S3 dev bucket prefix. + - Prod store: S3 prod bucket prefix. +- DuckDB is the analytical engine reading Parquet from the configured root; no Athena in MVP. +- **CI does not read or write live S3.** Integration workflows use dev bucket only. + +### CI workflow (every pull request) + +- Triggers on pull requests targeting `develop` or `main` (and optionally other long-lived branches if added). +- Steps: checkout → Python 3.11 + Poetry install → `make lint` → `make test` → `make test-smoke`. +- **Lint:** Ruff check and format check on application and test packages. +- **Unit tests:** pytest with markers excluding integration tests; all data from committed fixtures. +- **Smoke test:** end-to-end pipeline on a tiny fixture universe proving the vertical slice wiring (may initially stub modules until implemented). +- No AWS credentials configured on PR workflows. +- No live calls to SEC EDGAR, yfinance, or paid API tiers. + +### AWS integration workflow (ingest-smoke) + +- Separate workflow from PR CI; triggers: `workflow_dispatch` and weekly cron. +- Runs against **GitHub Environment `dev`** with OIDC assumption of the dev IAM role. +- Executes a minimal ingestion job (small ticker subset) writing to the dev bucket raw zone, then validates object presence and basic schema/count checks. +- Failures notify via workflow status; they do not block unrelated PR merges unless explicitly wired later. + +### Authentication and IAM + +- **GitHub OIDC → AWS IAM role assumption** is mandatory; static `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` in GitHub Secrets are out of scope. +- Two roles minimum: **dev GitHub Actions role** (dev bucket read/write, dev ECR push, dev ECS register task definition) and **prod GitHub Actions role** (prod bucket, prod ECR, prod ECS — tighter trust policy, main branch only). +- Runtime tasks use **task execution roles** distinct from GitHub deploy roles; secrets read from AWS Secrets Manager at container start. +- Trust policies restrict repository, environment, and branch refs. + +### Storage and registry + +- Two S3 buckets from day one: **dev** and **prod**, with consistent prefix conventions for lake zones, cache, and future MLflow artifacts. +- One ECR repository (or repository per image type later) for the pipeline image; tags include commit SHA and environment (`dev-`, `prod-`, optional `latest-dev` / `latest-prod`). + +### Runtime and orchestration + +- **ECS Fargate** is the first-class CD target for the pipeline container (Spot where appropriate for cost). +- AWS Batch is a future alternative if job queue semantics become necessary; not Phase 0. +- **Prefect** orchestration is deferred; initial runs may be one-off Fargate tasks triggered by deploy workflow or manual run-task until Prefect is introduced. + +### Branching and deployment mapping (GitFlow) + +- Feature branches → PR → CI only. +- Merge to **`develop`** → Phase 1+ deploy dev (build, push ECR, update ECS task). +- Merge to **`main`** → Phase 2 deploy prod with required reviewer on GitHub `production` environment. +- Hotfix flow may merge to `main` and back to `develop`; deploy workflows must respect branch protections. + +### Application modules touched or introduced + +Deep modules (simple interfaces, testable in isolation): + +| Module | Responsibility | Phase | +| --- | --- | --- | +| **Configuration** | Lake root URI, environment name, AWS region, non-secret resource names | M0 | +| **Lake I/O** | Read/write Parquet under raw/curated/pit prefixes; abstract file vs S3 | M1 | +| **Ingestion connectors** | SEC EDGAR and price providers → raw zone | M1–M2 | +| **Normalization + PIT** | Curated schema, as-of date tagging, incremental refresh | M2 | +| **Permanent loss filter** | Hard exclusions for fraud/bankruptcy | M1 | +| **Scoring** | ROC rank, EY rank, combined rank | M1 | +| **Portfolio selection** | Top combined rank → 15–30 name model portfolio | M1 | +| **Backtest runner** | Walk-forward on PIT data; pass/fail vs benchmarks | M2+ | +| **Sell-watch** | Daily signals on model holdings | M2+ | +| **Pipeline CLI** | Subcommands invoked locally, in CI smoke, and in container | M1 | + +Makefile targets wrap Poetry commands so CI and humans share entrypoints: install, lint, test, test-smoke, local ingest, docker build (pipeline), and later deploy helpers. + +### Poetry dependency strategy + +- Single Poetry project for the repository. +- **Dev group:** pytest, ruff; optional moto for S3 mock unit tests if needed. +- **Pipeline optional/extras group:** duckdb, boto3, pyarrow, sec-edgar-downloader, and other pipeline dependencies as modules land — avoid bloating dev-only installs. +- Enable package mode once the application package under `src/` exists. + +### Relationship to devcontainer + +- The existing devcontainer remains **development-only** (Docker-in-Docker, AWS CLI, Poetry, Jupyter, forwarded ports for future Streamlit/MLflow). +- CI uses GitHub-hosted runners with Makefile + Poetry; it does not build or run the devcontainer image. +- Production pipeline Dockerfile is slim and separate from the devcontainer Dockerfile. + +### Milestone sequence + +| Milestone | Deliverable | +| --- | --- | +| **M0** | Makefile, Poetry dev groups, PR CI workflow, empty package + passing smoke stub, fixture directory | +| **M1** | Core modules on fixtures; permanent loss + scoring + rank smoke green; package mode on | +| **M2** | Dev/prod buckets, OIDC roles, ingest-smoke workflow green against dev S3 | +| **M3** | Pipeline Dockerfile, ECR push on merge to `develop` | +| **M4** | ECS Fargate task runs pipeline in dev with `LAKE_URI` pointing at dev bucket | +| **M5** | Prod deploy on `main` with approval gate | +| **M6** | Prefect, dashboard image, MLflow, scheduled daily runs | + +## Testing Decisions + +### What makes a good test here + +- Test **observable behavior** at module boundaries: given fixture inputs and a run date, expect exclusions, ranks, portfolio membership, or sell signals — not internal function call order. +- Fixture data must respect **point-in-time** semantics: each fundamental row carries an as-of date; tests pass only when queries filter `as_of_date <= run_date`. +- CI tests must be **hermetic**: no network, no AWS, deterministic ordering where ranks are involved. +- Integration tests that hit S3 or live APIs live in a **separate workflow or marker** (`integration`) and are never required for every PR. + +### Modules to test in PR CI + +| Module | Test type | Notes | +| --- | --- | --- | +| Configuration | Unit | Default lake URI, env overrides | +| Permanent loss filter | Unit + regression | Enron, Lehman, WorldCom must be hard-excluded at correct as-of dates when module exists | +| ROC / EY scoring | Unit | Known EBIT, capital, EV → expected ranks on tiny cross-section | +| Combined rank + portfolio | Smoke | Fixture universe → expected top-N names | +| Pipeline CLI smoke | Smoke | Invokes wired stages sequentially on fixtures | +| Lake I/O | Unit | Optional moto or local temp dirs; not live S3 in PR CI | + +### Modules tested outside PR CI + +| Module | Test type | Notes | +| --- | --- | --- | +| Ingestion connectors | Integration (ingest-smoke workflow) | Tiny live pull → dev S3 raw zone | +| Full backtest | Scheduled / manual | 20+ year walk-forward too slow for PR | +| ECS deploy | Post-deploy smoke | Run task after dev deploy; assert exit code and logs | + +### Prior art + +- Devcontainer PRD validated tooling via manual smoke checks (Python, Poetry, AWS CLI, Ruff, pytest). +- Architecture doc specifies Enron / Lehman / WorldCom regression in CI for the permanent loss filter — adopt when that module is implemented. +- Legacy notebooks and `src/` exploration are not test prior art; new tests live under the package test tree with fixtures. + +## Out of Scope + +- **Prefect** orchestration and Prefect Cloud/server setup in Phase 0–1. +- **Streamlit dashboard** Docker image and dashboard CD. +- **MLflow tracking server** on EC2 and artifact promotion workflows. +- **Full continuous deployment on day one** (Phase 0 is CI + ingest-smoke only). +- **Kubernetes / EKS** and AWS Batch as primary runtime (Batch remains a later option). +- **Static AWS access keys** in GitHub Secrets. +- **Single fat Docker image** for dashboard + pipeline + MLflow. +- **CI ingestion or scoring against live SEC/yfinance on every PR.** +- **Prod bucket writes** before Phase 2 prod deploy exists. +- **Transaction costs, taxes, live broker execution** — product scope, not CI/CD scope. +- **Terraform vs CDK choice** — infrastructure-as-code tool may be chosen during M2; not blocking M0. +- **Ruff/pytest detailed rule configuration** beyond enabling tools in Phase 0 (may follow as chore). + +## Further Notes + +- This PRD captures decisions from the CI/CD design session (grill-me). It should be reflected in a future feature spec under `spec/features/` (e.g. `cicd-infrastructure.md`) and cross-linked from `spec/constitution/mission.md` when implementation starts — per spec-driven workflow, the feature spec becomes canonical for acceptance criteria and implementation status. +- The core product vertical slice remains: ingest → permanent loss filter → quality (ROC) → cheapness (EY) → combined rank → model portfolio (~30 names) → backtest → sell-watch. CI/CD serves that slice, not the reverse. +- Cost awareness: dev integration and ECS tasks should use minimal resource sizes and Spot where acceptable; align with the architecture soft budget (~low single-digit USD/month for control plane before storage growth). +- When implementation begins, update `spec/README.md` to index this PRD under an MVP PRDs section alongside the devcontainer PRD. +- GitHub issue creation with label `ready-for-agent` is recommended for tracking vertical implementation slices (`to-issues`), but this document is the saved PRD artifact at `spec/prds/ci-cd/ci-cd-prd.md` as requested. diff --git a/spec/prds/devcontainer/prd.md b/spec/prds/devcontainer/prd.md new file mode 100644 index 0000000..2490227 --- /dev/null +++ b/spec/prds/devcontainer/prd.md @@ -0,0 +1,131 @@ +# PRD: Development Container for SmartWealthAI + +## Problem Statement + +SmartWealthAI development currently depends on the developer's local machine configuration (macOS, Homebrew packages, Python version, Poetry, AWS CLI, etc.). This creates two problems: + +1. **Environment drift**: there is no guarantee that a fresh clone produces a working dev environment without manual setup steps. +2. **Cloud portability**: the architecture targets AWS (S3, ECS Fargate, ECR, Secrets Manager). The developer wants to be able to spin up an EC2 instance, clone the repo, open it in Cursor via SSH, and land in a fully functional dev environment identical to the local one -- with zero manual tool installation. + +## Solution + +Create a `.devcontainer/` configuration that packages the entire development toolchain (Python 3.11, Poetry, Docker, AWS CLI, GitHub CLI, system utilities, Cursor extensions) into a reproducible container. The developer opens the repo in Cursor (locally or via SSH to EC2), the container builds automatically, and all dependencies are ready. + +This is a **development-only** image. A separate, minimal production Dockerfile will be created later for ECS Fargate tasks. + +## User Stories + +1. As a developer, I want to open the repo in Cursor and have all Python dependencies installed automatically, so that I can start coding immediately without running setup scripts. +2. As a developer, I want the same dev environment on my Mac and on a remote EC2 instance, so that I never debug environment-specific issues. +3. As a developer, I want Docker available inside my dev container, so that I can build and test production Docker images locally before pushing to ECR. +4. As a developer, I want the AWS CLI pre-installed, so that I can interact with S3, ECR, Secrets Manager, and other AWS services during development. +5. As a developer, I want the GitHub CLI pre-installed, so that I can create PRs, manage issues, and check CI status from the terminal. +6. As a developer, I want `make`, `jq`, and `ripgrep` available, so that I have standard dev utilities for task automation, JSON inspection, and fast code search. +7. As a developer, I want Cursor to auto-detect the Poetry virtualenv as the Python interpreter, so that I never have to manually select the right Python. +8. As a developer, I want linting and formatting (Ruff) configured out of the box, so that code quality is enforced from day one. +9. As a developer, I want pytest available, so that I can run tests inside the container. +10. As a developer, I want Jupyter notebook support in Cursor, so that I can work with the existing `.ipynb` files in `notebooks/`. +11. As a developer, I want Streamlit (8501) and MLflow (5000) ports forwarded automatically, so that I can access dashboards from my browser when working remotely. +12. As a developer, I want AWS credentials handled via `~/.aws` mount (local) or IAM Instance Profile (EC2), so that no secrets are baked into the container image. +13. As a developer, I want to rebuild the container after changing its config and land in an updated environment, so that the setup evolves with the project. +14. As a developer, I want the container to use bash as the default shell, so that scripts behave consistently across dev and CI environments. + +## Implementation Decisions + +### Image strategy + +- **Dev-only container.** The devcontainer is not reused for CI/CD or production. A separate slim Dockerfile will be created later for ECS Fargate. +- **Base image:** `mcr.microsoft.com/devcontainers/python:3.11`. Provides a non-root `vscode` user, common utilities (git, curl, ssh, sudo), and native Cursor/VS Code remote compatibility. + +### Devcontainer features (pre-built add-ons) + +| Feature | Purpose | +|---|---| +| `ghcr.io/devcontainers/features/docker-in-docker` | Build and run Docker images inside the container | +| `ghcr.io/devcontainers/features/aws-cli` | Interact with AWS services (S3, ECR, Secrets Manager, etc.) | +| `ghcr.io/devcontainers/features/github-cli` | PR creation, issue management, CI status checks | + +### System packages (via Dockerfile) + +Installed on top of the base image via `apt-get`: + +- `make` -- task automation +- `jq` -- JSON processing (SEC EDGAR data, AWS CLI output) +- `ripgrep` -- fast codebase search + +### Python tooling + +- **Poetry** installed via `pipx` (already available in the MS base image). +- `poetry config virtualenvs.in-project true` so `.venv` lives inside the workspace. +- `poetry install` runs as `postCreateCommand` to auto-install all dependencies on container creation. +- **pytest** and **ruff** added as dev dependencies in `pyproject.toml`. + +### Cursor / VS Code customizations + +**Extensions:** + +| Extension ID | Purpose | +|---|---| +| `ms-python.python` | Python language support, IntelliSense, test discovery | +| `charliermarsh.ruff` | Linting + formatting | +| `ms-toolsai.jupyter` | Notebook support | + +**Settings:** + +| Setting | Value | Reason | +|---|---|---| +| `python.defaultInterpreterPath` | `${workspaceFolder}/.venv/bin/python` | Auto-select the Poetry venv | +| `python.terminal.activateEnvironment` | `true` | Auto-activate venv in terminals | + +### Port forwarding + +| Port | Service | +|---|---| +| 8501 | Streamlit dashboard | +| 5000 | MLflow tracking UI | + +### Credentials strategy + +- **Local (macOS):** mount `~/.aws` into the container (devcontainer mount config). +- **EC2:** IAM Instance Profile attached to the instance; AWS CLI picks it up via the metadata service automatically. +- **No secrets baked into the image.** Ever. + +### Shell + +- bash (Debian default). No zsh/oh-my-zsh customization. + +### Data directory + +- No special volume or mount config. `data/` is gitignored and stays empty on fresh clones. Data lives in S3 per the architecture; local `data/` is populated on demand by ETL bootstrap scripts. + +## Testing Decisions + +This is an infrastructure/tooling PRD, not a feature module. There is no application logic to unit-test. Validation is manual: + +- **Smoke test:** build the container locally (`Dev Containers: Rebuild Container` in Cursor), verify Python version, Poetry venv, installed tools (`docker --version`, `aws --version`, `gh --version`, `make --version`, `jq --version`, `rg --version`), and that `pytest` and `ruff` are importable. +- **EC2 test:** spin up an EC2 instance, install Docker, clone the repo, open via Cursor SSH, verify the same smoke checks pass. +- **Extension test:** confirm Cursor shows the correct Python interpreter and that Ruff linting is active on `.py` files. + +## Out of Scope + +- **Production Dockerfile.** That is a separate effort aligned with the ECS Fargate runtime decision in the architecture doc. +- **CI/CD integration.** GitHub Actions has its own runner environment; the devcontainer is not used there. +- **Data provisioning.** No EBS volumes, S3 sync scripts, or seed data in the container. +- **GPU support.** Not needed for the MVP (no ML training workloads). +- **Custom shell (zsh/oh-my-zsh).** Can be added later if desired. +- **Prefect / MLflow server setup.** Those are runtime services, not dev environment concerns. +- **Ruff / pytest configuration** (rules, pyproject sections). Adding the packages is in scope; configuring them is a follow-up. + +## Further Notes + +- The devcontainer config is fully version-controlled under `.devcontainer/` and evolves with the project. Any team member (or the developer on a new machine) gets the same environment by opening the repo. +- The architecture doc (`spec/constitution/mission.md`) references GitHub Actions + ECR for Docker image builds. The devcontainer's Docker-in-Docker feature allows local testing of those images before pushing. +- This PRD does not create a feature spec under `spec/features/` because the devcontainer is developer tooling, not an MVP feature module. It is tracked as a standalone PRD. + +## Files to Create or Modify + +| File | Action | +|---|---| +| `.devcontainer/devcontainer.json` | Create -- main devcontainer configuration | +| `.devcontainer/Dockerfile` | Create -- system packages on top of MS base image | +| `pyproject.toml` | Modify -- add `pytest` and `ruff` as dev dependencies | diff --git a/spec/prds/phase2/prd.md b/spec/prds/phase2/prd.md new file mode 100644 index 0000000..c7232e9 --- /dev/null +++ b/spec/prds/phase2/prd.md @@ -0,0 +1,322 @@ +# PRD: MVP Phase 2 — Quantitative Value, Cloud, Backtest, Sell-Watch + +**Status:** Ready for implementation +**Canonical architecture:** `spec/constitution/mission.md` +**Prior delivery:** June 30 demo slice (`spec/constitution/roadmap.md`, ADR-0002) +**Related specs:** ETL + data lake, permanent loss filter, backtesting, sell-watch, universe construction, dashboard reporting +**Related PRDs:** CI/CD (`spec/prds/ci-cd/ci-cd-prd.md`) +**Capacity assumption:** Solo developer, ~10–15 hours per week +**Estimated calendar:** Phase 2a ~13–16 weeks; Phase 2b ~8–12 weeks (~5–7 months total) + +--- + +## Problem Statement + +The June 30 demo slice delivers a working Greenblatt-style Magic Formula pipeline (ROC + Earnings Yield → combined rank → top-30 equal-weight model portfolio) on local SimFin data with a Streamlit dashboard and MLflow file-store logging. That slice proves ingestion, point-in-time fundamentals, cross-sectional ranking, and explainability — but it is not the investor's target strategy, not deployed to AWS, not historically validated, and does not monitor holdings for thesis breaks. + +The investor wants Phase 2 to: + +1. **Replace production scoring** with the *Quantitative Value* methodology (Wesley R. Gray / Tobias Carlisle): forensic screens, value funnel (EBIT/TEV), quality funnel (FS-Score), and a concentrated model portfolio (~50 names). +2. **Run the pipeline in AWS** so daily scoring is independent of a developer laptop. +3. **Backtest the strategy** to judge whether it is worth following — starting with a light historical run, then expanding to the full architecture spec. +4. **Alert on sell conditions** when a holding's QV thesis deteriorates — without auto-execution or paper trading in this phase. + +Without a single PRD tying these goals together, Phase 2 risks repeating the demo's scope creep in reverse: cloud work before the QV funnel exists, or backtests that still score ROC+EY while production claims to be Quantitative Value. + +## Solution + +Deliver Phase 2 in two increments: + +### Phase 2a (core) + +1. Extend the data lake for **multi-period fundamentals** and **daily prices** (5–10 year window) with point-in-time correctness preserved. +2. Implement **forensic / permanent-loss screening** including Beneish M-Score and the distress rules already specified, with a QVAL-style bottom-percentile gate on forensic models. +3. Replace the production scoring path with the **full QV funnel**: universe → forensic hard exclusion → top ~10% by EBIT/TEV → FS-Score on the value pool → top ~50 equal-weight model portfolio. +4. Keep **Magic Formula (ROC + EY + combined rank)** as a **benchmark module only** for backtest comparison — not production scoring. +5. Run a **light backtest** (5–10 years, annual rebalance, current SimFin US universe, S&P 500 CW + Magic Formula benchmarks) using a custom pandas/DuckDB engine (no Zipline). +6. Deploy **cloud phase 2a**: S3 data lake + artifacts, ECS Fargate Spot daily cron, Secrets Manager, MLflow tracking with S3 artifact store (extends CI/CD PRD milestones M1–M4). +7. Implement **sell-watch with QV-adapted triggers**; surface signals in the dashboard and curated parquet (email deferred to 2b). + +### Phase 2b (validation + operations) + +1. Historical **S&P 500 constituents including delisted** names (survivorship-bias mitigation). +2. **Full backtest** per `backtesting.md`: 20+ years, walk-forward 3–5 year windows, block-bootstrap Monte Carlo, crisis drawdown report, Sharpe gate vs four benchmarks. +3. **Cloud phase 2b**: Prefect or EventBridge orchestration, AWS SES email alerts, Streamlit dashboard hosted on AWS. + +Magic Formula remains the strategy's **benchmark comparator** for Sharpe pass/fail in 2b; production portfolio construction follows the QV funnel throughout. + +## User Stories + +### Strategy and scoring + +1. As an investor, I want the production pipeline to implement the Quantitative Value funnel (forensics → value → quality → portfolio), so that my model portfolio reflects the book's methodology rather than a Greenblatt placeholder. +2. As an investor, I want forensic accounting screens to hard-exclude companies at elevated fraud or bankruptcy risk before any value or quality score, so that permanent capital loss is filtered systematically. +3. As an investor, I want the Beneish M-Score included in forensic screening, so that earnings manipulation risk is part of the safety layer. +4. As an investor, I want companies in the bottom 5% of forensic models excluded (QVAL-style), so that the safety screen matches the published ETF process. +5. As an investor, I want the value screen to keep the top decile (~10%) of names by EBIT/TEV among survivors, so that I only quality-rank genuinely cheap stocks. +6. As an investor, I want quality ranked by the 10-point FS-Score (Gray/Carlisle variant) on the value pool, so that the final portfolio favors financially strong cheap names. +7. As an investor, I want the model portfolio to hold approximately 50 equal-weight long-only names after the quality screen, so that the portfolio matches QVAL concentration. +8. As an investor, I want every funnel stage to log how many names passed or failed, so that I can audit shrinkage from universe to portfolio. +9. As an investor, I want each score and exclusion to record formula version and inputs, so that any decision is reconstructible from the data lake. +10. As an investor, I want the dashboard to explain why a name is in the portfolio using QV stage outputs (forensic pass, EBIT/TEV rank, FS-Score components), so that the system stays explainable. +11. As a developer, I want Magic Formula ROC and EY scoring preserved as a separate benchmark path, so that backtests can compare QV against the Greenblatt replica without dual production logic. +12. As a developer, I want production and benchmark code paths named distinctly (QV vs MF), so that glossary terms in CONTEXT.md do not drift in implementation. + +### Data and point-in-time + +13. As a developer, I want annual and quarterly income, balance sheet, and cash flow stored in raw and curated zones, so that FS-Score year-over-year deltas are computable. +14. As a developer, I want the PIT fundamentals interface to return the correct historical filing rows for any decision date, so that backtests never leak future fundamentals. +15. As a developer, I want daily adjusted prices for at least a 5–10 year window in curated storage, so that light backtests and enterprise value history are supported. +16. As a developer, I want SimFin `shareprices/daily` as the primary price history source with a vendor fallback when needed, so that backtests are not blocked by free-tier snapshot lag. +17. As a developer, I want missing inputs for forensic or FS-Score rules routed to the review queue rather than silently dropped, so that data quality issues are visible. +18. As a developer, I want restatements to create new `version_id` rows with updated `as_of_date`, so that historical queries reflect what was knowable at each decision date. +19. As an investor, I want sector hard exclusions (banks, insurers, utilities) to remain upstream of QV scoring, so that incomparable financials never enter the funnel. + +### Permanent loss and forensics + +20. As an investor, I want Altman Z-score, interest coverage, net debt/EBITDA, negative equity, and delisting rules to remain available as distress signals, so that the permanent loss filter matches the existing spec where data allows. +21. As a developer, I want a CI regression test that forces Enron, Lehman, and WorldCom to be excluded at documented distress dates, so that bankruptcy screening cannot regress silently. +22. As a developer, I want each exclusion to store `rule_id`, `rule_version`, triggered values, and `as_of_date`, so that MLflow and the dashboard can show why a company was removed. +23. As a developer, I want fraud rules (restatement, auditor change, late filer) implemented where EDGAR data exists, with `unavailable` logged otherwise, so that the module is extensible without blocking on SEC ETL. + +### Backtesting + +24. As an investor, I want a light backtest over 5–10 years with annual rebalancing, so that I can see whether the QV strategy had acceptable risk-adjusted returns before investing further effort. +25. As an investor, I want the light backtest to recompute the full QV funnel at each rebalance date using only point-in-time data, so that results are not inflated by look-ahead bias. +26. As an investor, I want light backtest results compared to S&P 500 cap-weighted and Magic Formula replica benchmarks, so that I have familiar reference points. +27. As an investor, I want the light backtest universe limitation (current SimFin US, survivorship bias) clearly labeled in reports, so that I do not over-interpret early results. +28. As a developer, I want backtest runs logged as MLflow experiments with equity curve and trade ledger artifacts, so that each historical run is reproducible. +29. As a developer, I want long backtests to run outside PR CI (manual trigger or ECS ad-hoc task), so that commits are not blocked by 20-year simulations. +30. As an investor, I want Phase 2b to add a 20+ year walk-forward backtest with Monte Carlo and crisis drawdown reporting, so that the strategy meets the architecture validation bar. +31. As an investor, I want Phase 2b Sharpe compared against S&P 500 CW, S&P 500 EW, Russell 3000, and Magic Formula replica, so that pass/fail is objective when paper trading arrives later. +32. As a developer, I want delisted and bankrupt holdings handled with zero terminal price on delisting date in full backtests, so that NAV reflects realized losses. + +### Cloud and MLOps + +33. As a developer, I want the pipeline to run daily on ECS Fargate Spot without my laptop, so that the system is a real operational batch job. +34. As a developer, I want the data lake canonical store on S3 with a configurable lake root URI, so that the same code runs locally and in AWS. +35. As a developer, I want runtime secrets (e.g. `SIMFIN_API_KEY`) from AWS Secrets Manager, so that keys are not in the image or repository. +36. As a developer, I want MLflow run artifacts stored in S3, so that pipeline and backtest snapshots survive beyond a single machine. +37. As a developer, I want GitHub OIDC to deploy the pipeline image to ECR and update ECS task definitions on merge to `develop`, so that cloud deploys trace to git SHA. +38. As a developer, I want CloudWatch Logs for ECS task output, so that pipeline failures are debuggable. +39. As a portfolio reviewer, I want the README to document that Phase 2a cloud scope is pipeline-only (dashboard local), so that the MLOps story is honest about what runs where. +40. As a developer, I want Phase 2b to add SES email on sell-watch signals and host Streamlit on AWS, so that alerts and reporting work when I am not watching the dashboard. + +### Sell-watch + +41. As an investor, I want daily evaluation of model portfolio holdings for QV thesis breaks, so that I know when to consider exiting a position. +42. As an investor, I want a sell signal when forensic screening starts failing on a holding, so that fraud or distress triggers an alert. +43. As an investor, I want a sell signal when FS-Score drops materially (YoY or below threshold), so that quality deterioration is caught. +44. As an investor, I want a sell signal when a holding falls out of the EBIT/TEV value decile, so that overvaluation relative to the strategy is flagged. +45. As an investor, I want a sell signal when a watchlist name outranks a holding by a configurable margin on the QV composite rank, so that opportunity cost is monitored. +46. As an investor, I want sell signals to require manual confirmation before any future order build, so that the system never auto-sells. +47. As a developer, I want every holding evaluation logged (signal or no signal) for audit, so that false positive and false negative rates can be reviewed later. +48. As an investor, I want Phase 2b email alerts via AWS SES for new sell signals, so that I am notified without opening the dashboard. + +### Documentation and governance + +49. As a developer, I want a new feature spec `quantitative-value.md` as the canonical QV module before implementation, so that spec-driven workflow is preserved. +50. As a developer, I want CONTEXT.md updated when QV terms (quality, cheap, funnel rank) are resolved, so that agents and humans share one vocabulary. +51. As a developer, I want `roadmap.md` "After the demo" ordering updated to reflect QV-first production, so that docs do not contradict this PRD. + +## Implementation Decisions + +### Phasing + +| Increment | Scope | Exit criterion | +| --- | --- | --- | +| **2a** | Multi-period data, forensics + Beneish, QV funnel, light backtest, cloud pipeline (S3 + ECS + MLflow S3), sell-watch logic + dashboard | Daily QV portfolio on ECS; light backtest equity curve in MLflow; sell signals in dashboard | +| **2b** | S&P 500 historical universe, full backtest spec, Prefect/EventBridge, SES, Streamlit on AWS | 20y walk-forward backtest with Sharpe gate; email alerts; dashboard on AWS | + +Paper trading and broker execution are explicitly **out of Phase 2** (deferred until a passing full backtest exists in a later phase). + +### Deep modules (build or extend) + +These are intentionally **deep modules**: narrow public interfaces, substantial internal logic, stable contracts, testable in isolation. + +#### 1. Point-in-time fundamentals store (extend) + +- **Responsibility:** Given `decision_date` and `ticker` (or universe), return the latest fundamental rows per statement type with `as_of_date <= decision_date`; support multiple historical periods for YoY deltas. +- **Interface shape:** Query functions returning normalized provider-agnostic columns (`ebit`, `total_assets`, `cash`, etc.) plus metadata (`as_of_date`, `version_id`, `formula_version`). +- **Consumers:** Forensic evaluator, FS-Score calculator, EV/EBIT/TEV metrics, backtest engine. +- **Change frequency:** Low — extended for multi-period, not replaced. + +#### 2. Forensic evaluator (new) + +- **Responsibility:** Evaluate all fraud and distress rules per company at a run date; compute Beneish M-Score; compute cross-sectional percentiles; apply QVAL bottom-5% exclusion per model; emit hard `exclude` or `pass` with reasons. +- **Interface shape:** Input: universe pass list + PIT fundamentals (+ optional filing flags). Output: exclusions table keyed by `(cik, run_date)` with `rule_id`, `triggered_value`, `threshold`, `explanation`. +- **Consumers:** QV funnel stage 1, sell-watch `SW_FORENSIC`, CI regression fixtures. +- **Change frequency:** Medium — thresholds tuned, new rules versioned. + +#### 3. FS-Score calculator (new) + +- **Responsibility:** Compute the 10 binary FS-Score components (Gray/Carlisle variant: profitability, stability, recent operational improvements) from multi-period PIT inputs; sum to integer score 0–10. +- **Interface shape:** Input: PIT income/balance/cashflow history for one ticker at `decision_date`. Output: component dict, total score, `formula_version`. +- **Consumers:** QV funnel stage 3, sell-watch `SW_FS_SCORE_DROP`, dashboard explainability. +- **Change frequency:** Low — tied to published FS-Score definition. + +#### 4. QV funnel orchestrator (new) + +- **Responsibility:** Run sequential funnel stages with auditable counts; no scoring logic inside — delegates to forensic evaluator, value ranker, FS-Score ranker, portfolio constructor. +- **Stages:** + 1. Universe pass (from universe construction) + 2. Forensic hard exclusion + 3. Value screen: rank by EBIT/TEV descending; keep top decile (configurable count or fraction) + 4. Quality screen: FS-Score on value pool; keep top 50 (configurable) + 5. Portfolio: equal-weight selected names; optional market-cap tie-break on rank ties +- **Interface shape:** Input: `run_date`, lake root, config. Output: `ScoringResult`-like object with stage counts, ranked tables per stage, final `model_portfolio` rows, MLflow-ready metrics. +- **Consumers:** `score-universe` CLI, dashboard, backtest engine, sell-watch. +- **Change frequency:** Low — stage order fixed by QV methodology. + +#### 5. Value metrics (extend from cheapness module) + +- **Responsibility:** Compute EBIT, enterprise value, EBIT/TEV (same EV definition as Earnings Yield module); cross-sectional rank within forensic survivors. +- **Interface shape:** Reuse existing metrics building blocks where possible; separate production path from MF `EY rank`. +- **Consumers:** QV funnel stage 2, sell-watch `SW_VALUE_POOL_EXIT`. + +#### 6. Magic Formula benchmark path (preserve, demote) + +- **Responsibility:** ROC + EY + combined rank + top-N portfolio exactly as demo slice; used only for benchmark portfolio reconstruction in backtests. +- **Interface shape:** Existing ranking module interface unchanged; invoked only from backtest benchmark builder and smoke tests until smoke test updated to QV path. +- **Consumers:** Backtest benchmark comparator, CI smoke test (transition: add QV smoke, keep MF benchmark test). + +#### 7. Light backtest engine (new) + +- **Responsibility:** Loop rebalance dates; pin PIT data per date; invoke full QV funnel; simulate equal-weight holdings and daily NAV from curated prices; compare to benchmark series. +- **Interface shape:** Input: `start_date`, `end_date`, `rebalance_frequency`, config hash. Output: equity curves, trade ledger, holdings parquet, summary metrics → MLflow `backtesting` experiment. +- **Constraints:** No live network in run; read only curated parquet; custom loop (ADR-0002: no Zipline). +- **Consumers:** Manual/ECS ad-hoc runs, dashboard backtest panel (Phase 2a minimal). + +#### 8. Sell-watch evaluator (new) + +- **Responsibility:** For each model portfolio holding at `run_date`, evaluate QV triggers; dedupe against confirmed/dismissed history; write signals and full evaluation audit. +- **Trigger IDs:** `SW_FORENSIC`, `SW_FS_SCORE_DROP`, `SW_VALUE_POOL_EXIT`, `SW_QV_OPPORTUNITY` (replace ROC/EY triggers from sell-watch spec for production). +- **Interface shape:** Input: holdings, watchlist, latest scores, config thresholds. Output: signals parquet + evaluations parquet. +- **Consumers:** Dashboard, SES (2b), future broker module. + +#### 9. Lake root and cloud runtime (extend) + +- **Responsibility:** Abstract storage backend (local file vs S3 prefix); same zone layout (raw, curated, issues); DuckDB reads parquet from configured root. +- **Interface shape:** Single `LAKE_ROOT_URI` (or equivalent) consumed by all ingest and scoring CLIs. +- **Consumers:** All pipeline stages, ECS task entrypoint, ingest-smoke workflow. +- **Aligns with:** CI/CD PRD milestones M1–M4 for 2a; M4 uses EventBridge cron → ECS Fargate Spot → pipeline entrypoint. + +### Production vs benchmark separation + +| Concern | Production (Phase 2+) | Benchmark only | +| --- | --- | --- | +| Safety | Forensic evaluator + permanent loss rules | — | +| Value | EBIT/TEV value decile | EY rank (MF) | +| Quality | FS-Score on value pool | ROC rank (MF) | +| Ranking | QV funnel sequential rank | Combined rank = ROC + EY | +| Portfolio size | ~50 EW default | MF replica uses same universe/filters as configured for comparison | + +### Configuration and versioning + +- QV funnel parameters (decile fraction, portfolio size, FS-Score tie-break) live in versioned YAML under a `config/quantitative_value/` namespace. +- Forensic thresholds and Beneish coefficients versioned under `config/permanent_loss/`. +- Sell-watch thresholds versioned under `config/sell_watch/` with QV trigger IDs. +- Every MLflow run logs `git_sha`, config hashes, and stage counts. + +### Dashboard changes + +- Replace ROC/EY-centric explainability with QV stage breakdown per ticker. +- Add light backtest summary panel (equity curve, key metrics, survivorship bias warning). +- Add sell-watch signal list with trigger detail and confirm/dismiss actions (confirm does not build orders in Phase 2). + +### Glossary updates (CONTEXT.md) + +When implementation starts, resolve: + +- **Quality (production):** FS-Score composite, not ROC alone. +- **Cheap (production):** Membership in EBIT/TEV value pool, not EY rank alone. +- **QV funnel rank:** Order after quality screen within the value pool; supersedes **combined rank** for production. +- **Combined rank:** Retained for **Magic Formula replica** benchmark only. + +### Dependency order (2a) + +1. Feature spec `quantitative-value.md` +2. Multi-period ETL + PIT extension + daily prices +3. Forensic evaluator (+ Beneish) +4. FS-Score calculator +5. QV funnel orchestrator wired into scoring CLI +6. Light backtest engine +7. Cloud 2a (pipeline stable locally first) +8. Sell-watch evaluator + dashboard + +## Testing Decisions + +### Principles + +- Test **external behavior** (inputs → outputs, exclusions, ranks, portfolio membership) not internal implementation details. +- All PR CI tests use **pinned fixtures** — no SimFin, yfinance, or AWS calls in the default test job. +- Fixture design must respect **point-in-time correctness** (no row with `as_of_date > decision_date` in historical scenarios). +- Long-running backtests and full 20-year walk-forward run **outside PR CI** (manual or scheduled ECS). + +### Modules to test (priority) + +| Module | Priority | What to assert | +| --- | --- | --- | +| Forensic evaluator | **P0** | Enron, Lehman, WorldCom excluded at fixture dates; Beneish bottom-5% gate; exclusion reason columns populated | +| FS-Score calculator | **P0** | Known fixture company gets expected 0–10 score; each binary component matches hand-checked inputs | +| QV funnel orchestrator | **P0** | Fixture universe shrinks monotonically through stages; final portfolio size ≤ configured cap; excluded names never appear | +| Value metrics (EBIT/TEV) | **P1** | EV formula matches versioned config; negative EBIT routed to review queue | +| PIT fundamentals store | **P1** | Historical `decision_date` returns correct row; restatement version selection | +| Light backtest engine | **P1** | No look-ahead: fundamentals after rebalance date absent; turnover ledger balances | +| Sell-watch evaluator | **P1** | Each trigger fires on constructed holding; no signal when thresholds not met; dedupe of confirmed signals | +| Magic Formula benchmark | **P1** | Regression: demo slice ROC/EY/combined rank unchanged on fixtures (benchmark path not broken) | +| Lake root abstraction | **P2** | Local vs `s3://` prefix resolves same relative paths (mock or minio optional) | + +### Prior art in codebase + +- `fixture_lake.py` and `point_in_time_fundamentals()` for PIT fixture queries. +- `magic_formula_ranking.py` tests (if present) for rank assignment and portfolio selection patterns. +- CI/CD PRD user story 7: smoke test on fixture vertical slice — **update smoke to QV funnel** once forensic + funnel exist; keep MF benchmark unit tests separate. +- Permanent loss spec: Enron / Lehman / WorldCom regression cases under `tests/fixtures/permanent_loss/`. + +### CI vs integration + +| Tier | Runs | Scope | +| --- | --- | --- | +| PR CI | Every pull request | Lint, unit tests, QV smoke on fixtures, forensic regression | +| Ingest-smoke | Weekly / manual | S3 write from SimFin (existing CI/CD PRD workflow) | +| Light backtest | Manual / ECS ad-hoc | Real curated lake, 5–10 years | +| Full backtest (2b) | Manual / ECS / Batch | 20+ years, walk-forward, Monte Carlo | + +## Out of Scope + +- **Paper trading and broker execution** — no simulated or real orders in Phase 2. +- **Auto-execution of sell signals** — confirm/dismiss only; no order builder. +- **Corroborative signals** (buybacks, insider, short interest) — deferred. +- **Unstructured financial data** (LLM filings, going-concern NLP) — deferred; `BK_GOING_CONCERN` remains future. +- **SEC EDGAR normalizer as primary fundamentals source** — optional 2b+; SimFin remains primary for Phase 2. +- **Moat "pre-flight checklist"** and full forensic model zoo beyond Beneish + spec distress rules — optional 2b+. +- **Score-weighted and risk-parity portfolio weighting** — equal-weight only in 2a; weighting as backtest hyperparameter in 2b only. +- **Personal portfolio CSV evolution and NAV** — `portfolio-evolution.md` deferred. +- **Prefect server, Streamlit on AWS, SES** — Phase 2b only (not 2a). +- **Production deploy to prod ECS on `main`** — may follow 2a dev deploy; prod CD per CI/CD PRD Phase 2 when ready. +- **Zipline or third-party backtest frameworks** — rejected (ADR-0002). + +## Further Notes + +### Time and risk + +- **2a:** ~13–16 weeks at 10–15 h/week if SimFin data coverage is sufficient. +- **2b:** ~8–12 additional weeks. +- **Risks:** SimFin free-tier limits on multi-period and daily prices; survivorship bias in 2a light backtest (must be labeled); Beneish missing inputs shrinking the funnel; backtest compute cost (full funnel × rebalance dates) — plan DuckDB pushdown or materialized stage tables early. + +### Open question (not blocking PRD) + +- **Portfolio size:** QVAL uses ~50 names; demo uses 30. Default recommendation: **50** with configurable cap in QV config. Resolve in `quantitative-value.md` spec before coding. + +### Relationship to other documents + +- This PRD does **not** replace per-module feature specs; it coordinates them. Each module keeps acceptance criteria in `spec/features/`. +- Implementers should read ADR-0001 (SimFin fundamentals), ADR-0002 (demo scope cut / no Zipline), and the CI/CD PRD before cloud work. +- After Phase 2a ships, update `roadmap.md` "After the demo" ordering and `mission.md` functional flow to state QV as production scoring. + +### Suggested GitHub issue title + +`PRD: MVP Phase 2 — Quantitative Value, Cloud, Backtest, Sell-Watch` + +Link this PRD path in the issue body; label `ready-for-agent` when creating tracker entry. From b9c28ed3ac50a607b69da6b15722373d6a4066a0 Mon Sep 17 00:00:00 2001 From: JLaborda <15078416+JLaborda@users.noreply.github.com> Date: Wed, 1 Jul 2026 07:52:14 +0000 Subject: [PATCH 2/5] chore(agents): align rules and skills with spec/ layout Replace Notion task tracking with GitHub Issues workflow and update all agent path references from docs/mvp to spec/. Co-authored-by: Cursor --- .cursor/rules/domain.md | 22 ++++++------ .cursor/rules/github-issues.mdc | 24 +++++++++++++ .cursor/rules/issue-tracker.md | 10 +++--- .cursor/rules/mvp-docs.mdc | 8 ++--- .cursor/rules/notion-tasks.mdc | 35 ------------------- .cursor/rules/project-context.mdc | 16 ++++----- .cursor/rules/spec-driven-workflow.mdc | 6 ++-- .cursor/skills/commit-split/SKILL.md | 2 +- .cursor/skills/grill-with-docs/ADR-FORMAT.md | 6 ++-- .cursor/skills/grill-with-docs/SKILL.md | 6 ++-- .../improve-codebase-architecture/SKILL.md | 2 +- .../skills/setup-matt-pocock-skills/SKILL.md | 6 ++-- .../skills/setup-matt-pocock-skills/domain.md | 10 +++--- 13 files changed, 70 insertions(+), 83 deletions(-) create mode 100644 .cursor/rules/github-issues.mdc delete mode 100644 .cursor/rules/notion-tasks.mdc diff --git a/.cursor/rules/domain.md b/.cursor/rules/domain.md index 8025bba..35e3a19 100644 --- a/.cursor/rules/domain.md +++ b/.cursor/rules/domain.md @@ -14,11 +14,11 @@ How the engineering skills should consume this repo's domain documentation when ## Before exploring, read these - **`CONTEXT.md`** at the repo root (ubiquitous language; created or extended by `/grill-with-docs` when terms are resolved). -- **`docs/adr/`** — architectural decision records for cross-cutting choices. -- **`docs/mvp/architecture/architecture.md`** — MVP vision, closed decisions, and module map (canonical during spec-driven phase). -- **Relevant `docs/mvp/features/.md`** — feature scope and acceptance criteria for the area you are changing. +- **`spec/adr/`** — architectural decision records for cross-cutting choices. +- **`spec/constitution/mission.md`** — MVP vision, closed decisions, and module map (canonical during spec-driven phase). +- **Relevant `spec/features/00N-slug/spec.md`** — feature scope and acceptance criteria for the area you are changing. -If `CONTEXT.md` or `docs/adr/` do not exist yet, **proceed silently**. Do not flag their absence or suggest creating them upfront. Use `docs/mvp/` as the source of truth until `/grill-with-docs` materializes terms into `CONTEXT.md`. +If `CONTEXT.md` or `spec/adr/` do not exist yet, **proceed silently**. Do not flag their absence or suggest creating them upfront. Use `spec/` as the source of truth until `/grill-with-docs` materializes terms into `CONTEXT.md`. Do not treat `src/` or `notebooks/` as canonical architecture; they are legacy exploration per `AGENTS.md`. @@ -27,24 +27,22 @@ Do not treat `src/` or `notebooks/` as canonical architecture; they are legacy e ``` / ├── CONTEXT.md ← ubiquitous language (extend via grill-with-docs) -├── docs/ -│ ├── adr/ ← system-wide ADRs -│ ├── agents/ ← agent skill config (this folder) -│ └── mvp/ -│ ├── architecture/ -│ └── features/ +├── spec/ +│ ├── constitution/ ← mission, tech-stack, roadmap +│ ├── features/ ← 00N-slug/spec.md (+ plan.md, tasks.md) +│ └── adr/ └── src/ ← implementation (not planning truth) ``` ## Use CONTEXT vocabulary -When your output names a domain concept (in an issue title, a refactor proposal, a hypothesis, a test name), prefer terms from `CONTEXT.md` when defined there; otherwise use terms consistently with `docs/mvp/` (e.g. point-in-time, universe, EY rank, Magic Formula replica). +When your output names a domain concept (in an issue title, a refactor proposal, a hypothesis, a test name), prefer terms from `CONTEXT.md` when defined there; otherwise use terms consistently with `spec/` (e.g. point-in-time, universe, EY rank, Magic Formula replica). If the concept you need isn't in `CONTEXT.md` yet, either reconsider invented language or note the gap for `/grill-with-docs`. ## Flag ADR conflicts -If your output contradicts an existing ADR or a **closed decision** in `docs/mvp/architecture/architecture.md`, surface it explicitly rather than silently overriding: +If your output contradicts an existing ADR or a **closed decision** in `spec/constitution/mission.md`, surface it explicitly rather than silently overriding: > _Contradicts [decision or ADR] — but worth reopening because…_ diff --git a/.cursor/rules/github-issues.mdc b/.cursor/rules/github-issues.mdc new file mode 100644 index 0000000..219bf50 --- /dev/null +++ b/.cursor/rules/github-issues.mdc @@ -0,0 +1,24 @@ +--- +description: GitHub Issues vs Git specs +alwaysApply: true +--- + +# GitHub Issues workflow + +**Git specs in `spec/` are canonical.** GitHub Issues track execution. + +## Issue fields + +Each implementation issue should include: + +- **Title:** e.g. `Feature: Cheap Stocks — implement EY rank` +- **Spec path:** `spec/features/00N-slug/spec.md` +- **Body:** link to `plan.md` and `tasks.md` when they exist; copy acceptance criteria from the spec + +## Sync discipline + +1. Change MVP decisions in the Git spec first. +2. Create or update the GitHub issue (label `ready-for-agent` when ready). +3. On completion: update `spec.md` (implementation status, acceptance criteria), check off `tasks.md`, close the issue. + +See [`spec/meta/github-issues.md`](../../spec/meta/github-issues.md). diff --git a/.cursor/rules/issue-tracker.md b/.cursor/rules/issue-tracker.md index 8464a6a..6d24bbd 100644 --- a/.cursor/rules/issue-tracker.md +++ b/.cursor/rules/issue-tracker.md @@ -18,15 +18,15 @@ Issues and PRDs for this repo live as GitHub issues. Use the `gh` CLI for all op Infer the repo from `git remote -v` — `gh` does this automatically when run inside a clone (`JLaborda/SmartWealthAI`). -## Relationship to Git specs and Notion +## Relationship to Git specs | Layer | Where | Role | | --- | --- | --- | -| **MVP specs (canonical)** | `docs/mvp/features/*.md`, `docs/mvp/architecture/architecture.md` | Product and engineering truth; update before closing work | -| **GitHub issues** | This repo's GitHub Issues | PRDs, vertical slices, and triage for Matt Pocock engineering skills (`to-issues`, `to-prd`, `triage`) | -| **Notion tasks** | Board `Cursor Agent Tasks` | Optional execution tracking via Notion MCP; see `AGENTS.md` and `docs/mvp/NOTION_SETUP.md` | +| **MVP specs (canonical)** | `spec/features/00N-slug/spec.md`, `spec/constitution/mission.md` | Product and engineering truth; update before closing work | +| **GitHub issues** | This repo's GitHub Issues | Feature implementation tracking, PRDs, vertical slices, triage | +| **`tasks.md`** | `spec/features/00N-slug/tasks.md` | Versioned checklist when a feature is in progress | -When a skill publishes to the issue tracker, create or update a **GitHub issue**. Link the relevant `docs/mvp/features/...` path in the issue body. Do not treat Notion as the issue tracker for those skills unless the user explicitly asks to sync there. +When a skill publishes to the issue tracker, create or update a **GitHub issue**. Link the relevant `spec/features/.../spec.md` path in the issue body. See [`spec/meta/github-issues.md`](../../spec/meta/github-issues.md). ## When a skill says "publish to the issue tracker" diff --git a/.cursor/rules/mvp-docs.mdc b/.cursor/rules/mvp-docs.mdc index 5c9ff20..76390e0 100644 --- a/.cursor/rules/mvp-docs.mdc +++ b/.cursor/rules/mvp-docs.mdc @@ -1,16 +1,16 @@ --- description: MVP Markdown spec structure and editing conventions -globs: docs/** +globs: spec/** alwaysApply: false --- # MVP documentation conventions -When creating or editing specs under `docs/mvp/`: +When creating or editing specs under `spec/`: ## Feature specs -Follow the structure used in existing features (e.g. `docs/mvp/features/cheap-stocks.md`): +Follow the structure in [`spec/meta/feature-spec-template.md`](../../spec/meta/feature-spec-template.md) and existing features (e.g. `spec/features/003-cheap-stocks/spec.md`): - Objective - MVP scope / Out of MVP scope @@ -23,7 +23,7 @@ Add **Implementation status** when coding starts: `planned` | `in_progress` | `d ## Architecture spec -- Record closed decisions in tables (see "Decisions made so far" in `docs/mvp/architecture/architecture.md`). +- Record closed decisions in tables (see "Decisions made so far" in `spec/constitution/mission.md`). - Cross-link feature files from the module table. - Do not contradict a closed architecture decision without updating the architecture doc first. diff --git a/.cursor/rules/notion-tasks.mdc b/.cursor/rules/notion-tasks.mdc deleted file mode 100644 index 759d3c4..0000000 --- a/.cursor/rules/notion-tasks.mdc +++ /dev/null @@ -1,35 +0,0 @@ ---- -description: Notion task board vs Git specs -alwaysApply: true ---- - -# Notion task tracking - -**Git specs in `docs/mvp/` are canonical.** Notion is for tasks and status only. - -## Task fields - -Each Notion task should include: - -- **Title:** e.g. `Feature: Cheap Stocks — implement EY rank` -- **Spec path:** `docs/mvp/features/.md` -- **Notes:** copy acceptance criteria from the spec - -## Sync discipline - -1. Change MVP decisions in the Git spec first. -2. Then create or update Notion tasks. -3. On completion: update the Git spec (implementation status, acceptance criteria), then mark the Notion task done. - -## Default board - -**`Cursor Agent Tasks`** in this project's Notion workspace. Use Notion MCP `notion-search` with that name; do not require a board URL from the repo. See `AGENTS.md`. - -## Cursor skills (Notion MCP authenticated) - -- `spec-to-implementation` — break a spec into tasks -- `create-task` — add a single task -- `tasks-build` — implement from a Notion task URL -- `tasks-explain-diff` — document completed work in Notion - -If Notion MCP calls fail with auth errors, ask the user to authenticate the Notion plugin in Cursor Settings. diff --git a/.cursor/rules/project-context.mdc b/.cursor/rules/project-context.mdc index 7345673..6b9f80c 100644 --- a/.cursor/rules/project-context.mdc +++ b/.cursor/rules/project-context.mdc @@ -9,17 +9,17 @@ SmartWealthAI is a modular **quantitative value investing** system (MVP): US equ ## Current phase -**MVP planning and spec refinement** in `docs/mvp/`. **June 30, 2026 demo slice** ([`demo-slice.md`](../../docs/mvp/demo-slice.md)) is the active delivery target. Prefer updating specs over writing code unless the user explicitly moves to implementation. +**MVP planning and spec refinement** in `spec/`. **June 30, 2026 demo slice** ([`roadmap.md`](../../spec/constitution/roadmap.md)) is the active delivery target. Prefer updating specs over writing code unless the user explicitly moves to implementation. ## Canonical documentation -- Demo slice: `docs/mvp/demo-slice.md` -- ADRs: `docs/adr/` -- Architecture: `docs/mvp/architecture/architecture.md` -- Features: `docs/mvp/features/*.md` +- Constitution: `spec/constitution/` (`mission.md`, `tech-stack.md`, `roadmap.md`) +- ADRs: `spec/adr/` +- Features: `spec/features/00N-slug/spec.md` (optional `plan.md`, `tasks.md`) - Ubiquitous language: `CONTEXT.md` -- Requirements (historical): `docs/mvp/requirements/requirements.md` -- Backlog: `docs/mvp/backlog/backlog.md` +- Archive (historical): `spec/archive/requirements.md` +- Backlog: `spec/backlog/backlog.md` +- GitHub Issues workflow: `spec/meta/github-issues.md` ## Not canonical @@ -27,4 +27,4 @@ Do not use `src/` or `notebooks/` as reference architecture. Legacy exploration ## Code changes -Every implementation change must trace to a feature spec under `docs/mvp/features/` or the demo slice (`docs/mvp/demo-slice.md`). Read the architecture doc for cross-cutting rules before proposing design or code. +Every implementation change must trace to a feature spec under `spec/features/` or the roadmap (`spec/constitution/roadmap.md`). Read `spec/constitution/mission.md` for cross-cutting rules before proposing design or code. diff --git a/.cursor/rules/spec-driven-workflow.mdc b/.cursor/rules/spec-driven-workflow.mdc index 19a9e77..3ee5fcb 100644 --- a/.cursor/rules/spec-driven-workflow.mdc +++ b/.cursor/rules/spec-driven-workflow.mdc @@ -7,13 +7,13 @@ alwaysApply: true ## Before coding -1. Name the feature spec file (e.g. `docs/mvp/features/cheap-stocks.md`). -2. Read `docs/mvp/architecture/architecture.md` for global constraints: point-in-time data, hard exclusions, MLflow runs, paper trading, universe rules. +1. Name the feature spec file (e.g. `spec/features/003-cheap-stocks/spec.md`). +2. Read `spec/constitution/mission.md` for global constraints: point-in-time data, hard exclusions, MLflow runs, paper trading, universe rules. 3. If scope is unclear, ask the user and record the answer in the spec (open questions or decisions tables). ## During work -- Do not add features without a corresponding `docs/mvp/features/` document. Create or extend the spec first. +- Do not add features without a corresponding `spec/features/` document. Create or extend the spec first. - Keep changes aligned with MVP scope and "Out of MVP scope" sections in the spec. ## After implementation diff --git a/.cursor/skills/commit-split/SKILL.md b/.cursor/skills/commit-split/SKILL.md index 07cb502..799748e 100644 --- a/.cursor/skills/commit-split/SKILL.md +++ b/.cursor/skills/commit-split/SKILL.md @@ -54,7 +54,7 @@ Rules: imperative present tense; **lowercase** description; **no** trailing peri | --- | --- | | `docs` | `docs/`, `CONTEXT.md`, `AGENTS.md` | | `agents` | `.cursor/rules/` | -| `mvp` | `docs/mvp/**` specs | +| `mvp` | `spec/**` specs | | `skills` | `.cursor/skills/` | | `chore` | `.gitignore`, repo hygiene with no domain doc change | diff --git a/.cursor/skills/grill-with-docs/ADR-FORMAT.md b/.cursor/skills/grill-with-docs/ADR-FORMAT.md index da7e78e..dbf2274 100644 --- a/.cursor/skills/grill-with-docs/ADR-FORMAT.md +++ b/.cursor/skills/grill-with-docs/ADR-FORMAT.md @@ -1,8 +1,8 @@ # ADR Format -ADRs live in `docs/adr/` and use sequential numbering: `0001-slug.md`, `0002-slug.md`, etc. +ADRs live in `spec/adr/` and use sequential numbering: `0001-slug.md`, `0002-slug.md`, etc. -Create the `docs/adr/` directory lazily — only when the first ADR is needed. +Create the `spec/adr/` directory lazily — only when the first ADR is needed. ## Template @@ -24,7 +24,7 @@ Only include these when they add genuine value. Most ADRs won't need them. ## Numbering -Scan `docs/adr/` for the highest existing number and increment by one. +Scan `spec/adr/` for the highest existing number and increment by one. ## When to offer an ADR diff --git a/.cursor/skills/grill-with-docs/SKILL.md b/.cursor/skills/grill-with-docs/SKILL.md index 5ea0aa9..602b737 100644 --- a/.cursor/skills/grill-with-docs/SKILL.md +++ b/.cursor/skills/grill-with-docs/SKILL.md @@ -43,13 +43,13 @@ If a `CONTEXT-MAP.md` exists at the root, the repo has multiple contexts. The ma ├── src/ │ ├── ordering/ │ │ ├── CONTEXT.md -│ │ └── docs/adr/ ← context-specific decisions +│ │ └── spec/adr/ ← context-specific decisions │ └── billing/ │ ├── CONTEXT.md -│ └── docs/adr/ +│ └── spec/adr/ ``` -Create files lazily — only when you have something to write. If no `CONTEXT.md` exists, create one when the first term is resolved. If no `docs/adr/` exists, create it when the first ADR is needed. +Create files lazily — only when you have something to write. If no `CONTEXT.md` exists, create one when the first term is resolved. If no `spec/adr/` exists, create it when the first ADR is needed. ## During the session diff --git a/.cursor/skills/improve-codebase-architecture/SKILL.md b/.cursor/skills/improve-codebase-architecture/SKILL.md index c12b263..dcfb703 100644 --- a/.cursor/skills/improve-codebase-architecture/SKILL.md +++ b/.cursor/skills/improve-codebase-architecture/SKILL.md @@ -1,6 +1,6 @@ --- name: improve-codebase-architecture -description: Find deepening opportunities in a codebase, informed by the domain language in CONTEXT.md and the decisions in docs/adr/. Use when the user wants to improve architecture, find refactoring opportunities, consolidate tightly-coupled modules, or make a codebase more testable and AI-navigable. +description: Find deepening opportunities in a codebase, informed by the domain language in CONTEXT.md and the decisions in spec/adr/. Use when the user wants to improve architecture, find refactoring opportunities, consolidate tightly-coupled modules, or make a codebase more testable and AI-navigable. --- # Improve Codebase Architecture diff --git a/.cursor/skills/setup-matt-pocock-skills/SKILL.md b/.cursor/skills/setup-matt-pocock-skills/SKILL.md index b6948cb..df4949a 100644 --- a/.cursor/skills/setup-matt-pocock-skills/SKILL.md +++ b/.cursor/skills/setup-matt-pocock-skills/SKILL.md @@ -23,7 +23,7 @@ Look at the current repo to understand its starting state. Read whatever exists; - `git remote -v` and `.git/config` — is this a GitHub repo? Which one? - `AGENTS.md` and `CLAUDE.md` at the repo root — does either exist? Is there already an `## Agent skills` section in either? - `CONTEXT.md` and `CONTEXT-MAP.md` at the repo root -- `docs/adr/` and any `src/*/docs/adr/` directories +- `spec/adr/` and any `src/*/spec/adr/` directories - `.cursor/rules/` — do `issue-tracker.md`, `triage-labels.md`, and `domain.md` already exist? - `.scratch/` — sign that a local-markdown issue tracker convention is already in use @@ -60,11 +60,11 @@ Default: each role's string equals its name. Ask the user if they want to overri **Section C — Domain docs.** -> Explainer: Some skills (`improve-codebase-architecture`, `diagnose`, `tdd`) read a `CONTEXT.md` file to learn the project's domain language, and `docs/adr/` for past architectural decisions. They need to know whether the repo has one global context or multiple (e.g. a monorepo with separate frontend/backend contexts) so they look in the right place. +> Explainer: Some skills (`improve-codebase-architecture`, `diagnose`, `tdd`) read a `CONTEXT.md` file to learn the project's domain language, and `spec/adr/` for past architectural decisions. They need to know whether the repo has one global context or multiple (e.g. a monorepo with separate frontend/backend contexts) so they look in the right place. Confirm the layout: -- **Single-context** — one `CONTEXT.md` + `docs/adr/` at the repo root. Most repos are this. +- **Single-context** — one `CONTEXT.md` + `spec/adr/` at the repo root. Most repos are this. - **Multi-context** — `CONTEXT-MAP.md` at the root pointing to per-context `CONTEXT.md` files (typically a monorepo). ### 3. Confirm and edit diff --git a/.cursor/skills/setup-matt-pocock-skills/domain.md b/.cursor/skills/setup-matt-pocock-skills/domain.md index c97d6a6..c0d4d0d 100644 --- a/.cursor/skills/setup-matt-pocock-skills/domain.md +++ b/.cursor/skills/setup-matt-pocock-skills/domain.md @@ -6,7 +6,7 @@ How the engineering skills should consume this repo's domain documentation when - **`CONTEXT.md`** at the repo root, or - **`CONTEXT-MAP.md`** at the repo root if it exists — it points at one `CONTEXT.md` per context. Read each one relevant to the topic. -- **`docs/adr/`** — read ADRs that touch the area you're about to work in. In multi-context repos, also check `src//docs/adr/` for context-scoped decisions. +- **`spec/adr/`** — read ADRs that touch the area you're about to work in. In multi-context repos, also check `src//spec/adr/` for context-scoped decisions. If any of these files don't exist, **proceed silently**. Don't flag their absence; don't suggest creating them upfront. The producer skill (`/grill-with-docs`) creates them lazily when terms or decisions actually get resolved. @@ -17,7 +17,7 @@ Single-context repo (most repos): ``` / ├── CONTEXT.md -├── docs/adr/ +├── spec/adr/ │ ├── 0001-event-sourced-orders.md │ └── 0002-postgres-for-write-model.md └── src/ @@ -28,14 +28,14 @@ Multi-context repo (presence of `CONTEXT-MAP.md` at the root): ``` / ├── CONTEXT-MAP.md -├── docs/adr/ ← system-wide decisions +├── spec/adr/ ← system-wide decisions └── src/ ├── ordering/ │ ├── CONTEXT.md - │ └── docs/adr/ ← context-specific decisions + │ └── spec/adr/ ← context-specific decisions └── billing/ ├── CONTEXT.md - └── docs/adr/ + └── spec/adr/ ``` ## Use the glossary's vocabulary From 679dd50ca84f918bad8bad082705d03d51e0258d Mon Sep 17 00:00:00 2001 From: JLaborda <15078416+JLaborda@users.noreply.github.com> Date: Wed, 1 Jul 2026 07:52:15 +0000 Subject: [PATCH 3/5] refactor: point source docstrings to spec/ paths Co-authored-by: Cursor --- src/smartwealthai/download_fundamentals.py | 6 +++--- src/smartwealthai/download_simfin.py | 4 ++-- src/smartwealthai/edgartools_client.py | 2 +- src/smartwealthai/lake_paths.py | 2 +- src/smartwealthai/normalize_simfin.py | 2 +- src/smartwealthai/sec_client.py | 2 +- tests/test_download_fundamentals.py | 2 +- 7 files changed, 10 insertions(+), 10 deletions(-) diff --git a/src/smartwealthai/download_fundamentals.py b/src/smartwealthai/download_fundamentals.py index b027cd9..a3e489c 100644 --- a/src/smartwealthai/download_fundamentals.py +++ b/src/smartwealthai/download_fundamentals.py @@ -1,7 +1,7 @@ """CLI to download raw SEC companyfacts and edgartools statements for a universe. Orchestrates the fundamentals download spike documented in -``docs/mvp/features/etl-data-lake.md``. For each ``(ticker, cik)`` in the universe: +``spec/features/006-etl-data-lake/spec.md``. For each ``(ticker, cik)`` in the universe: 1. SEC REST — verbatim ``companyfacts`` JSON. 2. edgartools — annual income, balance, and cash-flow statements as parquet. @@ -11,7 +11,7 @@ export SEC_IDENTITY="Your Name your@email.com" poetry run download-fundamentals --universe dow30 -See ``docs/mvp/guides/download-fundamentals.md`` for the full operator guide. +See ``spec/guides/download-fundamentals.md`` for the full operator guide. """ from __future__ import annotations @@ -181,7 +181,7 @@ def run_download( @click.command( context_settings={"help_option_names": ["-h", "--help"]}, - epilog="Guide: docs/mvp/guides/download-fundamentals.md", + epilog="Guide: spec/guides/download-fundamentals.md", ) @click.option("--universe", help="Universe preset name (e.g. dow30).") @click.option( diff --git a/src/smartwealthai/download_simfin.py b/src/smartwealthai/download_simfin.py index 96d065d..c4541b6 100644 --- a/src/smartwealthai/download_simfin.py +++ b/src/smartwealthai/download_simfin.py @@ -1,7 +1,7 @@ """CLI to download SimFin bulk US fundamentals into the raw data lake. Orchestrates the demo SimFin connector documented in -``docs/mvp/features/etl-data-lake.md``. Downloads ``companies``, ``industries``, +``spec/features/006-etl-data-lake/spec.md``. Downloads ``companies``, ``industries``, ``income`` (TTM), ``balance`` (quarterly), ``cashflow`` (TTM), and ``shareprices`` (latest) for ``market=us``. @@ -10,7 +10,7 @@ export SIMFIN_API_KEY="" poetry run download-simfin -See ``docs/mvp/guides/download-simfin.md`` for the operator guide. +See ``spec/guides/download-simfin.md`` for the operator guide. """ from __future__ import annotations diff --git a/src/smartwealthai/edgartools_client.py b/src/smartwealthai/edgartools_client.py index c2ed9ff..b968ce6 100644 --- a/src/smartwealthai/edgartools_client.py +++ b/src/smartwealthai/edgartools_client.py @@ -7,7 +7,7 @@ Output parquets preserve the edgartools dataframe shape (``concept``, ``label``, ``section``, ``FY 20xx`` columns). -Operator guide: ``docs/mvp/guides/download-fundamentals.md``. +Operator guide: ``spec/guides/download-fundamentals.md``. """ from __future__ import annotations diff --git a/src/smartwealthai/lake_paths.py b/src/smartwealthai/lake_paths.py index c5064fa..f96e650 100644 --- a/src/smartwealthai/lake_paths.py +++ b/src/smartwealthai/lake_paths.py @@ -1,6 +1,6 @@ """Path builders for the local data lake raw zone. -Paths follow the layout documented in ``docs/mvp/features/etl-data-lake.md`` (Fundamentals +Paths follow the layout documented in ``spec/features/006-etl-data-lake/spec.md`` (Fundamentals download spike) so local runs can move to S3 without renaming partitions. Example:: diff --git a/src/smartwealthai/normalize_simfin.py b/src/smartwealthai/normalize_simfin.py index 4190e3e..0143004 100644 --- a/src/smartwealthai/normalize_simfin.py +++ b/src/smartwealthai/normalize_simfin.py @@ -54,7 +54,7 @@ def resolve_normalize_tickers( if ticker_set is None: msg = ( "Pass --universe-run-date (after build-universe) or --ticker to limit scope. " - "See docs/mvp/guides/download-simfin.md." + "See spec/guides/download-simfin.md." ) raise click.ClickException(msg) return ticker_set diff --git a/src/smartwealthai/sec_client.py b/src/smartwealthai/sec_client.py index d2bb8c6..52c773c 100644 --- a/src/smartwealthai/sec_client.py +++ b/src/smartwealthai/sec_client.py @@ -6,7 +6,7 @@ - Request throttling (~8 req/s). - Retries with exponential backoff on transient failures. -Operator guide: ``docs/mvp/guides/download-fundamentals.md``. +Operator guide: ``spec/guides/download-fundamentals.md``. """ from __future__ import annotations diff --git a/tests/test_download_fundamentals.py b/tests/test_download_fundamentals.py index 6c75b52..d748eb5 100644 --- a/tests/test_download_fundamentals.py +++ b/tests/test_download_fundamentals.py @@ -2,7 +2,7 @@ Covers universe loading, raw-zone path layout, and skip/force cache semantics. Network integration tests are intentionally excluded from PR CI; see -``docs/mvp/guides/download-fundamentals.md``. +``spec/guides/download-fundamentals.md``. """ from __future__ import annotations From bfdcffa99802e711d1bcf58b7d46e925905ff86b Mon Sep 17 00:00:00 2001 From: JLaborda <15078416+JLaborda@users.noreply.github.com> Date: Wed, 1 Jul 2026 07:52:49 +0000 Subject: [PATCH 4/5] docs(mvp): remove legacy docs tree and add portfolio-evolution spec Un-ignore spec/features/*portfolio*/ paths blocked by *portfolio* gitignore. Co-authored-by: Cursor --- .gitignore | 2 + docs/adr/0001-simfin-fundamentals-mvp.md | 15 - docs/adr/0002-june-demo-scope-cut.md | 15 - docs/mvp/NOTION_SETUP.md | 39 -- docs/mvp/architecture/architecture.md | 497 ------------------ docs/mvp/backlog/backlog.md | 16 - docs/mvp/demo-slice.md | 98 ---- docs/mvp/features/backtesting.md | 162 ------ docs/mvp/features/broker-execution.md | 127 ----- docs/mvp/features/cheap-stocks.md | 113 ---- docs/mvp/features/corroborative-signals.md | 86 --- docs/mvp/features/dashboard-reporting.md | 135 ----- docs/mvp/features/etl-data-lake.md | 451 ---------------- docs/mvp/features/high-quality-stocks.md | 111 ---- docs/mvp/features/permanent-loss-filter.md | 129 ----- docs/mvp/features/sell-watch.md | 132 ----- docs/mvp/features/universe-construction.md | 166 ------ .../features/unstructured-financial-data.md | 103 ---- docs/mvp/guides/download-fundamentals.md | 178 ------- docs/mvp/guides/download-simfin.md | 80 --- docs/mvp/prds/ci-cd/ci-cd-prd.md | 231 -------- docs/mvp/prds/devcontainer/prd.md | 131 ----- docs/mvp/prds/phase2/prd.md | 322 ------------ docs/mvp/requirements/requirements.md | 39 -- .../features/009-portfolio-evolution/spec.md | 4 +- 25 files changed, 4 insertions(+), 3378 deletions(-) delete mode 100644 docs/adr/0001-simfin-fundamentals-mvp.md delete mode 100644 docs/adr/0002-june-demo-scope-cut.md delete mode 100644 docs/mvp/NOTION_SETUP.md delete mode 100644 docs/mvp/architecture/architecture.md delete mode 100644 docs/mvp/backlog/backlog.md delete mode 100644 docs/mvp/demo-slice.md delete mode 100644 docs/mvp/features/backtesting.md delete mode 100644 docs/mvp/features/broker-execution.md delete mode 100644 docs/mvp/features/cheap-stocks.md delete mode 100644 docs/mvp/features/corroborative-signals.md delete mode 100644 docs/mvp/features/dashboard-reporting.md delete mode 100644 docs/mvp/features/etl-data-lake.md delete mode 100644 docs/mvp/features/high-quality-stocks.md delete mode 100644 docs/mvp/features/permanent-loss-filter.md delete mode 100644 docs/mvp/features/sell-watch.md delete mode 100644 docs/mvp/features/universe-construction.md delete mode 100644 docs/mvp/features/unstructured-financial-data.md delete mode 100644 docs/mvp/guides/download-fundamentals.md delete mode 100644 docs/mvp/guides/download-simfin.md delete mode 100644 docs/mvp/prds/ci-cd/ci-cd-prd.md delete mode 100644 docs/mvp/prds/devcontainer/prd.md delete mode 100644 docs/mvp/prds/phase2/prd.md delete mode 100644 docs/mvp/requirements/requirements.md rename docs/mvp/features/portfolio-evolution.md => spec/features/009-portfolio-evolution/spec.md (96%) diff --git a/.gitignore b/.gitignore index 30828cb..152de8e 100644 --- a/.gitignore +++ b/.gitignore @@ -13,6 +13,8 @@ data/* !data/reference/** *hacienda* *portfolio* +!spec/features/*portfolio*/ +!spec/features/*portfolio*/** .DS_Store backup* notebooks/* \ No newline at end of file diff --git a/docs/adr/0001-simfin-fundamentals-mvp.md b/docs/adr/0001-simfin-fundamentals-mvp.md deleted file mode 100644 index 287e94c..0000000 --- a/docs/adr/0001-simfin-fundamentals-mvp.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -status: accepted ---- - -# SimFin as MVP fundamentals source (SEC ETL deferred) - -For the June 30 demo and the near-term MVP pipeline, US fundamentals and **run-date share prices** come from **SimFin** (free tier, bulk download via the `simfin` Python package), not SEC EDGAR. Demo prices use SimFin bulk `shareprices/latest` joined to the universe by ticker. The existing SEC spike (`sec_client`, `edgartools_client`, `download-fundamentals`) stays in the repo **frozen** for phase 2; `yfinance` remains a possible fallback for phase 2 backtests and personal NAV, not the demo pipeline. - -**Why:** SEC ETL complexity and rate limits were blocking progress on the scoring pipeline. SimFin provides standardized income, balance, and cash-flow statements with `Publish Date` / `Restated Date` for point-in-time queries, ~20 years of US history on the free tier, and a separate industry taxonomy—enough to ship a Magic Formula demo by end of June. - -**Trade-offs:** `as_of_date` uses SimFin `Publish Date` (not EDGAR acceptance). Phase 2 SEC ingestion may require a reconciliation or re-backtest. SimFin free-tier datasets refresh roughly weekly, which is acceptable for annual-rebalance logic but not for intraday freshness. - -**Considered:** Continue with SEC `companyfacts` only (rejected for June deadline); paid vendors Bloomberg/FactSet (out of budget). - -**Consequences:** Update `etl-data-lake.md` and `universe-construction.md`; add SimFin connector + normalizer; keep `curated/fundamentals` schema provider-agnostic so scoring modules do not change. diff --git a/docs/adr/0002-june-demo-scope-cut.md b/docs/adr/0002-june-demo-scope-cut.md deleted file mode 100644 index 89bf763..0000000 --- a/docs/adr/0002-june-demo-scope-cut.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -status: accepted ---- - -# June 30 demo slice — simplest Magic Formula vertical - -The **June 30 deliverable** is a reduced vertical slice, not the full architecture vision. Pipeline: **SimFin ETL → universe (US market) → ROC/EY → combined rank → top-30 equal-weight model portfolio → Streamlit dashboard**. Permanent loss filter, backtesting, sell-watch, paper trading, watchlist, and corroborative/unstructured modules are **deferred to phase 2**. - -**Why:** The project is behind schedule. The portfolio demo must show an explainable, end-to-end Magic Formula run on real data—not every MLOps and risk gate in the full spec. - -**Trade-offs:** No backtest gate before orders (orders are out of scope anyway). Universe is all SimFin US companies minus banks/insurers/utilities—not historical S&P 500 with delisted names (survivorship bias mitigation waits for backtest phase). No permanent-loss filter in the demo path. - -**Considered:** Full MVP including 20-year walk-forward backtest (rejected for June); demo + minimal backtest (rejected—user chose fastest path); Zipline for backtests (rejected—incompatible with Python 3.11, unmaintained, poor fit for fundamental annual rebalance). - -**Consequences:** Documented in [`docs/mvp/demo-slice.md`](../mvp/demo-slice.md). Full feature specs remain the north star; modules marked deferred are unchanged in intent. MLflow logs **pipeline runs** in the demo; the `backtesting` experiment starts in phase 2. diff --git a/docs/mvp/NOTION_SETUP.md b/docs/mvp/NOTION_SETUP.md deleted file mode 100644 index 3faa90a..0000000 --- a/docs/mvp/NOTION_SETUP.md +++ /dev/null @@ -1,39 +0,0 @@ -# Notion setup for SmartWealthAI - -Git specs under `docs/mvp/` stay canonical. Notion tracks tasks only. - -## Default board - -The project task board in Notion is named **`Cursor Agent Tasks`** (in this repo’s Notion workspace). Agents locate it via **Notion MCP search** — no board URL is committed to Git. See [AGENTS.md](../../AGENTS.md). - -## 1. Board in Notion - -Use the existing **`Cursor Agent Tasks`** database/board in the project Notion space. If you need a fresh board, you can duplicate the [Code with Notion template](https://notion.notion.site/code-with-notion-board) and name it `Cursor Agent Tasks`. - -## 2. Project home page (optional) - -Create a Notion page with: - -- Link to this repository -- Link to [architecture.md](architecture/architecture.md) on GitHub -- List of [feature specs](features/) - -## 3. Connect Cursor - -1. Cursor **Settings → MCP** → ensure the Notion plugin is enabled. -2. Authenticate when prompted (or run `mcp_auth` for Notion if tool calls fail). - -## 4. First tasks from a spec - -In Agent mode, after MCP auth: - -- Ask to create tasks on **`Cursor Agent Tasks`** from e.g. `docs/mvp/features/universe-construction.md`, or -- Ask to use **spec-to-implementation** (point the agent at the Git spec path; specs live in this repo, not in Notion). - -## Task discipline - -| Step | Where | -| --- | --- | -| Change MVP decision | Update Git spec first | -| Track work | Notion task with `docs/mvp/features/...` path | -| Finish work | Update Git spec (status + acceptance criteria), then mark Notion done | diff --git a/docs/mvp/architecture/architecture.md b/docs/mvp/architecture/architecture.md deleted file mode 100644 index c3939c7..0000000 --- a/docs/mvp/architecture/architecture.md +++ /dev/null @@ -1,497 +0,0 @@ -# SmartWealthAI - MVP Architecture - -This document is a living specification for designing the SmartWealthAI MVP with a spec-driven development workflow. Its purpose is to capture decisions, open questions, assumptions, and acceptance criteria before writing implementation code. - -This is a portfolio project intended to showcase MLOps practices applied to a quantitative value investing system. The end user is a particular investor, but the system itself behaves as an automated agent that runs end-to-end without manual intervention. - -## June 30 demo slice (current delivery target) - -The **first shippable vertical** is narrower than the full vision below. See [`demo-slice.md`](../demo-slice.md) and [ADR-0002](../../adr/0002-june-demo-scope-cut.md): SimFin ETL → US-market universe → ROC/EY → top-30 equal-weight portfolio → Streamlit dashboard. Backtest, permanent loss filter, sell-watch, and paper trading are **phase 2**. The full architecture in this document remains the north star. - -## MVP vision - -Build a modular quantitative value investing system that: - -1. Retrieves financial data from **SimFin** (fundamentals, demo) and free price providers; SEC EDGAR deferred to phase 2 ([ADR-0001](../../adr/0001-simfin-fundamentals-mvp.md)). -2. Stores raw and curated data in an AWS-based, incrementally refreshed data lake. -3. Filters out companies with high risk of permanent capital loss (fraud and bankruptcy). -4. Identifies high-quality companies. -5. Identifies cheap companies. -6. Uses corroborative signals to strengthen or weaken investment theses. -7. Analyzes unstructured financial data (filings, transcripts, news). -8. Builds a ranked watchlist and a long-only model portfolio of 15 to 30 US stocks. -9. Backtests the strategy over 20+ years against benchmarks and historical crises. -10. Continuously monitors the model portfolio to detect sell signals. -11. Visualizes the evolution of the model portfolio and of the user's personal portfolio. -12. Prepares broker orders in paper trading mode only. -13. Surfaces every decision in a dashboard with auditable explanations. - -The MVP prioritizes traceability, reproducibility, point-in-time correctness, low operational cost, and a clean separation between data ingestion, rules, scoring, portfolio construction, monitoring, and execution. - -## Architecture principles - -- **Modularity**: each module evolves independently and can be replaced without rewriting downstream code. -- **Traceability and explainability**: every score, exclusion, buy, or sell decision is reconstructible from its input data and rule version. -- **Reproducibility**: a run over a given universe and date can be replayed bit-for-bit from versioned data and code. -- **Point-in-time correctness**: no module is allowed to use data that was not yet publicly available at the decision date. Look-ahead bias is treated as a critical defect. -- **Raw before transformed**: provider responses are stored verbatim in a raw zone before any normalization, so any bug downstream can be replayed from source. -- **Incremental data lake**: new filings or prices update only what changed; we never reprocess the full history unless we explicitly request it. -- **Provider abstraction**: SimFin, SEC EDGAR (phase 2), free price providers, and future sources are wrapped behind interchangeable connectors; `curated/fundamentals` schema is provider-agnostic. -- **AWS-first, cheapest-first**: data lake, compute, secrets, and dashboard all live in AWS, choosing the cheapest viable option at MVP scale. Heavier infrastructure (Kubernetes, paid data) is a growth path, not an MVP requirement. -- **MLOps and CI/CD by design**: every pipeline component is built, tested, packaged, deployed, scheduled, and observed. -- **Paper trading first**: the broker module never touches real money in the MVP, even by accident. -- **Backtesting before broker**: the strategy must pass a backtest before any order, even a paper one, is generated. **Does not apply to the June demo slice** (no broker in demo). -- **Specs before code**: this document and the feature specs are refined before implementation begins. - -## Architecture diagram - -```mermaid -flowchart LR - subgraph Sources["Data sources"] - SimFin["SimFin (fundamentals, MVP)"] - SEC["SEC EDGAR (phase 2)"] - Prices["Prices: SimFin shareprices/latest (demo) + yfinance / vendor fallback (phase 2)"] - UserPort["User portfolio CSV (data/clean/personal_finance/...)"] - News["News and transcripts (later)"] - end - - subgraph ETL["ETL + Data Lake (S3 + DuckDB, incremental)"] - Ingest["Ingestion connectors"] - Raw["Raw zone (immutable)"] - Normalized["Normalized / curated zone (versioned schema)"] - PITStore["Point-in-time store (as_of_date = SimFin Publish Date)"] - QualityChecks["Data quality + review queue"] - end - - subgraph Universe["Universe construction"] - SP500Hist["Universe: SimFin US (demo) / S&P 500 historical (phase 2)"] - UniFilters["Filters: IndustryId exclusion banks / insurers / utilities"] - end - - subgraph Analysis["Analysis engine"] - PermanentLoss["Permanent loss filter (fraud + bankruptcy)"] - Quality["Quality score (ROC)"] - Cheapness["Cheapness score (Earnings Yield)"] - Signals["Corroborative signals"] - Unstructured["Unstructured data analysis"] - end - - subgraph Decision["Investment decision"] - RiskGate["Risk gate (max 10% per name)"] - Ranking["Greenblatt-style ranking + market-cap tie-break"] - Sizing["Portfolio construction (15-30 names, EW/SW/RP)"] - Watchlist["Watchlist + model portfolio"] - end - - subgraph Live["Live monitoring (daily)"] - SellWatch["Sell-watch: quality drop + fraud/bankruptcy + overvaluation + opportunity cost"] - PortfolioEvo["Portfolio evolution (model vs personal vs benchmarks)"] - EmailAlerts["Dashboard + AWS SES email"] - end - - subgraph Validation["Validation"] - Backtest["Backtest (>= 20 years, annual rebalance, walk-forward 3-5y) + Monte Carlo"] - CrisisReport["Crisis drawdown report (informational)"] - Benchmarks["Benchmarks: S&P 500 CW + S&P 500 EW + Russell 3000 + Magic Formula"] - end - - subgraph Execution["Execution"] - OrderBuilder["Order builder (manual confirmation)"] - Paper["Paper trading"] - end - - subgraph Ops["MLOps + Reporting"] - Snapshots["MLflow runs + S3 artifacts (immutable snapshots)"] - Dashboard["Streamlit dashboard"] - CICD["GitHub Actions CI/CD + Prefect orchestration"] - Runtime["ECS Fargate Spot (or AWS Batch) tasks"] - Secrets["GitHub Secrets (build) + AWS Secrets Manager (runtime)"] - end - - SimFin --> Ingest - SEC -. "phase 2" .-> Ingest - Prices --> Ingest - UserPort --> Ingest - News --> Ingest - - Ingest --> Raw - Raw --> Normalized - Normalized --> PITStore - PITStore --> QualityChecks - - SP500Hist --> UniFilters - UniFilters --> PermanentLoss - QualityChecks --> PermanentLoss - QualityChecks --> Quality - QualityChecks --> Cheapness - QualityChecks --> Signals - QualityChecks --> Unstructured - - PermanentLoss --> RiskGate - Quality --> Ranking - Cheapness --> Ranking - Signals --> Ranking - Unstructured --> Ranking - RiskGate --> Ranking - Ranking --> Sizing - Sizing --> Watchlist - - PITStore --> Backtest - Ranking --> Backtest - Sizing --> Backtest - Benchmarks --> Backtest - Backtest --> CrisisReport - Backtest -. "Sharpe > all benchmarks" .-> OrderBuilder - - Watchlist --> OrderBuilder - OrderBuilder --> Paper - - Watchlist --> SellWatch - QualityChecks --> SellWatch - Ranking --> SellWatch - SellWatch --> EmailAlerts - SellWatch -. "after user confirmation" .-> OrderBuilder - - Paper --> PortfolioEvo - UserPort --> PortfolioEvo - Benchmarks --> PortfolioEvo - - Ranking --> Snapshots - Sizing --> Snapshots - SellWatch --> Snapshots - Backtest --> Snapshots - Snapshots --> Dashboard - - Secrets --> Ingest - Secrets --> Paper - CICD --> Dashboard -``` - -## MVP modules - -| Module | Spec | Main responsibility | -| --- | --- | --- | -| ETL + Data Lake | [../features/etl-data-lake.md](../features/etl-data-lake.md) | Download, version, validate, and store financial data with point-in-time guarantees and incremental refresh. | -| Universe construction | [../features/universe-construction.md](../features/universe-construction.md) | **Demo:** SimFin US minus banks / insurers / utilities. **Phase 2:** historical S&P 500 (incl. delisted), common-stock filters, share-class dedup. | -| Permanent loss filter | [../features/permanent-loss-filter.md](../features/permanent-loss-filter.md) | Hard-exclude companies with fraud or bankruptcy risk; include the Enron / Lehman / WorldCom regression test. | -| High-quality stocks | [../features/high-quality-stocks.md](../features/high-quality-stocks.md) | Score quality starting from Greenblatt's ROC. | -| Cheap stocks | [../features/cheap-stocks.md](../features/cheap-stocks.md) | Score valuation starting from Earnings Yield. | -| Corroborative signals | [../features/corroborative-signals.md](../features/corroborative-signals.md) | Buybacks, insider activity, and other confirming signals. | -| Unstructured financial data | [../features/unstructured-financial-data.md](../features/unstructured-financial-data.md) | Extract useful information from filings, transcripts, and news. | -| Backtesting + crisis report | [../features/backtesting.md](../features/backtesting.md) | Walk-forward backtest (3-5y windows) over 20+ years, Monte Carlo, benchmarks (S&P 500 CW/EW, Russell 3000, Magic Formula), crisis drawdown report. | -| Sell-watch / vigilance | [../features/sell-watch.md](../features/sell-watch.md) | Daily monitor of model portfolio for quality drop, fraud/bankruptcy, overvaluation, and opportunity cost. Emits signals (no auto-execution). | -| Portfolio evolution | [../features/portfolio-evolution.md](../features/portfolio-evolution.md) | Track the model portfolio and the user's personal portfolio over time and compare against configurable benchmarks. | -| Broker execution | [../features/broker-execution.md](../features/broker-execution.md) | Convert confirmed decisions into paper trading orders only. | -| Dashboard + reporting | [../features/dashboard-reporting.md](../features/dashboard-reporting.md) | Surface every input, score, decision, and explanation. Functional-first for the MVP. | - -## Functional flow — June 30 demo slice - -See [`demo-slice.md`](../demo-slice.md). Steps not listed here are **phase 2**. - -1. Pipeline run for a `run_date`; secrets from env / AWS Secrets Manager (`SIMFIN_API_KEY`, etc.). -2. Bulk-download SimFin US datasets if older than `refresh_days`; store verbatim under `raw/simfin/`. -3. Build the demo universe: SimFin US companies minus banks / insurers / utilities (`IndustryId` CSV + bank/insurance sanity check). -4. Build run-date prices from SimFin bulk `shareprices/latest`: join universe tickers, take the latest `Date <= run_date`, and store `curated/prices`. -5. Run the SimFin normalizer → `curated/fundamentals` with PIT `as_of_date` from SimFin `Publish Date`. -6. Calculate ROC and Earnings Yield; combined rank with market-cap tie-break. -7. Select top **30** names, equal-weight model portfolio. -8. Log an MLflow run (params, metrics, portfolio artifact, git SHA). -9. Publish the Streamlit dashboard: ranking table, portfolio, per-name ROC/EY explainability. - -## Functional flow — full MVP (phase 2) - -North-star end-to-end flow after the demo slice ships: - -1. CI/CD pipeline triggers a daily run (cron via Prefect / EventBridge) and pulls secrets. -2. Build the run-date universe from historical S&P 500 constituents, apply universe filters, deduplicate share classes. -3. Ingest fundamentals from SimFin and/or SEC EDGAR and prices from free providers, storing raw responses immutably. -4. Incrementally normalize new or restated data and write to the point-in-time store (`as_of_date` = provider publish or EDGAR acceptance). -5. Run data quality checks; failing rows go to the review queue and are excluded if not resolved. -6. Apply the permanent loss filter (fraud + bankruptcy) as a hard exclusion with stored reasons. CI runs the Enron / Lehman / WorldCom regression check. -7. Calculate ROC (quality) and Earnings Yield (cheapness). -8. Apply corroborative and unstructured signals. -9. Build a Greenblatt-style combined ranking; break ties by ascending market cap. -10. Select 15 to 30 long-only names, max 10% per name; portfolio weighting (EW, SW, RP) is a backtest hyperparameter. -11. Backtest the configuration on 20+ years of point-in-time data, walk-forward 3-5 year windows. If Sharpe does not beat all benchmarks (S&P 500 CW, S&P 500 EW, Russell 3000, Magic Formula), do not auto-promote any new configuration; the configuration that runs in production is the last one that passed. -12. The sell-watch module re-scores current holdings daily; any sell trigger creates a signal that goes to dashboard + email; only proceeds to the order builder after explicit user confirmation. -13. Generate paper trading orders only. -14. Track the model portfolio (as if it were traded) and the user's personal portfolio (from the cleaned CSV) and compare against configurable benchmarks. -15. Log every run as an MLflow run with artifacts in S3 (immutable snapshot). -16. Publish the dashboard with inputs, scores, decisions, and explanations. - -## Decisions made so far - -These items are now closed for the MVP. They can be reopened in later iterations. - -### Universe and data - -| Area | Decision | -| --- | --- | -| **June demo** | See [`demo-slice.md`](../demo-slice.md). SimFin → US universe → ROC/EY → top 30 EW → dashboard. | -| Markets | US only. Other markets deferred. | -| Universe (demo) | All SimFin US companies minus banks/insurers/utilities. | -| Universe (full MVP) | S&P 500 historical constituents (incl. delisted). Phase 2. | -| Sector classification (demo) | SimFin `IndustryId` + `load_industries()`; exclusions in `data/reference/simfin_industry_exclusions.csv`. | -| Sector classification (full MVP) | SIC from SEC EDGAR when SEC ETL ships. | -| Sectors excluded | Banks, insurers, and utilities (incomparable accounting for ROC/EY). | -| Sector limits | No sector / country / industry quotas. Out of MVP scope. | -| Share classes | Treat as the same company; keep the class with the highest average trading liquidity and drop the rest. Phase 2 for demo. | -| Market cap floor | Optional parameter. Off in demo. | -| Trading volume floor | Optional; off in demo. | -| Primary fundamentals source | **SimFin** (free tier, bulk download). [ADR-0001](../../adr/0001-simfin-fundamentals-mvp.md). | -| SEC ETL | Frozen spike in repo; phase 2 normalizer. | -| Primary price source | **Demo:** SimFin bulk `shareprices/latest`. **Phase 2:** `yfinance`; free tiers of FMP, Alpha Vantage, and EODHD as redundancy / fallback. | -| Data lake | S3 (raw + curated zones) + DuckDB as the analytical engine (`duckdb` reads parquet directly from S3, no Athena bill). | -| Data lake refresh | Bulk re-download on schedule (`refresh_days=7` on free tier); incremental normalize by `Publish Date` watermark. | -| Schema versioning | Normalized schemas are versioned with explicit migrations. | -| Raw data policy | Store provider responses verbatim in the raw zone (SimFin bulk files for variants the pipeline downloads). | -| Retention policy | Keep curated data long-term; purge raw data only once curated data has been validated. | -| Point-in-time | Required. SimFin `Publish Date` is `as_of_date`; `Restated Date` for new versions; `Report Date + lag` fallback → review queue. | -| Fundamentals periodicity | Income/cashflow TTM; balance sheet quarterly (latest PIT snapshot). | -| Missing data | Flag for review for the MVP. If review backlog grows, fall back to exclusion. | - -### Scoring and portfolio construction - -| Area | Decision | -| --- | --- | -| Ranking style | Greenblatt-style: ROC for quality + Earnings Yield for cheapness. Treated as a placeholder until replaced by a more practical model. | -| Tie-break | Sort ties by ascending market cap; smaller names have priority (more room to grow). | -| Permanent loss | Hard exclusion. Scope: fraud + bankruptcy only. | -| Portfolio size | **Demo:** top 30. **Full MVP:** 15 to 30 long-only positions. | -| Short positions | Not allowed. | -| Per-name cap | 10% of portfolio (full MVP; irrelevant for demo EW top 30). | -| Weighting | **Demo:** equal-weight only. **Full MVP:** EW / SW / RP as backtest hyperparameters. | -| Rebalancing | Annual fixed for the full MVP. Demo is single `run_date` snapshot. | -| Outputs | **Demo:** model portfolio + full ranking in dashboard. **Full MVP:** watchlist + model portfolio + evolution. | - -### Risk, explainability, and operations - -| Area | Decision | -| --- | --- | -| Explainability | Dashboard shows raw inputs + scores + explanations + the rules that fired. Functional-first; visual polish later. | -| Run snapshots | MLflow is enabled from day one. Every pipeline run and every backtest is an MLflow run with parameters, metrics, and artifacts in S3 (see "MLflow as the snapshot store" below). | -| Risk checks that block orders | Permanent loss filter must have flagged `pass`; no duplicate orders; data freshness within threshold; full scoring pipeline completed; cash availability within configured limit; backtest must beat all benchmark Sharpes. | -| FP/FN review | Backtest builds a confusion matrix per rule and a curated regression test forces the bankruptcy filter to flag Enron, Lehman, and WorldCom. | -| Overfitting controls | Walk-forward backtesting + hold-out years never used for tuning + an in-sample vs out-of-sample Sharpe divergence flag treated as a red signal. | -| Run frequency | Daily. Cheap by design. | -| Secrets management | GitHub Secrets at build / deploy time, AWS Secrets Manager at runtime. | -| Broker mode | Paper trading only. | -| Primary user | Particular investor consuming a dashboard; the system itself runs as an automated agent. | - -### Infrastructure (AWS, cheapest-first) - -| Area | Decision | -| --- | --- | -| Storage | S3 (raw + curated parquet) + DuckDB as the local query engine. No Athena bill. | -| Experiment tracking | MLflow from day one. Tracking server on a small EC2 (SQLite backend) with `s3://` as the artifact root. | -| Dashboard | Streamlit. | -| Email alerts | AWS SES (sporadic emails, very cheap). | -| Compute / runtime | ECS Fargate Spot tasks (or AWS Batch on Fargate Spot), whichever is cheaper for the daily run. Triggered by Prefect. | -| Orchestration | Prefect Core (self-hosted on the same EC2 as MLflow, or via Prefect Cloud free tier). | -| Scheduling | EventBridge cron triggers the Prefect deployment once per day. | -| CI/CD | GitHub Actions builds and pushes Docker images to ECR. | -| Observability | CloudWatch Logs + Prefect UI for the MVP. Per-module metrics added if/when needed. | - -### Backtesting - -| Area | Decision | -| --- | --- | -| Historical depth | At least 20 years (value-investing horizon). | -| Crisis scenarios | All major historical crises included (dotcom, GFC, COVID, 2022 rate shock). Drawdown per crisis is reported but the MVP does not require crisis pass / fail. | -| Monte Carlo | Yes, in addition to historical replay. See "Monte Carlo simulation" below. | -| Transaction costs | Not modeled in the MVP. | -| Taxes | Not modeled in the MVP. | -| Survivorship bias | Avoided by including delisted historical S&P 500 constituents. Bankruptcies remain in the universe as evidence. | -| Pass criterion | Strategy's Sharpe must beat all of: S&P 500 cap-weighted, S&P 500 equal-weighted, Russell 3000, and Greenblatt Magic Formula. | -| Walk-forward windows | 3 to 5 year train / validation splits, given the long horizon. | -| Backtest rebalancing | Annual (matches production for the MVP). | -| Benchmarks | S&P 500 CW, S&P 500 EW, Russell 3000, Greenblatt Magic Formula portfolio. | - -### Sell-watch - -| Area | Decision | -| --- | --- | -| Scope | Model portfolio only. The user's personal portfolio is not actively monitored (those are personal orders). | -| Signals | Quality deterioration, fraud / bankruptcy flag turning on after entry, overvaluation, and opportunity cost (a better candidate exists in the watchlist). | -| Overvaluation trigger | Earnings Yield below the cross-sectional 10th percentile **or** Earnings Yield below 5% absolute. Both thresholds are starting points and treated as hyperparameters. | -| Quality deterioration trigger | ROC YoY drop greater than 30% **or** the name dropping out of the ROC top decile. Both thresholds are starting points and treated as hyperparameters. | -| Opportunity cost trigger | A watchlist candidate must outrank the held name by more than 5 positions in the combined Greenblatt ranking before the holding is flagged. | -| Frequency | Daily. | -| States | Hard `sell` only for the MVP. `trim` / `hold-with-warning` deferred. | -| Auto-execution | None. Signals require manual user confirmation before any order is built. | -| Alerts | Dashboard badge + AWS SES email. | -| Rules | Fundamentals-based + opportunity cost. Price-based stops (trailing / drawdown) deferred. | - -### Portfolio evolution - -| Area | Decision | -| --- | --- | -| User portfolio source | `data/clean/personal_finance/operations/my_operations_eur.csv` (already in EUR, derived from two broker exports). | -| Views | Cumulative return, drawdown, rolling Sharpe, holdings over time, contribution / attribution, vs S&P 500, vs the model portfolio. | -| Paper-traded model | The model portfolio is simulated as if it were actually traded, so the user can see what they would have earned or lost by following it daily. | -| Benchmarks | Configurable (S&P 500, MSCI World, others). | -| Update frequency | Daily. | -| Diversification / clustering | Deferred to a later milestone (kept in the long-term wishlist). | - -## Clarifications captured from this iteration - -### Point-in-time data and look-ahead bias - -A backtest (or any historical scoring) must only use information that was publicly available at the decision date. If on `2019-03-31` the system uses Q4 2018 earnings to rank a stock, but those earnings were not filed until `2019-04-25`, the backtest is leaking future information into the past. The same applies to: - -- Restated financials. The "as-known-in-2018" version is what 2018 decisions must use, not today's restated version. -- Index reconstitution. Using today's S&P 500 constituents to backtest 2010 introduces survivorship bias. -- Corporate actions (splits, dividends, delistings) and ticker changes. - -The MVP's point-in-time store records, for every fundamental value, the `as_of_date` (SimFin `Publish Date` in the demo; EDGAR acceptance in phase 2) and a `version_id`. Any historical query is forced to filter by `as_of_date <= decision_date`. When the publish date is missing or unreliable, a conservative lag (period end + 45 days for 10-Q, + 90 days for 10-K) is used and the row is flagged for review. This is conservative enough for a long-term value strategy. - -### MLflow as the snapshot store - -User question: "are immutable snapshots like artifacts? What if we used MLflow?" - -Yes. MLflow is a natural fit here, and it covers three needs at once: - -- **Runs**: each pipeline execution (daily, plus every backtest) is logged as an MLflow run. Parameters (universe definition, rebalance frequency, weighting scheme, thresholds, git commit SHA) are logged via `mlflow.log_param`. Metrics (Sharpe, drawdown, CAGR, hit rate, alpha, number of exclusions, count of sell signals) are logged via `mlflow.log_metric`. -- **Artifacts**: the curated input slice (or its hash), the watchlist, the model portfolio, the backtest report, and the markdown / HTML dashboard snapshot are logged as artifacts. The artifact store points at S3 with versioning and / or object lock so a past run is byte-for-byte recoverable. -- **Model registry (optional, future)**: when the scoring model evolves beyond the Greenblatt placeholder, MLflow's model registry can promote a candidate from `staging` to `production` and tie that decision back to a backtest run. - -Practical MVP setup: - -- MLflow tracking server: a tiny EC2 (or AWS Fargate task on demand) with SQLite or RDS Postgres as the backend store. For the absolute cheapest setup, an MLflow tracking server is not strictly required: `mlflow.start_run(...)` with `file://` or `s3://` as the artifact root works for a single user. -- Artifact root: `s3://smartwealthai-mlflow-artifacts/`. -- Each run is tagged with the commit SHA and pipeline name, which makes the "immutable snapshot" effectively the MLflow run id. - -Treating this as nice-to-have for the MVP is fine: we can start by writing snapshots straight to S3 with predictable paths, and slot in MLflow once the pipeline stabilizes. - -### Monte Carlo simulation of fundamentals and prices - -User question: "we cannot simulate company results coherently with prices, right?" - -Three options, in increasing complexity: - -1. **Block bootstrap of historical paths (recommended for the MVP).** Resample contiguous blocks (e.g., 6 or 12 months) from the real historical dataset across all companies simultaneously. This preserves the joint distribution of prices and fundamentals because both come from the same period of real data. It generates new "alternate histories" without inventing relationships that did not exist. It is the standard technique in academic backtests. -2. **Factor-based simulation.** Estimate a small number of factor returns (market, value, quality, size) plus idiosyncratic noise, and re-simulate company returns from those factors. Fundamentals are then assumed to evolve along their historical AR(1) / random-walk paths conditional on the factor regime. More flexible than bootstrap, but requires estimating a factor model. -3. **Generative joint model (out of MVP).** A VAR / copula / GAN / diffusion model trained on (prices, fundamentals) per company. Very powerful but easy to misuse, and effectively impossible to validate at MVP scale. - -The MVP uses block bootstrap. The synthetic-data approach is captured as a stretch goal. - -### MLOps stack cost reality check - -User proposed: GitHub Actions + MLflow + Prefect + Kubernetes. - -For a daily run over the S&P 500 historical universe, Kubernetes is overkill and expensive. Recommended cost-aware mapping: - -| User goal | MVP-cheap option | Growth path | -| --- | --- | --- | -| Orchestration | Prefect Core running on a small EC2 (or Prefect Cloud free tier) | Prefect on EKS | -| Execution | AWS Batch or ECS Fargate spot tasks triggered by Prefect, or a small EC2 with cron + Docker | EKS with autoscaling | -| Experiment tracking | MLflow with SQLite + S3 artifact root, on the same EC2 | MLflow on RDS + EC2 / Fargate | -| CI/CD | GitHub Actions building Docker images and pushing to ECR | Same | -| Scheduling | EventBridge cron triggering a Prefect deployment | Same | -| Secrets | GitHub Secrets in CI; AWS Secrets Manager at runtime | Same | -| Observability | CloudWatch Logs + Prefect UI | CloudWatch Logs + Prefect Cloud + Grafana | - -This still showcases MLOps competence (CI/CD, container build, orchestrator, experiment tracking, secrets, observability) without paying for EKS in the MVP. - -### Risk checks confirmed for the MVP - -- Permanent loss filter has passed for every name in the target portfolio (already a hard exclusion upstream). -- Full scoring pipeline completed without partial / missing scores. -- Data freshness within threshold. -- No duplicate orders. -- Cash availability within configured limit. -- Backtest of the current configuration must beat all benchmark Sharpes. - -### False positives / false negatives review - -- Build a confusion matrix per rule from each historical backtest. -- Curated regression test in CI: the bankruptcy filter must flag Enron, Lehman, and WorldCom at the right `as_of_date`. -- Maintain a review queue dataset that contains every borderline decision with the triggered rule and its inputs. -- Shadow-mode new rules for one or two runs before they are allowed to influence decisions. - -### Overfitting controls - -- Walk-forward backtesting with 3 to 5 year windows. -- A separate hold-out window that the strategy never sees during tuning. -- Hard alert when in-sample Sharpe and out-of-sample Sharpe diverge beyond a threshold; treated as a red flag for the candidate configuration. - -### Personal portfolio CSV schema - -The consolidated personal portfolio file `data/clean/personal_finance/operations/my_operations_eur.csv` already exists and is generated from two brokers (XTB and IBKR) by `src/preprocessing/cleaning_operations.py` (notebook: `notebooks/portfolio_evolution.ipynb`). The schema is the contract that the portfolio evolution module must consume: - -| Column | Type | Notes | -| --- | --- | --- | -| `Date` | timestamp (with sub-second precision) | Operation timestamp. Used as the time index. | -| `Symbol` | string | Yahoo-Finance-compatible ticker after broker-to-yfinance mapping (e.g., `ITXe` / `ITX.ES` -> `ITX.MC`, `SPY5.UK` -> `SPY5.L`, `FB` -> `META`, `GOOGC` -> `GOOG`). | -| `Type` | enum | `BUY` or `SELL`. | -| `Volume` | float | Shares, fractional allowed. | -| `Price` | float | Per-share price in EUR (USD trades already FX-converted). | -| `Value` | float | Gross EUR notional of the trade. | -| `Commission` | float | EUR fee, negative when the user paid. | -| `Currency` | string | Always `EUR` after cleaning. | - -Open items implied by this schema (to be resolved in the portfolio evolution spec): - -- Dividends are not tracked in the CSV today. Out of MVP scope; tracked as a follow-up so the personal portfolio NAV can include the dividend contribution. -- FX is already applied to USD trades using `EUR=X` daily close at trade date. We document this as the assumption. -- Bankrupt or delisted holdings need a systematic handling: write the last available price as zero on the delisting date instead of dropping the position, so the personal NAV reflects the actual loss. (Note: the `IRBT` case visible in the legacy notebook is not a real loss in the user's history; the user sold IRBT in 2022, well before the bankruptcy. The rule still applies as a general defense.) -- Ticker remapping (FB -> META, GOOGC -> GOOG, SPY5.UK -> SPY5.L, ITXe / ITX.ES -> ITX.MC, etc.) is moved out of the notebook into a CSV at `data/reference/ticker_mapping.csv`, versioned in git. - -### Magic Formula replica (benchmark) - -The Greenblatt Magic Formula benchmark is implemented as a strict canonical replica: - -- Quality factor: `ROC = EBIT / (Net Working Capital + Net Fixed Assets)`. -- Cheapness factor: `EY = EBIT / Enterprise Value`. -- Combined rank: sum of the two cross-sectional ranks (lower is better). -- Same universe, same annual rebalance, same long-only construction as the production strategy. - -This benchmark is the placeholder while the user iterates on the quality and cheapness modules; future scoring variants are evaluated against it. - -### yfinance cache (phase 2) - -For phase 2 backtests and personal NAV, `yfinance` responses are cached on S3 under `s3://smartwealthai-cache/yfinance///.parquet`, with a configurable TTL per endpoint (e.g., prices: 1 day; corporate actions: 7 days; fundamentals: 90 days). Demo prices come from SimFin `shareprices/latest`. Cache misses trigger a live call; cache hits are read straight from S3. - -### Ticker mapping table - -A versioned CSV at `data/reference/ticker_mapping.csv` with columns `broker_symbol, yfinance_symbol, notes, valid_from, valid_to`. Stored in git for diff-able history. A small DuckDB view loads it on demand; we keep the source of truth in CSV because the table is small, infrequently updated, and benefits from PR-reviewable changes. - -## Open questions - -A short, targeted list. Everything else is now closed for the MVP. - -- Source of the S&P 500 historical constituents (including delisted): community dataset such as `github.com/fja05680/sp500`, periodic Wikipedia scrape, or a manually maintained CSV in `data/reference/`? Recommendation: import the community dataset once and pin a snapshot under `data/reference/sp500_constituents.csv`. To be confirmed in `universe-construction.md`. -- Final cost target: with `t4g.micro` for MLflow + Prefect, monthly bill targets around 6 USD plus S3 storage and SES. We track this as a soft budget. - -## Next steps for the spec-driven workflow - -Order proposed for refining the feature specs (each spec follows the same template: Objective, Scope, Out of scope, Inputs, Outputs, Mermaid diagram, Flow, Open questions, Acceptance criteria, Risks): - -1. `universe-construction.md` (new) - blocks every downstream module. -2. `etl-data-lake.md` — SimFin connector + normalizer (demo); SEC spike frozen for phase 2. ✅ Updated. -3. `permanent-loss-filter.md` (update) - fraud + bankruptcy, hard exclusion, Enron / Lehman / WorldCom regression test. -4. `high-quality-stocks.md` (update) - ROC + tie-break by market cap. -5. `cheap-stocks.md` (update) - Earnings Yield as primary cheapness signal. -6. `backtesting.md` (new) - walk-forward, 20+ years, Monte Carlo block bootstrap, benchmark suite, crisis drawdown report. -7. `sell-watch.md` (new) - daily triggers with the thresholds above, manual confirmation, AWS SES alert. -8. `portfolio-evolution.md` (new) - consume the personal CSV schema, paper-trade the model portfolio, configurable benchmarks. -9. `dashboard-reporting.md` (new) - Streamlit views per module, MLflow run links, explainability table. -10. `broker-execution.md` (update) - paper trading only, ECS Fargate Spot runtime, idempotent orders. -11. `corroborative-signals.md` (update) - light pass; deferred to a later iteration if needed. -12. `unstructured-financial-data.md` (update) - light pass; deferred to a later iteration if needed. - -## Architecture acceptance criteria - -- Every module has its own spec under `docs/mvp/features/`. -- Every spec includes objective, scope, inputs, outputs, flow, Mermaid diagram, open questions, and acceptance criteria. -- The architecture can run the pipeline end to end without a live broker. -- The architecture separates raw data, normalized data, point-in-time queries, scoring, portfolio construction, monitoring, backtesting, and orders. -- Investment decisions, exclusions, and sell signals can each be explained from versioned data and rules. -- Every run is logged as an MLflow run (parameters, metrics, artifacts) and / or as a versioned S3 snapshot, with a tag pointing to the git commit SHA. -- New data providers can be added without changing downstream scoring modules. -- The backtesting module can fail a run and block order generation when the strategy's Sharpe does not beat all benchmarks. -- The sell-watch module can emit signals that, after explicit user confirmation, flow into the broker module under the same audit trail as buy decisions. -- The portfolio evolution module can compare the model portfolio (as if traded) and the user's actual personal portfolio against at least one benchmark. -- The Enron / Lehman / WorldCom regression test runs in CI and fails the build if the bankruptcy filter stops flagging them. -- All secrets are sourced from GitHub Secrets (build) and / or AWS Secrets Manager (runtime). No secret is stored in the repo. -- Total MVP AWS bill stays under a small monthly budget (target to be set during infrastructure design). diff --git a/docs/mvp/backlog/backlog.md b/docs/mvp/backlog/backlog.md deleted file mode 100644 index 7a3fc89..0000000 --- a/docs/mvp/backlog/backlog.md +++ /dev/null @@ -1,16 +0,0 @@ -# Product backlog (informal) - -Informal ideas from the product owner. Each item should eventually map to one or more [feature specs](features/) and [architecture.md](../architecture/architecture.md). Notion tasks should reference the relevant spec path. - -| # | Idea | Likely MVP feature(s) | -| --- | --- | --- | -| 1 | Detect financial problems with holdings in my portfolio | [sell-watch.md](features/sell-watch.md), [permanent-loss-filter.md](features/permanent-loss-filter.md), [dashboard-reporting.md](features/dashboard-reporting.md) | -| 2 | Track portfolio evolution over time and compare to benchmarks (e.g. S&P 500) | [dashboard-reporting.md](features/dashboard-reporting.md), [backtesting.md](features/backtesting.md) | -| 3 | AI-assisted detector for undervalued stocks (Peter Lynch filters, Magic Formula) | [cheap-stocks.md](features/cheap-stocks.md), [high-quality-stocks.md](features/high-quality-stocks.md), [universe-construction.md](features/universe-construction.md) | -| 4 | Diversification analysis beyond sector labels (clustering in growth vs contraction regimes) | [corroborative-signals.md](features/corroborative-signals.md), [dashboard-reporting.md](features/dashboard-reporting.md) — may need a future spec | -| 5 | Detect overvalued positions where selling or trimming may make sense | [sell-watch.md](features/sell-watch.md), [cheap-stocks.md](features/cheap-stocks.md) | -| 6 | Rebalancing guidance (owner questions whether rebalance fits buy-cheap / sell-dear philosophy) | [architecture.md](../architecture/architecture.md) (annual rebalance decision), [broker-execution.md](features/broker-execution.md) | - -## Priority - -Ordering is not fixed here. During MVP planning, promote items into feature specs with acceptance criteria before implementation. diff --git a/docs/mvp/demo-slice.md b/docs/mvp/demo-slice.md deleted file mode 100644 index 0572334..0000000 --- a/docs/mvp/demo-slice.md +++ /dev/null @@ -1,98 +0,0 @@ -# June 30 demo slice — simplest Magic Formula - -**Status:** accepted (see [ADR-0002](../adr/0002-june-demo-scope-cut.md)) -**Target date:** 2026-06-30 -**North star:** Full MVP in [`architecture/architecture.md`](architecture/architecture.md) — this document defines only what ships first. - -## Objective - -Deliver a working, explainable Greenblatt-style Magic Formula pipeline on real US data: ingest fundamentals, rank the market, build a model portfolio, show results in Streamlit. No historical validation or execution in this slice. - -## In scope - -| Step | Module / spec | Notes | -| --- | --- | --- | -| 1 | SimFin ETL | [`etl-data-lake.md`](features/etl-data-lake.md) — bulk US download, raw zone, SimFin normalizer | -| 2 | Universe | [`universe-construction.md`](features/universe-construction.md) — demo mode: SimFin US minus sector exclusions | -| 3 | Quality | [`high-quality-stocks.md`](features/high-quality-stocks.md) — ROC | -| 4 | Cheapness | [`cheap-stocks.md`](features/cheap-stocks.md) — Earnings Yield | -| 5 | Ranking | Combined rank = ROC rank + EY rank (lower is better); tie-break ascending market cap | -| 6 | Model portfolio | Top **30** names, **equal-weight** only | -| 7 | Dashboard | [`dashboard-reporting.md`](features/dashboard-reporting.md) — ranking table, portfolio, per-name explainability | -| 8 | MLflow | Log each pipeline run (params, scoring metrics, portfolio artifact) — `src/smartwealthai/mlflow_run_logging.py`, wired in `score-universe` ([#61](https://github.com/JLaborda/SmartWealthAI/issues/61)) | - -## Out of scope (phase 2) - -- Permanent loss filter ([`permanent-loss-filter.md`](features/permanent-loss-filter.md)) -- Backtesting and crisis report ([`backtesting.md`](features/backtesting.md)) -- Sell-watch ([`sell-watch.md`](features/sell-watch.md)) -- Paper trading / broker ([`broker-execution.md`](features/broker-execution.md)) -- Watchlist (ranking table in dashboard is enough) -- Corroborative signals, unstructured data, portfolio evolution (personal CSV) -- SEC EDGAR ETL (frozen spike remains in repo — [ADR-0001](../adr/0001-simfin-fundamentals-mvp.md)) -- Historical S&P 500 universe with delisted names -- Score-weighted and risk-parity weighting - -## Data sources - -| Data | Source | Tier | -| --- | --- | --- | -| Fundamentals | SimFin bulk (`income` TTM, `balance` quarterly, `cashflow` TTM, `companies`, `industries`) | Free | -| Prices (demo) | SimFin bulk `shareprices/latest` | Free; same ticker namespace as universe | -| Prices (phase 2) | SimFin `shareprices/daily` or vendor fallback | Backtest and personal NAV | -| Industry exclusions | `data/reference/simfin_industry_exclusions.csv` + bank/insurance dataset sanity check | Versioned CSV | - -## Pipeline diagram - -```mermaid -flowchart LR - SimFin["SimFin bulk US"] --> Raw["raw/simfin/"] - Raw --> Norm["SimFin normalizer"] - Raw --> SharePx["shareprices/latest"] - Norm --> Fund["curated/fundamentals"] - SharePx --> Prices["curated/prices"] - Companies["SimFin companies + industries"] --> Uni["Universe (US − exclusions)"] - Fund --> Uni - Uni --> ROC["ROC rank"] - Uni --> EY["EY rank"] - Prices --> ROC - Prices --> EY - Fund --> ROC - Fund --> EY - ROC --> Rank["Combined rank"] - EY --> Rank - Rank --> Port["Top 30 EW portfolio"] - Port --> Dash["Streamlit dashboard"] - Port --> MLflow["MLflow run"] -``` - -## Key decisions (closed for demo) - -| Topic | Decision | -| --- | --- | -| Backfill | Full SimFin US bulk per dataset variant; normalizer filters to universe tickers | -| `as_of_date` | SimFin `Publish Date`; restatements → new `version_id` with `Restated Date`; missing publish → `Report Date + lag` → review queue | -| Fundamentals periodicity | Income/cashflow **TTM**; balance sheet **quarterly** (latest PIT row) | -| Raw vs curated | Raw verbatim for downloaded variants; curated minimal (provider-agnostic schema) | -| Universe | All SimFin `market=us` companies minus banks/insurers/utilities (`IndustryId` CSV + bank/insurance sanity check) | -| Share prices | SimFin `shareprices/latest`; `price_date` may lag `run_date` by ~30 days (free tier) — OK for demo | -| Portfolio | Top 30, equal-weight, market-cap tie-break on ranks | -| SEC code | Frozen, not called by demo pipeline | - -## Acceptance criteria - -- [x] One command (or Prefect flow) runs the full demo pipeline for a `run_date`. -- [x] Dashboard shows combined rank, ROC/EY inputs, and top-30 portfolio with explanations. -- [ ] Every curated fundamental row has `as_of_date <= run_date` when queried PIT. -- [ ] No bank/insurer/utility from the exclusion list appears in the ranked universe. -- [x] MLflow run exists with portfolio parquet artifact and git commit SHA tag. -- [x] Hermetic CI tests do not call SimFin or yfinance live. - -## After the demo (phase 2 order) - -1. Historical S&P 500 universe + permanent loss filter -2. Minimal backtest (annual rebalance, 20 years) — custom pandas/DuckDB loop, not Zipline -3. Walk-forward, Monte Carlo, benchmark gate -4. Sell-watch + paper trading -5. SEC EDGAR normalizer (optional PIT upgrade) -6. Quantitative Value metrics (multi-period fundamentals from raw SimFin archives) diff --git a/docs/mvp/features/backtesting.md b/docs/mvp/features/backtesting.md deleted file mode 100644 index b56f379..0000000 --- a/docs/mvp/features/backtesting.md +++ /dev/null @@ -1,162 +0,0 @@ -# Feature: Backtesting and Crisis Report - -## Implementation status - -**Deferred** for the June 30 demo slice ([ADR-0002](../../adr/0002-june-demo-scope-cut.md)). Spec remains the target for phase 2. - -## Objective - -Validate the strategy on at least 20 years of point-in-time data before any paper trading order is generated. Backtests must be reproducible, point-in-time correct, free of survivorship bias, and tracked end-to-end via MLflow. A backtest run that does not beat all benchmarks on Sharpe blocks order generation for that configuration. - -## MVP scope - -- Walk-forward backtest with 3 to 5 year train / validation windows. -- Annual rebalancing (matches production). -- Long-only, 15 to 30 names, max 10% per name (matches production). -- Weighting (equal-weight, score-weighted, risk-parity) as a hyperparameter; the winning weighting on validation Sharpe is used in production. -- Block-bootstrap Monte Carlo to generate alternate histories from the same data. -- Benchmark suite: S&P 500 cap-weighted, S&P 500 equal-weighted, Russell 3000, Greenblatt Magic Formula canonical replica. -- Crisis drawdown report (dotcom 2000-2002, GFC 2008-2009, COVID 2020, 2022 rate shock). Drawdowns reported, not gated. -- MLflow run per backtest with parameters, metrics, and artifacts. - -## Out of MVP scope - -- Transaction costs, slippage, bid-ask spreads. -- Capital-gains taxes. -- Currency hedging. -- Synthetic data via generative models (GAN / diffusion / VAR). -- Live optimization of the rebalance frequency (fixed annual for the MVP). -- Short positions, leverage. - -## Inputs - -| Input | Source | -| --- | --- | -| PIT fundamentals | `curated/fundamentals` | -| Adjusted prices | `curated/prices` | -| Historical universe per run date | `curated/universe` (rebuilt for each rebalance date) | -| Permanent loss exclusions | `curated/permanent_loss` | -| Quality and cheapness scores | `curated/scores/quality` + `curated/scores/cheap` (recomputed inside the backtest with PIT inputs) | -| Benchmarks reference | `data/reference/benchmarks/` with constituents of S&P 500 CW/EW and Russell 3000 over time; Magic Formula portfolio is rebuilt at every rebalance from the same universe and PIT data | -| Backtest config | `config/backtest/.yaml` (parameters: start date, end date, rebalance, weighting, MC settings, walk-forward windows) | - -## Outputs - -All outputs live as MLflow run artifacts under the `backtesting` experiment. - -| Output | Description | -| --- | --- | -| Equity curves parquet | Daily NAV per backtest variant, per benchmark | -| Trade ledger parquet | Every simulated trade at each rebalance | -| Holdings parquet | Holdings per name per date | -| Metrics dictionary | Sharpe, CAGR, max drawdown, turnover, hit rate, alpha vs each benchmark, in-sample vs out-of-sample Sharpe gap | -| Crisis report HTML | Drawdown per named crisis vs each benchmark | -| Monte Carlo distribution parquet | Per-trial returns, Sharpe distribution, drawdown distribution | -| Pass / fail flag | `True` only if strategy Sharpe strictly beats every benchmark | -| MLflow tags | `git_sha`, `config_hash`, `pit_data_hash` | - -## Pass criterion - -The backtest passes (and therefore allows production paper orders) only if all of the following hold on the out-of-sample window: - -- `Sharpe(strategy) > Sharpe(S&P 500 CW)` -- `Sharpe(strategy) > Sharpe(S&P 500 EW)` -- `Sharpe(strategy) > Sharpe(Russell 3000)` -- `Sharpe(strategy) > Sharpe(Magic Formula canonical replica)` - -Additionally, the **in-sample vs out-of-sample Sharpe gap** is logged. A gap larger than a configurable threshold (default: in-sample Sharpe more than 50% above out-of-sample Sharpe) flips a `overfit_risk` flag in the MLflow metrics. The flag does not block by itself, but the run is highlighted in the dashboard. - -Crisis windows have drawdowns reported but do not gate the run for the MVP. - -## Walk-forward design - -- Start date: backtest config (default: 20 years before the run date). -- End date: most recent year with full data, leaving the last 1 to 2 years as a frozen hold-out. -- Window length: 3 to 5 years (parameter). Hyperparameters (weighting, score thresholds) are tuned on the first part of each window, validated on the second. -- Rolling step: 1 year forward. -- Hold-out: an explicit final window the strategy never sees during tuning. Reported separately. - -## Monte Carlo (block bootstrap) - -- Resample contiguous blocks of `block_length` months (default 12) from the historical multi-asset return panel. -- Trial count `n_trials` (default 500). -- Same strategy logic runs on each synthetic path. -- Logged outputs: Sharpe distribution, max drawdown distribution, 5th / 50th / 95th percentile equity curves. -- Block bootstrap preserves the joint distribution of prices and fundamentals because both come from the same sampled blocks. - -## Mermaid diagram - -```mermaid -flowchart TD - Config["config/backtest/.yaml"] --> Engine["Backtest engine"] - PIT["curated/fundamentals (PIT)"] --> Engine - Prices["curated/prices"] --> Engine - UniverseHist["curated/universe (per rebalance date)"] --> Engine - PLoss["curated/permanent_loss (per rebalance date)"] --> Engine - - Engine --> Rebalance["For each rebalance date"] - Rebalance --> Scoring["Recompute ROC + EY (PIT)"] - Scoring --> RankBuild["Combined rank + portfolio construction"] - RankBuild --> Holdings["Holdings parquet"] - Holdings --> NAV["Daily NAV"] - - NAV --> Metrics["Sharpe / CAGR / drawdown / turnover / alpha"] - Metrics --> Benchmarks["vs S&P 500 CW + EW + Russell 3000 + Magic Formula"] - Benchmarks --> Pass{"Sharpe > all benchmarks?"} - Pass -->|Yes| PassFlag["pass = True"] - Pass -->|No| FailFlag["pass = False"] - - Engine --> MC["Block-bootstrap Monte Carlo"] - MC --> MCDist["Sharpe and drawdown distributions"] - - Metrics --> CrisisReport["Crisis drawdown report"] - - PassFlag --> MLflow["MLflow run"] - FailFlag --> MLflow - MCDist --> MLflow - CrisisReport --> MLflow -``` - -## Expected flow - -1. The pipeline triggers a backtest when the configuration changes, when a new release is built, or on a manual request from the dashboard. -2. The engine reads the config and resolves the start / end dates and walk-forward windows. -3. For each rebalance date (annually, starting at `start_date`): - 1. Build the universe via `universe-construction` at that date. - 2. Apply the permanent loss filter (PIT). - 3. Recompute ROC and EY on PIT fundamentals. - 4. Combine ranks, apply tie-break, select top 15 to 30 names with 10% cap. - 5. Compose the holding using the weighting variant under test. -4. Simulate daily NAV from rebalance to rebalance using adjusted prices. -5. Compute metrics overall, per walk-forward window, and per crisis window. -6. Run the same logic against the benchmark constructions to produce comparable Sharpe / drawdown. -7. Run the Monte Carlo block-bootstrap loop. -8. Decide pass / fail. Log everything to MLflow. - -## Acceptance criteria - -- A backtest run with the same config and the same PIT data hash produces metrics within numerical tolerance across two executions. -- Every backtest is logged as an MLflow run under the `backtesting` experiment with the `git_sha`, `config_hash`, and `pit_data_hash` tags. -- The engine never reads data more recent than the rebalance date during simulation; an automated check verifies this (e.g., max `as_of_date` per slice equals the rebalance date). -- The Magic Formula benchmark is built from the same universe and rebalanced annually like the strategy. -- The hold-out window is reported separately and is never touched by hyperparameter tuning. -- The crisis report includes at minimum dotcom 2000-2002, GFC 2008-2009, COVID 2020, 2022 rate shock. -- The pass flag is `True` only if the strategy Sharpe strictly beats every benchmark Sharpe. -- The Monte Carlo distribution has at least 500 trials by default. -- Production order generation reads the pass flag of the most recent backtest before emitting any order. - -## Open questions - -- Should we use the full S&P 500 CW total-return series as one benchmark and an equal-weight backtest of the same survivors as another? Recommendation: use a published total-return index (e.g., SPX TR via `^SP500TR` or a stitched series) for CW; build the EW from the historical constituents we already have. -- For Russell 3000 historical constituents we do not have a free reliable source. Recommendation: pin the Russell 3000 total-return index as a price series only (no constituent rebuild) for the MVP and revisit if it becomes the bottleneck. -- For Magic Formula benchmark, do we constrain it to the same exclusions (no banks / insurers / utilities) as the strategy, or run it on the full common-stock universe? Recommendation: same exclusions, so the comparison is fair to the strategy's universe. -- What is the right `block_length` for Monte Carlo: 6 or 12 months? Recommendation: 12 to capture annual cyclicality; expose as a parameter. -- Should we emit a pass / fail per weighting variant separately, or only for the winning variant? Recommendation: log every variant; promote only the winner. - -## Risks - -- Backtests on free data can have subtle PIT errors (missing restatements, late filings). The `as_of_date` check above is the main defense. -- Survivorship bias is removed by the historical universe but can creep back through benchmarks that only include current survivors. The benchmark sources matter. -- Monte Carlo block bootstrap underestimates extreme tails because it cannot generate scenarios outside historical experience. Documented as a known limitation. -- A strategy that beats all benchmarks on Sharpe can still be unstable in a single bad year. The crisis drawdown report is the user-facing warning. -- Without modeling transaction costs, the strategy looks better than it would in real life. Documented; revisit before any live trading. diff --git a/docs/mvp/features/broker-execution.md b/docs/mvp/features/broker-execution.md deleted file mode 100644 index 94260ad..0000000 --- a/docs/mvp/features/broker-execution.md +++ /dev/null @@ -1,127 +0,0 @@ -# Feature: Broker Execution - -## Implementation status - -**Deferred** for the June 30 demo slice ([ADR-0002](../../adr/0002-june-demo-scope-cut.md)). Spec remains the target for phase 2. - -## Objective - -Convert the model portfolio's target positions, plus confirmed sell-watch signals, into broker-compatible orders. For the MVP, every order is **paper-traded** in a simulator. Real broker connectivity is out of scope until the user explicitly opts in. - -## MVP scope - -- Translate target weights (from `portfolio-construction`) into target share counts using today's closing price. -- Compare target positions against the current paper book to compute required trades. -- Accept confirmed sell-watch signals (status `confirmed` in `curated/sell_watch/confirmations.parquet`) as forced sells. -- Run pre-trade risk checks (see "Pre-trade risk checks" below) and reject orders that violate them. -- Submit accepted orders to the internal paper trading simulator. -- Track order lifecycle (`proposed`, `accepted`, `rejected`, `submitted`, `filled`, `partially_filled`, `cancelled`). -- Reconcile the paper book against the expected target positions after fills. -- Idempotent: a given `(run_date, ticker, side, quantity, source)` tuple cannot produce two orders. -- Runs on ECS Fargate Spot once per trading day, after the daily pipeline finishes. - -## Out of MVP scope - -- Real broker connectivity (Alpaca, Interactive Brokers, etc.). -- Real money. -- Smart order routing. -- Limit orders, stop orders, options, futures, margin. -- Fractional-share semantics beyond what the simulator supports (the simulator accepts fractional volume to mirror the personal CSV). -- Tax-aware lot selection (FIFO at aggregate level). - -## Inputs - -| Input | Source | -| --- | --- | -| Target model portfolio | `curated/portfolio/model/holdings.parquet` (today's run) | -| Current paper book | `curated/broker/paper_book.parquet` | -| Confirmed sell signals | `curated/sell_watch/confirmations.parquet` (only `status = confirmed`) | -| Backtest pass flag | Latest MLflow run in `backtesting` experiment must have `pass = True` | -| Cash balance | `curated/broker/cash.parquet` | -| Today's adjusted close | `curated/prices` | -| Run date | Pipeline parameter | -| Broker config | `config/broker.yaml` (mode = `paper`, cash floor, per-order limit, daily turnover cap) | - -## Outputs - -| Output | Path / target | -| --- | --- | -| Proposed orders | `curated/broker/proposed_orders.parquet` | -| Risk-check log | `curated/broker/risk_checks.parquet` | -| Submitted orders + fills | `curated/broker/orders.parquet` (lifecycle) | -| Updated paper book | `curated/broker/paper_book.parquet` | -| Updated cash | `curated/broker/cash.parquet` | -| Reconciliation report | `curated/broker/reconciliation.parquet` | -| MLflow run | Parameters (config), metrics (orders proposed / submitted / rejected, turnover, cash before / after), artifacts (per-run report) | - -## Pre-trade risk checks (blocks order generation when any fails) - -| Check | Block? | -| --- | --- | -| Latest backtest `pass = True` | Yes | -| Permanent loss filter has flagged `pass` for every target ticker (not `exclude` or `unknown`) | Yes | -| Full scoring pipeline completed today (quality, cheapness, ranking parquets present) | Yes | -| Data freshness within threshold (every input price has a row dated today or the previous trading day) | Yes | -| No duplicate order in `curated/broker/orders.parquet` for the same `(run_date, ticker, side, quantity, source)` | Yes | -| Cash balance is above the configured floor after the trade | Yes | -| Per-order notional is below the configured per-order limit | Yes | -| Daily turnover (sum of trade notional) is below the configured cap | Yes | -| Position concentration after the trade keeps every name <= 10% of NAV | Yes | - -Any failing check moves the corresponding order to `rejected` with the failure reason. The pipeline continues with the orders that passed. - -## Mermaid diagram - -```mermaid -flowchart TD - Targets["curated/portfolio/model/holdings.parquet"] --> OrderBuilder["Compute required trades"] - PaperBook["curated/broker/paper_book.parquet"] --> OrderBuilder - Confirmed["confirmed sell-watch signals"] --> OrderBuilder - - OrderBuilder --> Proposed["proposed_orders.parquet"] - Proposed --> Checks["Pre-trade risk checks"] - - Checks --> Pass{"All checks pass?"} - Pass -->|No| Rejected["risk_checks.parquet (rejected)"] - Pass -->|Yes| Paper["Paper trading simulator"] - Paper --> Orders["orders.parquet (lifecycle)"] - Orders --> NewBook["Updated paper_book.parquet"] - Orders --> NewCash["Updated cash.parquet"] - NewBook --> Reconcile["reconciliation.parquet"] -``` - -## Expected flow - -1. Load target positions and the current paper book. -2. Compute required trades (target shares - current shares) using today's closing price. -3. Add forced sells for every confirmed (and not yet submitted) sell-watch signal. -4. Run pre-trade risk checks; mark each order `accepted` or `rejected`. -5. For accepted orders, submit them to the paper trading simulator at today's close. The simulator immediately marks them `filled` with the close price as the fill price. -6. Update the paper book and cash balance. -7. Run reconciliation: assert that `paper_book` equals `target_positions` for the accepted set; log any mismatch. -8. Persist all parquet outputs and log MLflow. - -## Acceptance criteria - -- Live execution code paths do not exist in the MVP. The simulator is the only execution backend. -- The pipeline cannot produce two identical orders. The deduplication key is documented and tested. -- Every rejection is logged with the failing check id; rejections never silently drop. -- The reconciliation report flags discrepancies between the paper book and the target positions. -- If the latest backtest `pass = False`, no order is submitted, even for confirmed sell-watch signals. (Sell signals remain confirmed and pending until a passing backtest exists.) -- The MLflow run logs at minimum: orders proposed, orders submitted, orders rejected (per check), and turnover. -- The module runs on ECS Fargate Spot triggered by Prefect, after the daily pipeline. - -## Open questions - -- Should confirmed sells be allowed to execute even when the latest backtest fails? Argument for yes: protecting capital is more important than waiting for a fresh backtest. Recommendation for the MVP: no, to keep the safety boundary clean; mark as open for review once we have data. -- Do we model intraday vs end-of-day fills? Recommendation: end-of-day close only for the MVP. -- Do we charge a simulated commission per trade? Recommendation: no for the MVP (matches backtest assumptions); revisit before any real trading. -- Where does the paper book start? Recommendation: configurable initial cash in `config/broker.yaml`, default 100,000 EUR. -- Does the simulator support fractional shares like the personal CSV? Recommendation: yes; matches the real broker behavior the user already experienced. - -## Risks - -- Once real-broker code exists, it can be enabled by accident. The MVP module should not import any live broker SDK; a separate module behind an explicit feature flag will be added later. -- A bug in deduplication can flood the paper book with phantom positions. The unit tests must cover replays and idempotency. -- Stale prices on a market holiday could produce incorrect fills. The freshness check is the primary defense. -- The user can ignore reconciliation failures. The dashboard surfaces them prominently. diff --git a/docs/mvp/features/cheap-stocks.md b/docs/mvp/features/cheap-stocks.md deleted file mode 100644 index 25fe902..0000000 --- a/docs/mvp/features/cheap-stocks.md +++ /dev/null @@ -1,113 +0,0 @@ -# Feature: Cheap Stocks - -## Implementation status - -done (demo cross-sectional slice) — EY scoring and ranks: `src/smartwealthai/magic_formula_ranking.py`, CLI `score-universe` ([#60](https://github.com/JLaborda/SmartWealthAI/issues/60)). Single-ticker tracer: `magic_formula_metrics.py`, `pit_fundamentals.py`, `compute-metrics` ([#44](https://github.com/JLaborda/SmartWealthAI/issues/44)). - -## Objective - -Score the cheapness of every company that survives the universe filter, the permanent loss filter, and the quality scoring step. For the MVP, cheapness is a strict Greenblatt-style **Earnings Yield (EY) = EBIT / Enterprise Value**. Future iterations can plug additional valuation signals (FCF yield, EV/EBITDA, shareholder yield) through the same interface. - -## MVP scope - -- Compute `EY = EBIT / EV` per the canonical Greenblatt definition. -- Use the most recent point-in-time fundamentals available on the decision date and the run-date market data for EV. -- Produce a cross-sectional cheapness rank (lower rank = cheaper) for every passing company. -- Validate denominators: rows with `EV <= 0` or `EBIT` missing are flagged for review and excluded from the ranking. -- Avoid blindly ranking value traps as attractive: rows with negative EBIT are routed to the review queue rather than being inverted into "expensive". -- Log MLflow metrics (count valid, count invalid, EY quantiles). -- Expose the inputs alongside the score so the dashboard can explain "why is this company cheap?". - -## Out of MVP scope - -- Multi-metric cheapness score (FCF yield, EV/EBITDA, P/B, P/S, shareholder yield). -- Sector-relative valuation. -- Value-trap defense beyond the permanent loss filter (no quality threshold required to enter the cheapness rank; quality is a separate, parallel rank that combines later). -- Cyclically adjusted earnings. -- Forward-looking estimates. - -## Inputs - -| Input | Source | Notes | -| --- | --- | --- | -| Passing universe + permanent loss filter pass list | `curated/universe` + `curated/permanent_loss` | Only `pass` rows are scored. | -| PIT fundamentals (income statement, balance sheet) | `curated/fundamentals` | Filtered by `as_of_date <= run_date`. | -| Run-date market cap | `curated/prices/run_date=/prices.parquet` join `curated/fundamentals` | `shares_outstanding * adj_close`; `price_date` is the latest trading day ≤ `run_date`. | -| Enterprise value components | `curated/fundamentals` | Total debt, preferred equity, minority interest, cash. | -| Run date | Pipeline parameter | | -| EY formula version | `config/cheap/ey.yaml` | Versioned. | - -## EY definition (canonical Greenblatt) - -``` -EY = EBIT / Enterprise Value -EV = Market Cap + Total Debt + Preferred Equity + Minority Interest - Cash and Equivalents -``` - -with: - -- `EBIT` = Operating income before interest and taxes. Trailing twelve months. Same definition as in `high-quality-stocks.md` so both scores share the same `EBIT`. -- `Market Cap` = `shares_outstanding * close` on `run_date`. -- All other components from the latest filing whose `as_of_date <= run_date`. - -The formula and its variants are versioned in `config/cheap/ey.yaml`. Any change requires a new version id so backtests on prior versions remain reproducible. - -## Outputs - -| Output | Path / target | -| --- | --- | -| Cheapness scores parquet | `curated/scores/cheap/run_date=/scores.parquet` with `cik, ticker, ebit, market_cap, total_debt, preferred_equity, minority_interest, cash, ev, ey, ey_rank, formula_version, as_of_date` | -| Review queue rows | `curated/issues/run_date=/cheap.parquet` for invalid denominators, negative EBIT, missing inputs | -| MLflow metrics | `cheap_n_valid`, `cheap_n_invalid`, EY quantiles | - -## Mermaid diagram - -```mermaid -flowchart TD - Passing["Universe pass + Permanent loss pass"] --> Loader["Load PIT fundamentals + market cap"] - Loader --> Components["EBIT, Market Cap, Total Debt, Preferred Equity, Minority Interest, Cash"] - Components --> EV["Compute EV"] - EV --> Validate{"EV > 0 and EBIT present?"} - - Validate -->|No| Review["Review queue (cheap.parquet)"] - Validate -->|Yes| EBITSign{"EBIT > 0?"} - - EBITSign -->|No| Review - EBITSign -->|Yes| EY["EY = EBIT / EV"] - EY --> Rank["Cross-sectional rank (descending EY)"] - Rank --> Output["cheap/scores.parquet"] - Output --> MLflow["MLflow metrics"] -``` - -## Expected flow - -1. Load the passing universe and join with PIT fundamentals + market cap. -2. Compute Enterprise Value from `market_cap + total_debt + preferred_equity + minority_interest - cash`. -3. Validate inputs: missing component rows are dropped; `EV <= 0` and `EBIT <= 0` rows are flagged for review. -4. Compute `EY`. -5. Produce a cross-sectional rank from highest EY (rank 1) to lowest. -6. Persist parquet and log MLflow metrics. -7. The combined Greenblatt rank (`roc_rank + ey_rank`) is built downstream by `portfolio-construction` (inside the same pipeline). Tie-break by market cap from `high-quality-stocks` carries over. - -## Acceptance criteria - -- Same `(universe, run_date, formula_version)` produces byte-identical output (hash-verifiable). -- The score is purely a function of curated PIT data; no network calls. -- Every row has the EY value and all components that produced it. -- Rows with invalid EV or negative EBIT are visible in the review queue and not silently flipped to "expensive". -- The formula version travels with each scored row. -- The MLflow run logs at minimum count of valid rows, count of invalid rows, EY median, and EY quantiles. - -## Open questions - -- For Enterprise Value, do we use `Long Term Debt + Short Term Debt + Capital Lease Obligations` for `Total Debt`, or a narrower definition? Recommendation: include capital leases under `Total Debt` and document as `formula_version = v1`. -- ~~Cash definition: `CashAndCashEquivalents` only, or `CashAndCashEquivalents + ShortTermInvestments`?~~ **Closed (v1):** include short-term investments — SimFin column `Cash, Cash Equivalents & Short Term Investments` maps to curated `cash`. -- Preferred equity: use book value or market value? Recommendation: book value (market is rarely available for free). -- Should the EY rank skip companies that fail to score on quality (i.e., invalid ROC denominator)? Recommendation: no; keep the two ranks independent so the combined score only excludes a name when both fail. - -## Risks - -- Negative-EBIT companies are silent value traps that easy EY models can mislabel as attractive. The MVP routes them to the review queue, which avoids the trap but may exclude legitimate turnarounds. -- One-off items in EBIT distort EY. Same caveat as in `high-quality-stocks`; accepted as a Greenblatt placeholder. -- `Total Debt` reported by EDGAR has multiple equally defensible definitions. Version locking is the only durable mitigation. -- Restated balance sheet items shift EV across versions. PIT versioning handles it; the test suite must cover restatements explicitly. diff --git a/docs/mvp/features/corroborative-signals.md b/docs/mvp/features/corroborative-signals.md deleted file mode 100644 index 6c2e483..0000000 --- a/docs/mvp/features/corroborative-signals.md +++ /dev/null @@ -1,86 +0,0 @@ -# Feature: Corroborative Signals - -> **Status for the MVP: deferred.** No corroborative signal is computed or used in the first version of the pipeline. The architecture leaves a slot for this module in the ranking diagram, but the MVP combined rank is exactly `quality_rank + cheapness_rank` (Greenblatt). This spec captures the design we will revisit once the MVP is validated. - -## Objective - -Add signals that corroborate or challenge the core quality and cheapness ranking. These signals are auxiliary; they never overrule the permanent loss filter and they should not dominate the value framework. - -## Out of MVP scope (entire module) - -The whole module is parked. The first iteration of the system runs without any corroborative input. The reasons: - -- The Greenblatt placeholder is intentionally minimal until the user finishes reading *Quantitative Value* and chooses the next factors. -- Insider transaction data, short interest data, and institutional ownership data all require either paid feeds or fragile scraping. The MVP's "free data only" constraint makes this hard to deliver reliably. -- A noisy corroborative score on top of a placeholder Greenblatt rank is more likely to hurt than help. - -## Candidate signals (future iterations) - -Listed for memory. None of them is implemented now. - -### Shareholder return - -- Net buyback yield. -- Share count reduction over time. -- Dividend yield. -- Dividend growth. -- Total shareholder yield. - -### Insider activity - -- Insider buying by executives or directors. -- Cluster buying. -- Insider selling after large price appreciation. -- Insider ownership level. - -### Market and ownership context - -- Short interest. -- Institutional ownership changes. -- Activist involvement. - -### Corporate events - -- Spin-offs. -- Tender offers. -- Debt refinancing. -- Management changes. - -## Mermaid diagram (future state) - -```mermaid -flowchart TD - CuratedData["Curated financial data"] --> Buybacks["Buyback signals"] - InsiderData["Insider transactions"] --> Insiders["Insider signals"] - MarketData["Market / ownership data"] --> Ownership["Ownership and short interest"] - Events["Corporate actions"] --> EventSignals["Event signals"] - - Buybacks --> SignalAggregator["Signal aggregator"] - Insiders --> SignalAggregator - Ownership --> SignalAggregator - EventSignals --> SignalAggregator - - SignalAggregator --> Adjustment["Adjustment to combined rank"] - SignalAggregator --> Flags["Review flags"] -``` - -## Open questions (for the iteration when this module is reactivated) - -- Which corroborative signal should be included first? Likely candidates: shareholder yield (gettable from EDGAR + prices for free) and SEC Form 4 insider transactions (also free). -- Should corroborative signals adjust the combined rank, or only appear as flags? -- How large must a buyback be to matter, and how do we strip out stock-based-compensation noise? -- How do we avoid double-counting dividends in cheapness and in shareholder yield? -- Should corroborative signals ever override a permanent loss exclusion? Recommendation locked: **no**. - -## Acceptance criteria for the future module - -- The module is opt-in via configuration; the MVP default keeps it off. -- It can be added without changing the ETL or the scoring modules: it consumes curated parquet and writes its own parquet. -- Its contribution to the combined rank is explicit, bounded, and documented. -- Missing signal data does not silently penalize a company. - -## Risks (future) - -- Insider data quality varies. Form 4 filings are reliable; aggregator interpretations are not. -- Short interest is reported with significant delay; pretending it is real-time invites bias. -- Buybacks can be value-destructive at high prices. The signal must condition on cheapness, not just on the existence of a buyback. diff --git a/docs/mvp/features/dashboard-reporting.md b/docs/mvp/features/dashboard-reporting.md deleted file mode 100644 index 7fbb43e..0000000 --- a/docs/mvp/features/dashboard-reporting.md +++ /dev/null @@ -1,135 +0,0 @@ -# Feature: Dashboard and Reporting - -## Implementation status - -**done** — demo slice three-page app ([#62](https://github.com/JLaborda/SmartWealthAI/issues/62)); full MVP pages in phase 2. - -### Demo run - -```bash -# After score-universe (or full demo pipeline) for a run_date: -poetry run run-dashboard --data-dir data --run-date 2026-06-18 -``` - -Environment variables: `SMARTWEALTHAI_DATA_DIR` (lake root, default `data`), `SMARTWEALTHAI_RUN_DATE` (optional override). - -Code: `apps/dashboard/` (Streamlit UI), `src/smartwealthai/dashboard_data.py` (parquet readers), `tests/test_dashboard_data.py`. - -## Objective - -Surface every input, score, decision, and audit trail produced by the pipeline in a single Streamlit dashboard. The dashboard is the primary product surface for the user. Visual polish is explicitly deferred; functional completeness comes first. - -## MVP scope - -### Demo slice (June 30) - -- Streamlit app (local or lightweight AWS deploy). -- Pages: **Overview**, **ETL & data quality**, **Universe**, **Quality (ROC)**, **Cheapness (EY)**, **Ranking + model portfolio** (top 30 EW). -- Per-name explainability: ROC/EY inputs and combined rank. -- MLflow run link per pipeline execution. -- Read-only views (no sell-watch confirmation in demo). - -### Full MVP (phase 2) - -- Streamlit app deployed on AWS (likely Fargate Spot behind an ALB, or App Runner if cheaper at MVP scale). -- Additional pages: permanent loss filter, sell-watch, backtests, portfolio evolution. -- Sell-watch confirmation writes back to `curated/sell_watch/confirmations.parquet`. -- Authentication: simple username + password from AWS Secrets Manager for the MVP (or Cognito if the cheapest path is similar). -- Reports rendered as HTML inside Streamlit and persisted as static HTML snapshots in S3 per run date. - -## Out of MVP scope - -- Custom domain + TLS beyond what App Runner / ALB provides by default. -- Multi-user workspaces, roles, or permissions. -- Real-time websocket updates (page refreshes are enough for a daily cadence). -- LLM-generated narratives. -- Mobile-optimized layouts. -- Editable model parameters from the UI (those live in YAML, version-controlled). - -## Inputs - -| Input | Source | -| --- | --- | -| Curated parquet from every module | `s3://smartwealthai-data-lake/curated/...` | -| MLflow tracking server | `http://:5000` | -| Latest run metadata | MLflow `latest_versions` per experiment | -| Sell-watch state | `curated/sell_watch/` | -| Personal and model NAV | `curated/portfolio_evolution/` | -| User credentials | AWS Secrets Manager | - -## Pages - -### Demo slice - -| Page | What it shows | -| --- | --- | -| **Overview** | Latest pipeline run timestamp, model portfolio headline, link to MLflow run. | -| **ETL & data quality** | Last successful SimFin / yfinance ingestion, freshness, review-queue count. | -| **Universe** | Today's universe and exclusion log (sector + bank/insurance sanity). | -| **Quality** | ROC distribution + top / bottom names + per-name component breakdown. | -| **Cheapness** | EY distribution + top / bottom names + per-name component breakdown. | -| **Ranking + model portfolio** | Combined Greenblatt rank with tie-break; top **30** equal-weight holdings. | -| **MLflow links** | Direct links to runs by date and `git_sha` tag. | - -### Full MVP (phase 2) - -| Page | What it shows | -| --- | --- | -| **Overview** | Headline KPIs (latest backtest Sharpe pass/fail, model NAV, personal NAV, open sell signals), latest pipeline run timestamp, links to MLflow runs. | -| **Permanent loss** | Today's exclusions with rule and value; trend line of count of exclusions over time; CI status of the Enron / Lehman / WorldCom regression. | -| **Sell-watch** | Open signals (proposed), confirmed history, dismissed history; each signal has a confirm and dismiss button. | -| **Backtests** | Equity curves, Sharpe table, crisis drawdown, Monte Carlo distribution, overfit flag, pass/fail. | -| **Portfolio evolution** | Personal NAV, model paper NAV, benchmark overlays, drawdown, rolling Sharpe, attribution. | - -All pages (demo and full MVP): every score row shows input components and rules that fired; every page links to the MLflow run id that produced the displayed data. - -## Mermaid diagram - -```mermaid -flowchart LR - User["User browser"] --> Streamlit["Streamlit app (Fargate Spot)"] - Streamlit --> S3["S3 (curated parquet + reports)"] - Streamlit --> MLflow["MLflow tracking server"] - Streamlit --> SES["AWS SES (confirm action triggers email follow-up)"] - Streamlit --> SellWatch["curated/sell_watch/confirmations.parquet"] - SellWatch --> Orders["Order builder (paper)"] -``` - -## Expected flow - -1. The user signs in. -2. The Streamlit app loads parquet directly from S3 via DuckDB for speed. -3. Each page queries the curated zone and renders the relevant tables and plots. -4. The sell-watch page lists `proposed` signals; clicking confirm or dismiss writes back to `confirmations.parquet` and the broker module picks up confirmed signals on its next run. -5. Each page footer shows the MLflow run id and a link. - -## Acceptance criteria - -### Demo slice - -- [x] Dashboard shows combined rank, ROC/EY inputs, and top-30 equal-weight portfolio with explanations. -- [x] Dashboard is readable from cached parquet; no live SimFin or yfinance calls for display. -- [x] Every numeric score traces to a curated parquet row. -- [x] Dashboard renders correctly when curated parquet for a module is missing (clear empty state). - -### Full MVP (phase 2) - -- The dashboard is fully readable from cached parquet; no network calls to the live pipeline are made for display. -- Every numeric score on the dashboard can be traced to a row in a curated parquet file. -- Sell-watch confirmation is the only write operation triggered by the dashboard. -- The static HTML snapshot per run date is stored in `s3://smartwealthai-reports/run_date=/index.html` and is browsable. -- Authentication blocks unauthenticated access. -- The dashboard build is published from GitHub Actions to ECR and deployed to Fargate Spot. - -## Open questions - -- Hosting choice: Fargate Spot behind ALB, App Runner, or even a small EC2 with Caddy. Recommendation: App Runner if the price difference at the MVP scale is small; otherwise Fargate Spot. -- Custom domain in the MVP, or just the default AWS URL? Recommendation: default URL for the MVP; custom domain is a follow-up. -- Persisting the HTML snapshot per run date: do we generate it from Streamlit (which is not natively static) or from a small jinja template fed by the same parquet? Recommendation: jinja template; Streamlit is the live interactive surface, the snapshot is the immutable record. - -## Risks - -- Streamlit reloads on each interaction; large parquet datasets must be cached aggressively in memory. -- App Runner / Fargate Spot can be killed mid-session by AWS; the user just refreshes. Documented as acceptable for the MVP. -- Authentication via Streamlit secrets is weak; the deployment must sit behind an AWS auth layer (Cognito, ALB auth, or IAM) before any real user lands on it. -- Sell-watch confirmations from the dashboard must be idempotent; double-clicking confirm should not create two orders. diff --git a/docs/mvp/features/etl-data-lake.md b/docs/mvp/features/etl-data-lake.md deleted file mode 100644 index 58a8c8e..0000000 --- a/docs/mvp/features/etl-data-lake.md +++ /dev/null @@ -1,451 +0,0 @@ -# Feature: ETL and Data Lake - -## Implementation status - -**done** (demo slice) — SimFin connector + normalizer + shareprices snapshot + end-to-end orchestrator `run-demo-pipeline` ([#63](https://github.com/JLaborda/SmartWealthAI/issues/63)). SEC spike frozen ([ADR-0001](../../adr/0001-simfin-fundamentals-mvp.md)). - -## Objective - -Build the module that downloads, validates, normalizes, and stores financial data so that every downstream module (universe construction, scoring, backtesting, sell-watch, portfolio evolution) can rely on a single trustworthy source. The data lake lives on AWS S3 and is queried with DuckDB. Point-in-time correctness and incremental refresh are mandatory. - -## MVP scope - -### Demo slice (June 30 — primary) - -- Ingest US fundamentals from **SimFin** bulk download (`simfin` Python package, free tier). -- Datasets: `companies`, `industries`, `income` (TTM), `balance` (quarterly), `cashflow` (TTM), `shareprices` (`latest`) for `market=us`. -- Store SimFin bulk responses verbatim under `raw/simfin/`. -- Normalize into provider-agnostic `curated/fundamentals` (same schema scoring modules expect). -- Point-in-time: `as_of_date` = SimFin `Publish Date`; restatements via `Restated Date` + new `version_id`. -- Build run-date prices from SimFin bulk `shareprices/latest` joined to the universe (one row per ticker). -- Run data quality checks; failing rows → review queue. -- Weekly bulk refresh on free tier (`refresh_days=7`); incremental normalize by publish-date watermark. -- DuckDB views on curated parquet. - -### Full MVP (phase 2 additions) - -- SEC EDGAR ETL (frozen spike: `sec_client`, `download-fundamentals`). -- Incremental per-CIK filing ingest when SEC normalizer ships. -- S&P 500–scoped backfill policies. - -## Out of MVP scope - -- Real-time streaming ingestion. -- Paid data providers (Bloomberg, FactSet, CRSP). -- Cross-currency data (only USD-denominated US issuers). -- Dividend history for the personal portfolio (tracked separately; see `portfolio-evolution.md`). -- Cross-region replication or HA setups for S3. - -## Inputs - -| Input | Source | Notes | -| --- | --- | --- | -| Income statement (TTM) | SimFin bulk `income` variant `ttm` | EBIT, interest, revenue, net income. | -| Balance sheet (quarterly) | SimFin bulk `balance` variant `quarterly` | NWC, PP&E, debt, cash, shares. | -| Cash flow (TTM) | SimFin bulk `cashflow` variant `ttm` | Phase 2 permanent-loss filter; ingest in demo for raw archive. | -| Publish / report / restated dates | SimFin statement rows | `Publish Date` → `as_of_date`. | -| Company metadata | SimFin `companies` | `Ticker`, `CIK`, `IndustryId`, `SimFinId`. | -| Industry labels | SimFin `industries` | Sector/industry names for exclusions CSV. | -| Share prices (demo) | SimFin bulk `shareprices` variant `latest` | Run-date close for market cap; same ticker namespace as universe. | -| Share prices (phase 2) | SimFin `shareprices/daily` or vendor fallback | Backtest and personal NAV. | -| Industry exclusions | `data/reference/simfin_industry_exclusions.csv` | Banks, insurers, utilities. | -| Reference ticker map | `data/reference/ticker_mapping.csv` | Broker symbol → yfinance symbol. | -| SEC `companyfacts` (frozen) | SEC EDGAR | Phase 2 only; spike under `raw/sec_edgar/`. | - -## Outputs - -All outputs live under `s3://smartwealthai-data-lake/` and are queryable from DuckDB. - -| Dataset | Path (S3) | Partitioning | Notes | -| --- | --- | --- | --- | -| Raw SimFin bulk | `raw/simfin/dataset=/variant=/market=us/as_of_date=/` | by dataset, variant, download date | Verbatim CSV/ZIP from SimFin bulk API. | -| Raw share prices | `raw/simfin/dataset=shareprices/variant=latest/market=us/as_of_date=/` | by download date | Verbatim SimFin bulk CSV. | -| Raw prices (phase 2) | `raw/yfinance/...` or `shareprices/daily` | by ticker / date | Vendor fallback for backtest. | -| Raw SEC (frozen) | `raw/sec_edgar/cik=/endpoint=companyfacts/...` | by CIK | Phase 2; existing spike layout. | -| Curated fundamentals (PIT) | `curated/fundamentals/cik=/period=/` | by CIK and fiscal period | Provider-agnostic schema; `as_of_date`, `version_id`, `fiscal_period_end`. | -| Curated prices (demo) | `curated/prices/run_date=/prices.parquet` | by run date | One row per universe ticker: `run_date`, `ticker`, `price_date`, `close`, `adj_close`, `volume`. | -| Curated prices (phase 2) | `curated/prices/ticker=/year=/` | by ticker and year | Full daily history for backtest and NAV. | -| Universe history | `curated/universe/run_date=/` | by run date | Built by `universe-construction`. | -| Issue registry | `curated/issues/run_date=/` | by run date | Rows that failed quality checks. | -| yfinance cache | `cache/yfinance///.parquet` | by ticker, endpoint, date | TTL per endpoint. | - -## Mermaid diagram - -```mermaid -flowchart TD - Scheduler["Pipeline run"] --> SimFinConn["SimFin bulk connector"] - - SimFinConn --> RawSF["raw/simfin/ (immutable)"] - RawSF --> SFNorm["SimFin fundamentals normalizer"] - RawSF --> SharePx["shareprices/latest"] - SharePx --> PriceNorm["Demo price snapshot builder"] - Scheduler -. "phase 2" .-> YFConn["yfinance / vendor fallback (cached)"] - YFConn --> Cache["yfinance cache (S3, TTL)"] - Cache --> RawYF["raw/yfinance/"] - RawYF --> PriceHist["Phase 2 price normalizer"] - - SFNorm --> Curated["Curated parquet (S3)"] - PriceNorm --> Curated - PriceHist --> Curated - Curated --> PITStore["PIT store (curated/fundamentals)"] - PITStore --> QC["Data quality checks"] - Curated --> QC - - QC --> ReviewQueue["Review queue (curated/issues)"] - QC --> DuckDB["DuckDB views"] - DuckDB --> Downstream["Downstream modules"] -``` - -## Expected flow (demo) - -1. Download SimFin bulk US datasets (`companies`, `industries`, `income-ttm`, `balance-quarterly`, `cashflow-ttm`, `shareprices-latest`) if older than `refresh_days`. Store verbatim under `raw/simfin/...`. -2. Build universe for `run_date` (see `universe-construction.md`). -3. Join universe tickers to `shareprices/latest`; for each ticker take the latest `Date <= run_date`; write `curated/prices/run_date=/prices.parquet`. Missing tickers → error summary, excluded from scoring join. -4. Run the **SimFin normalizer** on fundamentals bulk snapshots (see *SimFin normalizer* below). -5. Quality checks; failures → `curated/issues/`. -6. Publish DuckDB views. Downstream reads curated only. - -## Expected flow (SEC — phase 2, frozen spike) - -Existing `download-fundamentals` CLI and `sec_client` remain in repo for reference. Not invoked by the demo pipeline. When resumed: per-CIK `companyfacts` download, EDGAR `acceptance-datetime` as `as_of_date`, separate SEC normalizer path documented below. - -## Data quality checks (initial set) - -| Check | Severity | -| --- | --- | -| Mandatory fields present (revenue, EBIT, net income, total assets, total liabilities, shares outstanding) | Block | -| `as_of_date` exists and is not in the future relative to `run_date` | Block | -| `fiscal_period_end <= as_of_date` | Block | -| Reported currency is USD | Block (non-USD goes to review queue) | -| Restated values produce a new `version_id` for the same `(cik, fiscal_period_end)` | Warn | -| Price gap larger than configurable threshold without a corresponding corporate action | Warn | -| Volume zero across multiple consecutive trading days | Warn | -| Schema migration mismatch | Block | - -## Point-in-time semantics - -- Every curated fundamentals row has `(cik, fiscal_period_end, as_of_date, version_id)` as the natural key. -- A query "fundamentals as of decision date D" returns, per `(cik, fiscal_period_end)`, the row with the highest `as_of_date <= D` and, on tie, the highest `version_id`. -- The same logic applies when re-running historical backtests: the backtest engine pins `D = decision_date` for each rebalance and never sees a row with `as_of_date > D`. -- Restated financials are kept as new versions; the prior version is preserved for replay of past decisions. -- **Demo share prices:** curated `price_date` comes from SimFin `shareprices/latest` (free tier refreshes ~weekly). **`price_date` may trail `run_date` by up to ~30 days**; no block or review queue for staleness in the demo slice. Phase 2 uses `shareprices/daily` or vendor fallback when same-day accuracy matters. - -Phase 2 yfinance cache semantics: - -- Cache key: `(ticker, endpoint, as_of_date)`. -- TTL per endpoint: - - Daily prices (`history`): 1 day after market close. - - Corporate actions (`actions`): 7 days. - - Static company info (`info`): 30 days. - - Income statement / balance sheet / cash flow: 90 days (yfinance fundamentals are sanity cross-check only; SimFin is canonical). -- Cache miss triggers a live call and writes the response to both the cache and the raw zone. -- Cache hits never trigger network calls. - -## Incremental refresh strategy - -- **SimFin (demo):** Re-download bulk US files when on-disk age exceeds `refresh_days` (default `7` on free tier). Normalizer processes only rows with `Publish Date` newer than the last successful watermark per dataset. -- **Initial backfill:** One manual bulk download of all demo datasets; normalizer filters to universe tickers. -- **yfinance / vendor fallback (phase 2):** Per-ticker watermark as before. -- Curated zones are append-only. Restatements create new versions; we never overwrite a prior version. - -## Acceptance criteria - -- Raw and curated zones are clearly separated; raw is never read by scoring modules. -- Every fundamentals value can be traced to a source file under `raw/simfin/` and its SimFin publish metadata. -- A query for "fundamentals available on date D" never returns rows with `as_of_date > D`. -- The same ingest run can fail for one ticker without aborting the rest. -- Schema versions and migrations are explicit; downstream views do not break silently. -- `yfinance` is not called when a valid cache entry exists. -- A full daily incremental run for the demo universe completes inside the Fargate Spot task budget (target: under 30 minutes; to validate during implementation). Phase 2 S&P 500 historical universe may need a separate budget check. -- A backtest run never triggers fresh `yfinance` calls; it only reads curated parquet. -- Schema, partitioning, and DuckDB view names are documented in the spec, not only in code. - -### Progress notes - -- SimFin bulk connector implemented in `src/smartwealthai/download_simfin.py`, - `simfin_client.py`, and `lake_paths.simfin_bulk_path`. Operator guide: - [`docs/mvp/guides/download-simfin.md`](../guides/download-simfin.md). -- SimFin fundamentals normalizer in `src/smartwealthai/simfin_normalizer.py` and - `normalize_simfin.py` CLI; mapping at `config/fundamentals/simfin_mapping_v1.yaml`. - Hermetic tests in `tests/test_simfin_normalizer.py` with fixtures under - `tests/fixtures/lake/raw/simfin/`. -- Hermetic tests in `tests/test_download_simfin.py` (path layout, skip/force, - mocked download, per-dataset failure handling). -- A hermetic fixture lake contract is implemented for CI in - `tests/fixtures/lake/README.md` with raw SEC, raw SimFin, and raw yfinance - snapshots, curated derived fundamentals, and a provenance manifest with - checksums. -- Point-in-time selection and raw fixture loading behavior are covered by tests in - `tests/test_ci_baseline.py` via `smartwealthai.fixture_lake`. -- Fundamentals download spike modules are implemented under `src/smartwealthai/` - (`sec_client`, `edgartools_client`, `download_fundamentals`) — **frozen** for phase 2. - SimFin connector + normalizer are the active demo path ([ADR-0001](../../adr/0001-simfin-fundamentals-mvp.md)). -- SimFin shareprices snapshot in `price_ingest.py` and `download_prices.py` CLI - (`poetry run download-prices`). Requires `shareprices/latest` from `download-simfin`. - Writes `curated/prices/run_date=/prices.parquet`. Hermetic tests in - `tests/test_download_prices.py`. - -**Operator sequence (demo pipeline):** - -```bash -poetry run run-demo-pipeline --run-date 2026-06-19 -poetry run run-dashboard --data-dir data --run-date 2026-06-19 -``` - -Equivalent manual steps: - -```bash -poetry run download-simfin --as-of-date 2026-06-19 -poetry run build-universe --run-date 2026-06-19 -poetry run normalize-simfin --snapshot-date 2026-06-19 --universe-run-date 2026-06-19 -poetry run download-prices --run-date 2026-06-19 --snapshot-date 2026-06-19 -poetry run score-universe --run-date 2026-06-19 -``` - -## Decisions made (fundamentals) - -| Area | Decision | -| --- | --- | -| **Demo normalizer input** | SimFin bulk parquets from `raw/simfin/` (income TTM + balance quarterly + cashflow TTM). | -| **`as_of_date` (demo)** | SimFin `Publish Date`; `Restated Date` → new `version_id`. | -| **SEC normalizer (phase 2)** | `companyfacts` JSON from `raw/sec_edgar/...`; EDGAR acceptance as `as_of_date`. | -| **Mapping** | `config/fundamentals/simfin_mapping_v1.yaml` (demo); `mapping_v1.yaml` (SEC phase 2). | -| Canonical fields | Same ~12 curated columns for ROC, EY, and QC regardless of provider. | -| Provenance | Per-field source column + `mapping_version`. | -| Downstream contract | Scoring reads `curated/fundamentals` only — provider-agnostic schema. | -| SEC spike | Frozen in repo; not deleted. | - -## SimFin bulk connector (demo) - -Downloads US fundamentals via the `simfin` Python package into `raw/simfin/`. -Operator guide: [`docs/mvp/guides/download-simfin.md`](../guides/download-simfin.md). - -### CLI - -```bash -export SIMFIN_API_KEY="" -poetry run download-simfin -poetry run download-simfin --refresh-days 7 --force -``` - -### Module map - -| Module | Role | -| --- | --- | -| `smartwealthai.simfin_client` | API key config, safe bulk download (zip-slip guarded), cache CSV path. | -| `smartwealthai.download_simfin` | CLI orchestration, skip/force by `refresh_days`, run summary. | -| `smartwealthai.lake_paths` | `simfin_bulk_path`, `simfin_errors_path`. | - -### Acceptance criteria (SimFin connector) - -- [x] `simfin` dependency in `pyproject.toml`; API key from `SIMFIN_API_KEY`. -- [x] CLI downloads all six demo datasets into stable `raw/simfin/` partitions. -- [x] Re-run without `--force` skips datasets fresher than `refresh_days`; `--force` overwrites. -- [x] Per-dataset failures recorded in run summary; batch continues when possible. -- [x] Hermetic tests cover path building, skip/force logic, and mocked download. -- [x] Bulk ZIP extraction validates member paths (zip-slip guard); does not use simfin `load_*` extractall path. -- [x] Operator steps in [`download-simfin.md`](../guides/download-simfin.md). - -## SimFin normalizer (demo) - -Transforms SimFin bulk statements into curated canonical parquet. Joins income TTM with the latest quarterly balance row per ticker subject to PIT filters. Curated rows are accumulated in memory and written in bulk (one `to_parquet` per `cik`/`period` partition; deduped `mkdir`; parallel thread pool for I/O). Interactive runs show two Click progress bars: tickers during transform, partitions during write (`--quiet` to suppress; `--progress` to force on non-TTY). - -### Configuration - -| Setting | Source | Notes | -| --- | --- | --- | -| API key | `SIMFIN_API_KEY` env var | Required; AWS Secrets Manager at runtime. Never commit to repo. | -| Data root | `--data-dir` or S3 lake root | Local dev default `data/`. | -| Refresh | `refresh_days` | Default `7` for free tier. | -| Column mapping | `config/fundamentals/simfin_mapping_v1.yaml` | SimFin column → canonical field. | - -### Local raw layout (demo) - -| Dataset | Path | -| --- | --- | -| SimFin bulk snapshot | `raw/simfin/dataset=/variant=/market=us/as_of_date=/` | - -### Acceptance criteria (SimFin normalizer) - -- [x] Reads bulk files from `raw/simfin/` only. -- [x] Emits same curated schema as SEC path would (see canonical fields below). -- [x] PIT natural key `(cik, fiscal_period_end, as_of_date, version_id)`. -- [x] Hermetic tests with fixture SimFin CSV snippets. -- [x] `simfin_mapping_v1.yaml` drives column resolution. - -### Module map (SimFin normalizer) - -| Module | Role | -| --- | --- | -| `smartwealthai.simfin_normalizer` | Raw SimFin CSV join + PIT stamping + curated parquet writer. | -| `smartwealthai.normalize_simfin` | CLI entry point (`poetry run normalize-simfin`). | -| `config/fundamentals/simfin_mapping_v1.yaml` | SimFin column → canonical field mapping. | -| `smartwealthai.lake_paths` | `curated_fundamentals_path`, `curated_issues_path`, `fiscal_period_label`. | - -```bash -poetry run normalize-simfin --snapshot-date 2026-06-18 --universe-run-date 2026-06-18 -poetry run normalize-simfin --snapshot-date 2026-06-18 --ticker AAPL --ticker MSFT -poetry run normalize-simfin --snapshot-date 2026-06-18 --universe-run-date 2026-06-18 --quiet -``` - -## SEC fundamentals normalizer (phase 2) - -**GitHub issue:** [#52](https://github.com/JLaborda/SmartWealthAI/issues/52) — blocks [#44](https://github.com/JLaborda/SmartWealthAI/issues/44) (ROC/EY) and feeds [#50](https://github.com/JLaborda/SmartWealthAI/issues/50) (PIT selection). - -Transforms immutable `companyfacts` JSON into curated canonical parquet. Discovery logic from `notebooks/poc_metrics.ipynb` (JNJ EBIT walk-up, debt summation, NWC components) is productized here — not in scoring code. - -### Pipeline stages - -```mermaid -flowchart LR - Raw["raw/companyfacts JSON"] --> Long["Long facts table"] - Long --> Resolve["Resolve canonical fields (mapping_v1)"] - Resolve --> Curated["curated/fundamentals parquet"] - Resolve --> Issues["curated/issues (review queue)"] -``` - -1. **Ingest:** Parse `companyfacts` into a long table: `(cik, concept, fiscal_period_end, value_usd, as_of_date, form, accession)`. -2. **Resolve:** For each canonical field, apply `config/fundamentals/mapping_v1.yaml` rules: - - **Direct:** read a single XBRL concept when populated (e.g. `OperatingIncomeLoss` for EBIT). - - **Fallback chain:** try ordered alternative concepts (e.g. `InterestExpense`, then `InterestExpenseNonoperating`). - - **Derived:** compute from other resolved components (e.g. EBIT walk-up from net income + taxes + interest; `total_debt` as sum of components in scoring, not necessarily stored). - - **Review queue:** if resolution fails or QC blocks, write to `curated/issues/` — do not silently impute. -3. **Stamp:** Attach `as_of_date`, `fiscal_period_end`, `version_id`, `mapping_version`, and per-field provenance. -4. **Append:** Write append-only parquet under `curated/fundamentals/cik=/period=/`. - -### Canonical output fields (mapping v1) - -Fields required by `high-quality-stocks.md`, `cheap-stocks.md`, and ETL QC: - -| Canonical column | Used for | -| --- | --- | -| `ebit` | ROC, EY | -| `current_assets`, `current_liabilities`, `cash`, `short_term_debt` | Net working capital | -| `ppe_net` | ROC denominator | -| `long_term_debt`, `preferred_equity`, `minority_interest` | Enterprise value | -| `shares_outstanding` | Market cap join | -| `revenue`, `net_income`, `total_assets`, `total_liabilities` | Data quality checks | - -`total_debt`, `nwc`, `ev`, `roc`, and `ey` are computed in scoring modules from curated inputs plus prices — not stored in curated fundamentals unless a future spec revision says otherwise. - -### Configuration - -| Setting | Path | Notes | -| --- | --- | --- | -| XBRL → canonical mapping | `config/fundamentals/mapping_v1.yaml` | Versioned; new file for breaking mapping changes. | -| ROC formula | `config/quality/roc.yaml` | Scoring layer (downstream). | -| EY formula | `config/cheap/ey.yaml` | Scoring layer (downstream). | - -### Acceptance criteria (normalizer) - -- [ ] Reads only `companyfacts` from raw; no dependency on edgartools parquets. -- [ ] `mapping_v1.yaml` drives resolution; provenance columns on every output row. -- [ ] PIT natural key `(cik, fiscal_period_end, as_of_date, version_id)` on curated output. -- [ ] JNJ resolves EBIT via walk-up when `OperatingIncomeLoss` is blank (POC-validated). -- [ ] Hermetic tests with fixture `companyfacts` JSON; golden checks for JNJ + at least one direct-tag issuer. -- [ ] Per-field coverage summary for Dow 30 (`% direct` / `% fallback` / `% review queue`). -- [ ] CLI entry point documented in operator guide when implemented. - -### Out of normalizer scope (MVP) - -- Full US-GAAP taxonomy materialization. -- Per-ticker special cases (`if ticker == "JNJ"`). -- edgartools as a second normalization path. -- S3 upload (local `--data-dir` first; S3 follows CI/CD lake work). - -## Fundamentals download spike (local) - -First vertical slice: download and persist raw SEC `companyfacts` for a parameterized -universe. Optionally also download standardized annual statements from `edgartools` for -notebook exploration and cross-checks — **not** for the production normalizer. - -No `submissions` ingest, no curated parquet normalizer in this slice (normalizer: [#52](https://github.com/JLaborda/SmartWealthAI/issues/52)), and no S3 upload. - -**Operator guide:** [`docs/mvp/guides/download-fundamentals.md`](../guides/download-fundamentals.md) - -### Scope - -- Universe presets backed by versioned CSV files under `data/reference/universes/`. - Initial preset: `dow30` (30 tickers with fixed CIKs). Expand later to S&P 500, - Russell 3000, or Nasdaq as additional presets. -- SEC REST: verbatim `companyfacts` JSON per CIK (**required** for normalizer). -- `edgartools` (optional): `Company(ticker).get_facts()` → income, balance, and cash-flow - statements via `.income_statement()`, `.balance_sheet()`, and - `.cashflow_statement()` with `period="annual"` and configurable `periods` - (default 16). Retained for dev/QC; may be dropped from the CLI once [#52](https://github.com/JLaborda/SmartWealthAI/issues/52) is stable. -- Local raw zone only (`--data-dir`, default `data/`). Paths mirror the production - lake layout so the module can move to S3 later without renaming. - -### Configuration - -| Setting | Source | Notes | -| --- | --- | --- | -| SEC identity | `SEC_IDENTITY` env var (required) | Used for SEC REST `User-Agent` and `edgartools.set_identity()`. | -| Data root | `--data-dir` CLI flag | Default `data/`. | -| Universe | `--universe` preset or `--universe-file` | Preset `dow30` reads `data/reference/universes/dow30.csv`. | -| History depth | `--periods` | Default `16` annual columns from `edgartools`. | -| Snapshot date | `--as-of-date` | Default: UTC today. Partition key for immutable daily snapshots. | -| Re-download | `--force` | Ignore existing files for the chosen `as_of_date`. | - -### Local raw layout - -| Dataset | Path | -| --- | --- | -| SEC companyfacts | `raw/sec_edgar/cik=/endpoint=companyfacts/as_of_date=/response.json` | -| edgartools income | `raw/edgartools/cik=/as_of_date=/income_statement_annual.parquet` | -| edgartools balance | `raw/edgartools/cik=/as_of_date=/balance_sheet_annual.parquet` | -| edgartools cash flow | `raw/edgartools/cik=/as_of_date=/cashflow_statement_annual.parquet` | -| Run errors | `raw/download_runs/as_of_date=/errors.json` (written only when failures occur) | - -### Cache and refresh - -- Partition by `as_of_date`. If a target file for today already exists, skip the - network call unless `--force` is set. -- SEC requests are throttled (max ~8 req/s) and retried up to three times with - exponential backoff on transient errors (429, 5xx, timeouts). -- A failure for one CIK does not abort the run. Permanent errors (e.g. 404) are - not retried. Exit code is `1` when any CIK fails, `0` otherwise. - -### CLI - -```bash -export SEC_IDENTITY="Your Name your@email.com" -python -m smartwealthai.download_fundamentals --universe dow30 -python -m smartwealthai.download_fundamentals --universe dow30 --periods 16 --force -``` - -### Module map - -| Module | Role | -| --- | --- | -| `smartwealthai.sec_client` | SEC REST client (throttle, retry, `companyfacts` download). | -| `smartwealthai.edgartools_client` | Optional `get_facts()` statement extraction to parquet (dev/QC). | -| `smartwealthai.download_fundamentals` | CLI orchestration, universe loading, run summary. | -| `smartwealthai.normalize_fundamentals` (planned) | `companyfacts` → curated canonical parquet ([#52](https://github.com/JLaborda/SmartWealthAI/issues/52)). | - -### Acceptance criteria (spike) - -- [x] `dow30` preset loads 30 `(ticker, cik)` rows from a git-versioned CSV. -- [ ] Each successful CIK produces the four raw artifacts above for the run date. -- [x] Re-running without `--force` on the same day skips existing files. -- [ ] `--force` re-downloads and overwrites today's partition. -- [ ] One failing CIK does not stop the rest; failures are listed in `errors.json`. -- [x] Hermetic unit tests cover universe loading, path building, and skip/force logic. - -## Open questions - -- For SEC EDGAR, do we use `sec-edgar-downloader` (filings as files), the `sec_api` (paid), or the official EDGAR REST APIs (`/submissions`, `/companyfacts`)? **Closed:** official REST APIs for fundamentals (`/companyfacts`); `sec-edgar-downloader` only when full 10-K / 10-Q text is needed by `unstructured-financial-data`. -- ~~Should `edgartools` be a normalizer input alongside `companyfacts`?~~ **Closed:** `companyfacts` only; edgartools optional dev/QC ([#52](https://github.com/JLaborda/SmartWealthAI/issues/52)). -- Do we keep daily prices only, or also intraday OHLC? Recommendation: daily-only for the MVP. -- Do we need a separate metadata table tracking ingestion provenance (URL, response code, byte size, hash), or is the S3 path enough? -- What is the policy when the same field disagrees between EDGAR and yfinance? Recommendation: EDGAR wins for fundamentals; yfinance wins for prices and corporate actions; disagreements are logged. -- Should the DuckDB views materialize parquet artifacts or always read directly from S3? Recommendation: read directly from S3 for the MVP; materialize only if query latency becomes a bottleneck. -- Do we add a hash of the raw payload to detect silent provider changes? - -## Risks - -- yfinance is a community wrapper around an undocumented Yahoo endpoint. It can break with little warning. Mitigations: cache aggressively, treat fallbacks as first-class, log every miss. -- SEC EDGAR rate limits requests (10 req/sec, with a required User-Agent). Mitigations: respect headers, throttle, retry with backoff. -- Free-tier providers have monthly quotas. The cache and provider abstraction must make it easy to skip a provider when its quota is exhausted. -- Restated fundamentals are easy to miss if the normalizer overwrites rows instead of creating new versions. The unit tests must explicitly cover this case. -- Mishandled timezones can shift `as_of_date` by a day and create silent look-ahead bias. All timestamps are stored in UTC. diff --git a/docs/mvp/features/high-quality-stocks.md b/docs/mvp/features/high-quality-stocks.md deleted file mode 100644 index aeaf903..0000000 --- a/docs/mvp/features/high-quality-stocks.md +++ /dev/null @@ -1,111 +0,0 @@ -# Feature: High-Quality Stocks - -## Implementation status - -done (demo cross-sectional slice) — ROC scoring and ranks: `src/smartwealthai/magic_formula_ranking.py`, CLI `score-universe` ([#60](https://github.com/JLaborda/SmartWealthAI/issues/60)). Single-ticker tracer: `magic_formula_metrics.py`, `pit_fundamentals.py`, `compute-metrics` ([#44](https://github.com/JLaborda/SmartWealthAI/issues/44)). - -## Objective - -Score the economic quality of every company that survives the universe filter and the permanent loss filter. For the MVP, the quality factor is a strict Greenblatt-style **Return on Capital (ROC)** computed from point-in-time fundamentals. Future iterations can plug additional quality signals into the same interface. - -## MVP scope - -- Compute `ROC = EBIT / (Net Working Capital + Net Fixed Assets)` per the canonical Greenblatt definition. -- Use the most recent point-in-time fundamentals available on the decision date. -- Produce a cross-sectional quality rank (lower rank = higher quality) for every passing company. -- Apply a market-cap tie-break: when ROC ties, the smaller market cap wins. -- Validate denominator: rows with `Net Working Capital + Net Fixed Assets <= 0` are flagged for review and excluded from the ranking. -- Log MLflow metrics: distribution of ROC, count of valid vs invalid rows, percentile statistics. -- Expose the score, the input components, and the explanation downstream so the dashboard can show "why is this company high quality?". - -## Out of MVP scope - -- Multi-metric quality scores (ROIC, ROE, ROA, FCF margin, accruals, balance sheet sub-score). Captured as candidates for the next iteration. -- Sector-relative quality (sectors with unusual accounting are already excluded upstream). -- Earnings quality / accruals scoring. -- Capital allocation scoring. -- Forward-looking estimates. -- Machine learning quality prediction. - -## Inputs - -| Input | Source | Notes | -| --- | --- | --- | -| Passing universe + permanent loss filter pass list | `curated/universe` + `curated/permanent_loss` | Only `pass` rows are scored. | -| PIT fundamentals (income statement, balance sheet) | `curated/fundamentals` | Filtered by `as_of_date <= run_date`. | -| Market cap | `curated/prices/run_date=/prices.parquet` join `curated/fundamentals` | Tie-break uses `shares_outstanding * adj_close` on `run_date`. | -| Run date | Pipeline parameter | | -| ROC formula version | `config/quality/roc.yaml` | Versioned to allow future variants. | - -## ROC definition (canonical Greenblatt) - -``` -ROC = EBIT / (Net Working Capital + Net Fixed Assets) -``` - -with: - -- `EBIT` = Operating income before interest and taxes. Trailing twelve months. -- `Net Working Capital` = `max(Current Assets - Excess Cash - Current Liabilities + Short-Term Debt, 0)` (Greenblatt uses non-interest-bearing current liabilities; we use this approximation and version it). -- `Net Fixed Assets` = Total fixed assets (PP&E net of depreciation). -- All values from the latest filing whose `as_of_date <= run_date`. - -The formula and its variants are versioned in `config/quality/roc.yaml`. Any change requires a new version id so backtests on prior versions remain reproducible. - -## Outputs - -| Output | Path / target | -| --- | --- | -| Quality scores parquet | `curated/scores/quality/run_date=/scores.parquet` with `cik, ticker, ebit, nwc, net_fixed_assets, roc, roc_rank, market_cap, tiebreak_rank, formula_version, as_of_date` | -| Review queue rows | `curated/issues/run_date=/quality.parquet` for invalid denominators and other warnings | -| MLflow metrics | `quality_n_valid`, `quality_n_invalid`, ROC quantiles | - -## Mermaid diagram - -```mermaid -flowchart TD - Passing["Universe pass + Permanent loss pass"] --> Loader["Load PIT fundamentals + market cap"] - Loader --> Compute["Compute EBIT, NWC, Net Fixed Assets"] - Compute --> Validate{"Denominator > 0?"} - - Validate -->|No| Review["Review queue (quality.parquet)"] - Validate -->|Yes| ROC["ROC = EBIT / (NWC + Net Fixed Assets)"] - ROC --> Rank["Cross-sectional rank (descending ROC)"] - Rank --> TieBreak["Tie-break by ascending market cap"] - TieBreak --> Output["quality/scores.parquet"] - Output --> MLflow["MLflow metrics"] -``` - -## Expected flow - -1. Load the passing universe and join with PIT fundamentals. -2. Compute `EBIT`, `Net Working Capital`, and `Net Fixed Assets` using the formula version configured for the run. -3. Validate inputs: drop rows with missing components; flag rows with `denominator <= 0` and route them to the review queue. -4. Compute `ROC`. -5. Produce a cross-sectional rank from highest ROC (rank 1) to lowest. -6. Resolve ties by ascending market cap. -7. Persist the parquet output and log MLflow metrics. - -## Acceptance criteria - -- Same `(universe, run_date, formula_version)` produces byte-identical output (hash-verifiable). -- The score is purely a function of curated PIT data; no network calls. -- Every row has both the ROC value and the components that produced it. -- Rows with invalid denominators are visible in the review queue and not silently dropped or auto-scored. -- The ranking is stable: changing only the market cap of a non-tied row never changes the rank order. -- The MLflow run logs at minimum count of valid rows, count of invalid rows, ROC median, and ROC quantiles. -- The formula version travels with each scored row, so a backtest using a past `formula_version` is reproducible. - -## Open questions - -- Greenblatt himself uses Pre-Tax Operating Earnings; do we use `EBIT` straight from EDGAR (`OperatingIncomeLoss + InterestAndDebtExpense`) or compute Pre-Tax Operating Earnings explicitly? Recommendation: use `OperatingIncomeLoss` from EDGAR and document the choice as `formula_version = v1`. -- ~~Excess cash definition for `Net Working Capital`~~ **Closed (v1):** curated `cash` uses SimFin `Cash, Cash Equivalents & Short Term Investments` for both NWC and EV (known approximation — NWC excess-cash adjustment is slightly aggressive vs cash-equivalents-only). -- Should very small ROC differences (e.g., < 0.1 percentage point) be treated as ties for the market-cap tie-break? Recommendation: no in the MVP; revisit if rank stability becomes a problem. -- For companies with negative EBIT but positive denominator, ROC is negative. Do we exclude them, or rank them at the bottom? Recommendation: rank them at the bottom; they will likely never enter the top 30 anyway. - -## Risks - -- `Net Working Capital` and `Net Fixed Assets` definitions vary across textbooks and providers. Locking the formula version is the only way to keep backtests reproducible. -- Single-metric quality leaves the strategy exposed to capital-light tech businesses whose balance sheets distort ROC. We accept this in the MVP and document the limitation. -- Restatements can move ROC sharply between versions. The PIT store keeps both versions; backtests must pick the version available at `as_of_date`. -- One-off items in EBIT can produce false positives. The MVP does not adjust for them; this is a known weakness of the Greenblatt placeholder. diff --git a/docs/mvp/features/permanent-loss-filter.md b/docs/mvp/features/permanent-loss-filter.md deleted file mode 100644 index 821b89f..0000000 --- a/docs/mvp/features/permanent-loss-filter.md +++ /dev/null @@ -1,129 +0,0 @@ -# Feature: Permanent Loss Filter - -## Implementation status - -**Deferred** for the June 30 demo slice ([ADR-0002](../../adr/0002-june-demo-scope-cut.md)). Spec remains the target for phase 2. - -## Objective - -Identify companies in the investable universe with elevated risk of permanent capital loss and remove them from the ranking before any score is computed. For the MVP, "permanent loss" is defined narrowly as **fraud or bankruptcy / financial distress**. Companies flagged by either subfilter are hard-excluded. - -## MVP scope - -- Hard exclusion (not a score penalty). A flagged company never enters the ranking. -- Two subfilters: fraud signals and bankruptcy / distress signals. -- Inputs come from the curated point-in-time store; no live network calls. -- Every exclusion records the triggered rule, the inputs that fired it, and the `as_of_date`. -- Regression test in CI: the bankruptcy subfilter must flag Enron, Lehman, and WorldCom on the dates each company was already in clear distress (e.g., Enron Q3 2001 10-Q, Lehman Q2 2008 10-Q, WorldCom Q1 2002 10-Q). If any of these stops being flagged, the CI build fails. - -## Out of MVP scope - -- Manipulation / accruals models (Beneish M-score, accruals-based scores). Deferred. -- Machine learning fraud detection. -- LLM-driven qualitative analysis of filings (handled later by `unstructured-financial-data`). -- Sector-specific distress models (banks, insurers, REITs and utilities are already excluded upstream by `universe-construction`). -- `Penalize` and `unknown` states. Only `pass` and `exclude` for the MVP. - -## Inputs - -| Input | Source | -| --- | --- | -| Curated PIT fundamentals (income statement, balance sheet, cash flow) | `curated/fundamentals` | -| Adjusted prices and corporate actions | `curated/prices` | -| SEC filing index (form type, accession, acceptance datetime) | `curated/sec_edgar/submissions` | -| Auditor information (when extracted) | `curated/sec_edgar/auditor` (future) | -| Regression test fixtures | `tests/fixtures/permanent_loss/` (CIK + as_of_date + expected `exclude`) | -| Run date | Pipeline parameter | - -## Outputs - -| Output | Path / target | -| --- | --- | -| Exclusion table | `curated/permanent_loss/run_date=/exclusions.parquet` with columns `cik, ticker, subfilter, rule_id, rule_version, triggered_value, threshold, as_of_date, explanation` | -| Filter status | `pass` or `exclude` per `(cik, as_of_date)` | -| Logged metrics (MLflow) | Number of evaluations, number of exclusions per subfilter, list of newly excluded companies | - -## Bankruptcy / distress subfilter (MVP rules) - -The MVP implements a small but well-known set of distress indicators. Each rule has a versioned id so historical decisions can be replayed. - -| Rule id | Definition | Threshold (initial) | Source | -| --- | --- | --- | --- | -| `BK_ALTMAN_Z` | Altman Z-score for non-financials | Z < 1.81 | `curated/fundamentals` | -| `BK_INT_COVERAGE` | Interest coverage (EBIT / Interest Expense), TTM | < 1.0 | `curated/fundamentals` | -| `BK_NETDEBT_EBITDA` | Net debt to EBITDA, TTM | > 7.0 with negative FCF | `curated/fundamentals` | -| `BK_NEGATIVE_EQUITY` | Stockholders' equity | < 0 | `curated/fundamentals` | -| `BK_GOING_CONCERN` | "Going concern" language flag from latest 10-K (provided by `unstructured-financial-data` once available) | flag present | `curated/text_flags` (future) | -| `BK_DELISTED` | Listing status | `delisted` and `delisting_reason in {bankruptcy, regulatory}` | `curated/prices` | - -A company is excluded if **any** of the above rules fire. - -## Fraud subfilter (MVP rules) - -The MVP fraud signals are intentionally narrow. They detect structural / accounting events, not subjective judgments. - -| Rule id | Definition | Threshold (initial) | -| --- | --- | --- | -| `FRD_RESTATEMENT_RECENT` | Material restatement of prior reported figures in the last 12 months (e.g., 10-K/A or 10-Q/A filings) | `>= 1` filing | -| `FRD_AUDITOR_CHANGE_REPEATED` | Auditor change in 2 of the last 3 fiscal years | `>= 2` changes | -| `FRD_REGULATORY_ACTION` | Open SEC enforcement action against the issuer | `True` | -| `FRD_LATE_FILER` | Filed `NT 10-K` or `NT 10-Q` (late filing notification) in last 12 months | `>= 1` filing | - -A company is excluded if **any** rule fires. Rules that depend on data not yet available in the MVP (`FRD_REGULATORY_ACTION`, derived from EDGAR enforcement feeds) are coded but tolerated as `unavailable` until the data is wired. Their absence is logged. - -## Mermaid diagram - -```mermaid -flowchart TD - Universe["v_universe (today)"] --> Loader["Load PIT fundamentals + prices + filings"] - Loader --> Bankruptcy["Bankruptcy / distress rules"] - Loader --> Fraud["Fraud rules"] - - Bankruptcy --> Decision{"Any rule fired?"} - Fraud --> Decision - - Decision -->|Yes| Exclude["Exclude (hard)"] - Decision -->|No| Pass["Pass to scoring modules"] - - Exclude --> Output["curated/permanent_loss exclusions.parquet"] - Output --> MLflow["MLflow metrics + artifact"] - - subgraph Tests["CI regression"] - Enron["Enron Q3 2001"] --> RegTest["Must be excluded"] - Lehman["Lehman Q2 2008"] --> RegTest - WorldCom["WorldCom Q1 2002"] --> RegTest - end -``` - -## Expected flow - -1. Read the universe for the run date from `v_universe`. -2. Join with the latest PIT fundamentals (using `as_of_date <= run_date`). -3. Compute each bankruptcy and fraud rule. Rules with missing required inputs are recorded as `unavailable` and the row is sent to the review queue (not auto-excluded). -4. If any rule fires, mark the company as `exclude` with the rule id, threshold, and the values that triggered it. -5. Write the exclusion parquet and log MLflow metrics. -6. Hand the passing set of `(cik, ticker)` to the scoring modules. - -## Acceptance criteria - -- The module is a pure function of curated parquet + reference rules: same inputs produce byte-identical output (verifiable by hash). -- The regression CI test for Enron, Lehman, and WorldCom blocks the build if any of the three stops being flagged. -- Every excluded row carries the `rule_id`, `rule_version`, `triggered_value`, and `threshold`. -- Rule definitions live in code, but thresholds live in a YAML config under `config/permanent_loss/` so they can be tuned by backtests without code changes. -- The filter never queries network resources. -- Companies with `unavailable` rule outputs do not pass silently: they enter the review queue. -- Each MLflow run for the permanent loss filter logs the count of exclusions per rule. - -## Open questions - -- Should `BK_NETDEBT_EBITDA` be sector-relative even though banks / insurers / utilities are excluded? Recommendation: keep it absolute for the MVP; revisit when those sectors are reintroduced. -- Threshold for `BK_NETDEBT_EBITDA` should probably be revisited per backtest; the initial 7.0 is a placeholder. -- Where does the auditor-change history come from? Recommendation: parse `acceptedAccountingFirm` from EDGAR if available; otherwise wait for `unstructured-financial-data` to provide it. -- Do we want a "watchlist" state (`watch`) between `pass` and `exclude`? Recommendation: no for the MVP; that role is fulfilled by `sell-watch` once a company is held. - -## Risks - -- Excluding bankrupt companies after the fact is easy; excluding them *before* is the hard part. The Altman Z-score has known weaknesses for tech / asset-light companies. The MVP accepts this in exchange for simplicity. -- Restatements happen for innocent reasons (acquisitions, IFRS-to-GAAP changes). The MVP rule will produce some false positives; they are tracked in the FP/FN review process. -- Removing companies for "late filer" status can be aggressive. We log every `NT 10-K` and `NT 10-Q` so the FP review can tune the rule. -- `BK_DELISTED` is only useful historically; it cannot prevent a loss in real time. Its purpose is to make the historical backtest realistic (a delisted-for-bankruptcy company never enters the post-delisting universe). diff --git a/docs/mvp/features/sell-watch.md b/docs/mvp/features/sell-watch.md deleted file mode 100644 index a9b68d5..0000000 --- a/docs/mvp/features/sell-watch.md +++ /dev/null @@ -1,132 +0,0 @@ -# Feature: Sell-Watch / Vigilance - -## Implementation status - -**Deferred** for the June 30 demo slice ([ADR-0002](../../adr/0002-june-demo-scope-cut.md)). Spec remains the target for phase 2. - -## Objective - -Monitor every name held in the **model portfolio** every day and emit a hard `sell` signal when the thesis breaks. Signals never auto-execute: they appear in the dashboard and trigger an AWS SES email so the user can review and confirm. Sell-watch does not monitor the user's personal portfolio (those are personal decisions). - -## MVP scope - -- Daily run against the live model portfolio holdings. -- Four trigger families: quality deterioration, fraud / bankruptcy flag turning on after entry, overvaluation, opportunity cost. -- Hard `sell` only. No `trim` or `hold-with-warning` states. -- Manual confirmation required: a sell signal must be confirmed in the dashboard before the broker module builds an order. -- Alerts: dashboard badge + AWS SES email per signal. -- Logs every evaluation (signal or no signal) for audit and FP/FN review. -- MLflow run per daily evaluation with parameters, metrics, and artifacts. - -## Out of MVP scope - -- Monitoring of the user's personal portfolio. -- Price-based stops (trailing stop, drawdown stop). -- Time-based stops. -- Automatic execution. -- Multi-state output (`trim`, `hold-with-warning`). -- LLM-driven narrative explanation (deferred). - -## Inputs - -| Input | Source | -| --- | --- | -| Current model portfolio holdings | `curated/portfolio/model/holdings.parquet` (produced by portfolio-construction) | -| Latest PIT fundamentals | `curated/fundamentals` | -| Latest prices | `curated/prices` | -| Latest quality scores | `curated/scores/quality` | -| Latest cheapness scores | `curated/scores/cheap` | -| Latest watchlist (top-ranked names not yet held) | `curated/portfolio/watchlist.parquet` | -| Permanent loss filter output | `curated/permanent_loss` | -| Confirmed signals history | `curated/sell_watch/confirmations.parquet` (so we do not re-alert on the same signal day after day) | -| Sell-watch config | `config/sell_watch.yaml` (thresholds, opportunity-cost margin, lookback for ROC YoY) | - -## Trigger definitions (MVP) - -All thresholds are starting points and live in `config/sell_watch.yaml`. Each is hyperparameter-able by the backtest engine. - -| Trigger id | Rule | Default threshold | -| --- | --- | --- | -| `SW_PERMANENT_LOSS` | The permanent loss filter, evaluated on the holding today, flags `exclude` | n/a | -| `SW_QUALITY_DROP_YOY` | ROC YoY drop greater than `quality_yoy_drop` | 30% | -| `SW_QUALITY_DECILE_DROP` | The holding is no longer in the top decile of cross-sectional ROC | top 10% | -| `SW_OVERVALUATION_PCT` | EY below the cross-sectional `overvaluation_percentile` of the current universe | 10th percentile | -| `SW_OVERVALUATION_ABS` | EY below the absolute `overvaluation_floor` | 5.0% | -| `SW_OPPORTUNITY_COST` | A watchlist candidate outranks the holding by more than `opportunity_cost_margin` positions on the combined Greenblatt rank | 5 positions | - -A holding is flagged `sell` if `SW_PERMANENT_LOSS` fires, **or** any quality trigger fires (`SW_QUALITY_DROP_YOY` or `SW_QUALITY_DECILE_DROP`), **or** any overvaluation trigger fires (`SW_OVERVALUATION_PCT` or `SW_OVERVALUATION_ABS`), **or** `SW_OPPORTUNITY_COST` fires. - -## Outputs - -| Output | Path / target | -| --- | --- | -| Signals parquet | `curated/sell_watch/run_date=/signals.parquet` with `ticker, triggers, fired_thresholds, values, message_id, status` | -| Audit parquet | `curated/sell_watch/run_date=/evaluations.parquet` (every holding evaluated, signal or no signal) | -| Email payload (per signal) | Subject + body + dashboard deep link | -| MLflow metrics | Count of signals per trigger, daily count of evaluations, count of confirmed vs ignored signals | - -The `status` field starts as `proposed` and moves to `confirmed` or `dismissed` when the user acts on it from the dashboard. - -## Mermaid diagram - -```mermaid -flowchart TD - Holdings["Model portfolio holdings (today)"] --> Eval["Evaluate triggers"] - PLoss["Permanent loss filter today"] --> Eval - ROC["Quality score (today and 1y ago)"] --> Eval - EY["Cheapness score (today, cross-section)"] --> Eval - Watchlist["Watchlist (top-ranked non-holders)"] --> Eval - - Eval --> AnyTrigger{"Any trigger fired?"} - AnyTrigger -->|No| Audit["audit parquet only"] - AnyTrigger -->|Yes| Dedup["Dedup against confirmations history"] - Dedup --> SignalsOut["sell_watch/signals.parquet"] - SignalsOut --> Dashboard["Dashboard sell-watch panel"] - SignalsOut --> SES["AWS SES email"] - Dashboard --> User["User confirms or dismisses"] - User -->|Confirm| Orders["Order builder (paper)"] - User -->|Dismiss| Audit2["confirmations.parquet (dismissed)"] -``` - -## Expected flow - -1. Pull today's holdings from the model portfolio table. -2. For each holding: - 1. Look up its current permanent loss status. If `exclude`, fire `SW_PERMANENT_LOSS`. - 2. Compute ROC today and ROC 1 year ago from PIT data. Fire `SW_QUALITY_DROP_YOY` if drop > threshold. - 3. Locate the holding's ROC rank among the current universe. Fire `SW_QUALITY_DECILE_DROP` if it left the top decile. - 4. Locate the holding's EY in today's universe percentile. Fire `SW_OVERVALUATION_PCT` if below the configured percentile. - 5. Read the holding's absolute EY. Fire `SW_OVERVALUATION_ABS` if below the absolute floor. - 6. Compare the holding's combined Greenblatt rank against the best non-held watchlist candidate. Fire `SW_OPPORTUNITY_COST` if margin exceeds threshold. -3. If any trigger fired, check the confirmations history to avoid re-alerting on an already-active signal. If it is new, write a signal row and send an SES email. -4. Write the audit parquet covering every evaluation. -5. The dashboard exposes the open signals. The user clicks confirm or dismiss, which writes `confirmations.parquet`. -6. Confirmed signals flow into the broker module as sell orders. Dismissed signals are remembered so we do not re-fire the same signal until the underlying input changes materially. -7. Log MLflow metrics for the daily run. - -## Acceptance criteria - -- The module never fires an order autonomously. The broker module requires a confirmed signal. -- A signal is deduplicated against the confirmations history so the same trigger does not email the user every day. -- Every signal row contains the trigger ids, the input values, the thresholds in effect, and the `as_of_date`. -- The audit parquet contains a row for every holding evaluated, signal or no signal. -- Thresholds live in YAML and travel with the run; backtests can sweep them. -- Emails sent through AWS SES include a dashboard deep link to the signal. -- The pipeline is idempotent for a given run date: re-running produces the same signal set without duplicate emails (idempotent by `message_id`). -- The MLflow run logs at minimum: number of evaluations, number of signals, number per trigger. - -## Open questions - -- For the "no longer in top decile" rule, do we use the decile of the same universe used to enter the position, or today's universe? Recommendation: today's universe; matches the spirit of opportunity cost. -- For `SW_QUALITY_DROP_YOY`, how do we handle restatements that change ROC retroactively? Recommendation: compare today's PIT ROC against the ROC value used at entry (snapshot at purchase), not against today's "1 year ago" PIT slice. -- The opportunity-cost trigger requires the watchlist to be sorted by the same combined rank used to enter. Should the watchlist be recomputed daily, or only at rebalance? Recommendation: daily, cheap. -- Should we dampen the email frequency with a rate limit (e.g., max 5 signals per day)? Recommendation: yes, with an MLflow metric reporting the suppression count. -- For the dismissed-signal memory: how long do we wait before re-firing a dismissed signal? Recommendation: until either the trigger value changes by more than 10% from the dismissal value, or 90 days have passed, whichever comes first. - -## Risks - -- Daily signals can desensitize the user. The dedup + dismissal memory exists to prevent this. -- The opportunity-cost trigger is the most rank-sensitive: small ranking noise can produce churn. The 5-position margin is the dampener; the backtest must validate that it is not too aggressive. -- Restated fundamentals can produce false sells if we rely on today's PIT slice for "ROC 1 year ago". The snapshot-at-entry approach above mitigates this. -- AWS SES may rate-limit or land in spam if the sender domain is not verified. The infrastructure spec must include verifying the SES sender identity. -- The user can dismiss legitimate signals out of bias. The FP/FN review of dismissed signals is a follow-up improvement. diff --git a/docs/mvp/features/universe-construction.md b/docs/mvp/features/universe-construction.md deleted file mode 100644 index c40eff3..0000000 --- a/docs/mvp/features/universe-construction.md +++ /dev/null @@ -1,166 +0,0 @@ -# Feature: Universe Construction - -## Implementation status - -**done** (demo slice) — universe builder ([#58](https://github.com/JLaborda/SmartWealthAI/issues/58)); industry exclusions reference CSV ([#56](https://github.com/JLaborda/SmartWealthAI/issues/56)). Full S&P 500 historical mode in phase 2. - -## Objective - -Produce the investable universe of US common stocks for each decision date. This module is the single entry point for "which tickers does the strategy consider today?" and is the upstream dependency of every downstream module. It must be point-in-time correct and survivorship-bias-free. - -## MVP scope - -### Demo slice (June 30) - -- Seed universe: all SimFin US companies (`load_companies(market='us')`). -- Exclude banks, insurers, and utilities via `data/reference/simfin_industry_exclusions.csv` (`IndustryId` list built from `load_industries()`). -- **Regeneration rules** (applied by `build_exclusions` in `src/smartwealthai/simfin_industry_exclusions.py`): - - `bank`: SimFin industry name exactly `Banks` - - `insurer`: industry name contains `Insurance` - - `utility`: SimFin sector exactly `Utilities` -- Regenerate after SimFin industry label changes: `poetry run generate-simfin-industry-exclusions --industries ` -- Sanity check: exclude tickers present in SimFin `income_banks` or `income_insurance` bulk datasets even if `IndustryId` is missing from the CSV. -- No S&P 500 historical file required for demo. -- No market-cap or ADV floors in demo (optional parameters disabled). -- Produce daily snapshot under `curated/universe/run_date=/`. - -### Full MVP (phase 2) - -- S&P 500 historical constituents (incl. delisted) from `data/reference/sp500_constituents.csv`. -- Common-stock filters (exclude ADRs, REITs, BDCs, ETFs, preferred-only). -- SIC-based sector exclusions when SEC ETL is available. -- Share-class deduplication by ADV. -- Optional market-cap and volume floors. - -## Out of MVP scope - -- Non-US universes. -- Index families other than S&P 500 (Russell 3000, MSCI USA, etc.) as the seed. -- Liquidity rules beyond a static daily-volume threshold. -- Sector exposure limits (out of MVP scope per architecture decision). -- Automatic re-classification of issuers as they change SIC code. - -## Inputs - -| Input | Source | Notes | -| --- | --- | --- | -| US company list | SimFin `companies` (demo) | `Ticker`, `CIK`, `IndustryId`. | -| Industry metadata | SimFin `industries` + `simfin_industry_exclusions.csv` | Sector/industry names for audit. | -| Bank/insurance sanity | SimFin `income_banks` / `income_insurance` ticker index | Secondary exclusion signal. | -| Historical S&P 500 constituents | `data/reference/sp500_constituents.csv` | Phase 2 only. | -| SIC codes | SEC EDGAR submissions | Phase 2 only. | -| Daily prices and volume | `curated/prices` | For ADV dedup and floors (phase 2). | -| Market cap | `curated/fundamentals` join `curated/prices` | Tie-break and optional floors. | -| Run date | Pipeline parameter | PIT universe slice. | - -## Outputs - -| Dataset | Path | Schema | -| --- | --- | --- | -| Daily universe | `s3://smartwealthai-data-lake/curated/universe/run_date=/universe.parquet` | `run_date, ticker, cik, industry_id, sector, market_cap_usd, exclusion_reasons` (demo schema; `sic_code` added in phase 2) | -| Exclusion log | `s3://smartwealthai-data-lake/curated/universe/run_date=/exclusions.parquet` | One row per excluded ticker with the triggered rule(s). | -| DuckDB view | `v_universe` | Latest universe view, partitioned on `run_date`. | - -## Mermaid diagram (demo) - -```mermaid -flowchart TD - Companies["SimFin companies (market=us)"] --> Seed["Seed universe at run_date"] - ExclCSV["simfin_industry_exclusions.csv"] --> SectorFilter{"IndustryId excluded?"} - BankSanity["Bank / insurance statement indices"] --> SanityFilter{"Bank or insurer ticker?"} - Seed --> SectorFilter - SectorFilter -->|Yes| Excluded["exclusions.parquet"] - SectorFilter -->|No| SanityFilter - SanityFilter -->|Yes| Excluded - SanityFilter -->|No| Universe["universe.parquet"] - Universe --> DuckDBView["v_universe"] - Excluded --> ExclusionLog["exclusions.parquet"] -``` - -## Mermaid diagram (full MVP — phase 2) - -```mermaid -flowchart TD - SP500["data/reference/sp500_constituents.csv"] --> Seed["Build seed universe at run_date"] - Curated["curated/fundamentals + curated/prices"] --> Enrich["Enrich with SIC, market cap, ADV"] - Seed --> Enrich - - Enrich --> CommonOnly{"Common stock?"} - CommonOnly -->|No| Excluded["Excluded: non-common-stock"] - CommonOnly -->|Yes| SectorFilter{"SIC in banks / insurers / utilities?"} - - SectorFilter -->|Yes| Excluded2["Excluded: sector"] - SectorFilter -->|No| Dedup["Deduplicate share classes by ADV"] - Dedup --> Floors{"Market cap and volume floors"} - Floors -->|Below| Excluded3["Excluded: too small / illiquid"] - Floors -->|Above| Universe["Daily universe (curated/universe)"] - - Excluded --> ExclusionLog["exclusions.parquet"] - Excluded2 --> ExclusionLog - Excluded3 --> ExclusionLog - - Universe --> DuckDBView["v_universe"] -``` - -## Expected flow (demo) - -1. Load SimFin `companies` for `market=us` from curated or raw snapshot. -2. Join `IndustryId` to `simfin_industry_exclusions.csv`; excluded rows → `exclusions.parquet` with reason `sector`. -3. Drop tickers found in bank/insurance SimFin statement indices (sanity check). -4. Persist `universe.parquet` for `run_date`. - -## Expected flow (full MVP — phase 2) - -1. Read `data/reference/sp500_constituents.csv` and compute historical membership through `run_date`. -2. Map each ticker to its CIK and `sic_code` via the curated EDGAR submissions table. -3. Filter to common stocks. The MVP keeps only issuers whose SEC form types include `10-K` and `10-Q` filed on a standard schedule, and excludes: - - ETFs and ETN issuers (form `N-CSR`, `N-Q`, fund-specific filings). - - REITs (`SIC 6798`). - - BDCs (`SIC 6770` and explicit BDC registrants). - - Preferred-only listings. - - Foreign private issuers filing `20-F` instead of `10-K` (ADRs). -4. Exclude sectors by SIC code range: - - Banks: `6020-6199`. - - Insurers: `6311-6411`. - - Utilities: `4900-4999`. - The full SIC-to-bucket mapping table is materialized in the spec for review. -5. For each issuer with multiple share classes, compute the trailing-90-day average daily volume per class and keep the class with the highest figure. All other classes go to `exclusions.parquet` with reason `share_class_lower_liquidity`. -6. Apply optional floors: - - `market_cap_usd >= market_cap_floor` (parameter, default off in the MVP). - - `avg_daily_volume_usd_90d >= adv_floor` (parameter, default `1_000_000` USD). -7. Persist the universe and exclusion log for `run_date`, alongside the parameter values used. - -## Acceptance criteria - -### Demo - -- [x] `data/reference/simfin_industry_exclusions.csv` versioned with banks, insurers, utilities (`industry_id`, `industry_name`, `sector`, `exclusion_reason`). -- [x] Same `run_date` → byte-identical `universe.parquet`. -- [x] No excluded `IndustryId` appears in the universe. -- [x] No bank/insurance sanity-check ticker appears in the universe. -- [x] Module consumes only curated/raw SimFin snapshots (no network). - -**Code:** `src/smartwealthai/universe_builder.py`, CLI `poetry run build-universe`. - -### Full MVP (phase 2) - -- Bankrupt companies that were once in the index appear in past universe snapshots up to their delisting date and are excluded only after that date with reason `delisted`. -- No company whose SIC code is in the excluded sector ranges appears in any universe snapshot. -- Share class deduplication is reversible from the exclusion log. -- The schema of `universe.parquet` is versioned and documented. - -## Open questions - -- Source of the historical constituents file. Proposed: `github.com/fja05680/sp500` snapshot pinned in `data/reference/sp500_constituents.csv`. Need user confirmation. -- How do we handle additions / removals on the same day a ticker is also evaluated for inclusion? Recommendation: include the ticker if it was in the index at the close of the prior trading day. -- Do we want a manual override list (`data/reference/universe_overrides.csv`) so the user can pin or blacklist tickers for testing? Recommendation: yes, but only honored when an explicit flag is set on the run. -- For dual-class issuers, do we collapse activity from both classes for the personal portfolio module, or do we keep them separate? Recommendation: keep separate in `portfolio-evolution`, deduplicate only in the investment universe. -- Do we want to record, in the universe snapshot, the SIC code reclassifications that happen mid-history? Recommendation: yes, store both the current and the as-of-date SIC code. - -## Risks - -- The community S&P 500 constituents dataset can have errors (missing additions, wrong dates). Mitigation: pin a snapshot and add a smoke test that asserts a known set of historical events (e.g., Lehman removal 2008, Tesla addition 2020). -- SIC codes are not a perfect sector classifier. Some banks file under non-bank SIC codes and vice versa. Mitigation: keep an explicit override list per CIK and review it during exclusions analysis. -- Survivorship bias still creeps in if the constituents file is built from "currently listed" companies. Mitigation: verify a sample of known-bankrupt companies (Lehman, Enron, WorldCom) are present in the historical file. -- Share class deduplication based on liquidity can flip the kept class across days for low-liquidity issuers. Mitigation: smooth the volume metric over 90 days and require a margin before flipping. -- Excluding banks, insurers, and utilities removes a sizable chunk of the index. Documented as an MVP trade-off. diff --git a/docs/mvp/features/unstructured-financial-data.md b/docs/mvp/features/unstructured-financial-data.md deleted file mode 100644 index 92533f9..0000000 --- a/docs/mvp/features/unstructured-financial-data.md +++ /dev/null @@ -1,103 +0,0 @@ -# Feature: Unstructured Financial Data - -> **Status for the MVP: minimal.** The MVP does not run LLM analyses, summaries, or embeddings on filings. It only stores raw filing references and exposes one targeted text-flag pipeline that the permanent loss filter can consume: a **"going concern" detector** on the latest 10-K. Everything else (transcripts, news, sentiment, RAG) is parked until after the MVP is validated. - -## Objective - -Provide a thin text-processing layer that complements the structured pipeline. For the MVP, the only consumer is the permanent loss filter, which benefits from a high-precision "going concern" flag pulled from the latest 10-K. - -## MVP scope - -- Persist the raw 10-K text (or filing reference) in the data lake under `raw/sec_edgar/...` (already produced by `etl-data-lake`). -- Run a rule-based scanner that looks for "going concern" language patterns in the latest 10-K per company. -- Emit a boolean flag plus the matched passage(s) and the filing url. -- Expose results in a curated parquet (`curated/text_flags/going_concern.parquet`) consumed by `permanent-loss-filter`. -- Run weekly (filings do not change daily); incremental. -- No LLM, no embeddings, no summarization in the MVP. - -## Out of MVP scope - -- Summarization or LLM-driven narratives. -- Sentiment analysis. -- Year-over-year risk-factor diffs. -- Earnings call transcripts. -- News scraping. -- Multilingual filings. -- RAG / vector search infrastructure. -- Auditor-change extraction (handled by a different rule once the data is available). - -## Inputs - -| Input | Source | -| --- | --- | -| Latest 10-K filing per CIK | `raw/sec_edgar/.../form=10-K/...` | -| List of patterns to match | `config/text_flags/going_concern_patterns.yaml` (versioned) | - -## Outputs - -| Output | Path | -| --- | --- | -| Going concern flag | `curated/text_flags/going_concern.parquet` with `cik, accession, as_of_date, flag_bool, matched_phrases, source_url, pattern_version` | -| MLflow metrics | Count of CIKs scanned, count of flags raised | - -## Patterns (initial set) - -Stored in YAML, versioned. Match is case-insensitive, regex-based, restricted to the "Notes to Consolidated Financial Statements" and "Management's Discussion" sections when section markers can be found; otherwise applied to the full text. - -```yaml -patterns: - - "substantial doubt about (its|the company.s) ability to continue as a going concern" - - "substantial doubt regarding the company.s ability to continue as a going concern" - - "raise substantial doubt about (our|the company.s) ability to continue as a going concern" -``` - -A single match is enough to flag. - -## Mermaid diagram - -```mermaid -flowchart TD - Raw["raw/sec_edgar/.../form=10-K"] --> Reader["Filing reader (text extract)"] - Patterns["config/text_flags/going_concern_patterns.yaml"] --> Scanner["Regex scanner"] - Reader --> Scanner - Scanner --> Flag{"Match found?"} - Flag -->|Yes| Out["going_concern.parquet (flag = True)"] - Flag -->|No| OutNo["going_concern.parquet (flag = False)"] - Out --> PLF["permanent-loss-filter (BK_GOING_CONCERN)"] - OutNo --> PLF -``` - -## Expected flow - -1. Locate the latest 10-K per CIK whose `acceptance-datetime <= run_date`. -2. Extract plain text from the filing (HTML to text, no OCR; 10-Ks are HTML on EDGAR). -3. Run the regex scanner. -4. Persist a row per CIK with the flag, the matched phrase(s), the filing URL, the accession, and the pattern version. -5. The permanent loss filter joins this table on its `BK_GOING_CONCERN` rule. - -## Acceptance criteria - -- The going concern flag is reproducible for the same accession and the same pattern version (deterministic). -- A CIK without a recent 10-K is not silently flagged as `False`; it is marked `unknown` and routed to the review queue. -- The matched passage is stored alongside the flag for human review. -- Adding a new pattern requires a new `pattern_version`; old runs do not re-flag retroactively unless the user triggers a backfill. - -## Open questions - -- Do we want to also scan 10-Qs, or 10-Ks only? Recommendation: 10-Ks only for the MVP; going concern is mostly disclosed in the annual report. -- Section extraction: do we attempt to limit the search to specific 10-K items (Item 7, Item 8 notes), or scan the full filing? Recommendation: full filing for the MVP; precision is high enough. -- Should the flag have a TTL (e.g., expires 13 months after the filing date)? Recommendation: yes, default 400 days. - -## Risks - -- Some 10-Ks contain "going concern" language in a hypothetical or risk-factor context. The MVP rule will produce some false positives. Logged for FP/FN review. -- HTML parsing of EDGAR documents can fail on edge cases. The pipeline must capture and log parse errors instead of crashing. -- A pattern list in YAML is easy to break if patterns conflict. The `pattern_version` discipline is the only mitigation. - -## Future iterations (parked) - -- Auditor-change extraction (`FRD_AUDITOR_CHANGE_REPEATED` in permanent-loss-filter). -- Risk-factor year-over-year diff. -- Earnings call transcript ingestion + topic flags. -- LLM-driven summary with citation enforcement. -- Embeddings + RAG search inside the dashboard. diff --git a/docs/mvp/guides/download-fundamentals.md b/docs/mvp/guides/download-fundamentals.md deleted file mode 100644 index 3aa2cee..0000000 --- a/docs/mvp/guides/download-fundamentals.md +++ /dev/null @@ -1,178 +0,0 @@ -# Guide: Download fundamentals (local spike — frozen) - -> **Status:** This guide documents the **frozen SEC ETL spike** ([ADR-0001](../../adr/0001-simfin-fundamentals-mvp.md)). The June 30 demo pipeline uses **SimFin** instead — see [`demo-slice.md`](../demo-slice.md) and [`etl-data-lake.md`](../features/etl-data-lake.md). Do not delete this spike; it resumes in phase 2. - -Operator guide for the first ETL vertical slice: download raw SEC `companyfacts` and -standardized annual statements from `edgartools` for a parameterized universe. - -**Canonical spec:** [`../features/etl-data-lake.md`](../features/etl-data-lake.md) (section -*Fundamentals download spike*). - -## Prerequisites - -1. Python 3.11+ and Poetry installed. -2. Project dependencies installed: - - ```bash - poetry install - ``` - -3. **SEC identity** (required by SEC EDGAR and `edgartools`): - - ```bash - export SEC_IDENTITY="Your Name your@email.com" - ``` - - Use a real contact address. SEC may block requests with generic or invalid identities. - -4. Network access to `data.sec.gov` and SEC endpoints used by `edgartools`. - -## Quick start (Dow 30) - -```bash -export SEC_IDENTITY="Your Name your@email.com" -poetry run download-fundamentals --universe dow30 -``` - -Equivalent module invocation: - -```bash -poetry run python -m smartwealthai.download_fundamentals --universe dow30 -``` - -A full Dow 30 run downloads **120 artifacts** (4 per CIK: 1 JSON + 3 parquet files) and -typically takes several minutes because of SEC rate limits and `edgartools` parsing. - -## CLI reference - -Built with [Click](https://click.palletsprojects.com/). Run `download-fundamentals --help` for -auto-generated option docs. - -| Flag | Default | Description | -| --- | --- | --- | -| `--universe` | — | Preset name. Currently: `dow30`. | -| `--universe-file` | — | Path to a custom CSV (`ticker,cik`). Overrides preset when both are set. | -| `--data-dir` | `data` | Local data lake root. | -| `--periods` | `16` | Annual fiscal columns requested from `edgartools`. | -| `--as-of-date` | UTC today | Partition date (`YYYY-MM-DD`) for immutable daily snapshots. | -| `--force` | off | Re-download even when today's partition already exists. | - -### Examples - -```bash -# Custom universe CSV -poetry run download-fundamentals --universe-file data/reference/universes/dow30.csv - -# Pin snapshot date (reproducible backfill slice) -poetry run download-fundamentals --universe dow30 --as-of-date 2026-06-07 - -# Force refresh after a failed partial run -poetry run download-fundamentals --universe dow30 --force - -# Write to a temp lake (CI / experiments) -poetry run download-fundamentals --universe dow30 --data-dir /tmp/swai-lake -``` - -## Universe files - -Presets map to versioned CSV files under `data/reference/universes/`. See -[`../../../data/reference/universes/README.md`](../../../data/reference/universes/README.md). - -Format: - -```csv -ticker,cik -AAPL,0000320193 -MSFT,0000789019 -``` - -- `ticker` — trading symbol passed to `edgartools.Company(ticker)`. -- `cik` — 10-digit zero-padded SEC CIK used for `companyfacts` URLs. - -Fixed CIKs avoid ambiguity across share classes and ticker renames. - -## Output layout - -Under `{data-dir}/raw/`: - -```text -sec_edgar/cik=/endpoint=companyfacts/as_of_date=/response.json -edgartools/cik=/as_of_date=/income_statement_annual.parquet -edgartools/cik=/as_of_date=/balance_sheet_annual.parquet -edgartools/cik=/as_of_date=/cashflow_statement_annual.parquet -download_runs/as_of_date=/errors.json # only when failures occur -``` - -### Artifact summary - -| Artifact | Source | Contents | -| --- | --- | --- | -| `response.json` | SEC REST `/api/xbrl/companyfacts/CIK*.json` | Verbatim XBRL facts (raw zone). | -| `*_annual.parquet` | `edgartools` `get_facts()` | Parsed annual statements with `concept`, `label`, `section`, `FY 20xx` columns. | - -Raw downloads are **never** consumed directly by scoring modules in the MVP; a future -normalizer will produce curated parquet with point-in-time semantics. - -## Cache and re-runs - -```mermaid -flowchart TD - Start["Run download_fundamentals"] --> Load["Load universe CSV"] - Load --> Loop["For each ticker/CIK"] - Loop --> Check{"Today's file exists?"} - Check -->|yes, no --force| Skip["Skip network call"] - Check -->|no or --force| Fetch["Download from SEC / edgartools"] - Fetch --> Write["Write under raw/.../as_of_date=today/"] - Skip --> Next["Next CIK"] - Write --> Next - Next --> Loop - Loop --> Summary["Print summary; exit 1 if any failures"] -``` - -- Re-running on the **same day** without `--force` skips existing files. -- `--force` overwrites today's partition only. -- Prior dates remain immutable (append-only by `as_of_date`). - -## Error handling - -- One failing CIK does **not** abort the run. -- Transient SEC errors (429, 5xx, timeouts) retry up to 3 times with exponential backoff. -- Permanent errors (404, invalid CIK) fail immediately for that issuer. -- Failures are written to `raw/download_runs/as_of_date=/errors.json`. -- Exit code: `0` if all issuers succeed, `1` if any fail. - -## Module map - -| Module | Responsibility | -| --- | --- | -| `smartwealthai.download_fundamentals` | CLI orchestration and run summary. | -| `smartwealthai.universe` | Preset resolution and CSV loading. | -| `smartwealthai.lake_paths` | Path builders for the local raw zone. | -| `smartwealthai.sec_client` | SEC REST client (throttle, retry, `companyfacts`). | -| `smartwealthai.edgartools_client` | `get_facts()` statement extraction to parquet. | - -## Tests - -Hermetic unit tests (no network): - -```bash -poetry run pytest tests/test_download_fundamentals.py -q -``` - -Integration smoke test (requires `SEC_IDENTITY` and network) is intentionally **not** part -of PR CI. Run locally on a small CSV when validating credentials. - -## Expanding universes - -1. Add `data/reference/universes/.csv` with `ticker,cik` rows. -2. Register the preset in `UNIVERSE_PRESETS` inside `src/smartwealthai/universe.py`. -3. Run: `poetry run download-fundamentals --universe `. - -Planned expansions: S&P 500, Russell 3000, Nasdaq — same CSV + preset pattern. - -## Out of scope (this spike) - -- `submissions` ingest (SIC, filing index). -- Curated parquet / point-in-time normalizer. -- S3 upload and DuckDB views. -- Quarterly statements (`period="quarterly"`). diff --git a/docs/mvp/guides/download-simfin.md b/docs/mvp/guides/download-simfin.md deleted file mode 100644 index d94cff1..0000000 --- a/docs/mvp/guides/download-simfin.md +++ /dev/null @@ -1,80 +0,0 @@ -# Download SimFin bulk fundamentals and prices (demo) - -Operator guide for the **SimFin bulk connector** on the June 30 demo path. Canonical spec: [`etl-data-lake.md`](../features/etl-data-lake.md). - -## Prerequisites - -- Poetry environment installed (`poetry install`) -- `SIMFIN_API_KEY` in the environment (free tier from [simfin.com](https://simfin.com); never commit) - -```bash -export SIMFIN_API_KEY="" -``` - -## Download US bulk datasets - -Downloads six demo datasets into the raw lake under `data/raw/simfin/`: - -| Dataset | Variant | Lake partition | -| --- | --- | --- | -| `companies` | `default` | `dataset=companies/variant=default/market=us/` | -| `industries` | `default` | `dataset=industries/variant=default/market=us/` | -| `income` | `ttm` | `dataset=income/variant=ttm/market=us/` | -| `balance` | `quarterly` | `dataset=balance/variant=quarterly/market=us/` | -| `cashflow` | `ttm` | `dataset=cashflow/variant=ttm/market=us/` | -| `shareprices` | `latest` | `dataset=shareprices/variant=latest/market=us/` | - -Each partition also includes `as_of_date=/` and the verbatim SimFin CSV filename (e.g. `us-income-ttm.csv`). - -```bash -poetry run download-simfin -poetry run download-simfin --data-dir data --refresh-days 7 -poetry run download-simfin --as-of-date 2026-06-18 --force -``` - -This command writes raw snapshots only. Run the full demo pipeline in order: - -```bash -poetry run download-simfin --as-of-date 2026-06-18 -poetry run build-universe --run-date 2026-06-18 -poetry run normalize-simfin --snapshot-date 2026-06-18 --universe-run-date 2026-06-18 -poetry run download-prices --run-date 2026-06-18 --snapshot-date 2026-06-18 -poetry run compute-metrics --ticker AAPL --as-of-date 2026-06-18 -``` - -Or run ingest → score in one command (then launch the dashboard): - -```bash -poetry run run-demo-pipeline --run-date 2026-06-18 -poetry run run-demo-pipeline --run-date 2026-06-18 --skip-download -poetry run run-dashboard --data-dir data --run-date 2026-06-18 -``` - -Pass `--ticker` to limit the normalize step to specific names (intersect universe). Use `compute-metrics` for single-ticker ROC/EY smoke tests without a full scoring run. - -`normalize-simfin` requires `--universe-run-date` (after `build-universe`) or `--ticker` for smoke tests. It does not process the full SimFin US table by default. - -## Refresh and cache behaviour - -- **Skip:** Re-run without `--force` when the on-disk lake copy is younger than `--refresh-days` (default `7`). -- **Force:** `--force` re-downloads from SimFin and overwrites today's partition regardless of age. -- **SimFin package cache:** Intermediate downloads land in `data/cache/simfin/` before being copied into `raw/simfin/`. - -## Failures - -- A failure for one dataset does not stop the rest. -- Non-critical dataset (`cashflow`) failure still exits `0` when critical datasets succeed. -- Exit code `1` when all critical datasets (`companies`, `industries`, `income`, `balance`, `shareprices`) fail, or when `SIMFIN_API_KEY` is missing. -- Per-run errors are written to `raw/simfin/download_runs/as_of_date=/errors.json` when any dataset fails. - -## Module map - -| Module | Role | -| --- | --- | -| `smartwealthai.simfin_client` | Configure API key, safe bulk download (zip-slip guarded), resolve cache CSV path. | -| `smartwealthai.download_simfin` | CLI orchestration, skip/force logic, run summary. | -| `smartwealthai.lake_paths` | Raw lake path builders for `raw/simfin/`. | - -## Tests - -Hermetic tests live in `tests/test_download_simfin.py`. They mock SimFin network calls; PR CI does not require a live API key. diff --git a/docs/mvp/prds/ci-cd/ci-cd-prd.md b/docs/mvp/prds/ci-cd/ci-cd-prd.md deleted file mode 100644 index 5951ba3..0000000 --- a/docs/mvp/prds/ci-cd/ci-cd-prd.md +++ /dev/null @@ -1,231 +0,0 @@ -# PRD: CI/CD and MLOps Infrastructure (Phase 0) - -**Status:** Ready for implementation -**Canonical architecture:** `docs/mvp/architecture/architecture.md` -**Related specs:** ETL + data lake, permanent loss filter, backtesting, sell-watch (pipeline vertical slice) - ---- - -## Problem Statement - -SmartWealthAI is a portfolio-grade quantitative value-investing MVP that must demonstrate MLOps competence on AWS: reproducible builds, automated quality gates, containerized pipeline execution, and a clear path from development to production. Today the repository has Poetry dependencies and a development container, but no Makefile, no production Dockerfile, no GitHub Actions workflows, and no defined contract for how local development, CI tests, AWS data lake access, and deployment relate to each other. - -The developer also needs a phased approach that does not over-build infrastructure before the core investment pipeline exists (data ingestion → permanent loss filter → quality and cheapness scoring → ranking → model portfolio → backtest → sell-watch). Without an explicit CI/CD plan, work on the data lake and AWS risks becoming confusing: it is unclear what runs locally, what runs in CI, what touches S3, and when full continuous deployment should begin. - -## Solution - -Establish a **Phase 0 CI/CD foundation** that separates three concerns: - -1. **Fast, deterministic PR CI** — lint and unit/smoke tests against pinned fixtures; no network, no AWS, no live SEC or price provider calls. -2. **Independent AWS integration tier** — a manual and scheduled workflow that proves ingestion can write to the dev S3 data lake using GitHub OIDC (no long-lived AWS keys). -3. **Deferred full CD (Phase 1–2)** — after the pipeline container and business logic exist, deploy to ECS Fargate on merge to `develop` (dev) and `main` (prod with approval), using multiple Docker images over time but shipping only the **pipeline image** first. - -The data lake uses the **same layout everywhere** (raw, curated, point-in-time zones) with a configurable lake root URI: local file mirror for optional offline work, S3 dev bucket as the canonical store for real ingestion, S3 prod bucket for promoted runs. DuckDB reads Parquet from either backend. - -GitFlow maps environments: pull requests run CI on all branches; merge to `develop` eventually deploys dev; merge to `main` eventually deploys prod behind a GitHub Environment approval gate. - -## User Stories - -1. As a developer, I want a single Makefile with standard targets for install, lint, test, and local ingestion, so that dev, CI, and documentation all reference the same commands. -2. As a developer, I want Poetry to manage Python 3.11 dependencies and an in-project virtualenv, so that the environment matches the devcontainer and CI runners. -3. As a developer, I want PR CI to run automatically on every pull request, so that broken changes are caught before merge. -4. As a developer, I want PR CI to complete quickly without external network calls, so that feedback is reliable and merges are not blocked by SEC or yfinance outages. -5. As a developer, I want unit and smoke tests to use pinned fixtures representing universe, fundamentals, and prices, so that scoring and filtering logic is testable without a live data lake. -6. As a developer, I want Ruff to enforce lint and format checks in CI, so that code quality is consistent from day one. -7. As a developer, I want a smoke test that exercises the core pipeline path on fixtures (permanent loss → ROC → EY → combined rank → portfolio selection), so that regressions in the vertical slice are caught early. -8. As a developer, I want real data ingestion to target AWS S3 in a dev bucket, so that the project demonstrates cloud-native data lake practice rather than local-only storage. -9. As a developer, I want ingestion tested independently from PR CI via a separate workflow, so that AWS integration is proven without making every PR flaky or slow. -10. As a developer, I want the ingest integration workflow to be triggerable manually and on a weekly schedule, so that I can validate connectors after changes without waiting for a release. -11. As a developer, I want separate dev and prod S3 buckets from the start, so that test artifacts never mix with production data and IAM can be scoped correctly. -12. As a developer, I want GitHub Actions to authenticate to AWS via OIDC role assumption, so that no long-lived AWS access keys are stored in GitHub Secrets. -13. As a developer, I want distinct IAM roles for dev and prod GitHub environments, so that production permissions are tighter and auditable. -14. As a developer, I want a pipeline Docker image that is separate from future dashboard and control-plane images, so that batch jobs stay minimal and deploy boundaries are clear. -15. As a developer, I want only the pipeline image built and deployed in Phase 0–1, so that infrastructure work stays proportional to existing application code. -16. As a developer, I want the pipeline container to run on ECS Fargate (Spot where viable), so that daily batch execution is cost-effective and aligned with the architecture doc. -17. As a developer, I want merge to `develop` to eventually deploy to the dev environment (ECR tag + ECS task revision), so that integrated changes are runnable in AWS before production. -18. As a developer, I want merge to `main` to eventually deploy to prod with a required GitHub Environment approval, so that production promotion is deliberate. -19. As a developer, I want runtime secrets (API keys, broker credentials) sourced from AWS Secrets Manager at task runtime, so that secrets never live in the repository or container image. -20. As a developer, I want build-time configuration limited to non-secret environment identifiers (bucket names, regions, cluster names), so that the security model matches architecture decisions. -21. As a developer, I want a configurable lake root URI so the same ingestion and query code works against local mirrors and S3, so that I understand the data lake as layout plus Parquet, not as “local vs cloud” code forks. -22. As a developer, I want an optional gitignored local lake mirror for speed or offline work, so that I am not blocked when AWS is unavailable, without making local storage the source of truth. -23. As a developer, I want the permanent loss filter regression cases (Enron, Lehman, WorldCom) to run in CI once that module exists, so that bankruptcy/fraud exclusions remain auditable. -24. As a developer, I want full 20-year walk-forward backtests to run outside PR CI (manual or scheduled), so that long-running validation does not block every commit. -25. As a developer, I want CI to respect point-in-time correctness in fixture design, so that tests reinforce the project’s core constraint against look-ahead bias. -26. As a portfolio reviewer, I want the README and docs to explain the three-tier testing model (fixtures / AWS ingest-smoke / deploy), so that the MLOps story is interview-ready. -27. As a developer, I want Prefect orchestration deferred until the ECS task path works, so that scheduling complexity does not block the first end-to-end AWS run. -28. As a developer, I want the Streamlit dashboard image and CD deferred until dashboard code exists, so that deploy pipelines are not empty scaffolding. -29. As a developer, I want MLflow experiment tracking integrated after the pipeline stabilizes, so that run snapshots do not slow initial delivery. -30. As a developer, I want infrastructure definitions (buckets, OIDC provider, IAM roles, ECR repository, ECS cluster skeleton) versioned alongside the application, so that AWS setup is reproducible. -31. As a developer, I want conventional commit and GitFlow branch conventions documented and followed, so that `develop` and `main` map cleanly to dev and prod deploy workflows. -32. As a developer, I want the devcontainer to remain development-only and not used as the CI runner image, so that dev ergonomics and production slim images stay separate concerns. -33. As a developer, I want ECR images tagged with git commit SHA and environment labels, so that any ECS run is traceable to source control. -34. As a developer, I want CloudWatch Logs as the initial observability sink for ECS tasks, so that pipeline failures are debuggable without heavier tooling. -35. As a developer, I want a clear milestone sequence from M0 (Makefile + CI) through M4 (Fargate runs pipeline in dev) before prod CD, so that implementation order is unambiguous. - -## Implementation Decisions - -### Scope and phasing - -- **Phase 0 (now):** Makefile, Poetry groups, PR CI workflow, fixture layout, pipeline package skeleton, ingest-smoke workflow to dev S3, OIDC + bucket provisioning. No full deploy on every merge yet. -- **Phase 1:** Pipeline Dockerfile, ECR push on merge to `develop`, ECS Fargate task definition update for dev. -- **Phase 2:** Prod deploy on merge to `main` with GitHub Environment approval; prod OIDC role and prod bucket writes restricted to promoted tasks. -- **Phase 3+:** Prefect orchestration, dashboard image, MLflow server, EventBridge daily schedule — explicitly later. - -### Deployable units (multi-image, pipeline first) - -- Target architecture uses **multiple Docker images** over the MVP lifetime: pipeline worker (batch), dashboard (Streamlit), and optionally separate control-plane services. -- **Phase 0–1 ships only the pipeline image.** Dashboard and Prefect worker images are out of scope until their application modules exist. -- The pipeline image entrypoint runs batch stages (ingest, normalize, score, rank, backtest slice, sell-watch) driven by CLI subcommands or a single orchestrated command — exact CLI shape to be defined during M1 implementation. - -### Data lake and environment configuration - -- Lake layout follows architecture zones: **raw** (immutable provider payloads), **curated** (normalized Parquet), **pit** (point-in-time store keyed by as-of date), plus **curated/issues** for the review queue. -- A single configuration value, **lake root URI**, selects the storage backend: - - Local optional mirror: file-backed root under a gitignored directory for developer convenience. - - Dev canonical store: S3 dev bucket prefix. - - Prod store: S3 prod bucket prefix. -- DuckDB is the analytical engine reading Parquet from the configured root; no Athena in MVP. -- **CI does not read or write live S3.** Integration workflows use dev bucket only. - -### CI workflow (every pull request) - -- Triggers on pull requests targeting `develop` or `main` (and optionally other long-lived branches if added). -- Steps: checkout → Python 3.11 + Poetry install → `make lint` → `make test` → `make test-smoke`. -- **Lint:** Ruff check and format check on application and test packages. -- **Unit tests:** pytest with markers excluding integration tests; all data from committed fixtures. -- **Smoke test:** end-to-end pipeline on a tiny fixture universe proving the vertical slice wiring (may initially stub modules until implemented). -- No AWS credentials configured on PR workflows. -- No live calls to SEC EDGAR, yfinance, or paid API tiers. - -### AWS integration workflow (ingest-smoke) - -- Separate workflow from PR CI; triggers: `workflow_dispatch` and weekly cron. -- Runs against **GitHub Environment `dev`** with OIDC assumption of the dev IAM role. -- Executes a minimal ingestion job (small ticker subset) writing to the dev bucket raw zone, then validates object presence and basic schema/count checks. -- Failures notify via workflow status; they do not block unrelated PR merges unless explicitly wired later. - -### Authentication and IAM - -- **GitHub OIDC → AWS IAM role assumption** is mandatory; static `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` in GitHub Secrets are out of scope. -- Two roles minimum: **dev GitHub Actions role** (dev bucket read/write, dev ECR push, dev ECS register task definition) and **prod GitHub Actions role** (prod bucket, prod ECR, prod ECS — tighter trust policy, main branch only). -- Runtime tasks use **task execution roles** distinct from GitHub deploy roles; secrets read from AWS Secrets Manager at container start. -- Trust policies restrict repository, environment, and branch refs. - -### Storage and registry - -- Two S3 buckets from day one: **dev** and **prod**, with consistent prefix conventions for lake zones, cache, and future MLflow artifacts. -- One ECR repository (or repository per image type later) for the pipeline image; tags include commit SHA and environment (`dev-`, `prod-`, optional `latest-dev` / `latest-prod`). - -### Runtime and orchestration - -- **ECS Fargate** is the first-class CD target for the pipeline container (Spot where appropriate for cost). -- AWS Batch is a future alternative if job queue semantics become necessary; not Phase 0. -- **Prefect** orchestration is deferred; initial runs may be one-off Fargate tasks triggered by deploy workflow or manual run-task until Prefect is introduced. - -### Branching and deployment mapping (GitFlow) - -- Feature branches → PR → CI only. -- Merge to **`develop`** → Phase 1+ deploy dev (build, push ECR, update ECS task). -- Merge to **`main`** → Phase 2 deploy prod with required reviewer on GitHub `production` environment. -- Hotfix flow may merge to `main` and back to `develop`; deploy workflows must respect branch protections. - -### Application modules touched or introduced - -Deep modules (simple interfaces, testable in isolation): - -| Module | Responsibility | Phase | -| --- | --- | --- | -| **Configuration** | Lake root URI, environment name, AWS region, non-secret resource names | M0 | -| **Lake I/O** | Read/write Parquet under raw/curated/pit prefixes; abstract file vs S3 | M1 | -| **Ingestion connectors** | SEC EDGAR and price providers → raw zone | M1–M2 | -| **Normalization + PIT** | Curated schema, as-of date tagging, incremental refresh | M2 | -| **Permanent loss filter** | Hard exclusions for fraud/bankruptcy | M1 | -| **Scoring** | ROC rank, EY rank, combined rank | M1 | -| **Portfolio selection** | Top combined rank → 15–30 name model portfolio | M1 | -| **Backtest runner** | Walk-forward on PIT data; pass/fail vs benchmarks | M2+ | -| **Sell-watch** | Daily signals on model holdings | M2+ | -| **Pipeline CLI** | Subcommands invoked locally, in CI smoke, and in container | M1 | - -Makefile targets wrap Poetry commands so CI and humans share entrypoints: install, lint, test, test-smoke, local ingest, docker build (pipeline), and later deploy helpers. - -### Poetry dependency strategy - -- Single Poetry project for the repository. -- **Dev group:** pytest, ruff; optional moto for S3 mock unit tests if needed. -- **Pipeline optional/extras group:** duckdb, boto3, pyarrow, sec-edgar-downloader, and other pipeline dependencies as modules land — avoid bloating dev-only installs. -- Enable package mode once the application package under `src/` exists. - -### Relationship to devcontainer - -- The existing devcontainer remains **development-only** (Docker-in-Docker, AWS CLI, Poetry, Jupyter, forwarded ports for future Streamlit/MLflow). -- CI uses GitHub-hosted runners with Makefile + Poetry; it does not build or run the devcontainer image. -- Production pipeline Dockerfile is slim and separate from the devcontainer Dockerfile. - -### Milestone sequence - -| Milestone | Deliverable | -| --- | --- | -| **M0** | Makefile, Poetry dev groups, PR CI workflow, empty package + passing smoke stub, fixture directory | -| **M1** | Core modules on fixtures; permanent loss + scoring + rank smoke green; package mode on | -| **M2** | Dev/prod buckets, OIDC roles, ingest-smoke workflow green against dev S3 | -| **M3** | Pipeline Dockerfile, ECR push on merge to `develop` | -| **M4** | ECS Fargate task runs pipeline in dev with `LAKE_URI` pointing at dev bucket | -| **M5** | Prod deploy on `main` with approval gate | -| **M6** | Prefect, dashboard image, MLflow, scheduled daily runs | - -## Testing Decisions - -### What makes a good test here - -- Test **observable behavior** at module boundaries: given fixture inputs and a run date, expect exclusions, ranks, portfolio membership, or sell signals — not internal function call order. -- Fixture data must respect **point-in-time** semantics: each fundamental row carries an as-of date; tests pass only when queries filter `as_of_date <= run_date`. -- CI tests must be **hermetic**: no network, no AWS, deterministic ordering where ranks are involved. -- Integration tests that hit S3 or live APIs live in a **separate workflow or marker** (`integration`) and are never required for every PR. - -### Modules to test in PR CI - -| Module | Test type | Notes | -| --- | --- | --- | -| Configuration | Unit | Default lake URI, env overrides | -| Permanent loss filter | Unit + regression | Enron, Lehman, WorldCom must be hard-excluded at correct as-of dates when module exists | -| ROC / EY scoring | Unit | Known EBIT, capital, EV → expected ranks on tiny cross-section | -| Combined rank + portfolio | Smoke | Fixture universe → expected top-N names | -| Pipeline CLI smoke | Smoke | Invokes wired stages sequentially on fixtures | -| Lake I/O | Unit | Optional moto or local temp dirs; not live S3 in PR CI | - -### Modules tested outside PR CI - -| Module | Test type | Notes | -| --- | --- | --- | -| Ingestion connectors | Integration (ingest-smoke workflow) | Tiny live pull → dev S3 raw zone | -| Full backtest | Scheduled / manual | 20+ year walk-forward too slow for PR | -| ECS deploy | Post-deploy smoke | Run task after dev deploy; assert exit code and logs | - -### Prior art - -- Devcontainer PRD validated tooling via manual smoke checks (Python, Poetry, AWS CLI, Ruff, pytest). -- Architecture doc specifies Enron / Lehman / WorldCom regression in CI for the permanent loss filter — adopt when that module is implemented. -- Legacy notebooks and `src/` exploration are not test prior art; new tests live under the package test tree with fixtures. - -## Out of Scope - -- **Prefect** orchestration and Prefect Cloud/server setup in Phase 0–1. -- **Streamlit dashboard** Docker image and dashboard CD. -- **MLflow tracking server** on EC2 and artifact promotion workflows. -- **Full continuous deployment on day one** (Phase 0 is CI + ingest-smoke only). -- **Kubernetes / EKS** and AWS Batch as primary runtime (Batch remains a later option). -- **Static AWS access keys** in GitHub Secrets. -- **Single fat Docker image** for dashboard + pipeline + MLflow. -- **CI ingestion or scoring against live SEC/yfinance on every PR.** -- **Prod bucket writes** before Phase 2 prod deploy exists. -- **Transaction costs, taxes, live broker execution** — product scope, not CI/CD scope. -- **Terraform vs CDK choice** — infrastructure-as-code tool may be chosen during M2; not blocking M0. -- **Ruff/pytest detailed rule configuration** beyond enabling tools in Phase 0 (may follow as chore). - -## Further Notes - -- This PRD captures decisions from the CI/CD design session (grill-me). It should be reflected in a future feature spec under `docs/mvp/features/` (e.g. `cicd-infrastructure.md`) and cross-linked from `docs/mvp/architecture/architecture.md` when implementation starts — per spec-driven workflow, the feature spec becomes canonical for acceptance criteria and implementation status. -- The core product vertical slice remains: ingest → permanent loss filter → quality (ROC) → cheapness (EY) → combined rank → model portfolio (~30 names) → backtest → sell-watch. CI/CD serves that slice, not the reverse. -- Cost awareness: dev integration and ECS tasks should use minimal resource sizes and Spot where acceptable; align with the architecture soft budget (~low single-digit USD/month for control plane before storage growth). -- When implementation begins, update `docs/README.md` to index this PRD under an MVP PRDs section alongside the devcontainer PRD. -- GitHub issue creation with label `ready-for-agent` is recommended for tracking vertical implementation slices (`to-issues`), but this document is the saved PRD artifact at `docs/mvp/prds/ci-cd/ci-cd-prd.md` as requested. diff --git a/docs/mvp/prds/devcontainer/prd.md b/docs/mvp/prds/devcontainer/prd.md deleted file mode 100644 index 8b8ee9b..0000000 --- a/docs/mvp/prds/devcontainer/prd.md +++ /dev/null @@ -1,131 +0,0 @@ -# PRD: Development Container for SmartWealthAI - -## Problem Statement - -SmartWealthAI development currently depends on the developer's local machine configuration (macOS, Homebrew packages, Python version, Poetry, AWS CLI, etc.). This creates two problems: - -1. **Environment drift**: there is no guarantee that a fresh clone produces a working dev environment without manual setup steps. -2. **Cloud portability**: the architecture targets AWS (S3, ECS Fargate, ECR, Secrets Manager). The developer wants to be able to spin up an EC2 instance, clone the repo, open it in Cursor via SSH, and land in a fully functional dev environment identical to the local one -- with zero manual tool installation. - -## Solution - -Create a `.devcontainer/` configuration that packages the entire development toolchain (Python 3.11, Poetry, Docker, AWS CLI, GitHub CLI, system utilities, Cursor extensions) into a reproducible container. The developer opens the repo in Cursor (locally or via SSH to EC2), the container builds automatically, and all dependencies are ready. - -This is a **development-only** image. A separate, minimal production Dockerfile will be created later for ECS Fargate tasks. - -## User Stories - -1. As a developer, I want to open the repo in Cursor and have all Python dependencies installed automatically, so that I can start coding immediately without running setup scripts. -2. As a developer, I want the same dev environment on my Mac and on a remote EC2 instance, so that I never debug environment-specific issues. -3. As a developer, I want Docker available inside my dev container, so that I can build and test production Docker images locally before pushing to ECR. -4. As a developer, I want the AWS CLI pre-installed, so that I can interact with S3, ECR, Secrets Manager, and other AWS services during development. -5. As a developer, I want the GitHub CLI pre-installed, so that I can create PRs, manage issues, and check CI status from the terminal. -6. As a developer, I want `make`, `jq`, and `ripgrep` available, so that I have standard dev utilities for task automation, JSON inspection, and fast code search. -7. As a developer, I want Cursor to auto-detect the Poetry virtualenv as the Python interpreter, so that I never have to manually select the right Python. -8. As a developer, I want linting and formatting (Ruff) configured out of the box, so that code quality is enforced from day one. -9. As a developer, I want pytest available, so that I can run tests inside the container. -10. As a developer, I want Jupyter notebook support in Cursor, so that I can work with the existing `.ipynb` files in `notebooks/`. -11. As a developer, I want Streamlit (8501) and MLflow (5000) ports forwarded automatically, so that I can access dashboards from my browser when working remotely. -12. As a developer, I want AWS credentials handled via `~/.aws` mount (local) or IAM Instance Profile (EC2), so that no secrets are baked into the container image. -13. As a developer, I want to rebuild the container after changing its config and land in an updated environment, so that the setup evolves with the project. -14. As a developer, I want the container to use bash as the default shell, so that scripts behave consistently across dev and CI environments. - -## Implementation Decisions - -### Image strategy - -- **Dev-only container.** The devcontainer is not reused for CI/CD or production. A separate slim Dockerfile will be created later for ECS Fargate. -- **Base image:** `mcr.microsoft.com/devcontainers/python:3.11`. Provides a non-root `vscode` user, common utilities (git, curl, ssh, sudo), and native Cursor/VS Code remote compatibility. - -### Devcontainer features (pre-built add-ons) - -| Feature | Purpose | -|---|---| -| `ghcr.io/devcontainers/features/docker-in-docker` | Build and run Docker images inside the container | -| `ghcr.io/devcontainers/features/aws-cli` | Interact with AWS services (S3, ECR, Secrets Manager, etc.) | -| `ghcr.io/devcontainers/features/github-cli` | PR creation, issue management, CI status checks | - -### System packages (via Dockerfile) - -Installed on top of the base image via `apt-get`: - -- `make` -- task automation -- `jq` -- JSON processing (SEC EDGAR data, AWS CLI output) -- `ripgrep` -- fast codebase search - -### Python tooling - -- **Poetry** installed via `pipx` (already available in the MS base image). -- `poetry config virtualenvs.in-project true` so `.venv` lives inside the workspace. -- `poetry install` runs as `postCreateCommand` to auto-install all dependencies on container creation. -- **pytest** and **ruff** added as dev dependencies in `pyproject.toml`. - -### Cursor / VS Code customizations - -**Extensions:** - -| Extension ID | Purpose | -|---|---| -| `ms-python.python` | Python language support, IntelliSense, test discovery | -| `charliermarsh.ruff` | Linting + formatting | -| `ms-toolsai.jupyter` | Notebook support | - -**Settings:** - -| Setting | Value | Reason | -|---|---|---| -| `python.defaultInterpreterPath` | `${workspaceFolder}/.venv/bin/python` | Auto-select the Poetry venv | -| `python.terminal.activateEnvironment` | `true` | Auto-activate venv in terminals | - -### Port forwarding - -| Port | Service | -|---|---| -| 8501 | Streamlit dashboard | -| 5000 | MLflow tracking UI | - -### Credentials strategy - -- **Local (macOS):** mount `~/.aws` into the container (devcontainer mount config). -- **EC2:** IAM Instance Profile attached to the instance; AWS CLI picks it up via the metadata service automatically. -- **No secrets baked into the image.** Ever. - -### Shell - -- bash (Debian default). No zsh/oh-my-zsh customization. - -### Data directory - -- No special volume or mount config. `data/` is gitignored and stays empty on fresh clones. Data lives in S3 per the architecture; local `data/` is populated on demand by ETL bootstrap scripts. - -## Testing Decisions - -This is an infrastructure/tooling PRD, not a feature module. There is no application logic to unit-test. Validation is manual: - -- **Smoke test:** build the container locally (`Dev Containers: Rebuild Container` in Cursor), verify Python version, Poetry venv, installed tools (`docker --version`, `aws --version`, `gh --version`, `make --version`, `jq --version`, `rg --version`), and that `pytest` and `ruff` are importable. -- **EC2 test:** spin up an EC2 instance, install Docker, clone the repo, open via Cursor SSH, verify the same smoke checks pass. -- **Extension test:** confirm Cursor shows the correct Python interpreter and that Ruff linting is active on `.py` files. - -## Out of Scope - -- **Production Dockerfile.** That is a separate effort aligned with the ECS Fargate runtime decision in the architecture doc. -- **CI/CD integration.** GitHub Actions has its own runner environment; the devcontainer is not used there. -- **Data provisioning.** No EBS volumes, S3 sync scripts, or seed data in the container. -- **GPU support.** Not needed for the MVP (no ML training workloads). -- **Custom shell (zsh/oh-my-zsh).** Can be added later if desired. -- **Prefect / MLflow server setup.** Those are runtime services, not dev environment concerns. -- **Ruff / pytest configuration** (rules, pyproject sections). Adding the packages is in scope; configuring them is a follow-up. - -## Further Notes - -- The devcontainer config is fully version-controlled under `.devcontainer/` and evolves with the project. Any team member (or the developer on a new machine) gets the same environment by opening the repo. -- The architecture doc (`docs/mvp/architecture/architecture.md`) references GitHub Actions + ECR for Docker image builds. The devcontainer's Docker-in-Docker feature allows local testing of those images before pushing. -- This PRD does not create a feature spec under `docs/mvp/features/` because the devcontainer is developer tooling, not an MVP feature module. It is tracked as a standalone PRD. - -## Files to Create or Modify - -| File | Action | -|---|---| -| `.devcontainer/devcontainer.json` | Create -- main devcontainer configuration | -| `.devcontainer/Dockerfile` | Create -- system packages on top of MS base image | -| `pyproject.toml` | Modify -- add `pytest` and `ruff` as dev dependencies | diff --git a/docs/mvp/prds/phase2/prd.md b/docs/mvp/prds/phase2/prd.md deleted file mode 100644 index b03e11b..0000000 --- a/docs/mvp/prds/phase2/prd.md +++ /dev/null @@ -1,322 +0,0 @@ -# PRD: MVP Phase 2 — Quantitative Value, Cloud, Backtest, Sell-Watch - -**Status:** Ready for implementation -**Canonical architecture:** `docs/mvp/architecture/architecture.md` -**Prior delivery:** June 30 demo slice (`docs/mvp/demo-slice.md`, ADR-0002) -**Related specs:** ETL + data lake, permanent loss filter, backtesting, sell-watch, universe construction, dashboard reporting -**Related PRDs:** CI/CD (`docs/mvp/prds/ci-cd/ci-cd-prd.md`) -**Capacity assumption:** Solo developer, ~10–15 hours per week -**Estimated calendar:** Phase 2a ~13–16 weeks; Phase 2b ~8–12 weeks (~5–7 months total) - ---- - -## Problem Statement - -The June 30 demo slice delivers a working Greenblatt-style Magic Formula pipeline (ROC + Earnings Yield → combined rank → top-30 equal-weight model portfolio) on local SimFin data with a Streamlit dashboard and MLflow file-store logging. That slice proves ingestion, point-in-time fundamentals, cross-sectional ranking, and explainability — but it is not the investor's target strategy, not deployed to AWS, not historically validated, and does not monitor holdings for thesis breaks. - -The investor wants Phase 2 to: - -1. **Replace production scoring** with the *Quantitative Value* methodology (Wesley R. Gray / Tobias Carlisle): forensic screens, value funnel (EBIT/TEV), quality funnel (FS-Score), and a concentrated model portfolio (~50 names). -2. **Run the pipeline in AWS** so daily scoring is independent of a developer laptop. -3. **Backtest the strategy** to judge whether it is worth following — starting with a light historical run, then expanding to the full architecture spec. -4. **Alert on sell conditions** when a holding's QV thesis deteriorates — without auto-execution or paper trading in this phase. - -Without a single PRD tying these goals together, Phase 2 risks repeating the demo's scope creep in reverse: cloud work before the QV funnel exists, or backtests that still score ROC+EY while production claims to be Quantitative Value. - -## Solution - -Deliver Phase 2 in two increments: - -### Phase 2a (core) - -1. Extend the data lake for **multi-period fundamentals** and **daily prices** (5–10 year window) with point-in-time correctness preserved. -2. Implement **forensic / permanent-loss screening** including Beneish M-Score and the distress rules already specified, with a QVAL-style bottom-percentile gate on forensic models. -3. Replace the production scoring path with the **full QV funnel**: universe → forensic hard exclusion → top ~10% by EBIT/TEV → FS-Score on the value pool → top ~50 equal-weight model portfolio. -4. Keep **Magic Formula (ROC + EY + combined rank)** as a **benchmark module only** for backtest comparison — not production scoring. -5. Run a **light backtest** (5–10 years, annual rebalance, current SimFin US universe, S&P 500 CW + Magic Formula benchmarks) using a custom pandas/DuckDB engine (no Zipline). -6. Deploy **cloud phase 2a**: S3 data lake + artifacts, ECS Fargate Spot daily cron, Secrets Manager, MLflow tracking with S3 artifact store (extends CI/CD PRD milestones M1–M4). -7. Implement **sell-watch with QV-adapted triggers**; surface signals in the dashboard and curated parquet (email deferred to 2b). - -### Phase 2b (validation + operations) - -1. Historical **S&P 500 constituents including delisted** names (survivorship-bias mitigation). -2. **Full backtest** per `backtesting.md`: 20+ years, walk-forward 3–5 year windows, block-bootstrap Monte Carlo, crisis drawdown report, Sharpe gate vs four benchmarks. -3. **Cloud phase 2b**: Prefect or EventBridge orchestration, AWS SES email alerts, Streamlit dashboard hosted on AWS. - -Magic Formula remains the strategy's **benchmark comparator** for Sharpe pass/fail in 2b; production portfolio construction follows the QV funnel throughout. - -## User Stories - -### Strategy and scoring - -1. As an investor, I want the production pipeline to implement the Quantitative Value funnel (forensics → value → quality → portfolio), so that my model portfolio reflects the book's methodology rather than a Greenblatt placeholder. -2. As an investor, I want forensic accounting screens to hard-exclude companies at elevated fraud or bankruptcy risk before any value or quality score, so that permanent capital loss is filtered systematically. -3. As an investor, I want the Beneish M-Score included in forensic screening, so that earnings manipulation risk is part of the safety layer. -4. As an investor, I want companies in the bottom 5% of forensic models excluded (QVAL-style), so that the safety screen matches the published ETF process. -5. As an investor, I want the value screen to keep the top decile (~10%) of names by EBIT/TEV among survivors, so that I only quality-rank genuinely cheap stocks. -6. As an investor, I want quality ranked by the 10-point FS-Score (Gray/Carlisle variant) on the value pool, so that the final portfolio favors financially strong cheap names. -7. As an investor, I want the model portfolio to hold approximately 50 equal-weight long-only names after the quality screen, so that the portfolio matches QVAL concentration. -8. As an investor, I want every funnel stage to log how many names passed or failed, so that I can audit shrinkage from universe to portfolio. -9. As an investor, I want each score and exclusion to record formula version and inputs, so that any decision is reconstructible from the data lake. -10. As an investor, I want the dashboard to explain why a name is in the portfolio using QV stage outputs (forensic pass, EBIT/TEV rank, FS-Score components), so that the system stays explainable. -11. As a developer, I want Magic Formula ROC and EY scoring preserved as a separate benchmark path, so that backtests can compare QV against the Greenblatt replica without dual production logic. -12. As a developer, I want production and benchmark code paths named distinctly (QV vs MF), so that glossary terms in CONTEXT.md do not drift in implementation. - -### Data and point-in-time - -13. As a developer, I want annual and quarterly income, balance sheet, and cash flow stored in raw and curated zones, so that FS-Score year-over-year deltas are computable. -14. As a developer, I want the PIT fundamentals interface to return the correct historical filing rows for any decision date, so that backtests never leak future fundamentals. -15. As a developer, I want daily adjusted prices for at least a 5–10 year window in curated storage, so that light backtests and enterprise value history are supported. -16. As a developer, I want SimFin `shareprices/daily` as the primary price history source with a vendor fallback when needed, so that backtests are not blocked by free-tier snapshot lag. -17. As a developer, I want missing inputs for forensic or FS-Score rules routed to the review queue rather than silently dropped, so that data quality issues are visible. -18. As a developer, I want restatements to create new `version_id` rows with updated `as_of_date`, so that historical queries reflect what was knowable at each decision date. -19. As an investor, I want sector hard exclusions (banks, insurers, utilities) to remain upstream of QV scoring, so that incomparable financials never enter the funnel. - -### Permanent loss and forensics - -20. As an investor, I want Altman Z-score, interest coverage, net debt/EBITDA, negative equity, and delisting rules to remain available as distress signals, so that the permanent loss filter matches the existing spec where data allows. -21. As a developer, I want a CI regression test that forces Enron, Lehman, and WorldCom to be excluded at documented distress dates, so that bankruptcy screening cannot regress silently. -22. As a developer, I want each exclusion to store `rule_id`, `rule_version`, triggered values, and `as_of_date`, so that MLflow and the dashboard can show why a company was removed. -23. As a developer, I want fraud rules (restatement, auditor change, late filer) implemented where EDGAR data exists, with `unavailable` logged otherwise, so that the module is extensible without blocking on SEC ETL. - -### Backtesting - -24. As an investor, I want a light backtest over 5–10 years with annual rebalancing, so that I can see whether the QV strategy had acceptable risk-adjusted returns before investing further effort. -25. As an investor, I want the light backtest to recompute the full QV funnel at each rebalance date using only point-in-time data, so that results are not inflated by look-ahead bias. -26. As an investor, I want light backtest results compared to S&P 500 cap-weighted and Magic Formula replica benchmarks, so that I have familiar reference points. -27. As an investor, I want the light backtest universe limitation (current SimFin US, survivorship bias) clearly labeled in reports, so that I do not over-interpret early results. -28. As a developer, I want backtest runs logged as MLflow experiments with equity curve and trade ledger artifacts, so that each historical run is reproducible. -29. As a developer, I want long backtests to run outside PR CI (manual trigger or ECS ad-hoc task), so that commits are not blocked by 20-year simulations. -30. As an investor, I want Phase 2b to add a 20+ year walk-forward backtest with Monte Carlo and crisis drawdown reporting, so that the strategy meets the architecture validation bar. -31. As an investor, I want Phase 2b Sharpe compared against S&P 500 CW, S&P 500 EW, Russell 3000, and Magic Formula replica, so that pass/fail is objective when paper trading arrives later. -32. As a developer, I want delisted and bankrupt holdings handled with zero terminal price on delisting date in full backtests, so that NAV reflects realized losses. - -### Cloud and MLOps - -33. As a developer, I want the pipeline to run daily on ECS Fargate Spot without my laptop, so that the system is a real operational batch job. -34. As a developer, I want the data lake canonical store on S3 with a configurable lake root URI, so that the same code runs locally and in AWS. -35. As a developer, I want runtime secrets (e.g. `SIMFIN_API_KEY`) from AWS Secrets Manager, so that keys are not in the image or repository. -36. As a developer, I want MLflow run artifacts stored in S3, so that pipeline and backtest snapshots survive beyond a single machine. -37. As a developer, I want GitHub OIDC to deploy the pipeline image to ECR and update ECS task definitions on merge to `develop`, so that cloud deploys trace to git SHA. -38. As a developer, I want CloudWatch Logs for ECS task output, so that pipeline failures are debuggable. -39. As a portfolio reviewer, I want the README to document that Phase 2a cloud scope is pipeline-only (dashboard local), so that the MLOps story is honest about what runs where. -40. As a developer, I want Phase 2b to add SES email on sell-watch signals and host Streamlit on AWS, so that alerts and reporting work when I am not watching the dashboard. - -### Sell-watch - -41. As an investor, I want daily evaluation of model portfolio holdings for QV thesis breaks, so that I know when to consider exiting a position. -42. As an investor, I want a sell signal when forensic screening starts failing on a holding, so that fraud or distress triggers an alert. -43. As an investor, I want a sell signal when FS-Score drops materially (YoY or below threshold), so that quality deterioration is caught. -44. As an investor, I want a sell signal when a holding falls out of the EBIT/TEV value decile, so that overvaluation relative to the strategy is flagged. -45. As an investor, I want a sell signal when a watchlist name outranks a holding by a configurable margin on the QV composite rank, so that opportunity cost is monitored. -46. As an investor, I want sell signals to require manual confirmation before any future order build, so that the system never auto-sells. -47. As a developer, I want every holding evaluation logged (signal or no signal) for audit, so that false positive and false negative rates can be reviewed later. -48. As an investor, I want Phase 2b email alerts via AWS SES for new sell signals, so that I am notified without opening the dashboard. - -### Documentation and governance - -49. As a developer, I want a new feature spec `quantitative-value.md` as the canonical QV module before implementation, so that spec-driven workflow is preserved. -50. As a developer, I want CONTEXT.md updated when QV terms (quality, cheap, funnel rank) are resolved, so that agents and humans share one vocabulary. -51. As a developer, I want `demo-slice.md` "After the demo" ordering updated to reflect QV-first production, so that docs do not contradict this PRD. - -## Implementation Decisions - -### Phasing - -| Increment | Scope | Exit criterion | -| --- | --- | --- | -| **2a** | Multi-period data, forensics + Beneish, QV funnel, light backtest, cloud pipeline (S3 + ECS + MLflow S3), sell-watch logic + dashboard | Daily QV portfolio on ECS; light backtest equity curve in MLflow; sell signals in dashboard | -| **2b** | S&P 500 historical universe, full backtest spec, Prefect/EventBridge, SES, Streamlit on AWS | 20y walk-forward backtest with Sharpe gate; email alerts; dashboard on AWS | - -Paper trading and broker execution are explicitly **out of Phase 2** (deferred until a passing full backtest exists in a later phase). - -### Deep modules (build or extend) - -These are intentionally **deep modules**: narrow public interfaces, substantial internal logic, stable contracts, testable in isolation. - -#### 1. Point-in-time fundamentals store (extend) - -- **Responsibility:** Given `decision_date` and `ticker` (or universe), return the latest fundamental rows per statement type with `as_of_date <= decision_date`; support multiple historical periods for YoY deltas. -- **Interface shape:** Query functions returning normalized provider-agnostic columns (`ebit`, `total_assets`, `cash`, etc.) plus metadata (`as_of_date`, `version_id`, `formula_version`). -- **Consumers:** Forensic evaluator, FS-Score calculator, EV/EBIT/TEV metrics, backtest engine. -- **Change frequency:** Low — extended for multi-period, not replaced. - -#### 2. Forensic evaluator (new) - -- **Responsibility:** Evaluate all fraud and distress rules per company at a run date; compute Beneish M-Score; compute cross-sectional percentiles; apply QVAL bottom-5% exclusion per model; emit hard `exclude` or `pass` with reasons. -- **Interface shape:** Input: universe pass list + PIT fundamentals (+ optional filing flags). Output: exclusions table keyed by `(cik, run_date)` with `rule_id`, `triggered_value`, `threshold`, `explanation`. -- **Consumers:** QV funnel stage 1, sell-watch `SW_FORENSIC`, CI regression fixtures. -- **Change frequency:** Medium — thresholds tuned, new rules versioned. - -#### 3. FS-Score calculator (new) - -- **Responsibility:** Compute the 10 binary FS-Score components (Gray/Carlisle variant: profitability, stability, recent operational improvements) from multi-period PIT inputs; sum to integer score 0–10. -- **Interface shape:** Input: PIT income/balance/cashflow history for one ticker at `decision_date`. Output: component dict, total score, `formula_version`. -- **Consumers:** QV funnel stage 3, sell-watch `SW_FS_SCORE_DROP`, dashboard explainability. -- **Change frequency:** Low — tied to published FS-Score definition. - -#### 4. QV funnel orchestrator (new) - -- **Responsibility:** Run sequential funnel stages with auditable counts; no scoring logic inside — delegates to forensic evaluator, value ranker, FS-Score ranker, portfolio constructor. -- **Stages:** - 1. Universe pass (from universe construction) - 2. Forensic hard exclusion - 3. Value screen: rank by EBIT/TEV descending; keep top decile (configurable count or fraction) - 4. Quality screen: FS-Score on value pool; keep top 50 (configurable) - 5. Portfolio: equal-weight selected names; optional market-cap tie-break on rank ties -- **Interface shape:** Input: `run_date`, lake root, config. Output: `ScoringResult`-like object with stage counts, ranked tables per stage, final `model_portfolio` rows, MLflow-ready metrics. -- **Consumers:** `score-universe` CLI, dashboard, backtest engine, sell-watch. -- **Change frequency:** Low — stage order fixed by QV methodology. - -#### 5. Value metrics (extend from cheapness module) - -- **Responsibility:** Compute EBIT, enterprise value, EBIT/TEV (same EV definition as Earnings Yield module); cross-sectional rank within forensic survivors. -- **Interface shape:** Reuse existing metrics building blocks where possible; separate production path from MF `EY rank`. -- **Consumers:** QV funnel stage 2, sell-watch `SW_VALUE_POOL_EXIT`. - -#### 6. Magic Formula benchmark path (preserve, demote) - -- **Responsibility:** ROC + EY + combined rank + top-N portfolio exactly as demo slice; used only for benchmark portfolio reconstruction in backtests. -- **Interface shape:** Existing ranking module interface unchanged; invoked only from backtest benchmark builder and smoke tests until smoke test updated to QV path. -- **Consumers:** Backtest benchmark comparator, CI smoke test (transition: add QV smoke, keep MF benchmark test). - -#### 7. Light backtest engine (new) - -- **Responsibility:** Loop rebalance dates; pin PIT data per date; invoke full QV funnel; simulate equal-weight holdings and daily NAV from curated prices; compare to benchmark series. -- **Interface shape:** Input: `start_date`, `end_date`, `rebalance_frequency`, config hash. Output: equity curves, trade ledger, holdings parquet, summary metrics → MLflow `backtesting` experiment. -- **Constraints:** No live network in run; read only curated parquet; custom loop (ADR-0002: no Zipline). -- **Consumers:** Manual/ECS ad-hoc runs, dashboard backtest panel (Phase 2a minimal). - -#### 8. Sell-watch evaluator (new) - -- **Responsibility:** For each model portfolio holding at `run_date`, evaluate QV triggers; dedupe against confirmed/dismissed history; write signals and full evaluation audit. -- **Trigger IDs:** `SW_FORENSIC`, `SW_FS_SCORE_DROP`, `SW_VALUE_POOL_EXIT`, `SW_QV_OPPORTUNITY` (replace ROC/EY triggers from sell-watch spec for production). -- **Interface shape:** Input: holdings, watchlist, latest scores, config thresholds. Output: signals parquet + evaluations parquet. -- **Consumers:** Dashboard, SES (2b), future broker module. - -#### 9. Lake root and cloud runtime (extend) - -- **Responsibility:** Abstract storage backend (local file vs S3 prefix); same zone layout (raw, curated, issues); DuckDB reads parquet from configured root. -- **Interface shape:** Single `LAKE_ROOT_URI` (or equivalent) consumed by all ingest and scoring CLIs. -- **Consumers:** All pipeline stages, ECS task entrypoint, ingest-smoke workflow. -- **Aligns with:** CI/CD PRD milestones M1–M4 for 2a; M4 uses EventBridge cron → ECS Fargate Spot → pipeline entrypoint. - -### Production vs benchmark separation - -| Concern | Production (Phase 2+) | Benchmark only | -| --- | --- | --- | -| Safety | Forensic evaluator + permanent loss rules | — | -| Value | EBIT/TEV value decile | EY rank (MF) | -| Quality | FS-Score on value pool | ROC rank (MF) | -| Ranking | QV funnel sequential rank | Combined rank = ROC + EY | -| Portfolio size | ~50 EW default | MF replica uses same universe/filters as configured for comparison | - -### Configuration and versioning - -- QV funnel parameters (decile fraction, portfolio size, FS-Score tie-break) live in versioned YAML under a `config/quantitative_value/` namespace. -- Forensic thresholds and Beneish coefficients versioned under `config/permanent_loss/`. -- Sell-watch thresholds versioned under `config/sell_watch/` with QV trigger IDs. -- Every MLflow run logs `git_sha`, config hashes, and stage counts. - -### Dashboard changes - -- Replace ROC/EY-centric explainability with QV stage breakdown per ticker. -- Add light backtest summary panel (equity curve, key metrics, survivorship bias warning). -- Add sell-watch signal list with trigger detail and confirm/dismiss actions (confirm does not build orders in Phase 2). - -### Glossary updates (CONTEXT.md) - -When implementation starts, resolve: - -- **Quality (production):** FS-Score composite, not ROC alone. -- **Cheap (production):** Membership in EBIT/TEV value pool, not EY rank alone. -- **QV funnel rank:** Order after quality screen within the value pool; supersedes **combined rank** for production. -- **Combined rank:** Retained for **Magic Formula replica** benchmark only. - -### Dependency order (2a) - -1. Feature spec `quantitative-value.md` -2. Multi-period ETL + PIT extension + daily prices -3. Forensic evaluator (+ Beneish) -4. FS-Score calculator -5. QV funnel orchestrator wired into scoring CLI -6. Light backtest engine -7. Cloud 2a (pipeline stable locally first) -8. Sell-watch evaluator + dashboard - -## Testing Decisions - -### Principles - -- Test **external behavior** (inputs → outputs, exclusions, ranks, portfolio membership) not internal implementation details. -- All PR CI tests use **pinned fixtures** — no SimFin, yfinance, or AWS calls in the default test job. -- Fixture design must respect **point-in-time correctness** (no row with `as_of_date > decision_date` in historical scenarios). -- Long-running backtests and full 20-year walk-forward run **outside PR CI** (manual or scheduled ECS). - -### Modules to test (priority) - -| Module | Priority | What to assert | -| --- | --- | --- | -| Forensic evaluator | **P0** | Enron, Lehman, WorldCom excluded at fixture dates; Beneish bottom-5% gate; exclusion reason columns populated | -| FS-Score calculator | **P0** | Known fixture company gets expected 0–10 score; each binary component matches hand-checked inputs | -| QV funnel orchestrator | **P0** | Fixture universe shrinks monotonically through stages; final portfolio size ≤ configured cap; excluded names never appear | -| Value metrics (EBIT/TEV) | **P1** | EV formula matches versioned config; negative EBIT routed to review queue | -| PIT fundamentals store | **P1** | Historical `decision_date` returns correct row; restatement version selection | -| Light backtest engine | **P1** | No look-ahead: fundamentals after rebalance date absent; turnover ledger balances | -| Sell-watch evaluator | **P1** | Each trigger fires on constructed holding; no signal when thresholds not met; dedupe of confirmed signals | -| Magic Formula benchmark | **P1** | Regression: demo slice ROC/EY/combined rank unchanged on fixtures (benchmark path not broken) | -| Lake root abstraction | **P2** | Local vs `s3://` prefix resolves same relative paths (mock or minio optional) | - -### Prior art in codebase - -- `fixture_lake.py` and `point_in_time_fundamentals()` for PIT fixture queries. -- `magic_formula_ranking.py` tests (if present) for rank assignment and portfolio selection patterns. -- CI/CD PRD user story 7: smoke test on fixture vertical slice — **update smoke to QV funnel** once forensic + funnel exist; keep MF benchmark unit tests separate. -- Permanent loss spec: Enron / Lehman / WorldCom regression cases under `tests/fixtures/permanent_loss/`. - -### CI vs integration - -| Tier | Runs | Scope | -| --- | --- | --- | -| PR CI | Every pull request | Lint, unit tests, QV smoke on fixtures, forensic regression | -| Ingest-smoke | Weekly / manual | S3 write from SimFin (existing CI/CD PRD workflow) | -| Light backtest | Manual / ECS ad-hoc | Real curated lake, 5–10 years | -| Full backtest (2b) | Manual / ECS / Batch | 20+ years, walk-forward, Monte Carlo | - -## Out of Scope - -- **Paper trading and broker execution** — no simulated or real orders in Phase 2. -- **Auto-execution of sell signals** — confirm/dismiss only; no order builder. -- **Corroborative signals** (buybacks, insider, short interest) — deferred. -- **Unstructured financial data** (LLM filings, going-concern NLP) — deferred; `BK_GOING_CONCERN` remains future. -- **SEC EDGAR normalizer as primary fundamentals source** — optional 2b+; SimFin remains primary for Phase 2. -- **Moat "pre-flight checklist"** and full forensic model zoo beyond Beneish + spec distress rules — optional 2b+. -- **Score-weighted and risk-parity portfolio weighting** — equal-weight only in 2a; weighting as backtest hyperparameter in 2b only. -- **Personal portfolio CSV evolution and NAV** — `portfolio-evolution.md` deferred. -- **Prefect server, Streamlit on AWS, SES** — Phase 2b only (not 2a). -- **Production deploy to prod ECS on `main`** — may follow 2a dev deploy; prod CD per CI/CD PRD Phase 2 when ready. -- **Zipline or third-party backtest frameworks** — rejected (ADR-0002). - -## Further Notes - -### Time and risk - -- **2a:** ~13–16 weeks at 10–15 h/week if SimFin data coverage is sufficient. -- **2b:** ~8–12 additional weeks. -- **Risks:** SimFin free-tier limits on multi-period and daily prices; survivorship bias in 2a light backtest (must be labeled); Beneish missing inputs shrinking the funnel; backtest compute cost (full funnel × rebalance dates) — plan DuckDB pushdown or materialized stage tables early. - -### Open question (not blocking PRD) - -- **Portfolio size:** QVAL uses ~50 names; demo uses 30. Default recommendation: **50** with configurable cap in QV config. Resolve in `quantitative-value.md` spec before coding. - -### Relationship to other documents - -- This PRD does **not** replace per-module feature specs; it coordinates them. Each module keeps acceptance criteria in `docs/mvp/features/`. -- Implementers should read ADR-0001 (SimFin fundamentals), ADR-0002 (demo scope cut / no Zipline), and the CI/CD PRD before cloud work. -- After Phase 2a ships, update `demo-slice.md` "After the demo" ordering and `architecture.md` functional flow to state QV as production scoring. - -### Suggested GitHub issue title - -`PRD: MVP Phase 2 — Quantitative Value, Cloud, Backtest, Sell-Watch` - -Link this PRD path in the issue body; label `ready-for-agent` when creating tracker entry. diff --git a/docs/mvp/requirements/requirements.md b/docs/mvp/requirements/requirements.md deleted file mode 100644 index f5fc26f..0000000 --- a/docs/mvp/requirements/requirements.md +++ /dev/null @@ -1,39 +0,0 @@ -# Sprint 0 requirements: Magic Formula screener spike - -> **Status:** Historical reference only. The full MVP is defined in [architecture.md](../architecture/architecture.md) and [features/](../features/). Do not treat this document as the current MVP scope. - -## Primary goal - -Build a minimal Python pipeline that downloads financial data for a very small set of tickers, computes a simple ranking, and prints the result to the console. - -## Functional requirements - -### Input - -- Start from a **static hardcoded list** of 5–10 known tickers (e.g. `["AAPL", "MSFT", "GOOGL", "JNJ", "KO"]`). -- Do not download the full S&P 500 in this spike (API rate limits and runtime). - -### Processing - -- Connect to a free API (recommended: `yfinance`). -- Fetch proxy metrics for the Magic Formula: - - **Return on Capital (ROC)**, or fallback **ROE** / **ROA** - - **Earnings Yield**, or fallback inverse **P/E** -- Rank each metric from 1 to N across the universe and **sum ranks** for a final Magic Rank. - -### Output - -- Print the final ranking to the console (plain `print` or a small pandas table), best to worst. - -## Technical requirements - -- **Language:** Python 3.x (project now standardizes on 3.11+ via Poetry) -- **Libraries:** `yfinance`, `pandas` -- **Version control:** Git with a few local commits - -## Explicitly out of scope for Sprint 0 - -- Databases, GUI, Docker, machine learning -- Downloading thousands of tickers - -Those belong to later MVP modules documented under `docs/mvp/features/`. diff --git a/docs/mvp/features/portfolio-evolution.md b/spec/features/009-portfolio-evolution/spec.md similarity index 96% rename from docs/mvp/features/portfolio-evolution.md rename to spec/features/009-portfolio-evolution/spec.md index 27cf29a..0181335 100644 --- a/docs/mvp/features/portfolio-evolution.md +++ b/spec/features/009-portfolio-evolution/spec.md @@ -12,7 +12,7 @@ The user must be able to answer the questions "have my own decisions been worth ## MVP scope - Consume the consolidated personal-operations CSV (already in EUR) as the single source of truth for the personal portfolio. -- Use the closed schema documented in `docs/mvp/architecture/architecture.md` (Date, Symbol, Type, Volume, Price, Value, Commission, Currency). +- Use the closed schema documented in `spec/constitution/mission.md` (Date, Symbol, Type, Volume, Price, Value, Commission, Currency). - Reconstruct daily NAV in EUR using adjusted prices from the curated price store. - Apply the broker-to-yfinance ticker mapping (`data/reference/ticker_mapping.csv`). - Treat bankrupt or delisted holdings systematically: write a closing price of zero on the delisting date instead of dropping the position. @@ -113,5 +113,5 @@ flowchart TD - The CSV schema is the contract; any silent change in the cleaning pipeline (`src/preprocessing/cleaning_operations.py`) can break the portfolio evolution module. The CI should include a schema test. - Ticker remapping in code paths can drift from the CSV file in `data/reference/`. The mapping must be loaded from the file only. - Bankrupt-ticker handling can mask data errors as real losses. The pipeline must log a clear warning when a ticker disappears, and require an explicit "delisted_on" entry in `data/reference/delistings.csv` before zeroing the price. -- Free yfinance prices can be wrong or missing for older tickers (especially European listings). The fallback chain in `etl-data-lake` must cover this. +- Free yfinance prices can be wrong or missing for older tickers (especially European listings). The fallback chain in [`../006-etl-data-lake/spec.md`](../006-etl-data-lake/spec.md) must cover this. - Paper-trading the model assumes execution at the close of the rebalance date. This is optimistic; future iterations should model open-next-day execution and a basic spread cost. From e80acffcf3a1ba1c479807a489ae11e333fe9f3a Mon Sep 17 00:00:00 2001 From: JLaborda <15078416+JLaborda@users.noreply.github.com> Date: Wed, 1 Jul 2026 07:59:45 +0000 Subject: [PATCH 5/5] docs(mvp): generalize feature spec template sections Use Delivery + In scope / Out of scope instead of MVP-prefixed headings. Co-authored-by: Cursor --- .cursor/rules/mvp-docs.mdc | 3 ++- spec/meta/feature-spec-template.md | 15 +++++++++++---- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/.cursor/rules/mvp-docs.mdc b/.cursor/rules/mvp-docs.mdc index 76390e0..fe18432 100644 --- a/.cursor/rules/mvp-docs.mdc +++ b/.cursor/rules/mvp-docs.mdc @@ -13,7 +13,8 @@ When creating or editing specs under `spec/`: Follow the structure in [`spec/meta/feature-spec-template.md`](../../spec/meta/feature-spec-template.md) and existing features (e.g. `spec/features/003-cheap-stocks/spec.md`): - Objective -- MVP scope / Out of MVP scope +- Delivery (phase / roadmap link) +- In scope / Out of scope - Inputs (tables where helpful) - Flow or logic (include at least one **Mermaid** diagram when the flow changes) - Acceptance criteria diff --git a/spec/meta/feature-spec-template.md b/spec/meta/feature-spec-template.md index beccb85..77f954e 100644 --- a/spec/meta/feature-spec-template.md +++ b/spec/meta/feature-spec-template.md @@ -35,17 +35,24 @@ spec/features/00N-slug/ +## Delivery + + + + +**Phase:** + ## Objective -## MVP scope +## In scope - + -## Out of MVP scope +## Out of scope - + ## Inputs