Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 75 additions & 0 deletions .github/workflows/ingest-smoke.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
name: Ingest smoke (dev S3)

# Separate from PR CI: proves GitHub OIDC → AWS dev bucket access.
# Requires GitHub Environment "dev" with repository variables (see docs/mvp/guides/cloud-foundation.md).

on:
workflow_dispatch:
schedule:
- cron: "0 6 * * 1"

permissions:
id-token: write
contents: read

jobs:
ingest-smoke:
runs-on: ubuntu-latest
environment: dev
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Configure AWS credentials (OIDC)
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ vars.AWS_ROLE_ARN }}
aws-region: ${{ vars.AWS_REGION }}

- name: Write smoke marker to dev lake raw zone
env:
DEV_LAKE_BUCKET: ${{ vars.DEV_LAKE_BUCKET }}
DEV_LAKE_PREFIX: ${{ vars.DEV_LAKE_PREFIX }}
run: |
set -euo pipefail
if [ -z "${DEV_LAKE_BUCKET:-}" ]; then
echo "Missing DEV_LAKE_BUCKET. See docs/mvp/guides/cloud-foundation.md"
exit 1
fi
prefix="${DEV_LAKE_PREFIX:-}"
key="${prefix}raw/_smoke/ingest-smoke.txt"
echo "smartwealthai ingest-smoke $(date -u +%Y-%m-%dT%H:%M:%SZ)" > smoke.txt
aws s3 cp smoke.txt "s3://${DEV_LAKE_BUCKET}/${key}"

- name: Validate smoke object
env:
DEV_LAKE_BUCKET: ${{ vars.DEV_LAKE_BUCKET }}
DEV_LAKE_PREFIX: ${{ vars.DEV_LAKE_PREFIX }}
run: |
set -euo pipefail
prefix="${DEV_LAKE_PREFIX:-}"
key="${prefix}raw/_smoke/ingest-smoke.txt"
aws s3api head-object --bucket "${DEV_LAKE_BUCKET}" --key "${key}"
aws s3 cp "s3://${DEV_LAKE_BUCKET}/${key}" -
echo
echo "ingest-smoke OK: s3://${DEV_LAKE_BUCKET}/${key}"

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.11"

- name: Install Poetry
uses: snok/install-poetry@76e04a911780d5b312d89783f7b1cd627778900a
with:
virtualenvs-in-project: true

- name: Validate lake root URI parsing (no live S3 I/O)
run: |
poetry install --with dev
poetry run python -c "
from smartwealthai.lake_root import resolve_lake_root
root = resolve_lake_root('s3://${{ vars.DEV_LAKE_BUCKET }}/${{ vars.DEV_LAKE_PREFIX }}')
assert root.backend == 's3'
print('lake root:', root.uri, 'bucket=', root.s3_bucket)
"
49 changes: 40 additions & 9 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,12 +77,40 @@ _Avoid_: dividend yield, earnings/price without EV
_Avoid_: market cap alone as “value”

**Combined rank**:
Sum of ROC rank and EY rank; lower is better. Used for portfolio selection and sell-watch opportunity cost.
_Avoid_: average of ranks, z-score blend (not MVP)
Sum of ROC rank and EY rank; lower is better. **Benchmark only** (Magic Formula replica) — not production portfolio selection after Phase 2. Sell-watch opportunity-cost triggers on MF path are superseded by QV triggers in production.
_Avoid_: production ranking term post–Phase 2; average of ranks, z-score blend

**Magic Formula replica**:
Canonical benchmark portfolio using the same ROC, EY, combined rank, universe, and annual rebalance as production. Used to gate backtest pass vs strategy Sharpe.
_Avoid_: live Greenblatt fund, generic “value factor”
Canonical **benchmark** portfolio using ROC, EY, combined rank, universe, and annual rebalance. Used to gate backtest pass vs strategy Sharpe. **Not** the Phase 2+ production scoring path.
_Avoid_: live Greenblatt fund, generic “value factor”, conflating with Quantitative Value production

**Quantitative Value (QV) funnel**:
Phase 2+ **production** scoring: universe → forensic hard exclusion (incl. Beneish bottom-5%) → EBIT/TEV value decile → FS-Score quality screen → ~50-name equal-weight model portfolio. Spec: `docs/mvp/features/quantitative-value.md`.
_Avoid_: Magic Formula path, ROC+EY combined rank for production

**Quality (production)**:
**FS-Score** composite (0–10, Gray/Carlisle variant) on the EBIT/TEV value pool — not ROC rank alone.
_Avoid_: ROC rank as production quality after Phase 2; ESG or subjective moat

**Cheap (production)**:
Membership in the **EBIT/TEV value pool** (top decile among forensic survivors) — not EY rank alone.
_Avoid_: EY rank as production cheapness after Phase 2; low P/E without EV

**EBIT/TEV**:
`EBIT / Enterprise Value`; value-screen metric for QV production. Same EV definition as **Earnings yield**; ranked within forensic survivors to form the value pool. Spec: `docs/mvp/features/quantitative-value.md`.
_Avoid_: MF EY rank, market cap alone

**FS-Score**:
Ten binary financial-strength components (profitability, stability, recent operational improvements) summed to 0–10. Production quality factor after Phase 2. `formula_version` on every scored row.
_Avoid_: Piotroski F-Score (different formula), ROC as production quality

**QV funnel rank**:
Order within the value pool after the FS-Score quality screen; determines portfolio membership. Supersedes **combined rank** for production.
_Avoid_: combined rank, ROC rank + EY rank for production selection

**Forensic evaluator**:
Hard `exclude` / `pass` before value or quality scoring; distress and fraud rules from permanent-loss filter plus Beneish M-Score bottom-5% gate. Every exclusion carries `rule_id`, `rule_version`, `triggered_value`, `threshold`, `explanation`.
_Avoid_: soft penalty, scoring before forensics

**Cross-sectional rank**:
Rank across all passing companies on one run date. Not comparable across dates without re-running the pipeline.
Expand All @@ -93,7 +121,7 @@ Version id for ROC, EY, or filter rules so runs and backtests stay reproducible.
_Avoid_: “latest formula”, implicit default

**Model portfolio**:
Target long-only holdings from the pipeline; **June 30 demo:** top 30 names by combined rank, equal-weight only, market-cap tie-break on ranks. No watchlist in demo slice. Paper-traded in full MVP (phase 2).
Target long-only holdings from the pipeline. **June 30 demo:** top 30 by MF combined rank, equal-weight, market-cap tie-break. **Phase 2+ production (QV):** ~50 names by FS-Score within the EBIT/TEV value pool, equal-weight, market-cap tie-break (configurable cap). Paper-traded in full MVP (phase 2).
_Avoid_: personal portfolio, watchlist (demo slice)

**Watchlist**:
Expand Down Expand Up @@ -130,7 +158,9 @@ _Avoid_: ad-hoc snapshot without run id

## Relationships

- A **run date** drives **universe** → **permanent loss filter** → **ROC** and **EY** ranks → **combined rank** → **model portfolio**
- **Demo:** **run date** → **universe** → **ROC** and **EY** ranks → **combined rank** → **model portfolio** (top 30)
- **Phase 2+ production:** **run date** → **universe** → **forensic evaluator** → **EBIT/TEV value pool** → **FS-Score** → **QV funnel rank** → **model portfolio** (~50)
- **Benchmark (all phases):** MF replica path (ROC + EY + combined rank) for backtest Sharpe gate — parallel to production, not mixed into QV funnel
- **As-of date** tags each fundamental row; PIT queries filter `as_of_date <= run_date`
- **Watchlist** superset of names that may enter the **model portfolio** on rebalance
- **Magic Formula replica** is the strategy’s primary benchmark comparator for Sharpe pass/fail
Expand All @@ -142,11 +172,12 @@ Resolved scope cuts (see ADRs and [`docs/mvp/demo-slice.md`](docs/mvp/demo-slice

- **June 30 demo MVP:** SimFin bulk US → raw → normalizer → **universe (US market)** → ROC/EY → combined rank → top-30 EW model portfolio → Streamlit dashboard. No permanent loss filter, backtest, sell-watch, or paper trading in this slice.
- SEC ETL spike (`sec_client`, `edgartools_client`, `download-fundamentals`) is **frozen** in repo for phase 2; demo pipeline uses SimFin bulk for fundamentals and run-date prices (`shareprices/latest`).
- **Phase 2 (Quantitative Value):** will need multi-period fundamentals (not only TTM snapshots)—lake design should not block adding annual/quarterly income history later.
- **Phase 2 (Quantitative Value):** production scoring follows the **QV funnel** (`docs/mvp/features/quantitative-value.md`); requires multi-period fundamentals for FS-Score YoY deltas.

Terminology reminders:

- “Cheap” means high **EY**, not low P/E—use **EY rank** in issues and code names.
- “Quality” means high **ROC**, not ESG or subjective moat—use **ROC rank**.
- **Demo / MF benchmark:** “cheap” = high **EY**; “quality” = high **ROC** — use **EY rank** and **ROC rank** in MF code paths.
- **Production (Phase 2+):** “cheap” = **EBIT/TEV value pool** membership; “quality” = **FS-Score** — do not use ROC/EY ranks for production portfolio selection.
- **Combined rank** is **benchmark-only** after Phase 2; production uses **QV funnel rank**.
- “Value trap” in specs means negative EBIT routed to **review queue**, not a separate score.
- MVP specs in `docs/mvp/` remain canonical until an ADR or architecture decision supersedes them; update `CONTEXT.md` when `/grill-with-docs` resolves a term conflict.
23 changes: 23 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Pipeline batch image for ingest, normalize, and score-universe on AWS or locally.
# Build: make docker-build
# Run: docker run --rm -e LAKE_ROOT_URI=file:///lake -v "$PWD/data:/lake" smartwealthai-pipeline score-universe --help

FROM python:3.11-slim-bookworm

WORKDIR /app

ENV POETRY_VERSION=2.1.1 \
POETRY_VIRTUALENVS_CREATE=false \
POETRY_NO_INTERACTION=1 \
LAKE_ROOT_URI=file:///lake

RUN pip install --no-cache-dir "poetry==${POETRY_VERSION}"

COPY pyproject.toml poetry.lock ./
COPY src ./src
COPY config ./config

RUN poetry install --only main

ENTRYPOINT ["score-universe"]
CMD ["--help"]
5 changes: 4 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: install lint test download-fundamentals
.PHONY: install lint test download-fundamentals docker-build

install:
poetry install --with dev
Expand All @@ -10,6 +10,9 @@ lint:
test:
poetry run pytest --cov=smartwealthai --cov-report=term-missing

docker-build:
docker build -f Dockerfile -t smartwealthai-pipeline .

# Requires SEC_IDENTITY in the environment. See docs/mvp/guides/download-fundamentals.md
download-fundamentals:
poetry run download-fundamentals --universe dow30
Expand Down
13 changes: 13 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,19 @@ Canonical specs: [`docs/mvp/`](docs/mvp/) · Ubiquitous language: [`CONTEXT.md`]

Guide: [`docs/mvp/guides/download-fundamentals.md`](docs/mvp/guides/download-fundamentals.md).

## Cloud foundation (Phase 2a)

Pipeline batch work targets AWS with a **configurable lake root**; the Streamlit dashboard stays local for now.

| Concern | Local dev | AWS integration |
| --- | --- | --- |
| Lake root | `LAKE_ROOT_URI=file://…` or `--data-dir data` | `LAKE_ROOT_URI=s3://dev-bucket/prefix/` (I/O slice: file backend in pytest) |
| PR CI | `make lint` + `make test` on fixtures | No credentials |
| Ingest-smoke | N/A | [`.github/workflows/ingest-smoke.yml`](.github/workflows/ingest-smoke.yml) via OIDC |
| Pipeline image | `make docker-build` | ECR/ECS in [#95](https://github.com/JLaborda/SmartWealthAI/issues/95) |

Guide: [`docs/mvp/guides/cloud-foundation.md`](docs/mvp/guides/cloud-foundation.md).

## Roadmap

See [`docs/mvp/demo-slice.md`](docs/mvp/demo-slice.md) for the **June 30, 2026** delivery target and [`docs/mvp/architecture/architecture.md`](docs/mvp/architecture/architecture.md) for the full MVP north star.
Expand Down
18 changes: 12 additions & 6 deletions docs/mvp/demo-slice.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,9 +90,15 @@ flowchart LR

## After the demo (phase 2 order)

1. Historical S&P 500 universe + permanent loss filter
2. Minimal backtest (annual rebalance, 20 years) — custom pandas/DuckDB loop, not Zipline
3. Walk-forward, Monte Carlo, benchmark gate
4. Sell-watch + paper trading
5. SEC EDGAR normalizer (optional PIT upgrade)
6. Quantitative Value metrics (multi-period fundamentals from raw SimFin archives)
Per [`prds/phase2/prd.md`](prds/phase2/prd.md) — **Quantitative Value is production scoring**; Magic Formula (ROC + EY + combined rank) remains **benchmark-only**.

1. **QV feature spec** — [`quantitative-value.md`](features/quantitative-value.md) (canonical funnel; blocks scoring implementation)
2. Multi-period fundamentals + daily prices (FS-Score YoY deltas, backtest NAV)
3. Forensic evaluator + Beneish bottom-5% gate (extends [`permanent-loss-filter.md`](features/permanent-loss-filter.md))
4. **QV funnel** — EBIT/TEV value decile → FS-Score → ~50-name EW model portfolio
5. Light backtest (5–10 years, annual rebalance) vs S&P 500 CW + MF replica
6. Cloud pipeline (S3 lake, ECS, MLflow S3 artifacts)
7. Sell-watch with QV-adapted triggers + dashboard
8. Historical S&P 500 universe (delisted names) + full backtest (20+ years, walk-forward, Sharpe gate)
9. SEC EDGAR normalizer (optional PIT upgrade)
10. Paper trading + broker (after passing full backtest)
Loading
Loading