From e2d3245d7be55778d4a11fc7d45b2626e5871b27 Mon Sep 17 00:00:00 2001 From: JLaborda <15078416+JLaborda@users.noreply.github.com> Date: Wed, 1 Jul 2026 09:05:00 +0000 Subject: [PATCH] docs(spec): add QV, CI/CD infra specs and Terraform ADR MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add missing spec artifacts referenced by open GitHub issues (#85–#112) after the spec/ migration: ADR-0003, terraform PRD, features 013 and 014, and roadmap/README registry updates. Co-authored-by: Cursor --- spec/README.md | 12 +++ spec/adr/0003-terraform-for-aws-iac.md | 34 +++++++ spec/constitution/roadmap.md | 18 ++-- spec/features/013-quantitative-value/spec.md | 98 +++++++++++++++++++ spec/features/014-cicd-infrastructure/spec.md | 98 +++++++++++++++++++ spec/prds/terraform/terraform-prd.md | 62 ++++++++++++ 6 files changed, 316 insertions(+), 6 deletions(-) create mode 100644 spec/adr/0003-terraform-for-aws-iac.md create mode 100644 spec/features/013-quantitative-value/spec.md create mode 100644 spec/features/014-cicd-infrastructure/spec.md create mode 100644 spec/prds/terraform/terraform-prd.md diff --git a/spec/README.md b/spec/README.md index 0c8b1bc..6ff6d4e 100644 --- a/spec/README.md +++ b/spec/README.md @@ -42,6 +42,16 @@ spec/ | --- | --- | | [0001](adr/0001-simfin-fundamentals-mvp.md) | SimFin fundamentals for MVP; SEC ETL phase 2 | | [0002](adr/0002-june-demo-scope-cut.md) | June 30 demo slice scope cut | +| [0003](adr/0003-terraform-for-aws-iac.md) | Terraform for AWS infrastructure (M2) | + +## PRDs + +| PRD | Scope | +| --- | --- | +| [devcontainer/prd.md](prds/devcontainer/prd.md) | Reproducible dev environment (Cursor / EC2) | +| [ci-cd/ci-cd-prd.md](prds/ci-cd/ci-cd-prd.md) | Phase 0 CI/CD, AWS integration, ECS deploy path | +| [terraform/terraform-prd.md](prds/terraform/terraform-prd.md) | M2 Terraform: S3, OIDC, Secrets Manager | +| [phase2/prd.md](prds/phase2/prd.md) | Phase 2: Quantitative Value, cloud, backtest, sell-watch | ## Operator guides @@ -66,6 +76,8 @@ spec/ | 010 | Sell-watch | [spec.md](features/010-sell-watch/spec.md) | | 011 | Universe construction | [spec.md](features/011-universe-construction/spec.md) | | 012 | Unstructured financial data | [spec.md](features/012-unstructured-financial-data/spec.md) | +| 013 | Quantitative Value | [spec.md](features/013-quantitative-value/spec.md) | +| 014 | CI/CD infrastructure | [spec.md](features/014-cicd-infrastructure/spec.md) | Start with [`constitution/mission.md`](constitution/mission.md) for global constraints, then open the feature spec for the area you are changing. diff --git a/spec/adr/0003-terraform-for-aws-iac.md b/spec/adr/0003-terraform-for-aws-iac.md new file mode 100644 index 0000000..43ccee5 --- /dev/null +++ b/spec/adr/0003-terraform-for-aws-iac.md @@ -0,0 +1,34 @@ +--- +status: accepted +--- + +# Terraform for AWS infrastructure (M2) + +SmartWealthAI needs versioned, reproducible AWS resources (S3 data lake, GitHub OIDC IAM, Secrets Manager) before ingest-smoke and cloud pipeline deploys. Manual console setup does not scale across dev/prod or survive team handoff. + +## Decision + +Use **Terraform** under `infra/terraform/` as the sole infrastructure-as-code tool for **CI/CD milestone M2**: + +- Bootstrap module: S3 remote state bucket + DynamoDB lock table (one-time manual apply, local bootstrap state). +- Environment roots: `environments/dev` and `environments/prod` composing storage, IAM (GitHub OIDC), and Secrets Manager modules. +- **Apply via `workflow_dispatch` only** — no auto-apply on merge. Prod requires GitHub Environment `production` approval. +- **Single AWS account**; dev/prod isolated by bucket name and IAM role trust policies. + +ECR, ECS, EC2, EventBridge, and MLflow server provisioning are **deferred to M3–M6** per [`spec/prds/ci-cd/ci-cd-prd.md`](../prds/ci-cd/ci-cd-prd.md). + +## Considered + +| Option | Why not (for M2) | +| --- | --- | +| AWS CDK | Heavier bootstrap; team already aligned on HCL for ops handoff | +| Manual console | Not reproducible; blocks PR review of infra changes | +| OpenTofu / Terraform Cloud | Out of scope for v1; S3 backend is sufficient | +| CDK + Terraform mix | Two IaC tools increase cognitive load | + +## Consequences + +- Feature spec [`spec/features/014-cicd-infrastructure/spec.md`](../features/014-cicd-infrastructure/spec.md) owns acceptance criteria for M2 slices. +- PRD [`spec/prds/terraform/terraform-prd.md`](../prds/terraform/terraform-prd.md) captures user stories and vertical slices ([#105](https://github.com/JLaborda/SmartWealthAI/issues/105)–[#112](https://github.com/JLaborda/SmartWealthAI/issues/112)). +- Application `LAKE_URI` / boto3 lake I/O is a **separate code slice** ([#112](https://github.com/JLaborda/SmartWealthAI/issues/112)), not Terraform scope. +- Operators run bootstrap once; child environments use remote backend outputs documented in `infra/terraform/bootstrap/README.md`. diff --git a/spec/constitution/roadmap.md b/spec/constitution/roadmap.md index a578190..7a0ff6b 100644 --- a/spec/constitution/roadmap.md +++ b/spec/constitution/roadmap.md @@ -92,12 +92,16 @@ flowchart LR ## After the demo (phase 2 order) -1. Historical S&P 500 universe + permanent loss filter -2. Minimal backtest (annual rebalance, 20 years) — custom pandas/DuckDB loop, not Zipline -3. Walk-forward, Monte Carlo, benchmark gate -4. Sell-watch + paper trading -5. SEC EDGAR normalizer (optional PIT upgrade) -6. Quantitative Value metrics (multi-period fundamentals from raw SimFin archives) +1. Quantitative Value feature spec + glossary ([#86](https://github.com/JLaborda/SmartWealthAI/issues/86)) +2. Multi-period fundamentals ETL + PIT history ([#87](https://github.com/JLaborda/SmartWealthAI/issues/87)) +3. Forensic evaluator + permanent loss filter ([#89](https://github.com/JLaborda/SmartWealthAI/issues/89)) +4. QV funnel orchestrator (production scoring) ([#91](https://github.com/JLaborda/SmartWealthAI/issues/91)) +5. Light backtest (5–10y QV) ([#96](https://github.com/JLaborda/SmartWealthAI/issues/96)) +6. CI/CD M2 Terraform + S3 lake ([#105](https://github.com/JLaborda/SmartWealthAI/issues/105)–[#112](https://github.com/JLaborda/SmartWealthAI/issues/112)) +7. Sell-watch + dashboard QV views ([#97](https://github.com/JLaborda/SmartWealthAI/issues/97), [#93](https://github.com/JLaborda/SmartWealthAI/issues/93)) +8. Historical S&P 500 universe + full backtest (2b) ([#99](https://github.com/JLaborda/SmartWealthAI/issues/99), [#100](https://github.com/JLaborda/SmartWealthAI/issues/100)) +9. SEC EDGAR normalizer (optional PIT upgrade) +10. Paper trading (deferred) ## Feature registry (stable IDs) @@ -117,6 +121,8 @@ Chronological spec IDs (creation order). **Priority** is defined by this roadmap | 010 | Sell Watch | [`spec/features/010-sell-watch/spec.md`](../features/010-sell-watch/spec.md) | | 011 | Universe Construction | [`spec/features/011-universe-construction/spec.md`](../features/011-universe-construction/spec.md) | | 012 | Unstructured Financial Data | [`spec/features/012-unstructured-financial-data/spec.md`](../features/012-unstructured-financial-data/spec.md) | +| 013 | Quantitative Value | [`spec/features/013-quantitative-value/spec.md`](../features/013-quantitative-value/spec.md) | +| 014 | CI/CD Infrastructure | [`spec/features/014-cicd-infrastructure/spec.md`](../features/014-cicd-infrastructure/spec.md) | ## Informal backlog diff --git a/spec/features/013-quantitative-value/spec.md b/spec/features/013-quantitative-value/spec.md new file mode 100644 index 0000000..ab46ffd --- /dev/null +++ b/spec/features/013-quantitative-value/spec.md @@ -0,0 +1,98 @@ +# Feature: Quantitative Value + +## Implementation status + +**planned** + +## Delivery + +**Phase:** phase 2 (production scoring replaces Greenblatt demo path) + +## Objective + +Implement the *Quantitative Value* funnel (Gray/Carlisle): forensic hard exclusion → top ~10% by EBIT/TEV among survivors → FS-Score on the value pool → ~50 equal-weight model portfolio. Magic Formula (ROC + EY) remains a **benchmark-only** path for backtest comparison, not production scoring. + +## In scope + +- QV funnel orchestrator invoked by `score-universe` (or successor CLI) for a `run_date`. +- Value screen: EBIT/TEV cross-sectional rank; keep top decile (~10%). +- Quality screen: 10-point FS-Score (binary components per Gray/Carlisle) on the value pool. +- Model portfolio: ~50 equal-weight names (configurable cap); default **50**. +- Stage-count metrics logged to MLflow; scored artifacts per funnel stage in curated lake. +- Forensic bottom-5% cross-sectional exclusion gate (QVAL-style) integrated with [`008-permanent-loss-filter`](../008-permanent-loss-filter/spec.md). +- Distinct `qv_*` vs `mf_*` code paths; production never calls MF combined rank. + +## Out of scope + +- Paper trading and broker execution. +- Full 20-year walk-forward backtest (see [`001-backtesting`](../001-backtesting/spec.md)). +- SEC EDGAR fraud rules that require EDGAR ETL (log `unavailable` until wired). +- Score-weighted or risk-parity portfolio construction. + +## Inputs + +| Input | Source | Notes | +| --- | --- | --- | +| Universe | `curated/universe/` | Sector exclusions upstream ([`011-universe-construction`](../011-universe-construction/spec.md)) | +| PIT fundamentals (multi-period) | `curated/fundamentals/` | Annual/quarterly for FS-Score and Beneish | +| Prices | `curated/prices/` | EV / market cap on `run_date` | +| Forensic exclusions | `curated/permanent_loss/` | Hard exclude before value screen | +| Run date | Pipeline parameter | Strict PIT: `as_of_date <= run_date` | + +## Outputs + +| Output | Path / target | +| --- | --- | +| Funnel stage counts | MLflow metrics | +| Value pool ranks | `curated/scores/value/` | +| FS-Score breakdown | `curated/scores/quality/` | +| Model portfolio | `curated/portfolio/run_date=/portfolio.parquet` | +| Review queue rows | `curated/issues/` for missing FS-Score / forensic inputs | + +## Mermaid diagram + +```mermaid +flowchart TD + Uni["Universe"] --> Forensic["Forensic hard exclusion"] + Forensic --> Value["Top ~10% EBIT/TEV"] + Value --> FS["FS-Score on value pool"] + FS --> Port["Top ~50 EW portfolio"] + MF["MF benchmark path (ROC+EY)"] -.->|"backtest only"| Bench["Benchmark portfolios"] +``` + +## Expected flow + +1. Load universe for `run_date`; apply sector exclusions (already done upstream). +2. Run forensic evaluator; drop hard exclusions and bottom-5% forensic percentile gate. +3. Rank survivors by EBIT/TEV; keep top decile (value pool). +4. Compute FS-Score (0–10) for each value-pool name; rank by score. +5. Select top ~50 names; equal-weight; write portfolio parquet + MLflow artifact. +6. Log stage counts: universe → forensic → value → quality → portfolio. + +## Acceptance criteria + +- [ ] `spec/features/013-quantitative-value/spec.md` exists with objective, scope, funnel stages, and acceptance criteria (this document). +- [ ] FS-Score components and forensic bottom-5% gate documented with formula versions. +- [ ] Portfolio size default (~50) and configurable cap recorded as a closed decision. +- [ ] `CONTEXT.md` updated with production QV terms (quality = FS-Score, cheap = value pool, QV funnel rank); MF terms marked benchmark-only. +- [ ] `spec/constitution/roadmap.md` "After the demo" section reflects QV-first production order. +- [ ] `score-universe` runs full QV funnel for a `run_date` when implementation completes ([#91](https://github.com/JLaborda/SmartWealthAI/issues/91)). +- [ ] Excluded names never appear in final portfolio; stage shrinkage monotonic on fixtures. + +## Open questions + +- **Closed:** Portfolio default size = **50** names, configurable via versioned QV config. + +## Risks + +- Multi-period fundamentals not available until ETL extension ([#87](https://github.com/JLaborda/SmartWealthAI/issues/87)) — funnel blocked until PIT history exists. +- FS-Score and Beneish need aligned fiscal-period joins; missing inputs must route to review queue, not silent drop. + +## Related specs + +- [`spec/prds/phase2/prd.md`](../../prds/phase2/prd.md) — parent PRD ([#85](https://github.com/JLaborda/SmartWealthAI/issues/85)) +- [`../008-permanent-loss-filter/spec.md`](../008-permanent-loss-filter/spec.md) — forensic evaluator ([#89](https://github.com/JLaborda/SmartWealthAI/issues/89)) +- [`../006-etl-data-lake/spec.md`](../006-etl-data-lake/spec.md) — multi-period ETL ([#87](https://github.com/JLaborda/SmartWealthAI/issues/87)) +- [`../007-high-quality-stocks/spec.md`](../007-high-quality-stocks/spec.md) — MF ROC benchmark ([#92](https://github.com/JLaborda/SmartWealthAI/issues/92)) +- [`../003-cheap-stocks/spec.md`](../003-cheap-stocks/spec.md) — MF EY benchmark ([#92](https://github.com/JLaborda/SmartWealthAI/issues/92)) +- [`../005-dashboard-reporting/spec.md`](../005-dashboard-reporting/spec.md) — QV explainability ([#93](https://github.com/JLaborda/SmartWealthAI/issues/93)) diff --git a/spec/features/014-cicd-infrastructure/spec.md b/spec/features/014-cicd-infrastructure/spec.md new file mode 100644 index 0000000..f569780 --- /dev/null +++ b/spec/features/014-cicd-infrastructure/spec.md @@ -0,0 +1,98 @@ +# Feature: CI/CD Infrastructure + +## Implementation status + +**planned** (M2 Terraform v1) + +## Delivery + +**Phase:** phase 0–2 infrastructure (M0 PR CI done; M2 Terraform in progress per [#105](https://github.com/JLaborda/SmartWealthAI/issues/105)) + +## Objective + +Versioned AWS infrastructure and CI/CD workflows so the investment pipeline can run against S3, authenticate via GitHub OIDC, and read runtime secrets from Secrets Manager — without long-lived AWS keys in the repository. + +## In scope (M2 v1) + +| Component | Description | +| --- | --- | +| Bootstrap | S3 remote state + DynamoDB lock (`infra/terraform/bootstrap/`) | +| Storage | `smartwealthai-data-lake-dev` and `smartwealthai-data-lake-prod` buckets | +| IAM | GitHub OIDC provider; `smartwealthai-github-actions-dev` and `-prod` roles | +| Secrets | `smartwealthai/simfin-api-key` placeholder in Secrets Manager | +| Terraform CI | PR workflow: `fmt -check`, `validate`, `plan` on `infra/terraform/` changes | +| Ingest-smoke | Manual/weekly workflow: OIDC → dev S3 write (post-apply) | +| Lake I/O | `LAKE_URI` env selects file or `s3://` backend ([#112](https://github.com/JLaborda/SmartWealthAI/issues/112)) | + +Lake layout follows [`spec/constitution/mission.md`](../../constitution/mission.md): `raw/`, `curated/`, `pit/`, `curated/issues/`. + +## Out of scope (M2 v1) + +- ECR, ECS Fargate, EventBridge daily schedule (M3–M4; [#95](https://github.com/JLaborda/SmartWealthAI/issues/95)) +- MLflow tracking server on EC2 +- Streamlit dashboard image and CD +- Prefect orchestration +- Auto-`apply` on merge to `develop` or `main` +- Multi-account AWS, OpenTofu, Terraform Cloud + +## Inputs + +| Input | Source | Notes | +| --- | --- | --- | +| Terraform modules | `infra/terraform/` | HCL under version control | +| GitHub repo/ref | OIDC trust policy | `JLaborda/SmartWealthAI` | +| `SIMFIN_API_KEY` | Secrets Manager (manual post-apply) | Never in Terraform variables or git | + +## Outputs + +| Output | Path / target | +| --- | --- | +| Dev/prod bucket ARNs | Terraform module outputs → GitHub Environment variables | +| OIDC role ARNs | GitHub Actions `aws-role-to-assume` | +| Ingest-smoke objects | `s3://smartwealthai-data-lake-dev/raw/...` | + +## Mermaid diagram + +```mermaid +flowchart LR + GH["GitHub Actions"] -->|OIDC| IAM["IAM roles dev/prod"] + IAM --> S3dev["S3 dev bucket"] + IAM --> S3prod["S3 prod bucket"] + TF["Terraform apply workflow_dispatch"] --> S3state["State bucket"] + App["Pipeline LAKE_URI"] --> S3dev +``` + +## Expected flow + +1. Operator runs bootstrap apply once (local credentials). +2. Configure dev/prod backends; `terraform plan` via PR CI. +3. Manual `workflow_dispatch` apply per environment; prod requires approval. +4. Set Secrets Manager secret value via Console/CLI. +5. Ingest-smoke workflow validates OIDC → dev S3 write. +6. Application reads/writes Parquet via `LAKE_URI` (local `data/` or `s3://`). + +## Acceptance criteria + +- [ ] Bootstrap module creates state bucket + lock table; README documents apply order ([#106](https://github.com/JLaborda/SmartWealthAI/issues/106)). +- [ ] Dev and prod storage modules with distinct bucket names ([#107](https://github.com/JLaborda/SmartWealthAI/issues/107)). +- [ ] OIDC roles with least-privilege S3 policies scoped to correct bucket ([#108](https://github.com/JLaborda/SmartWealthAI/issues/108)). +- [ ] Secrets Manager resource exists; no plaintext API key in Terraform state ([#109](https://github.com/JLaborda/SmartWealthAI/issues/109)). +- [ ] PR CI fails on Terraform fmt/validate errors ([#110](https://github.com/JLaborda/SmartWealthAI/issues/110)). +- [ ] Ingest-smoke writes to dev bucket via OIDC; does not block unrelated PRs ([#111](https://github.com/JLaborda/SmartWealthAI/issues/111)). +- [ ] `LAKE_URI` selects storage backend; local demo path still works ([#112](https://github.com/JLaborda/SmartWealthAI/issues/112)). +- [ ] PR CI remains hermetic (no live AWS calls in unit tests). + +## Open questions + +- None for M2 v1 (closed in [ADR-0003](../../adr/0003-terraform-for-aws-iac.md)). + +## Risks + +- First bootstrap apply requires human AWS credentials (HITL). +- Overlap between [#94](https://github.com/JLaborda/SmartWealthAI/issues/94) (Phase 2a tracer: Dockerfile + ingest-smoke) and [#112](https://github.com/JLaborda/SmartWealthAI/issues/112) — coordinate `LAKE_URI` implementation once. + +## Related specs + +- [`spec/prds/ci-cd/ci-cd-prd.md`](../../prds/ci-cd/ci-cd-prd.md) — full CI/CD milestones M0–M6 +- [`spec/prds/terraform/terraform-prd.md`](../../prds/terraform/terraform-prd.md) — M2 vertical slices +- [`spec/adr/0003-terraform-for-aws-iac.md`](../../adr/0003-terraform-for-aws-iac.md) diff --git a/spec/prds/terraform/terraform-prd.md b/spec/prds/terraform/terraform-prd.md new file mode 100644 index 0000000..0a61bb9 --- /dev/null +++ b/spec/prds/terraform/terraform-prd.md @@ -0,0 +1,62 @@ +# PRD: Terraform AWS Infrastructure (M2) + +**Status:** Ready for implementation +**Parent PRD:** [`spec/prds/ci-cd/ci-cd-prd.md`](../ci-cd/ci-cd-prd.md) (milestone M2) +**Feature spec:** [`spec/features/014-cicd-infrastructure/spec.md`](../../features/014-cicd-infrastructure/spec.md) +**ADR:** [`spec/adr/0003-terraform-for-aws-iac.md`](../../adr/0003-terraform-for-aws-iac.md) +**GitHub epic:** [#105](https://github.com/JLaborda/SmartWealthAI/issues/105) + +--- + +## Problem Statement + +SmartWealthAI needs reproducible AWS infrastructure (S3 data lake, GitHub OIDC IAM, Secrets Manager) but has no IaC in the repo. M2 Terraform unblocks ingest-smoke without provisioning empty ECS before the pipeline Docker image exists (M3). + +## Solution + +Terraform v1 scoped to **CI/CD milestone M2**: bootstrap remote state, dev/prod S3 buckets, GitHub OIDC roles, Secrets Manager placeholder for `SIMFIN_API_KEY`. ECR/ECS/MLflow deferred to M3–M6. + +## Implementation decisions + +| Topic | Decision | +| --- | --- | +| v1 scope | M2 only — S3 dev+prod, OIDC IAM, Secrets Manager skeleton | +| Tool | Terraform (`infra/terraform/`) | +| State | S3 + DynamoDB (bootstrap module, one-time manual apply) | +| Accounts | Single AWS account; dev/prod isolated by bucket + IAM role | +| OIDC | Repo `JLaborda/SmartWealthAI`; dev = `develop` + Environment `dev`; prod = `main` + Environment `production` | +| Apply | `workflow_dispatch` only; prod requires GitHub Environment approval | +| Out of v1 | ECR, ECS, EC2, EventBridge; application `LAKE_URI`/boto3 ([#112](https://github.com/JLaborda/SmartWealthAI/issues/112)) | + +## User stories (M2) + +1. As a developer, I want Terraform remote state in S3 with DynamoDB locking, so that infra changes are collaborative and auditable. +2. As a developer, I want separate dev and prod S3 data lake buckets with versioning and encryption, so that test data never mixes with production. +3. As a developer, I want GitHub Actions to assume AWS roles via OIDC (no static keys), so that CI secrets stay minimal. +4. As a developer, I want dev and prod OIDC roles with least-privilege S3 policies, so that a dev workflow cannot write prod buckets. +5. As a developer, I want `SIMFIN_API_KEY` in Secrets Manager (placeholder at apply time), so that runtime tasks can read keys without repo commits. +6. As a developer, I want `terraform fmt -check`, `validate`, and `plan` on PRs touching `infra/terraform/`, so that infra drift is caught in review. +7. As a developer, I want a manual apply workflow with prod approval gate, so that production promotion is deliberate. +8. As a developer, I want an ingest-smoke workflow (OIDC → dev S3 write) after M2 apply, so that AWS integration is proven outside PR CI. + +## Vertical slices (GitHub issues) + +| Issue | Slice | +| --- | --- | +| [#106](https://github.com/JLaborda/SmartWealthAI/issues/106) | Bootstrap: remote state bucket + DynamoDB lock | +| [#107](https://github.com/JLaborda/SmartWealthAI/issues/107) | Storage module: dev/prod S3 buckets | +| [#108](https://github.com/JLaborda/SmartWealthAI/issues/108) | IAM: GitHub OIDC provider + dev/prod roles | +| [#109](https://github.com/JLaborda/SmartWealthAI/issues/109) | Secrets Manager: SimFin API key placeholder | +| [#110](https://github.com/JLaborda/SmartWealthAI/issues/110) | Terraform CI workflow: fmt, validate, plan | +| [#111](https://github.com/JLaborda/SmartWealthAI/issues/111) | Ingest-smoke workflow: OIDC auth + dev S3 write | +| [#112](https://github.com/JLaborda/SmartWealthAI/issues/112) | Application: `LAKE_URI` config + S3 lake I/O | + +## Testing + +- **PR CI:** `terraform fmt -check`, `validate`, `plan` — no live `apply` on merge. +- **Post-apply:** ingest-smoke workflow proves OIDC → dev S3 write. +- **Hermetic:** application S3 tests use local temp dirs or moto; no live AWS in PR CI. + +## Out of scope + +Multi-account AWS, OpenTofu, Terraform Cloud, auto-apply on merge, ECS/ECR/EC2 in v1.