Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions spec/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,16 @@ spec/
| --- | --- |
| [0001](adr/0001-simfin-fundamentals-mvp.md) | SimFin fundamentals for MVP; SEC ETL phase 2 |
| [0002](adr/0002-june-demo-scope-cut.md) | June 30 demo slice scope cut |
| [0003](adr/0003-terraform-for-aws-iac.md) | Terraform for AWS infrastructure (M2) |

## PRDs

| PRD | Scope |
| --- | --- |
| [devcontainer/prd.md](prds/devcontainer/prd.md) | Reproducible dev environment (Cursor / EC2) |
| [ci-cd/ci-cd-prd.md](prds/ci-cd/ci-cd-prd.md) | Phase 0 CI/CD, AWS integration, ECS deploy path |
| [terraform/terraform-prd.md](prds/terraform/terraform-prd.md) | M2 Terraform: S3, OIDC, Secrets Manager |
| [phase2/prd.md](prds/phase2/prd.md) | Phase 2: Quantitative Value, cloud, backtest, sell-watch |

## Operator guides

Expand All @@ -66,6 +76,8 @@ spec/
| 010 | Sell-watch | [spec.md](features/010-sell-watch/spec.md) |
| 011 | Universe construction | [spec.md](features/011-universe-construction/spec.md) |
| 012 | Unstructured financial data | [spec.md](features/012-unstructured-financial-data/spec.md) |
| 013 | Quantitative Value | [spec.md](features/013-quantitative-value/spec.md) |
| 014 | CI/CD infrastructure | [spec.md](features/014-cicd-infrastructure/spec.md) |

Start with [`constitution/mission.md`](constitution/mission.md) for global constraints, then open the feature spec for the area you are changing.

Expand Down
34 changes: 34 additions & 0 deletions spec/adr/0003-terraform-for-aws-iac.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
---
status: accepted
---

# Terraform for AWS infrastructure (M2)

SmartWealthAI needs versioned, reproducible AWS resources (S3 data lake, GitHub OIDC IAM, Secrets Manager) before ingest-smoke and cloud pipeline deploys. Manual console setup does not scale across dev/prod or survive team handoff.

## Decision

Use **Terraform** under `infra/terraform/` as the sole infrastructure-as-code tool for **CI/CD milestone M2**:

- Bootstrap module: S3 remote state bucket + DynamoDB lock table (one-time manual apply, local bootstrap state).
- Environment roots: `environments/dev` and `environments/prod` composing storage, IAM (GitHub OIDC), and Secrets Manager modules.
- **Apply via `workflow_dispatch` only** — no auto-apply on merge. Prod requires GitHub Environment `production` approval.
- **Single AWS account**; dev/prod isolated by bucket name and IAM role trust policies.

ECR, ECS, EC2, EventBridge, and MLflow server provisioning are **deferred to M3–M6** per [`spec/prds/ci-cd/ci-cd-prd.md`](../prds/ci-cd/ci-cd-prd.md).

## Considered

| Option | Why not (for M2) |
| --- | --- |
| AWS CDK | Heavier bootstrap; team already aligned on HCL for ops handoff |
| Manual console | Not reproducible; blocks PR review of infra changes |
| OpenTofu / Terraform Cloud | Out of scope for v1; S3 backend is sufficient |
| CDK + Terraform mix | Two IaC tools increase cognitive load |

## Consequences

- Feature spec [`spec/features/014-cicd-infrastructure/spec.md`](../features/014-cicd-infrastructure/spec.md) owns acceptance criteria for M2 slices.
- PRD [`spec/prds/terraform/terraform-prd.md`](../prds/terraform/terraform-prd.md) captures user stories and vertical slices ([#105](https://github.com/JLaborda/SmartWealthAI/issues/105)–[#112](https://github.com/JLaborda/SmartWealthAI/issues/112)).
- Application `LAKE_URI` / boto3 lake I/O is a **separate code slice** ([#112](https://github.com/JLaborda/SmartWealthAI/issues/112)), not Terraform scope.
- Operators run bootstrap once; child environments use remote backend outputs documented in `infra/terraform/bootstrap/README.md`.
18 changes: 12 additions & 6 deletions spec/constitution/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,12 +92,16 @@ flowchart LR

## After the demo (phase 2 order)

1. Historical S&P 500 universe + permanent loss filter
2. Minimal backtest (annual rebalance, 20 years) — custom pandas/DuckDB loop, not Zipline
3. Walk-forward, Monte Carlo, benchmark gate
4. Sell-watch + paper trading
5. SEC EDGAR normalizer (optional PIT upgrade)
6. Quantitative Value metrics (multi-period fundamentals from raw SimFin archives)
1. Quantitative Value feature spec + glossary ([#86](https://github.com/JLaborda/SmartWealthAI/issues/86))
2. Multi-period fundamentals ETL + PIT history ([#87](https://github.com/JLaborda/SmartWealthAI/issues/87))
3. Forensic evaluator + permanent loss filter ([#89](https://github.com/JLaborda/SmartWealthAI/issues/89))
4. QV funnel orchestrator (production scoring) ([#91](https://github.com/JLaborda/SmartWealthAI/issues/91))
5. Light backtest (5–10y QV) ([#96](https://github.com/JLaborda/SmartWealthAI/issues/96))
6. CI/CD M2 Terraform + S3 lake ([#105](https://github.com/JLaborda/SmartWealthAI/issues/105)–[#112](https://github.com/JLaborda/SmartWealthAI/issues/112))
7. Sell-watch + dashboard QV views ([#97](https://github.com/JLaborda/SmartWealthAI/issues/97), [#93](https://github.com/JLaborda/SmartWealthAI/issues/93))
8. Historical S&P 500 universe + full backtest (2b) ([#99](https://github.com/JLaborda/SmartWealthAI/issues/99), [#100](https://github.com/JLaborda/SmartWealthAI/issues/100))
9. SEC EDGAR normalizer (optional PIT upgrade)
10. Paper trading (deferred)

## Feature registry (stable IDs)

Expand All @@ -117,6 +121,8 @@ Chronological spec IDs (creation order). **Priority** is defined by this roadmap
| 010 | Sell Watch | [`spec/features/010-sell-watch/spec.md`](../features/010-sell-watch/spec.md) |
| 011 | Universe Construction | [`spec/features/011-universe-construction/spec.md`](../features/011-universe-construction/spec.md) |
| 012 | Unstructured Financial Data | [`spec/features/012-unstructured-financial-data/spec.md`](../features/012-unstructured-financial-data/spec.md) |
| 013 | Quantitative Value | [`spec/features/013-quantitative-value/spec.md`](../features/013-quantitative-value/spec.md) |
| 014 | CI/CD Infrastructure | [`spec/features/014-cicd-infrastructure/spec.md`](../features/014-cicd-infrastructure/spec.md) |

## Informal backlog

Expand Down
98 changes: 98 additions & 0 deletions spec/features/013-quantitative-value/spec.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Feature: Quantitative Value

## Implementation status

**planned**

## Delivery

**Phase:** phase 2 (production scoring replaces Greenblatt demo path)

## Objective

Implement the *Quantitative Value* funnel (Gray/Carlisle): forensic hard exclusion → top ~10% by EBIT/TEV among survivors → FS-Score on the value pool → ~50 equal-weight model portfolio. Magic Formula (ROC + EY) remains a **benchmark-only** path for backtest comparison, not production scoring.

## In scope

- QV funnel orchestrator invoked by `score-universe` (or successor CLI) for a `run_date`.
- Value screen: EBIT/TEV cross-sectional rank; keep top decile (~10%).
- Quality screen: 10-point FS-Score (binary components per Gray/Carlisle) on the value pool.
- Model portfolio: ~50 equal-weight names (configurable cap); default **50**.
- Stage-count metrics logged to MLflow; scored artifacts per funnel stage in curated lake.
- Forensic bottom-5% cross-sectional exclusion gate (QVAL-style) integrated with [`008-permanent-loss-filter`](../008-permanent-loss-filter/spec.md).
- Distinct `qv_*` vs `mf_*` code paths; production never calls MF combined rank.

## Out of scope

- Paper trading and broker execution.
- Full 20-year walk-forward backtest (see [`001-backtesting`](../001-backtesting/spec.md)).
- SEC EDGAR fraud rules that require EDGAR ETL (log `unavailable` until wired).
- Score-weighted or risk-parity portfolio construction.

## Inputs

| Input | Source | Notes |
| --- | --- | --- |
| Universe | `curated/universe/` | Sector exclusions upstream ([`011-universe-construction`](../011-universe-construction/spec.md)) |
| PIT fundamentals (multi-period) | `curated/fundamentals/` | Annual/quarterly for FS-Score and Beneish |
| Prices | `curated/prices/` | EV / market cap on `run_date` |
| Forensic exclusions | `curated/permanent_loss/` | Hard exclude before value screen |
| Run date | Pipeline parameter | Strict PIT: `as_of_date <= run_date` |

## Outputs

| Output | Path / target |
| --- | --- |
| Funnel stage counts | MLflow metrics |
| Value pool ranks | `curated/scores/value/` |
| FS-Score breakdown | `curated/scores/quality/` |
| Model portfolio | `curated/portfolio/run_date=<YYYY-MM-DD>/portfolio.parquet` |
| Review queue rows | `curated/issues/` for missing FS-Score / forensic inputs |

## Mermaid diagram

```mermaid
flowchart TD
Uni["Universe"] --> Forensic["Forensic hard exclusion"]
Forensic --> Value["Top ~10% EBIT/TEV"]
Value --> FS["FS-Score on value pool"]
FS --> Port["Top ~50 EW portfolio"]
MF["MF benchmark path (ROC+EY)"] -.->|"backtest only"| Bench["Benchmark portfolios"]
```

## Expected flow

1. Load universe for `run_date`; apply sector exclusions (already done upstream).
2. Run forensic evaluator; drop hard exclusions and bottom-5% forensic percentile gate.
3. Rank survivors by EBIT/TEV; keep top decile (value pool).
4. Compute FS-Score (0–10) for each value-pool name; rank by score.
5. Select top ~50 names; equal-weight; write portfolio parquet + MLflow artifact.
6. Log stage counts: universe → forensic → value → quality → portfolio.

## Acceptance criteria

- [ ] `spec/features/013-quantitative-value/spec.md` exists with objective, scope, funnel stages, and acceptance criteria (this document).
- [ ] FS-Score components and forensic bottom-5% gate documented with formula versions.
- [ ] Portfolio size default (~50) and configurable cap recorded as a closed decision.
- [ ] `CONTEXT.md` updated with production QV terms (quality = FS-Score, cheap = value pool, QV funnel rank); MF terms marked benchmark-only.
- [ ] `spec/constitution/roadmap.md` "After the demo" section reflects QV-first production order.
- [ ] `score-universe` runs full QV funnel for a `run_date` when implementation completes ([#91](https://github.com/JLaborda/SmartWealthAI/issues/91)).
- [ ] Excluded names never appear in final portfolio; stage shrinkage monotonic on fixtures.

## Open questions

- **Closed:** Portfolio default size = **50** names, configurable via versioned QV config.

## Risks

- Multi-period fundamentals not available until ETL extension ([#87](https://github.com/JLaborda/SmartWealthAI/issues/87)) — funnel blocked until PIT history exists.
- FS-Score and Beneish need aligned fiscal-period joins; missing inputs must route to review queue, not silent drop.

## Related specs

- [`spec/prds/phase2/prd.md`](../../prds/phase2/prd.md) — parent PRD ([#85](https://github.com/JLaborda/SmartWealthAI/issues/85))
- [`../008-permanent-loss-filter/spec.md`](../008-permanent-loss-filter/spec.md) — forensic evaluator ([#89](https://github.com/JLaborda/SmartWealthAI/issues/89))
- [`../006-etl-data-lake/spec.md`](../006-etl-data-lake/spec.md) — multi-period ETL ([#87](https://github.com/JLaborda/SmartWealthAI/issues/87))
- [`../007-high-quality-stocks/spec.md`](../007-high-quality-stocks/spec.md) — MF ROC benchmark ([#92](https://github.com/JLaborda/SmartWealthAI/issues/92))
- [`../003-cheap-stocks/spec.md`](../003-cheap-stocks/spec.md) — MF EY benchmark ([#92](https://github.com/JLaborda/SmartWealthAI/issues/92))
- [`../005-dashboard-reporting/spec.md`](../005-dashboard-reporting/spec.md) — QV explainability ([#93](https://github.com/JLaborda/SmartWealthAI/issues/93))
98 changes: 98 additions & 0 deletions spec/features/014-cicd-infrastructure/spec.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Feature: CI/CD Infrastructure

## Implementation status

**planned** (M2 Terraform v1)

## Delivery

**Phase:** phase 0–2 infrastructure (M0 PR CI done; M2 Terraform in progress per [#105](https://github.com/JLaborda/SmartWealthAI/issues/105))

## Objective

Versioned AWS infrastructure and CI/CD workflows so the investment pipeline can run against S3, authenticate via GitHub OIDC, and read runtime secrets from Secrets Manager — without long-lived AWS keys in the repository.

## In scope (M2 v1)

| Component | Description |
| --- | --- |
| Bootstrap | S3 remote state + DynamoDB lock (`infra/terraform/bootstrap/`) |
| Storage | `smartwealthai-data-lake-dev` and `smartwealthai-data-lake-prod` buckets |
| IAM | GitHub OIDC provider; `smartwealthai-github-actions-dev` and `-prod` roles |
| Secrets | `smartwealthai/simfin-api-key` placeholder in Secrets Manager |
| Terraform CI | PR workflow: `fmt -check`, `validate`, `plan` on `infra/terraform/` changes |
| Ingest-smoke | Manual/weekly workflow: OIDC → dev S3 write (post-apply) |
| Lake I/O | `LAKE_URI` env selects file or `s3://` backend ([#112](https://github.com/JLaborda/SmartWealthAI/issues/112)) |

Lake layout follows [`spec/constitution/mission.md`](../../constitution/mission.md): `raw/`, `curated/`, `pit/`, `curated/issues/`.

## Out of scope (M2 v1)

- ECR, ECS Fargate, EventBridge daily schedule (M3–M4; [#95](https://github.com/JLaborda/SmartWealthAI/issues/95))
- MLflow tracking server on EC2
- Streamlit dashboard image and CD
- Prefect orchestration
- Auto-`apply` on merge to `develop` or `main`
- Multi-account AWS, OpenTofu, Terraform Cloud

## Inputs

| Input | Source | Notes |
| --- | --- | --- |
| Terraform modules | `infra/terraform/` | HCL under version control |
| GitHub repo/ref | OIDC trust policy | `JLaborda/SmartWealthAI` |
| `SIMFIN_API_KEY` | Secrets Manager (manual post-apply) | Never in Terraform variables or git |

## Outputs

| Output | Path / target |
| --- | --- |
| Dev/prod bucket ARNs | Terraform module outputs → GitHub Environment variables |
| OIDC role ARNs | GitHub Actions `aws-role-to-assume` |
| Ingest-smoke objects | `s3://smartwealthai-data-lake-dev/raw/...` |

## Mermaid diagram

```mermaid
flowchart LR
GH["GitHub Actions"] -->|OIDC| IAM["IAM roles dev/prod"]
IAM --> S3dev["S3 dev bucket"]
IAM --> S3prod["S3 prod bucket"]
TF["Terraform apply workflow_dispatch"] --> S3state["State bucket"]
App["Pipeline LAKE_URI"] --> S3dev
```

## Expected flow

1. Operator runs bootstrap apply once (local credentials).
2. Configure dev/prod backends; `terraform plan` via PR CI.
3. Manual `workflow_dispatch` apply per environment; prod requires approval.
4. Set Secrets Manager secret value via Console/CLI.
5. Ingest-smoke workflow validates OIDC → dev S3 write.
6. Application reads/writes Parquet via `LAKE_URI` (local `data/` or `s3://`).

## Acceptance criteria

- [ ] Bootstrap module creates state bucket + lock table; README documents apply order ([#106](https://github.com/JLaborda/SmartWealthAI/issues/106)).
- [ ] Dev and prod storage modules with distinct bucket names ([#107](https://github.com/JLaborda/SmartWealthAI/issues/107)).
- [ ] OIDC roles with least-privilege S3 policies scoped to correct bucket ([#108](https://github.com/JLaborda/SmartWealthAI/issues/108)).
- [ ] Secrets Manager resource exists; no plaintext API key in Terraform state ([#109](https://github.com/JLaborda/SmartWealthAI/issues/109)).
- [ ] PR CI fails on Terraform fmt/validate errors ([#110](https://github.com/JLaborda/SmartWealthAI/issues/110)).
- [ ] Ingest-smoke writes to dev bucket via OIDC; does not block unrelated PRs ([#111](https://github.com/JLaborda/SmartWealthAI/issues/111)).
- [ ] `LAKE_URI` selects storage backend; local demo path still works ([#112](https://github.com/JLaborda/SmartWealthAI/issues/112)).
- [ ] PR CI remains hermetic (no live AWS calls in unit tests).

## Open questions

- None for M2 v1 (closed in [ADR-0003](../../adr/0003-terraform-for-aws-iac.md)).

## Risks

- First bootstrap apply requires human AWS credentials (HITL).
- Overlap between [#94](https://github.com/JLaborda/SmartWealthAI/issues/94) (Phase 2a tracer: Dockerfile + ingest-smoke) and [#112](https://github.com/JLaborda/SmartWealthAI/issues/112) — coordinate `LAKE_URI` implementation once.

## Related specs

- [`spec/prds/ci-cd/ci-cd-prd.md`](../../prds/ci-cd/ci-cd-prd.md) — full CI/CD milestones M0–M6
- [`spec/prds/terraform/terraform-prd.md`](../../prds/terraform/terraform-prd.md) — M2 vertical slices
- [`spec/adr/0003-terraform-for-aws-iac.md`](../../adr/0003-terraform-for-aws-iac.md)
62 changes: 62 additions & 0 deletions spec/prds/terraform/terraform-prd.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# PRD: Terraform AWS Infrastructure (M2)

**Status:** Ready for implementation
**Parent PRD:** [`spec/prds/ci-cd/ci-cd-prd.md`](../ci-cd/ci-cd-prd.md) (milestone M2)
**Feature spec:** [`spec/features/014-cicd-infrastructure/spec.md`](../../features/014-cicd-infrastructure/spec.md)
**ADR:** [`spec/adr/0003-terraform-for-aws-iac.md`](../../adr/0003-terraform-for-aws-iac.md)
**GitHub epic:** [#105](https://github.com/JLaborda/SmartWealthAI/issues/105)

---

## Problem Statement

SmartWealthAI needs reproducible AWS infrastructure (S3 data lake, GitHub OIDC IAM, Secrets Manager) but has no IaC in the repo. M2 Terraform unblocks ingest-smoke without provisioning empty ECS before the pipeline Docker image exists (M3).

## Solution

Terraform v1 scoped to **CI/CD milestone M2**: bootstrap remote state, dev/prod S3 buckets, GitHub OIDC roles, Secrets Manager placeholder for `SIMFIN_API_KEY`. ECR/ECS/MLflow deferred to M3–M6.

## Implementation decisions

| Topic | Decision |
| --- | --- |
| v1 scope | M2 only — S3 dev+prod, OIDC IAM, Secrets Manager skeleton |
| Tool | Terraform (`infra/terraform/`) |
| State | S3 + DynamoDB (bootstrap module, one-time manual apply) |
| Accounts | Single AWS account; dev/prod isolated by bucket + IAM role |
| OIDC | Repo `JLaborda/SmartWealthAI`; dev = `develop` + Environment `dev`; prod = `main` + Environment `production` |
| Apply | `workflow_dispatch` only; prod requires GitHub Environment approval |
| Out of v1 | ECR, ECS, EC2, EventBridge; application `LAKE_URI`/boto3 ([#112](https://github.com/JLaborda/SmartWealthAI/issues/112)) |

## User stories (M2)

1. As a developer, I want Terraform remote state in S3 with DynamoDB locking, so that infra changes are collaborative and auditable.
2. As a developer, I want separate dev and prod S3 data lake buckets with versioning and encryption, so that test data never mixes with production.
3. As a developer, I want GitHub Actions to assume AWS roles via OIDC (no static keys), so that CI secrets stay minimal.
4. As a developer, I want dev and prod OIDC roles with least-privilege S3 policies, so that a dev workflow cannot write prod buckets.
5. As a developer, I want `SIMFIN_API_KEY` in Secrets Manager (placeholder at apply time), so that runtime tasks can read keys without repo commits.
6. As a developer, I want `terraform fmt -check`, `validate`, and `plan` on PRs touching `infra/terraform/`, so that infra drift is caught in review.
7. As a developer, I want a manual apply workflow with prod approval gate, so that production promotion is deliberate.
8. As a developer, I want an ingest-smoke workflow (OIDC → dev S3 write) after M2 apply, so that AWS integration is proven outside PR CI.

## Vertical slices (GitHub issues)

| Issue | Slice |
| --- | --- |
| [#106](https://github.com/JLaborda/SmartWealthAI/issues/106) | Bootstrap: remote state bucket + DynamoDB lock |
| [#107](https://github.com/JLaborda/SmartWealthAI/issues/107) | Storage module: dev/prod S3 buckets |
| [#108](https://github.com/JLaborda/SmartWealthAI/issues/108) | IAM: GitHub OIDC provider + dev/prod roles |
| [#109](https://github.com/JLaborda/SmartWealthAI/issues/109) | Secrets Manager: SimFin API key placeholder |
| [#110](https://github.com/JLaborda/SmartWealthAI/issues/110) | Terraform CI workflow: fmt, validate, plan |
| [#111](https://github.com/JLaborda/SmartWealthAI/issues/111) | Ingest-smoke workflow: OIDC auth + dev S3 write |
| [#112](https://github.com/JLaborda/SmartWealthAI/issues/112) | Application: `LAKE_URI` config + S3 lake I/O |

## Testing

- **PR CI:** `terraform fmt -check`, `validate`, `plan` — no live `apply` on merge.
- **Post-apply:** ingest-smoke workflow proves OIDC → dev S3 write.
- **Hermetic:** application S3 tests use local temp dirs or moto; no live AWS in PR CI.

## Out of scope

Multi-account AWS, OpenTofu, Terraform Cloud, auto-apply on merge, ECS/ECR/EC2 in v1.
Loading