diff --git a/.cursor/rules/domain.md b/.cursor/rules/domain.md index 8025bba..35e3a19 100644 --- a/.cursor/rules/domain.md +++ b/.cursor/rules/domain.md @@ -14,11 +14,11 @@ How the engineering skills should consume this repo's domain documentation when ## Before exploring, read these - **`CONTEXT.md`** at the repo root (ubiquitous language; created or extended by `/grill-with-docs` when terms are resolved). -- **`docs/adr/`** — architectural decision records for cross-cutting choices. -- **`docs/mvp/architecture/architecture.md`** — MVP vision, closed decisions, and module map (canonical during spec-driven phase). -- **Relevant `docs/mvp/features/.md`** — feature scope and acceptance criteria for the area you are changing. +- **`spec/adr/`** — architectural decision records for cross-cutting choices. +- **`spec/constitution/mission.md`** — MVP vision, closed decisions, and module map (canonical during spec-driven phase). +- **Relevant `spec/features/00N-slug/spec.md`** — feature scope and acceptance criteria for the area you are changing. -If `CONTEXT.md` or `docs/adr/` do not exist yet, **proceed silently**. Do not flag their absence or suggest creating them upfront. Use `docs/mvp/` as the source of truth until `/grill-with-docs` materializes terms into `CONTEXT.md`. +If `CONTEXT.md` or `spec/adr/` do not exist yet, **proceed silently**. Do not flag their absence or suggest creating them upfront. Use `spec/` as the source of truth until `/grill-with-docs` materializes terms into `CONTEXT.md`. Do not treat `src/` or `notebooks/` as canonical architecture; they are legacy exploration per `AGENTS.md`. @@ -27,24 +27,22 @@ Do not treat `src/` or `notebooks/` as canonical architecture; they are legacy e ``` / ├── CONTEXT.md ← ubiquitous language (extend via grill-with-docs) -├── docs/ -│ ├── adr/ ← system-wide ADRs -│ ├── agents/ ← agent skill config (this folder) -│ └── mvp/ -│ ├── architecture/ -│ └── features/ +├── spec/ +│ ├── constitution/ ← mission, tech-stack, roadmap +│ ├── features/ ← 00N-slug/spec.md (+ plan.md, tasks.md) +│ └── adr/ └── src/ ← implementation (not planning truth) ``` ## Use CONTEXT vocabulary -When your output names a domain concept (in an issue title, a refactor proposal, a hypothesis, a test name), prefer terms from `CONTEXT.md` when defined there; otherwise use terms consistently with `docs/mvp/` (e.g. point-in-time, universe, EY rank, Magic Formula replica). +When your output names a domain concept (in an issue title, a refactor proposal, a hypothesis, a test name), prefer terms from `CONTEXT.md` when defined there; otherwise use terms consistently with `spec/` (e.g. point-in-time, universe, EY rank, Magic Formula replica). If the concept you need isn't in `CONTEXT.md` yet, either reconsider invented language or note the gap for `/grill-with-docs`. ## Flag ADR conflicts -If your output contradicts an existing ADR or a **closed decision** in `docs/mvp/architecture/architecture.md`, surface it explicitly rather than silently overriding: +If your output contradicts an existing ADR or a **closed decision** in `spec/constitution/mission.md`, surface it explicitly rather than silently overriding: > _Contradicts [decision or ADR] — but worth reopening because…_ diff --git a/.cursor/rules/github-issues.mdc b/.cursor/rules/github-issues.mdc new file mode 100644 index 0000000..219bf50 --- /dev/null +++ b/.cursor/rules/github-issues.mdc @@ -0,0 +1,24 @@ +--- +description: GitHub Issues vs Git specs +alwaysApply: true +--- + +# GitHub Issues workflow + +**Git specs in `spec/` are canonical.** GitHub Issues track execution. + +## Issue fields + +Each implementation issue should include: + +- **Title:** e.g. `Feature: Cheap Stocks — implement EY rank` +- **Spec path:** `spec/features/00N-slug/spec.md` +- **Body:** link to `plan.md` and `tasks.md` when they exist; copy acceptance criteria from the spec + +## Sync discipline + +1. Change MVP decisions in the Git spec first. +2. Create or update the GitHub issue (label `ready-for-agent` when ready). +3. On completion: update `spec.md` (implementation status, acceptance criteria), check off `tasks.md`, close the issue. + +See [`spec/meta/github-issues.md`](../../spec/meta/github-issues.md). diff --git a/.cursor/rules/issue-tracker.md b/.cursor/rules/issue-tracker.md index 8464a6a..6d24bbd 100644 --- a/.cursor/rules/issue-tracker.md +++ b/.cursor/rules/issue-tracker.md @@ -18,15 +18,15 @@ Issues and PRDs for this repo live as GitHub issues. Use the `gh` CLI for all op Infer the repo from `git remote -v` — `gh` does this automatically when run inside a clone (`JLaborda/SmartWealthAI`). -## Relationship to Git specs and Notion +## Relationship to Git specs | Layer | Where | Role | | --- | --- | --- | -| **MVP specs (canonical)** | `docs/mvp/features/*.md`, `docs/mvp/architecture/architecture.md` | Product and engineering truth; update before closing work | -| **GitHub issues** | This repo's GitHub Issues | PRDs, vertical slices, and triage for Matt Pocock engineering skills (`to-issues`, `to-prd`, `triage`) | -| **Notion tasks** | Board `Cursor Agent Tasks` | Optional execution tracking via Notion MCP; see `AGENTS.md` and `docs/mvp/NOTION_SETUP.md` | +| **MVP specs (canonical)** | `spec/features/00N-slug/spec.md`, `spec/constitution/mission.md` | Product and engineering truth; update before closing work | +| **GitHub issues** | This repo's GitHub Issues | Feature implementation tracking, PRDs, vertical slices, triage | +| **`tasks.md`** | `spec/features/00N-slug/tasks.md` | Versioned checklist when a feature is in progress | -When a skill publishes to the issue tracker, create or update a **GitHub issue**. Link the relevant `docs/mvp/features/...` path in the issue body. Do not treat Notion as the issue tracker for those skills unless the user explicitly asks to sync there. +When a skill publishes to the issue tracker, create or update a **GitHub issue**. Link the relevant `spec/features/.../spec.md` path in the issue body. See [`spec/meta/github-issues.md`](../../spec/meta/github-issues.md). ## When a skill says "publish to the issue tracker" diff --git a/.cursor/rules/mvp-docs.mdc b/.cursor/rules/mvp-docs.mdc index 5c9ff20..fe18432 100644 --- a/.cursor/rules/mvp-docs.mdc +++ b/.cursor/rules/mvp-docs.mdc @@ -1,19 +1,20 @@ --- description: MVP Markdown spec structure and editing conventions -globs: docs/** +globs: spec/** alwaysApply: false --- # MVP documentation conventions -When creating or editing specs under `docs/mvp/`: +When creating or editing specs under `spec/`: ## Feature specs -Follow the structure used in existing features (e.g. `docs/mvp/features/cheap-stocks.md`): +Follow the structure in [`spec/meta/feature-spec-template.md`](../../spec/meta/feature-spec-template.md) and existing features (e.g. `spec/features/003-cheap-stocks/spec.md`): - Objective -- MVP scope / Out of MVP scope +- Delivery (phase / roadmap link) +- In scope / Out of scope - Inputs (tables where helpful) - Flow or logic (include at least one **Mermaid** diagram when the flow changes) - Acceptance criteria @@ -23,7 +24,7 @@ Add **Implementation status** when coding starts: `planned` | `in_progress` | `d ## Architecture spec -- Record closed decisions in tables (see "Decisions made so far" in `docs/mvp/architecture/architecture.md`). +- Record closed decisions in tables (see "Decisions made so far" in `spec/constitution/mission.md`). - Cross-link feature files from the module table. - Do not contradict a closed architecture decision without updating the architecture doc first. diff --git a/.cursor/rules/notion-tasks.mdc b/.cursor/rules/notion-tasks.mdc deleted file mode 100644 index 759d3c4..0000000 --- a/.cursor/rules/notion-tasks.mdc +++ /dev/null @@ -1,35 +0,0 @@ ---- -description: Notion task board vs Git specs -alwaysApply: true ---- - -# Notion task tracking - -**Git specs in `docs/mvp/` are canonical.** Notion is for tasks and status only. - -## Task fields - -Each Notion task should include: - -- **Title:** e.g. `Feature: Cheap Stocks — implement EY rank` -- **Spec path:** `docs/mvp/features/.md` -- **Notes:** copy acceptance criteria from the spec - -## Sync discipline - -1. Change MVP decisions in the Git spec first. -2. Then create or update Notion tasks. -3. On completion: update the Git spec (implementation status, acceptance criteria), then mark the Notion task done. - -## Default board - -**`Cursor Agent Tasks`** in this project's Notion workspace. Use Notion MCP `notion-search` with that name; do not require a board URL from the repo. See `AGENTS.md`. - -## Cursor skills (Notion MCP authenticated) - -- `spec-to-implementation` — break a spec into tasks -- `create-task` — add a single task -- `tasks-build` — implement from a Notion task URL -- `tasks-explain-diff` — document completed work in Notion - -If Notion MCP calls fail with auth errors, ask the user to authenticate the Notion plugin in Cursor Settings. diff --git a/.cursor/rules/project-context.mdc b/.cursor/rules/project-context.mdc index 7345673..6b9f80c 100644 --- a/.cursor/rules/project-context.mdc +++ b/.cursor/rules/project-context.mdc @@ -9,17 +9,17 @@ SmartWealthAI is a modular **quantitative value investing** system (MVP): US equ ## Current phase -**MVP planning and spec refinement** in `docs/mvp/`. **June 30, 2026 demo slice** ([`demo-slice.md`](../../docs/mvp/demo-slice.md)) is the active delivery target. Prefer updating specs over writing code unless the user explicitly moves to implementation. +**MVP planning and spec refinement** in `spec/`. **June 30, 2026 demo slice** ([`roadmap.md`](../../spec/constitution/roadmap.md)) is the active delivery target. Prefer updating specs over writing code unless the user explicitly moves to implementation. ## Canonical documentation -- Demo slice: `docs/mvp/demo-slice.md` -- ADRs: `docs/adr/` -- Architecture: `docs/mvp/architecture/architecture.md` -- Features: `docs/mvp/features/*.md` +- Constitution: `spec/constitution/` (`mission.md`, `tech-stack.md`, `roadmap.md`) +- ADRs: `spec/adr/` +- Features: `spec/features/00N-slug/spec.md` (optional `plan.md`, `tasks.md`) - Ubiquitous language: `CONTEXT.md` -- Requirements (historical): `docs/mvp/requirements/requirements.md` -- Backlog: `docs/mvp/backlog/backlog.md` +- Archive (historical): `spec/archive/requirements.md` +- Backlog: `spec/backlog/backlog.md` +- GitHub Issues workflow: `spec/meta/github-issues.md` ## Not canonical @@ -27,4 +27,4 @@ Do not use `src/` or `notebooks/` as reference architecture. Legacy exploration ## Code changes -Every implementation change must trace to a feature spec under `docs/mvp/features/` or the demo slice (`docs/mvp/demo-slice.md`). Read the architecture doc for cross-cutting rules before proposing design or code. +Every implementation change must trace to a feature spec under `spec/features/` or the roadmap (`spec/constitution/roadmap.md`). Read `spec/constitution/mission.md` for cross-cutting rules before proposing design or code. diff --git a/.cursor/rules/spec-driven-workflow.mdc b/.cursor/rules/spec-driven-workflow.mdc index 19a9e77..3ee5fcb 100644 --- a/.cursor/rules/spec-driven-workflow.mdc +++ b/.cursor/rules/spec-driven-workflow.mdc @@ -7,13 +7,13 @@ alwaysApply: true ## Before coding -1. Name the feature spec file (e.g. `docs/mvp/features/cheap-stocks.md`). -2. Read `docs/mvp/architecture/architecture.md` for global constraints: point-in-time data, hard exclusions, MLflow runs, paper trading, universe rules. +1. Name the feature spec file (e.g. `spec/features/003-cheap-stocks/spec.md`). +2. Read `spec/constitution/mission.md` for global constraints: point-in-time data, hard exclusions, MLflow runs, paper trading, universe rules. 3. If scope is unclear, ask the user and record the answer in the spec (open questions or decisions tables). ## During work -- Do not add features without a corresponding `docs/mvp/features/` document. Create or extend the spec first. +- Do not add features without a corresponding `spec/features/` document. Create or extend the spec first. - Keep changes aligned with MVP scope and "Out of MVP scope" sections in the spec. ## After implementation diff --git a/.cursor/skills/commit-split/SKILL.md b/.cursor/skills/commit-split/SKILL.md index 07cb502..799748e 100644 --- a/.cursor/skills/commit-split/SKILL.md +++ b/.cursor/skills/commit-split/SKILL.md @@ -54,7 +54,7 @@ Rules: imperative present tense; **lowercase** description; **no** trailing peri | --- | --- | | `docs` | `docs/`, `CONTEXT.md`, `AGENTS.md` | | `agents` | `.cursor/rules/` | -| `mvp` | `docs/mvp/**` specs | +| `mvp` | `spec/**` specs | | `skills` | `.cursor/skills/` | | `chore` | `.gitignore`, repo hygiene with no domain doc change | diff --git a/.cursor/skills/grill-with-docs/ADR-FORMAT.md b/.cursor/skills/grill-with-docs/ADR-FORMAT.md index da7e78e..dbf2274 100644 --- a/.cursor/skills/grill-with-docs/ADR-FORMAT.md +++ b/.cursor/skills/grill-with-docs/ADR-FORMAT.md @@ -1,8 +1,8 @@ # ADR Format -ADRs live in `docs/adr/` and use sequential numbering: `0001-slug.md`, `0002-slug.md`, etc. +ADRs live in `spec/adr/` and use sequential numbering: `0001-slug.md`, `0002-slug.md`, etc. -Create the `docs/adr/` directory lazily — only when the first ADR is needed. +Create the `spec/adr/` directory lazily — only when the first ADR is needed. ## Template @@ -24,7 +24,7 @@ Only include these when they add genuine value. Most ADRs won't need them. ## Numbering -Scan `docs/adr/` for the highest existing number and increment by one. +Scan `spec/adr/` for the highest existing number and increment by one. ## When to offer an ADR diff --git a/.cursor/skills/grill-with-docs/SKILL.md b/.cursor/skills/grill-with-docs/SKILL.md index 5ea0aa9..602b737 100644 --- a/.cursor/skills/grill-with-docs/SKILL.md +++ b/.cursor/skills/grill-with-docs/SKILL.md @@ -43,13 +43,13 @@ If a `CONTEXT-MAP.md` exists at the root, the repo has multiple contexts. The ma ├── src/ │ ├── ordering/ │ │ ├── CONTEXT.md -│ │ └── docs/adr/ ← context-specific decisions +│ │ └── spec/adr/ ← context-specific decisions │ └── billing/ │ ├── CONTEXT.md -│ └── docs/adr/ +│ └── spec/adr/ ``` -Create files lazily — only when you have something to write. If no `CONTEXT.md` exists, create one when the first term is resolved. If no `docs/adr/` exists, create it when the first ADR is needed. +Create files lazily — only when you have something to write. If no `CONTEXT.md` exists, create one when the first term is resolved. If no `spec/adr/` exists, create it when the first ADR is needed. ## During the session diff --git a/.cursor/skills/improve-codebase-architecture/SKILL.md b/.cursor/skills/improve-codebase-architecture/SKILL.md index c12b263..dcfb703 100644 --- a/.cursor/skills/improve-codebase-architecture/SKILL.md +++ b/.cursor/skills/improve-codebase-architecture/SKILL.md @@ -1,6 +1,6 @@ --- name: improve-codebase-architecture -description: Find deepening opportunities in a codebase, informed by the domain language in CONTEXT.md and the decisions in docs/adr/. Use when the user wants to improve architecture, find refactoring opportunities, consolidate tightly-coupled modules, or make a codebase more testable and AI-navigable. +description: Find deepening opportunities in a codebase, informed by the domain language in CONTEXT.md and the decisions in spec/adr/. Use when the user wants to improve architecture, find refactoring opportunities, consolidate tightly-coupled modules, or make a codebase more testable and AI-navigable. --- # Improve Codebase Architecture diff --git a/.cursor/skills/setup-matt-pocock-skills/SKILL.md b/.cursor/skills/setup-matt-pocock-skills/SKILL.md index b6948cb..df4949a 100644 --- a/.cursor/skills/setup-matt-pocock-skills/SKILL.md +++ b/.cursor/skills/setup-matt-pocock-skills/SKILL.md @@ -23,7 +23,7 @@ Look at the current repo to understand its starting state. Read whatever exists; - `git remote -v` and `.git/config` — is this a GitHub repo? Which one? - `AGENTS.md` and `CLAUDE.md` at the repo root — does either exist? Is there already an `## Agent skills` section in either? - `CONTEXT.md` and `CONTEXT-MAP.md` at the repo root -- `docs/adr/` and any `src/*/docs/adr/` directories +- `spec/adr/` and any `src/*/spec/adr/` directories - `.cursor/rules/` — do `issue-tracker.md`, `triage-labels.md`, and `domain.md` already exist? - `.scratch/` — sign that a local-markdown issue tracker convention is already in use @@ -60,11 +60,11 @@ Default: each role's string equals its name. Ask the user if they want to overri **Section C — Domain docs.** -> Explainer: Some skills (`improve-codebase-architecture`, `diagnose`, `tdd`) read a `CONTEXT.md` file to learn the project's domain language, and `docs/adr/` for past architectural decisions. They need to know whether the repo has one global context or multiple (e.g. a monorepo with separate frontend/backend contexts) so they look in the right place. +> Explainer: Some skills (`improve-codebase-architecture`, `diagnose`, `tdd`) read a `CONTEXT.md` file to learn the project's domain language, and `spec/adr/` for past architectural decisions. They need to know whether the repo has one global context or multiple (e.g. a monorepo with separate frontend/backend contexts) so they look in the right place. Confirm the layout: -- **Single-context** — one `CONTEXT.md` + `docs/adr/` at the repo root. Most repos are this. +- **Single-context** — one `CONTEXT.md` + `spec/adr/` at the repo root. Most repos are this. - **Multi-context** — `CONTEXT-MAP.md` at the root pointing to per-context `CONTEXT.md` files (typically a monorepo). ### 3. Confirm and edit diff --git a/.cursor/skills/setup-matt-pocock-skills/domain.md b/.cursor/skills/setup-matt-pocock-skills/domain.md index c97d6a6..c0d4d0d 100644 --- a/.cursor/skills/setup-matt-pocock-skills/domain.md +++ b/.cursor/skills/setup-matt-pocock-skills/domain.md @@ -6,7 +6,7 @@ How the engineering skills should consume this repo's domain documentation when - **`CONTEXT.md`** at the repo root, or - **`CONTEXT-MAP.md`** at the repo root if it exists — it points at one `CONTEXT.md` per context. Read each one relevant to the topic. -- **`docs/adr/`** — read ADRs that touch the area you're about to work in. In multi-context repos, also check `src//docs/adr/` for context-scoped decisions. +- **`spec/adr/`** — read ADRs that touch the area you're about to work in. In multi-context repos, also check `src//spec/adr/` for context-scoped decisions. If any of these files don't exist, **proceed silently**. Don't flag their absence; don't suggest creating them upfront. The producer skill (`/grill-with-docs`) creates them lazily when terms or decisions actually get resolved. @@ -17,7 +17,7 @@ Single-context repo (most repos): ``` / ├── CONTEXT.md -├── docs/adr/ +├── spec/adr/ │ ├── 0001-event-sourced-orders.md │ └── 0002-postgres-for-write-model.md └── src/ @@ -28,14 +28,14 @@ Multi-context repo (presence of `CONTEXT-MAP.md` at the root): ``` / ├── CONTEXT-MAP.md -├── docs/adr/ ← system-wide decisions +├── spec/adr/ ← system-wide decisions └── src/ ├── ordering/ │ ├── CONTEXT.md - │ └── docs/adr/ ← context-specific decisions + │ └── spec/adr/ ← context-specific decisions └── billing/ ├── CONTEXT.md - └── docs/adr/ + └── spec/adr/ ``` ## Use the glossary's vocabulary diff --git a/.gitignore b/.gitignore index 30828cb..152de8e 100644 --- a/.gitignore +++ b/.gitignore @@ -13,6 +13,8 @@ data/* !data/reference/** *hacienda* *portfolio* +!spec/features/*portfolio*/ +!spec/features/*portfolio*/** .DS_Store backup* notebooks/* \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index 4b84e11..701b97b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,7 +4,7 @@ Instructions for AI agents working in the SmartWealthAI repository. ## Project phase -SmartWealthAI is in **MVP planning and spec refinement**, with a **June 30, 2026 demo slice** as the current delivery target. Architecture and features are defined in Markdown under `docs/mvp/`. Implementation work must align with a feature spec in `docs/mvp/features/` and respect cross-cutting rules in `docs/mvp/architecture/architecture.md`. +SmartWealthAI is in **MVP planning and spec refinement**, with a **June 30, 2026 demo slice** as the current delivery target. Specifications live under `spec/`. Implementation work must align with a feature `spec.md` under `spec/features/` and respect cross-cutting rules in `spec/constitution/mission.md`. Do not treat `src/` or `notebooks/` as canonical architecture; they are legacy exploration and out of scope for planning context. @@ -13,13 +13,15 @@ Do not treat `src/` or `notebooks/` as canonical architecture; they are legacy e | Path | Purpose | | --- | --- | | `CONTEXT.md` | Ubiquitous language (domain terms; extend via `/grill-with-docs`) | -| `docs/adr/*.md` | Architecture Decision Records (hard-to-reverse choices) | -| `docs/mvp/demo-slice.md` | June 30 delivery target (narrow vertical slice) | -| `docs/mvp/architecture/architecture.md` | MVP vision, principles, module table, closed decisions | -| `docs/mvp/features/*.md` | Per-module specs (scope, acceptance criteria, diagrams) | -| `docs/mvp/requirements/requirements.md` | Sprint 0 spike (superseded by MVP specs; historical reference) | -| `docs/mvp/backlog/backlog.md` | Informal product ideas mapped to MVP features | -| `docs/README.md` | Index of the docs tree | +| `spec/adr/*.md` | Architecture Decision Records (hard-to-reverse choices) | +| `spec/constitution/roadmap.md` | June 30 delivery target (narrow vertical slice) | +| `spec/constitution/mission.md` | MVP vision, principles, module table, closed decisions | +| `spec/features/00N-slug/spec.md` | Per-module specs (scope, acceptance criteria); optional `plan.md` / `tasks.md` | +| `spec/constitution/tech-stack.md` | Technologies, infrastructure, runtime conventions | +| `spec/meta/github-issues.md` | GitHub Issues workflow for feature implementation | +| `spec/archive/requirements.md` | Sprint 0 spike (superseded by MVP specs; historical reference) | +| `spec/backlog/backlog.md` | Informal product ideas mapped to MVP features | +| `spec/README.md` | Index of the docs tree | See also `.cursor/rules/*.mdc` for persistent agent guidance. @@ -29,8 +31,8 @@ Write all code, comments, docstrings, documentation, commits, and PR text in **E ## Spec-driven workflow -1. Identify the feature spec (e.g. `docs/mvp/features/cheap-stocks.md`). -2. Read `docs/mvp/architecture/architecture.md` for constraints (point-in-time data, exclusions, MLflow, paper trading, etc.). +1. Identify the feature spec (e.g. `spec/features/003-cheap-stocks/spec.md`). +2. Read `spec/constitution/mission.md` for constraints (point-in-time data, exclusions, MLflow, paper trading, etc.). 3. Resolve open questions in the spec; record decisions in the spec or architecture doc. 4. Implement only what the spec allows for the current phase. 5. After implementation, update the feature spec (implementation status, acceptance criteria, links to code when it exists). @@ -72,7 +74,7 @@ export SIMFIN_API_KEY="" poetry run download-simfin ``` -Spec: [`docs/mvp/features/etl-data-lake.md`](docs/mvp/features/etl-data-lake.md). Operator guide: [`docs/mvp/guides/download-simfin.md`](docs/mvp/guides/download-simfin.md). Delivery target: [`docs/mvp/demo-slice.md`](docs/mvp/demo-slice.md). +Spec: [`spec/features/006-etl-data-lake/spec.md`](spec/features/006-etl-data-lake/spec.md). Operator guide: [`spec/guides/download-simfin.md`](spec/guides/download-simfin.md). Delivery target: [`spec/constitution/roadmap.md`](spec/constitution/roadmap.md). ### Fundamentals — SEC spike (frozen, phase 2) @@ -81,7 +83,7 @@ export SEC_IDENTITY="Your Name your@email.com" poetry run download-fundamentals --universe dow30 ``` -Guide: [`docs/mvp/guides/download-fundamentals.md`](docs/mvp/guides/download-fundamentals.md). +Guide: [`spec/guides/download-fundamentals.md`](spec/guides/download-fundamentals.md). ## Gotchas @@ -91,23 +93,14 @@ Guide: [`docs/mvp/guides/download-fundamentals.md`](docs/mvp/guides/download-fun - `data/` is gitignored except `data/reference/**` (versioned universe and mapping CSVs). Never commit personal finance files or raw broker exports. -## Notion (task tracking) +## GitHub Issues (task tracking) -Git specs in `docs/mvp/` are canonical. Notion tracks execution tasks only. Setup: [`docs/mvp/NOTION_SETUP.md`](docs/mvp/NOTION_SETUP.md). +Git specs in `spec/` are canonical. **GitHub Issues** track feature implementation. See [`spec/meta/github-issues.md`](spec/meta/github-issues.md). -### Default task board - -| Setting | Value | -| --- | --- | -| **Board name** | `Cursor Agent Tasks` | -| **Location** | This project's Notion workspace (connected via Cursor Notion MCP) | -| **Board URL in repo** | Not stored — use MCP OAuth and search by board name | - -Agents must use the Notion MCP server (authenticated in **Cursor → Settings → MCP**) to find and update this board. Search the workspace for `Cursor Agent Tasks` before creating or editing tasks. Do not ask the user to commit a Notion URL to the repository. - -Each task should reference a spec path (e.g. `docs/mvp/features/universe-construction.md`) and copy acceptance criteria from that spec. When a task completes, update the Git spec first, then mark the Notion task done. - -**Skills (after MCP auth):** `spec-to-implementation`, `create-task`, `tasks-build`, `tasks-explain-diff`. For `tasks-build`, the user supplies a single task URL in chat (not in this file). +- **One issue per feature** when implementation starts. +- Issue body must link `spec/features/00N-slug/spec.md` (and `plan.md` / `tasks.md` when they exist). +- Label `ready-for-agent` when the spec is complete and work can start. +- On completion: update `spec.md` and `tasks.md`, then close the issue. **Commit workflow:** `/commit_split` — project skill [`.cursor/skills/commit-split/SKILL.md`](.cursor/skills/commit-split/SKILL.md); splits into conventional commits; out-of-scope paths follow [`.gitignore`](.gitignore). @@ -117,7 +110,7 @@ Matt Pocock engineering skills ([`mattpocock/skills`](https://github.com/mattpoc ### Issue tracker -GitHub Issues on `JLaborda/SmartWealthAI` via the `gh` CLI. MVP specs in `docs/mvp/` remain canonical; Notion is optional for execution only. See [`.cursor/rules/issue-tracker.md`](.cursor/rules/issue-tracker.md). +GitHub Issues on `JLaborda/SmartWealthAI` via the `gh` CLI. MVP specs in `spec/` remain canonical. See [`.cursor/rules/issue-tracker.md`](.cursor/rules/issue-tracker.md) and [`spec/meta/github-issues.md`](spec/meta/github-issues.md). ### Triage labels @@ -125,4 +118,4 @@ Default vocabulary: `needs-triage`, `needs-info`, `ready-for-agent`, `ready-for- ### Domain docs -Single-context: `CONTEXT.md` + `docs/adr/` at repo root; MVP specs in `docs/mvp/` during planning. See [`.cursor/rules/domain.md`](.cursor/rules/domain.md). +Single-context: `CONTEXT.md` + `spec/adr/`; MVP specs in `spec/`. See [`.cursor/rules/domain.md`](.cursor/rules/domain.md). diff --git a/CHANGELOG.md b/CHANGELOG.md index 943c42a..3bd2247 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,12 +19,12 @@ First public release: the **June 30 demo slice** — a runnable Greenblatt-style - **Streamlit dashboard** — Overview, Ranking, and Portfolio pages (`run-dashboard`). - **MLflow run logging** — params, metrics, portfolio parquet artifact, and git commit SHA tag per pipeline run. - **Hermetic CI** — pytest fixtures; no live SimFin or yfinance calls in PR workflows. -- **MVP specs and ADRs** under `docs/mvp/` and `docs/adr/`. +- **MVP specs and ADRs** under `spec/` and `spec/adr/`. ### Requirements - Python 3.11+, Poetry. -- `SIMFIN_API_KEY` for live data download (see [download-simfin guide](docs/mvp/guides/download-simfin.md)). +- `SIMFIN_API_KEY` for live data download (see [download-simfin guide](spec/guides/download-simfin.md)). ### Quickstart diff --git a/CONTEXT.md b/CONTEXT.md index abc9705..41fc2d1 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -1,6 +1,6 @@ # SmartWealthAI -Ubiquitous language for the quantitative value-investing MVP. Canonical formulas and acceptance criteria live in `docs/mvp/`; this file is the concise vocabulary agents and humans share. Extend via `/grill-with-docs` when terms are resolved. +Ubiquitous language for the quantitative value-investing MVP. Canonical formulas and acceptance criteria live in `spec/`; this file is the concise vocabulary agents and humans share. Extend via `/grill-with-docs` when terms are resolved. ## Language @@ -29,7 +29,7 @@ Backtesting only companies that still exist today, overstating returns. Mitigate _Avoid_: living-universe backtest **Universe**: -Investable tickers for a run date. **June 30 demo:** all SimFin US companies minus banks/insurers/utilities (`IndustryId` exclusions + bank/insurance sanity check). **Phase 2:** historical S&P 500 constituents including delisted names. Spec: `docs/mvp/features/universe-construction.md`. +Investable tickers for a run date. **June 30 demo:** all SimFin US companies minus banks/insurers/utilities (`IndustryId` exclusions + bank/insurance sanity check). **Phase 2:** historical S&P 500 constituents including delisted names. Spec: `spec/features/011-universe-construction/spec.md`. _Avoid_: watchlist, portfolio, benchmark index today **Industry classification**: @@ -45,7 +45,7 @@ A company removed entirely from ranking (not a score penalty). Permanent loss an _Avoid_: penalize, down-rank, soft filter **Permanent loss filter**: -Hard exclusion for fraud or bankruptcy/distress risk before any score. Spec: `docs/mvp/features/permanent-loss-filter.md`. +Hard exclusion for fraud or bankruptcy/distress risk before any score. Spec: `spec/features/008-permanent-loss-filter/spec.md`. _Avoid_: risk score, stop-loss, drawdown rule **Review queue**: @@ -61,7 +61,7 @@ Income and cash-flow statements use SimFin **TTM**; balance sheet uses the lates _Avoid_: mixing balance-sheet TTM into ROC denominators, using annual income for ranking between rebalance dates **Return on capital (ROC)**: -`EBIT / (Net Working Capital + Net Fixed Assets)`. Quality factor; higher is better; cross-sectional **ROC rank** (1 = best). Spec: `docs/mvp/features/high-quality-stocks.md`. +`EBIT / (Net Working Capital + Net Fixed Assets)`. Quality factor; higher is better; cross-sectional **ROC rank** (1 = best). Spec: `spec/features/007-high-quality-stocks/spec.md`. _Avoid_: ROE, ROIC (unless explicitly that metric) **Net working capital (NWC)**: @@ -69,7 +69,7 @@ _Avoid_: ROE, ROIC (unless explicitly that metric) _Avoid_: total working capital without the excess-cash adjustment **Earnings yield (EY)**: -`EBIT / Enterprise Value`. Cheapness factor; higher is cheaper; cross-sectional **EY rank** (1 = cheapest). Spec: `docs/mvp/features/cheap-stocks.md`. +`EBIT / Enterprise Value`. Cheapness factor; higher is cheaper; cross-sectional **EY rank** (1 = cheapest). Spec: `spec/features/003-cheap-stocks/spec.md`. _Avoid_: dividend yield, earnings/price without EV **Enterprise value (EV)**: @@ -105,7 +105,7 @@ Simulated orders only; no real capital in the MVP. _Avoid_: live trading, shadow trading with real broker **Sell-watch**: -Daily monitor of model holdings for quality drop, fraud/bankruptcy, overvaluation, or opportunity cost; emits signals, no auto-execution. Spec: `docs/mvp/features/sell-watch.md`. +Daily monitor of model holdings for quality drop, fraud/bankruptcy, overvaluation, or opportunity cost; emits signals, no auto-execution. Spec: `spec/features/010-sell-watch/spec.md`. _Avoid_: stop-loss, trailing stop (deferred) **Sell signal**: @@ -113,7 +113,7 @@ Recommendation to exit a holding; requires explicit user confirmation before ord _Avoid_: auto-sell, trim (deferred state) **Walk-forward backtest**: -Rolling train/validation windows (3–5 years) over 20+ years of PIT data; annual rebalance. Spec: `docs/mvp/features/backtesting.md`. **Deferred to phase 2** for the June 30 demo MVP; demo slice stops at ranked model portfolio + dashboard. +Rolling train/validation windows (3–5 years) over 20+ years of PIT data; annual rebalance. Spec: `spec/features/001-backtesting/spec.md`. **Deferred to phase 2** for the June 30 demo MVP; demo slice stops at ranked model portfolio + dashboard. _Avoid_: single in-sample fit, peeking at hold-out (when backtest ships) **Block bootstrap**: @@ -138,7 +138,7 @@ _Avoid_: ad-hoc snapshot without run id ## Flagged ambiguities -Resolved scope cuts (see ADRs and [`docs/mvp/demo-slice.md`](docs/mvp/demo-slice.md)): +Resolved scope cuts (see ADRs and [`spec/constitution/roadmap.md`](spec/constitution/roadmap.md)): - **June 30 demo MVP:** SimFin bulk US → raw → normalizer → **universe (US market)** → ROC/EY → combined rank → top-30 EW model portfolio → Streamlit dashboard. No permanent loss filter, backtest, sell-watch, or paper trading in this slice. - SEC ETL spike (`sec_client`, `edgartools_client`, `download-fundamentals`) is **frozen** in repo for phase 2; demo pipeline uses SimFin bulk for fundamentals and run-date prices (`shareprices/latest`). @@ -149,4 +149,4 @@ Terminology reminders: - “Cheap” means high **EY**, not low P/E—use **EY rank** in issues and code names. - “Quality” means high **ROC**, not ESG or subjective moat—use **ROC rank**. - “Value trap” in specs means negative EBIT routed to **review queue**, not a separate score. -- MVP specs in `docs/mvp/` remain canonical until an ADR or architecture decision supersedes them; update `CONTEXT.md` when `/grill-with-docs` resolves a term conflict. +- MVP specs in `spec/` remain canonical until an ADR or architecture decision supersedes them; update `CONTEXT.md` when `/grill-with-docs` resolves a term conflict. diff --git a/Makefile b/Makefile index 665216f..6479df9 100644 --- a/Makefile +++ b/Makefile @@ -10,7 +10,7 @@ lint: test: poetry run pytest --cov=smartwealthai --cov-report=term-missing -# Requires SEC_IDENTITY in the environment. See docs/mvp/guides/download-fundamentals.md +# Requires SEC_IDENTITY in the environment. See spec/guides/download-fundamentals.md download-fundamentals: poetry run download-fundamentals --universe dow30 diff --git a/README.md b/README.md index d69378d..8fdc9a4 100644 --- a/README.md +++ b/README.md @@ -5,13 +5,13 @@ **A quantitative value-investing MVP: Greenblatt-style ranking on US equities.** -*Status: **v0.1.0** — [June 30 demo slice](docs/mvp/demo-slice.md) runnable from CLI (Poetry + SimFin API key).* +*Status: **v0.1.0** — [June 30 demo slice](spec/constitution/roadmap.md) runnable from CLI (Poetry + SimFin API key).* ## Project vision SmartWealthAI is a modular quantitative value investing system: SimFin fundamentals, point-in-time correctness, explainable ROC/EY ranking, and a Streamlit dashboard. The full architecture (backtest, sell-watch, paper trading) is the north star; the demo slice ships a narrower vertical first. -Canonical specs: [`docs/mvp/`](docs/mvp/) · Ubiquitous language: [`CONTEXT.md`](CONTEXT.md) · ADRs: [`docs/adr/`](docs/adr/) +Canonical specs: [`spec/`](spec/) · Ubiquitous language: [`CONTEXT.md`](CONTEXT.md) · ADRs: [`spec/adr/`](spec/adr/) ## Tech stack @@ -20,7 +20,7 @@ Canonical specs: [`docs/mvp/`](docs/mvp/) · Ubiquitous language: [`CONTEXT.md`] * **Data (demo):** SimFin bulk fundamentals and `shareprices/latest` (`simfin`) * **Data (phase 2):** `yfinance` and free vendor fallbacks for prices / personal NAV * **Core libraries:** `pandas`, `simfin`, `yfinance`, `requests` (SEC spike: `edgartools` — frozen) -* **MVP specs:** `docs/mvp/` (architecture + per-module features) +* **MVP specs:** `spec/` (architecture + per-module features) ## 🚀 Quickstart @@ -35,7 +35,7 @@ Canonical specs: [`docs/mvp/`](docs/mvp/) · Ubiquitous language: [`CONTEXT.md`] ``` 3. **Demo slice docs** — start here before coding: - [`docs/mvp/demo-slice.md`](docs/mvp/demo-slice.md) + [`spec/constitution/roadmap.md`](spec/constitution/roadmap.md) 4. **Run the full demo pipeline** (one command for a `run_date`): @@ -44,7 +44,7 @@ Canonical specs: [`docs/mvp/`](docs/mvp/) · Ubiquitous language: [`CONTEXT.md`] poetry run run-demo-pipeline --run-date 2026-06-18 ``` - Individual stages (`download-simfin`, `normalize-simfin`, `build-universe`, `score-universe`, …) are also available. Guide: [`docs/mvp/guides/download-simfin.md`](docs/mvp/guides/download-simfin.md). + Individual stages (`download-simfin`, `normalize-simfin`, `build-universe`, `score-universe`, …) are also available. Guide: [`spec/guides/download-simfin.md`](spec/guides/download-simfin.md). 5. **Demo dashboard** (after the pipeline for the same `run_date`): @@ -52,7 +52,7 @@ Canonical specs: [`docs/mvp/`](docs/mvp/) · Ubiquitous language: [`CONTEXT.md`] poetry run run-dashboard --data-dir data --run-date 2026-06-18 ``` - Spec: [`docs/mvp/features/dashboard-reporting.md`](docs/mvp/features/dashboard-reporting.md). + Spec: [`spec/features/005-dashboard-reporting/spec.md`](spec/features/005-dashboard-reporting/spec.md). 6. **SEC fundamentals spike (frozen, phase 2):** ```bash @@ -60,11 +60,11 @@ Canonical specs: [`docs/mvp/`](docs/mvp/) · Ubiquitous language: [`CONTEXT.md`] poetry run download-fundamentals --universe dow30 ``` - Guide: [`docs/mvp/guides/download-fundamentals.md`](docs/mvp/guides/download-fundamentals.md). + Guide: [`spec/guides/download-fundamentals.md`](spec/guides/download-fundamentals.md). ## Roadmap -See [`docs/mvp/demo-slice.md`](docs/mvp/demo-slice.md) for the **June 30, 2026** delivery target and [`docs/mvp/architecture/architecture.md`](docs/mvp/architecture/architecture.md) for the full MVP north star. +See [`spec/constitution/roadmap.md`](spec/constitution/roadmap.md) for the **June 30, 2026** delivery target and [`spec/constitution/mission.md`](spec/constitution/mission.md) for the full MVP north star. ### Demo slice (v0.1.0) diff --git a/docs/README.md b/docs/README.md index 6eae17a..f93d3ee 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,84 +1,9 @@ -# SmartWealthAI documentation +# Documentation moved -All MVP architecture and feature specifications live here. This tree is the **versioned source of truth** for the project. Notion is used only for task tracking (see [AGENTS.md](../AGENTS.md)). +All specifications moved to [`spec/`](../spec/README.md). -## Ubiquitous language +- Constitution: [`spec/constitution/`](../spec/constitution/) +- Feature specs: [`spec/features/`](../spec/features/) +- ADRs: [`spec/adr/`](../spec/adr/) -Domain vocabulary lives in [`CONTEXT.md`](../CONTEXT.md) at the repo root (Matt Pocock [single-context](https://github.com/mattpocock/skills) pattern). Extend it with `/grill-with-docs`; agent consumption rules are in [`.cursor/rules/domain.md`](../.cursor/rules/domain.md). - -## Layout - -```text -docs/ - adr/ # Architecture Decision Records (why, not how) - mvp/ - demo-slice.md # June 30 delivery target (narrow vertical) - architecture/architecture.md # MVP vision, principles, module table, decisions - features/ # One spec per module - guides/ # Operator guides for implemented slices - prds/ # Implementation PRDs (tooling and cross-cutting) - requirements/requirements.md # Sprint 0 spike (historical; superseded by MVP specs) - backlog/backlog.md # Informal ideas mapped to features -``` - -## Architecture Decision Records (ADRs) - -Short notes on hard-to-reverse choices. See [`docs/adr/`](../adr/). - -| ADR | Decision | -| --- | --- | -| [0001](../adr/0001-simfin-fundamentals-mvp.md) | SimFin fundamentals for MVP; SEC ETL phase 2 | -| [0002](../adr/0002-june-demo-scope-cut.md) | June 30 demo slice scope cut | - -## Demo slice - -**Current delivery target:** [`demo-slice.md`](mvp/demo-slice.md) — SimFin → US universe → ROC/EY → top-30 portfolio → dashboard. - -## Operator guides - -| Guide | When to use | -| --- | --- | -| [download-simfin.md](mvp/guides/download-simfin.md) | **Active demo path:** download SimFin bulk fundamentals and `shareprices/latest`, then build universe and run-date price snapshots. | -| [download-fundamentals.md](mvp/guides/download-fundamentals.md) | **Frozen SEC spike** (phase 2): download `companyfacts` + `edgartools` for a universe. Active demo path uses SimFin — see [`demo-slice.md`](mvp/demo-slice.md) and [`etl-data-lake.md`](mvp/features/etl-data-lake.md). | - -## PRDs - -| PRD | Scope | -| --- | --- | -| [devcontainer/prd.md](mvp/prds/devcontainer/prd.md) | Reproducible dev environment (Cursor / EC2) | -| [ci-cd/ci-cd-prd.md](mvp/prds/ci-cd/ci-cd-prd.md) | Phase 0 CI/CD, AWS integration, ECS deploy path | -| [phase2/prd.md](mvp/prds/phase2/prd.md) | Phase 2: Quantitative Value funnel, cloud deploy, light/full backtest, sell-watch | - -## Feature specs - -| Spec | Module | -| --- | --- | -| [etl-data-lake.md](mvp/features/etl-data-lake.md) | ETL and data lake | -| [universe-construction.md](mvp/features/universe-construction.md) | Universe construction | -| [permanent-loss-filter.md](mvp/features/permanent-loss-filter.md) | Permanent loss filter | -| [high-quality-stocks.md](mvp/features/high-quality-stocks.md) | Quality scoring | -| [cheap-stocks.md](mvp/features/cheap-stocks.md) | Cheapness scoring | -| [corroborative-signals.md](mvp/features/corroborative-signals.md) | Corroborative signals | -| [unstructured-financial-data.md](mvp/features/unstructured-financial-data.md) | Unstructured data | -| [backtesting.md](mvp/features/backtesting.md) | Backtesting | -| [sell-watch.md](mvp/features/sell-watch.md) | Sell watch | -| [portfolio-evolution.md](mvp/features/portfolio-evolution.md) | Portfolio evolution (model vs personal vs benchmarks) | -| [dashboard-reporting.md](mvp/features/dashboard-reporting.md) | Dashboard and reporting | -| [broker-execution.md](mvp/features/broker-execution.md) | Broker execution (paper) | - -Start with [architecture.md](mvp/architecture/architecture.md) for global constraints, then open the feature spec for the area you are changing. - -## Spec-driven workflow - -1. Read or update the relevant feature spec. -2. Align with architecture decisions (point-in-time data, exclusions, paper trading, etc.). -3. Implement only what the spec allows for the current phase. -4. Update the spec after implementation (status, acceptance criteria, code links). - -## Notion tasks - -- Duplicate the [Code with Notion board template](https://notion.notion.site/code-with-notion-board). -- Link each task to a spec path under `docs/mvp/features/`. -- Task board: **`Cursor Agent Tasks`** in Notion (MCP OAuth; see [NOTION_SETUP.md](mvp/NOTION_SETUP.md) and [AGENTS.md](../AGENTS.md)). - -Cursor agent rules: `.cursor/rules/*.mdc`. +Ubiquitous language remains in [`CONTEXT.md`](../CONTEXT.md) at the repo root. diff --git a/docs/mvp/NOTION_SETUP.md b/docs/mvp/NOTION_SETUP.md deleted file mode 100644 index 3faa90a..0000000 --- a/docs/mvp/NOTION_SETUP.md +++ /dev/null @@ -1,39 +0,0 @@ -# Notion setup for SmartWealthAI - -Git specs under `docs/mvp/` stay canonical. Notion tracks tasks only. - -## Default board - -The project task board in Notion is named **`Cursor Agent Tasks`** (in this repo’s Notion workspace). Agents locate it via **Notion MCP search** — no board URL is committed to Git. See [AGENTS.md](../../AGENTS.md). - -## 1. Board in Notion - -Use the existing **`Cursor Agent Tasks`** database/board in the project Notion space. If you need a fresh board, you can duplicate the [Code with Notion template](https://notion.notion.site/code-with-notion-board) and name it `Cursor Agent Tasks`. - -## 2. Project home page (optional) - -Create a Notion page with: - -- Link to this repository -- Link to [architecture.md](architecture/architecture.md) on GitHub -- List of [feature specs](features/) - -## 3. Connect Cursor - -1. Cursor **Settings → MCP** → ensure the Notion plugin is enabled. -2. Authenticate when prompted (or run `mcp_auth` for Notion if tool calls fail). - -## 4. First tasks from a spec - -In Agent mode, after MCP auth: - -- Ask to create tasks on **`Cursor Agent Tasks`** from e.g. `docs/mvp/features/universe-construction.md`, or -- Ask to use **spec-to-implementation** (point the agent at the Git spec path; specs live in this repo, not in Notion). - -## Task discipline - -| Step | Where | -| --- | --- | -| Change MVP decision | Update Git spec first | -| Track work | Notion task with `docs/mvp/features/...` path | -| Finish work | Update Git spec (status + acceptance criteria), then mark Notion done | diff --git a/docs/mvp/backlog/backlog.md b/docs/mvp/backlog/backlog.md deleted file mode 100644 index 7a3fc89..0000000 --- a/docs/mvp/backlog/backlog.md +++ /dev/null @@ -1,16 +0,0 @@ -# Product backlog (informal) - -Informal ideas from the product owner. Each item should eventually map to one or more [feature specs](features/) and [architecture.md](../architecture/architecture.md). Notion tasks should reference the relevant spec path. - -| # | Idea | Likely MVP feature(s) | -| --- | --- | --- | -| 1 | Detect financial problems with holdings in my portfolio | [sell-watch.md](features/sell-watch.md), [permanent-loss-filter.md](features/permanent-loss-filter.md), [dashboard-reporting.md](features/dashboard-reporting.md) | -| 2 | Track portfolio evolution over time and compare to benchmarks (e.g. S&P 500) | [dashboard-reporting.md](features/dashboard-reporting.md), [backtesting.md](features/backtesting.md) | -| 3 | AI-assisted detector for undervalued stocks (Peter Lynch filters, Magic Formula) | [cheap-stocks.md](features/cheap-stocks.md), [high-quality-stocks.md](features/high-quality-stocks.md), [universe-construction.md](features/universe-construction.md) | -| 4 | Diversification analysis beyond sector labels (clustering in growth vs contraction regimes) | [corroborative-signals.md](features/corroborative-signals.md), [dashboard-reporting.md](features/dashboard-reporting.md) — may need a future spec | -| 5 | Detect overvalued positions where selling or trimming may make sense | [sell-watch.md](features/sell-watch.md), [cheap-stocks.md](features/cheap-stocks.md) | -| 6 | Rebalancing guidance (owner questions whether rebalance fits buy-cheap / sell-dear philosophy) | [architecture.md](../architecture/architecture.md) (annual rebalance decision), [broker-execution.md](features/broker-execution.md) | - -## Priority - -Ordering is not fixed here. During MVP planning, promote items into feature specs with acceptance criteria before implementation. diff --git a/spec/README.md b/spec/README.md new file mode 100644 index 0000000..0c8b1bc --- /dev/null +++ b/spec/README.md @@ -0,0 +1,82 @@ +# SmartWealthAI specifications + +All MVP architecture and feature specifications live here. This tree is the **versioned source of truth** for the project. GitHub Issues track execution (see [`meta/github-issues.md`](meta/github-issues.md)). + +## Ubiquitous language + +Domain vocabulary lives in [`CONTEXT.md`](../CONTEXT.md) at the repo root. Extend it with `/grill-with-docs`; agent consumption rules are in [`.cursor/rules/domain.md`](../.cursor/rules/domain.md). + +## Layout + +```text +spec/ + README.md + constitution/ + mission.md # Vision, principles, module map, closed domain decisions + tech-stack.md # Technologies, infrastructure, runtime conventions + roadmap.md # Delivery phases, demo slice, feature priority + features/ + 00N-slug/ + spec.md # Scope and acceptance criteria (always) + plan.md # Implementation plan (when in progress) + tasks.md # Versioned checklist (when in progress) + adr/ # Architecture Decision Records + guides/ # Operator guides for implemented slices + prds/ # Tooling and cross-cutting PRDs + meta/ # Agent workflow conventions + backlog/ # Informal product ideas + archive/ # Superseded historical specs +``` + +## Constitution + +| Document | Purpose | +| --- | --- | +| [`mission.md`](constitution/mission.md) | What we build, for whom, principles, module map | +| [`tech-stack.md`](constitution/tech-stack.md) | Python, AWS, SimFin, MLflow, CI/CD | +| [`roadmap.md`](constitution/roadmap.md) | June 30 demo slice, phase 2 order, feature registry | + +## Architecture Decision Records + +| ADR | Decision | +| --- | --- | +| [0001](adr/0001-simfin-fundamentals-mvp.md) | SimFin fundamentals for MVP; SEC ETL phase 2 | +| [0002](adr/0002-june-demo-scope-cut.md) | June 30 demo slice scope cut | + +## Operator guides + +| Guide | When to use | +| --- | --- | +| [download-simfin.md](guides/download-simfin.md) | **Active demo path:** SimFin bulk fundamentals and prices | +| [download-fundamentals.md](guides/download-fundamentals.md) | **Frozen SEC spike** (phase 2) | + +## Feature specs + +| ID | Module | Spec | +| --- | --- | --- | +| 001 | Backtesting | [spec.md](features/001-backtesting/spec.md) | +| 002 | Broker execution | [spec.md](features/002-broker-execution/spec.md) | +| 003 | Cheap stocks | [spec.md](features/003-cheap-stocks/spec.md) | +| 004 | Corroborative signals | [spec.md](features/004-corroborative-signals/spec.md) | +| 005 | Dashboard reporting | [spec.md](features/005-dashboard-reporting/spec.md) | +| 006 | ETL + data lake | [spec.md](features/006-etl-data-lake/spec.md) | +| 007 | High-quality stocks | [spec.md](features/007-high-quality-stocks/spec.md) | +| 008 | Permanent loss filter | [spec.md](features/008-permanent-loss-filter/spec.md) | +| 009 | Portfolio evolution | [spec.md](features/009-portfolio-evolution/spec.md) | +| 010 | Sell-watch | [spec.md](features/010-sell-watch/spec.md) | +| 011 | Universe construction | [spec.md](features/011-universe-construction/spec.md) | +| 012 | Unstructured financial data | [spec.md](features/012-unstructured-financial-data/spec.md) | + +Start with [`constitution/mission.md`](constitution/mission.md) for global constraints, then open the feature spec for the area you are changing. + +## Spec-driven workflow + +1. Read or update the relevant feature `spec.md`. +2. Align with [`constitution/mission.md`](constitution/mission.md) (point-in-time data, exclusions, paper trading, etc.). +3. When implementation starts, add `plan.md` and `tasks.md` under the feature folder. +4. Open **one GitHub issue per feature** (see [`meta/github-issues.md`](meta/github-issues.md)). +5. Update the spec after implementation (status, acceptance criteria, code links). + +## New feature specs + +Copy [`meta/feature-spec-template.md`](meta/feature-spec-template.md) into `spec/features/00N-slug/spec.md`. Register the ID in [`constitution/roadmap.md`](constitution/roadmap.md). diff --git a/docs/adr/0001-simfin-fundamentals-mvp.md b/spec/adr/0001-simfin-fundamentals-mvp.md similarity index 100% rename from docs/adr/0001-simfin-fundamentals-mvp.md rename to spec/adr/0001-simfin-fundamentals-mvp.md diff --git a/docs/adr/0002-june-demo-scope-cut.md b/spec/adr/0002-june-demo-scope-cut.md similarity index 81% rename from docs/adr/0002-june-demo-scope-cut.md rename to spec/adr/0002-june-demo-scope-cut.md index 89bf763..9231a0d 100644 --- a/docs/adr/0002-june-demo-scope-cut.md +++ b/spec/adr/0002-june-demo-scope-cut.md @@ -12,4 +12,4 @@ The **June 30 deliverable** is a reduced vertical slice, not the full architectu **Considered:** Full MVP including 20-year walk-forward backtest (rejected for June); demo + minimal backtest (rejected—user chose fastest path); Zipline for backtests (rejected—incompatible with Python 3.11, unmaintained, poor fit for fundamental annual rebalance). -**Consequences:** Documented in [`docs/mvp/demo-slice.md`](../mvp/demo-slice.md). Full feature specs remain the north star; modules marked deferred are unchanged in intent. MLflow logs **pipeline runs** in the demo; the `backtesting` experiment starts in phase 2. +**Consequences:** Documented in [`spec/constitution/roadmap.md`](../constitution/roadmap.md). Full feature specs remain the north star; modules marked deferred are unchanged in intent. MLflow logs **pipeline runs** in the demo; the `backtesting` experiment starts in phase 2. diff --git a/docs/mvp/requirements/requirements.md b/spec/archive/requirements.md similarity index 81% rename from docs/mvp/requirements/requirements.md rename to spec/archive/requirements.md index f5fc26f..7d23520 100644 --- a/docs/mvp/requirements/requirements.md +++ b/spec/archive/requirements.md @@ -1,6 +1,8 @@ +> **Superseded.** Historical Sprint 0 spike. Canonical specs live under `spec/`. + # Sprint 0 requirements: Magic Formula screener spike -> **Status:** Historical reference only. The full MVP is defined in [architecture.md](../architecture/architecture.md) and [features/](../features/). Do not treat this document as the current MVP scope. +> **Status:** Historical reference only. The full MVP is defined in [mission.md](../constitution/mission.md) and [features/](../features/). Do not treat this document as the current MVP scope. ## Primary goal @@ -36,4 +38,4 @@ Build a minimal Python pipeline that downloads financial data for a very small s - Databases, GUI, Docker, machine learning - Downloading thousands of tickers -Those belong to later MVP modules documented under `docs/mvp/features/`. +Those belong to later MVP modules documented under `spec/features/`. diff --git a/spec/backlog/backlog.md b/spec/backlog/backlog.md new file mode 100644 index 0000000..f6d6b40 --- /dev/null +++ b/spec/backlog/backlog.md @@ -0,0 +1,16 @@ +# Product backlog (informal) + +Informal ideas from the product owner. Each item should eventually map to one or more [feature specs](features/) and [mission.md](../constitution/mission.md). GitHub issues should reference the relevant spec path. + +| # | Idea | Likely MVP feature(s) | +| --- | --- | --- | +| 1 | Detect financial problems with holdings in my portfolio | [sell-watch.md](../010-sell-watch/spec.md), [permanent-loss-filter.md](../008-permanent-loss-filter/spec.md), [dashboard-reporting.md](../005-dashboard-reporting/spec.md) | +| 2 | Track portfolio evolution over time and compare to benchmarks (e.g. S&P 500) | [dashboard-reporting.md](../005-dashboard-reporting/spec.md), [backtesting.md](../001-backtesting/spec.md) | +| 3 | AI-assisted detector for undervalued stocks (Peter Lynch filters, Magic Formula) | [cheap-stocks.md](../003-cheap-stocks/spec.md), [high-quality-stocks.md](../007-high-quality-stocks/spec.md), [universe-construction.md](../011-universe-construction/spec.md) | +| 4 | Diversification analysis beyond sector labels (clustering in growth vs contraction regimes) | [corroborative-signals.md](../004-corroborative-signals/spec.md), [dashboard-reporting.md](../005-dashboard-reporting/spec.md) — may need a future spec | +| 5 | Detect overvalued positions where selling or trimming may make sense | [sell-watch.md](../010-sell-watch/spec.md), [cheap-stocks.md](../003-cheap-stocks/spec.md) | +| 6 | Rebalancing guidance (owner questions whether rebalance fits buy-cheap / sell-dear philosophy) | [mission.md](../constitution/mission.md) (annual rebalance decision), [broker-execution.md](../002-broker-execution/spec.md) | + +## Priority + +Ordering is not fixed here. During MVP planning, promote items into feature specs with acceptance criteria before implementation. diff --git a/docs/mvp/architecture/architecture.md b/spec/constitution/mission.md similarity index 64% rename from docs/mvp/architecture/architecture.md rename to spec/constitution/mission.md index c3939c7..ffbd4fe 100644 --- a/docs/mvp/architecture/architecture.md +++ b/spec/constitution/mission.md @@ -1,18 +1,18 @@ -# SmartWealthAI - MVP Architecture +# Mission -This document is a living specification for designing the SmartWealthAI MVP with a spec-driven development workflow. Its purpose is to capture decisions, open questions, assumptions, and acceptance criteria before writing implementation code. +This document captures product vision, architecture principles, module map, and closed domain decisions for the SmartWealthAI MVP. This is a portfolio project intended to showcase MLOps practices applied to a quantitative value investing system. The end user is a particular investor, but the system itself behaves as an automated agent that runs end-to-end without manual intervention. ## June 30 demo slice (current delivery target) -The **first shippable vertical** is narrower than the full vision below. See [`demo-slice.md`](../demo-slice.md) and [ADR-0002](../../adr/0002-june-demo-scope-cut.md): SimFin ETL → US-market universe → ROC/EY → top-30 equal-weight portfolio → Streamlit dashboard. Backtest, permanent loss filter, sell-watch, and paper trading are **phase 2**. The full architecture in this document remains the north star. +The **first shippable vertical** is narrower than the full vision below. See [`roadmap.md`](roadmap.md) and [ADR-0002](../adr/0002-june-demo-scope-cut.md): SimFin ETL → US-market universe → ROC/EY → top-30 equal-weight portfolio → Streamlit dashboard. Backtest, permanent loss filter, sell-watch, and paper trading are **phase 2**. The full architecture in this document remains the north star. ## MVP vision Build a modular quantitative value investing system that: -1. Retrieves financial data from **SimFin** (fundamentals, demo) and free price providers; SEC EDGAR deferred to phase 2 ([ADR-0001](../../adr/0001-simfin-fundamentals-mvp.md)). +1. Retrieves financial data from **SimFin** (fundamentals, demo) and free price providers; SEC EDGAR deferred to phase 2 ([ADR-0001](../adr/0001-simfin-fundamentals-mvp.md)). 2. Stores raw and curated data in an AWS-based, incrementally refreshed data lake. 3. Filters out companies with high risk of permanent capital loss (fraud and bankruptcy). 4. Identifies high-quality companies. @@ -171,22 +171,22 @@ flowchart LR | Module | Spec | Main responsibility | | --- | --- | --- | -| ETL + Data Lake | [../features/etl-data-lake.md](../features/etl-data-lake.md) | Download, version, validate, and store financial data with point-in-time guarantees and incremental refresh. | -| Universe construction | [../features/universe-construction.md](../features/universe-construction.md) | **Demo:** SimFin US minus banks / insurers / utilities. **Phase 2:** historical S&P 500 (incl. delisted), common-stock filters, share-class dedup. | -| Permanent loss filter | [../features/permanent-loss-filter.md](../features/permanent-loss-filter.md) | Hard-exclude companies with fraud or bankruptcy risk; include the Enron / Lehman / WorldCom regression test. | -| High-quality stocks | [../features/high-quality-stocks.md](../features/high-quality-stocks.md) | Score quality starting from Greenblatt's ROC. | -| Cheap stocks | [../features/cheap-stocks.md](../features/cheap-stocks.md) | Score valuation starting from Earnings Yield. | -| Corroborative signals | [../features/corroborative-signals.md](../features/corroborative-signals.md) | Buybacks, insider activity, and other confirming signals. | -| Unstructured financial data | [../features/unstructured-financial-data.md](../features/unstructured-financial-data.md) | Extract useful information from filings, transcripts, and news. | -| Backtesting + crisis report | [../features/backtesting.md](../features/backtesting.md) | Walk-forward backtest (3-5y windows) over 20+ years, Monte Carlo, benchmarks (S&P 500 CW/EW, Russell 3000, Magic Formula), crisis drawdown report. | -| Sell-watch / vigilance | [../features/sell-watch.md](../features/sell-watch.md) | Daily monitor of model portfolio for quality drop, fraud/bankruptcy, overvaluation, and opportunity cost. Emits signals (no auto-execution). | -| Portfolio evolution | [../features/portfolio-evolution.md](../features/portfolio-evolution.md) | Track the model portfolio and the user's personal portfolio over time and compare against configurable benchmarks. | -| Broker execution | [../features/broker-execution.md](../features/broker-execution.md) | Convert confirmed decisions into paper trading orders only. | -| Dashboard + reporting | [../features/dashboard-reporting.md](../features/dashboard-reporting.md) | Surface every input, score, decision, and explanation. Functional-first for the MVP. | +| ETL + Data Lake | [../features/006-etl-data-lake/spec.md](../features/006-etl-data-lake/spec.md) | Download, version, validate, and store financial data with point-in-time guarantees and incremental refresh. | +| Universe construction | [../features/011-universe-construction/spec.md](../features/011-universe-construction/spec.md) | **Demo:** SimFin US minus banks / insurers / utilities. **Phase 2:** historical S&P 500 (incl. delisted), common-stock filters, share-class dedup. | +| Permanent loss filter | [../features/008-permanent-loss-filter/spec.md](../features/008-permanent-loss-filter/spec.md) | Hard-exclude companies with fraud or bankruptcy risk; include the Enron / Lehman / WorldCom regression test. | +| High-quality stocks | [../features/007-high-quality-stocks/spec.md](../features/007-high-quality-stocks/spec.md) | Score quality starting from Greenblatt's ROC. | +| Cheap stocks | [../features/003-cheap-stocks/spec.md](../features/003-cheap-stocks/spec.md) | Score valuation starting from Earnings Yield. | +| Corroborative signals | [../features/004-corroborative-signals/spec.md](../features/004-corroborative-signals/spec.md) | Buybacks, insider activity, and other confirming signals. | +| Unstructured financial data | [../features/012-unstructured-financial-data/spec.md](../features/012-unstructured-financial-data/spec.md) | Extract useful information from filings, transcripts, and news. | +| Backtesting + crisis report | [../features/001-backtesting/spec.md](../features/001-backtesting/spec.md) | Walk-forward backtest (3-5y windows) over 20+ years, Monte Carlo, benchmarks (S&P 500 CW/EW, Russell 3000, Magic Formula), crisis drawdown report. | +| Sell-watch / vigilance | [../features/010-sell-watch/spec.md](../features/010-sell-watch/spec.md) | Daily monitor of model portfolio for quality drop, fraud/bankruptcy, overvaluation, and opportunity cost. Emits signals (no auto-execution). | +| Portfolio evolution | [../features/009-portfolio-evolution/spec.md](../features/009-portfolio-evolution/spec.md) | Track the model portfolio and the user's personal portfolio over time and compare against configurable benchmarks. | +| Broker execution | [../features/002-broker-execution/spec.md](../features/002-broker-execution/spec.md) | Convert confirmed decisions into paper trading orders only. | +| Dashboard + reporting | [../features/005-dashboard-reporting/spec.md](../features/005-dashboard-reporting/spec.md) | Surface every input, score, decision, and explanation. Functional-first for the MVP. | ## Functional flow — June 30 demo slice -See [`demo-slice.md`](../demo-slice.md). Steps not listed here are **phase 2**. +See [`roadmap.md`](roadmap.md). Steps not listed here are **phase 2**. 1. Pipeline run for a `run_date`; secrets from env / AWS Secrets Manager (`SIMFIN_API_KEY`, etc.). 2. Bulk-download SimFin US datasets if older than `refresh_days`; store verbatim under `raw/simfin/`. @@ -227,7 +227,7 @@ These items are now closed for the MVP. They can be reopened in later iterations | Area | Decision | | --- | --- | -| **June demo** | See [`demo-slice.md`](../demo-slice.md). SimFin → US universe → ROC/EY → top 30 EW → dashboard. | +| **June demo** | See [`roadmap.md`](roadmap.md). SimFin → US universe → ROC/EY → top 30 EW → dashboard. | | Markets | US only. Other markets deferred. | | Universe (demo) | All SimFin US companies minus banks/insurers/utilities. | | Universe (full MVP) | S&P 500 historical constituents (incl. delisted). Phase 2. | @@ -238,7 +238,7 @@ These items are now closed for the MVP. They can be reopened in later iterations | Share classes | Treat as the same company; keep the class with the highest average trading liquidity and drop the rest. Phase 2 for demo. | | Market cap floor | Optional parameter. Off in demo. | | Trading volume floor | Optional; off in demo. | -| Primary fundamentals source | **SimFin** (free tier, bulk download). [ADR-0001](../../adr/0001-simfin-fundamentals-mvp.md). | +| Primary fundamentals source | **SimFin** (free tier, bulk download). [ADR-0001](../adr/0001-simfin-fundamentals-mvp.md). | | SEC ETL | Frozen spike in repo; phase 2 normalizer. | | Primary price source | **Demo:** SimFin bulk `shareprices/latest`. **Phase 2:** `yfinance`; free tiers of FMP, Alpha Vantage, and EODHD as redundancy / fallback. | | Data lake | S3 (raw + curated zones) + DuckDB as the analytical engine (`duckdb` reads parquet directly from S3, no Athena bill). | @@ -375,123 +375,10 @@ Three options, in increasing complexity: The MVP uses block bootstrap. The synthetic-data approach is captured as a stretch goal. -### MLOps stack cost reality check -User proposed: GitHub Actions + MLflow + Prefect + Kubernetes. +## Related documents -For a daily run over the S&P 500 historical universe, Kubernetes is overkill and expensive. Recommended cost-aware mapping: - -| User goal | MVP-cheap option | Growth path | -| --- | --- | --- | -| Orchestration | Prefect Core running on a small EC2 (or Prefect Cloud free tier) | Prefect on EKS | -| Execution | AWS Batch or ECS Fargate spot tasks triggered by Prefect, or a small EC2 with cron + Docker | EKS with autoscaling | -| Experiment tracking | MLflow with SQLite + S3 artifact root, on the same EC2 | MLflow on RDS + EC2 / Fargate | -| CI/CD | GitHub Actions building Docker images and pushing to ECR | Same | -| Scheduling | EventBridge cron triggering a Prefect deployment | Same | -| Secrets | GitHub Secrets in CI; AWS Secrets Manager at runtime | Same | -| Observability | CloudWatch Logs + Prefect UI | CloudWatch Logs + Prefect Cloud + Grafana | - -This still showcases MLOps competence (CI/CD, container build, orchestrator, experiment tracking, secrets, observability) without paying for EKS in the MVP. - -### Risk checks confirmed for the MVP - -- Permanent loss filter has passed for every name in the target portfolio (already a hard exclusion upstream). -- Full scoring pipeline completed without partial / missing scores. -- Data freshness within threshold. -- No duplicate orders. -- Cash availability within configured limit. -- Backtest of the current configuration must beat all benchmark Sharpes. - -### False positives / false negatives review - -- Build a confusion matrix per rule from each historical backtest. -- Curated regression test in CI: the bankruptcy filter must flag Enron, Lehman, and WorldCom at the right `as_of_date`. -- Maintain a review queue dataset that contains every borderline decision with the triggered rule and its inputs. -- Shadow-mode new rules for one or two runs before they are allowed to influence decisions. - -### Overfitting controls - -- Walk-forward backtesting with 3 to 5 year windows. -- A separate hold-out window that the strategy never sees during tuning. -- Hard alert when in-sample Sharpe and out-of-sample Sharpe diverge beyond a threshold; treated as a red flag for the candidate configuration. - -### Personal portfolio CSV schema - -The consolidated personal portfolio file `data/clean/personal_finance/operations/my_operations_eur.csv` already exists and is generated from two brokers (XTB and IBKR) by `src/preprocessing/cleaning_operations.py` (notebook: `notebooks/portfolio_evolution.ipynb`). The schema is the contract that the portfolio evolution module must consume: - -| Column | Type | Notes | -| --- | --- | --- | -| `Date` | timestamp (with sub-second precision) | Operation timestamp. Used as the time index. | -| `Symbol` | string | Yahoo-Finance-compatible ticker after broker-to-yfinance mapping (e.g., `ITXe` / `ITX.ES` -> `ITX.MC`, `SPY5.UK` -> `SPY5.L`, `FB` -> `META`, `GOOGC` -> `GOOG`). | -| `Type` | enum | `BUY` or `SELL`. | -| `Volume` | float | Shares, fractional allowed. | -| `Price` | float | Per-share price in EUR (USD trades already FX-converted). | -| `Value` | float | Gross EUR notional of the trade. | -| `Commission` | float | EUR fee, negative when the user paid. | -| `Currency` | string | Always `EUR` after cleaning. | - -Open items implied by this schema (to be resolved in the portfolio evolution spec): - -- Dividends are not tracked in the CSV today. Out of MVP scope; tracked as a follow-up so the personal portfolio NAV can include the dividend contribution. -- FX is already applied to USD trades using `EUR=X` daily close at trade date. We document this as the assumption. -- Bankrupt or delisted holdings need a systematic handling: write the last available price as zero on the delisting date instead of dropping the position, so the personal NAV reflects the actual loss. (Note: the `IRBT` case visible in the legacy notebook is not a real loss in the user's history; the user sold IRBT in 2022, well before the bankruptcy. The rule still applies as a general defense.) -- Ticker remapping (FB -> META, GOOGC -> GOOG, SPY5.UK -> SPY5.L, ITXe / ITX.ES -> ITX.MC, etc.) is moved out of the notebook into a CSV at `data/reference/ticker_mapping.csv`, versioned in git. - -### Magic Formula replica (benchmark) - -The Greenblatt Magic Formula benchmark is implemented as a strict canonical replica: - -- Quality factor: `ROC = EBIT / (Net Working Capital + Net Fixed Assets)`. -- Cheapness factor: `EY = EBIT / Enterprise Value`. -- Combined rank: sum of the two cross-sectional ranks (lower is better). -- Same universe, same annual rebalance, same long-only construction as the production strategy. - -This benchmark is the placeholder while the user iterates on the quality and cheapness modules; future scoring variants are evaluated against it. - -### yfinance cache (phase 2) - -For phase 2 backtests and personal NAV, `yfinance` responses are cached on S3 under `s3://smartwealthai-cache/yfinance///.parquet`, with a configurable TTL per endpoint (e.g., prices: 1 day; corporate actions: 7 days; fundamentals: 90 days). Demo prices come from SimFin `shareprices/latest`. Cache misses trigger a live call; cache hits are read straight from S3. - -### Ticker mapping table - -A versioned CSV at `data/reference/ticker_mapping.csv` with columns `broker_symbol, yfinance_symbol, notes, valid_from, valid_to`. Stored in git for diff-able history. A small DuckDB view loads it on demand; we keep the source of truth in CSV because the table is small, infrequently updated, and benefits from PR-reviewable changes. - -## Open questions - -A short, targeted list. Everything else is now closed for the MVP. - -- Source of the S&P 500 historical constituents (including delisted): community dataset such as `github.com/fja05680/sp500`, periodic Wikipedia scrape, or a manually maintained CSV in `data/reference/`? Recommendation: import the community dataset once and pin a snapshot under `data/reference/sp500_constituents.csv`. To be confirmed in `universe-construction.md`. -- Final cost target: with `t4g.micro` for MLflow + Prefect, monthly bill targets around 6 USD plus S3 storage and SES. We track this as a soft budget. - -## Next steps for the spec-driven workflow - -Order proposed for refining the feature specs (each spec follows the same template: Objective, Scope, Out of scope, Inputs, Outputs, Mermaid diagram, Flow, Open questions, Acceptance criteria, Risks): - -1. `universe-construction.md` (new) - blocks every downstream module. -2. `etl-data-lake.md` — SimFin connector + normalizer (demo); SEC spike frozen for phase 2. ✅ Updated. -3. `permanent-loss-filter.md` (update) - fraud + bankruptcy, hard exclusion, Enron / Lehman / WorldCom regression test. -4. `high-quality-stocks.md` (update) - ROC + tie-break by market cap. -5. `cheap-stocks.md` (update) - Earnings Yield as primary cheapness signal. -6. `backtesting.md` (new) - walk-forward, 20+ years, Monte Carlo block bootstrap, benchmark suite, crisis drawdown report. -7. `sell-watch.md` (new) - daily triggers with the thresholds above, manual confirmation, AWS SES alert. -8. `portfolio-evolution.md` (new) - consume the personal CSV schema, paper-trade the model portfolio, configurable benchmarks. -9. `dashboard-reporting.md` (new) - Streamlit views per module, MLflow run links, explainability table. -10. `broker-execution.md` (update) - paper trading only, ECS Fargate Spot runtime, idempotent orders. -11. `corroborative-signals.md` (update) - light pass; deferred to a later iteration if needed. -12. `unstructured-financial-data.md` (update) - light pass; deferred to a later iteration if needed. - -## Architecture acceptance criteria - -- Every module has its own spec under `docs/mvp/features/`. -- Every spec includes objective, scope, inputs, outputs, flow, Mermaid diagram, open questions, and acceptance criteria. -- The architecture can run the pipeline end to end without a live broker. -- The architecture separates raw data, normalized data, point-in-time queries, scoring, portfolio construction, monitoring, backtesting, and orders. -- Investment decisions, exclusions, and sell signals can each be explained from versioned data and rules. -- Every run is logged as an MLflow run (parameters, metrics, artifacts) and / or as a versioned S3 snapshot, with a tag pointing to the git commit SHA. -- New data providers can be added without changing downstream scoring modules. -- The backtesting module can fail a run and block order generation when the strategy's Sharpe does not beat all benchmarks. -- The sell-watch module can emit signals that, after explicit user confirmation, flow into the broker module under the same audit trail as buy decisions. -- The portfolio evolution module can compare the model portfolio (as if traded) and the user's actual personal portfolio against at least one benchmark. -- The Enron / Lehman / WorldCom regression test runs in CI and fails the build if the bankruptcy filter stops flagging them. -- All secrets are sourced from GitHub Secrets (build) and / or AWS Secrets Manager (runtime). No secret is stored in the repo. -- Total MVP AWS bill stays under a small monthly budget (target to be set during infrastructure design). +- [`roadmap.md`](roadmap.md) — current delivery target and feature priority +- [`tech-stack.md`](tech-stack.md) — technologies, infrastructure, and runtime conventions +- [`../features/`](../features/) — per-module specs (`spec.md`, optional `plan.md` / `tasks.md`) +- [`../adr/`](../adr/) — architecture decision records diff --git a/docs/mvp/demo-slice.md b/spec/constitution/roadmap.md similarity index 58% rename from docs/mvp/demo-slice.md rename to spec/constitution/roadmap.md index 0572334..a578190 100644 --- a/docs/mvp/demo-slice.md +++ b/spec/constitution/roadmap.md @@ -1,8 +1,10 @@ -# June 30 demo slice — simplest Magic Formula +# Roadmap + +## June 30 demo slice — simplest Magic Formula **Status:** accepted (see [ADR-0002](../adr/0002-june-demo-scope-cut.md)) **Target date:** 2026-06-30 -**North star:** Full MVP in [`architecture/architecture.md`](architecture/architecture.md) — this document defines only what ships first. +**North star:** Full MVP in [`mission.md`](mission.md) — this document defines only what ships first. ## Objective @@ -12,21 +14,21 @@ Deliver a working, explainable Greenblatt-style Magic Formula pipeline on real U | Step | Module / spec | Notes | | --- | --- | --- | -| 1 | SimFin ETL | [`etl-data-lake.md`](features/etl-data-lake.md) — bulk US download, raw zone, SimFin normalizer | -| 2 | Universe | [`universe-construction.md`](features/universe-construction.md) — demo mode: SimFin US minus sector exclusions | -| 3 | Quality | [`high-quality-stocks.md`](features/high-quality-stocks.md) — ROC | -| 4 | Cheapness | [`cheap-stocks.md`](features/cheap-stocks.md) — Earnings Yield | +| 1 | SimFin ETL | [`etl-data-lake.md`](features/006-etl-data-lake/spec.md) — bulk US download, raw zone, SimFin normalizer | +| 2 | Universe | [`universe-construction.md`](features/011-universe-construction/spec.md) — demo mode: SimFin US minus sector exclusions | +| 3 | Quality | [`high-quality-stocks.md`](features/007-high-quality-stocks/spec.md) — ROC | +| 4 | Cheapness | [`cheap-stocks.md`](features/003-cheap-stocks/spec.md) — Earnings Yield | | 5 | Ranking | Combined rank = ROC rank + EY rank (lower is better); tie-break ascending market cap | | 6 | Model portfolio | Top **30** names, **equal-weight** only | -| 7 | Dashboard | [`dashboard-reporting.md`](features/dashboard-reporting.md) — ranking table, portfolio, per-name explainability | +| 7 | Dashboard | [`dashboard-reporting.md`](features/005-dashboard-reporting/spec.md) — ranking table, portfolio, per-name explainability | | 8 | MLflow | Log each pipeline run (params, scoring metrics, portfolio artifact) — `src/smartwealthai/mlflow_run_logging.py`, wired in `score-universe` ([#61](https://github.com/JLaborda/SmartWealthAI/issues/61)) | ## Out of scope (phase 2) -- Permanent loss filter ([`permanent-loss-filter.md`](features/permanent-loss-filter.md)) -- Backtesting and crisis report ([`backtesting.md`](features/backtesting.md)) -- Sell-watch ([`sell-watch.md`](features/sell-watch.md)) -- Paper trading / broker ([`broker-execution.md`](features/broker-execution.md)) +- Permanent loss filter ([`permanent-loss-filter.md`](features/008-permanent-loss-filter/spec.md)) +- Backtesting and crisis report ([`backtesting.md`](features/001-backtesting/spec.md)) +- Sell-watch ([`sell-watch.md`](features/010-sell-watch/spec.md)) +- Paper trading / broker ([`broker-execution.md`](features/002-broker-execution/spec.md)) - Watchlist (ranking table in dashboard is enough) - Corroborative signals, unstructured data, portfolio evolution (personal CSV) - SEC EDGAR ETL (frozen spike remains in repo — [ADR-0001](../adr/0001-simfin-fundamentals-mvp.md)) @@ -96,3 +98,26 @@ flowchart LR 4. Sell-watch + paper trading 5. SEC EDGAR normalizer (optional PIT upgrade) 6. Quantitative Value metrics (multi-period fundamentals from raw SimFin archives) + +## Feature registry (stable IDs) + +Chronological spec IDs (creation order). **Priority** is defined by this roadmap and [`../backlog/backlog.md`](../backlog/backlog.md), not by the number. + +| ID | Feature | Spec | +| --- | --- | --- | +| 001 | Backtesting | [`spec/features/001-backtesting/spec.md`](../features/001-backtesting/spec.md) | +| 002 | Broker Execution | [`spec/features/002-broker-execution/spec.md`](../features/002-broker-execution/spec.md) | +| 003 | Cheap Stocks | [`spec/features/003-cheap-stocks/spec.md`](../features/003-cheap-stocks/spec.md) | +| 004 | Corroborative Signals | [`spec/features/004-corroborative-signals/spec.md`](../features/004-corroborative-signals/spec.md) | +| 005 | Dashboard Reporting | [`spec/features/005-dashboard-reporting/spec.md`](../features/005-dashboard-reporting/spec.md) | +| 006 | Etl Data Lake | [`spec/features/006-etl-data-lake/spec.md`](../features/006-etl-data-lake/spec.md) | +| 007 | High Quality Stocks | [`spec/features/007-high-quality-stocks/spec.md`](../features/007-high-quality-stocks/spec.md) | +| 008 | Permanent Loss Filter | [`spec/features/008-permanent-loss-filter/spec.md`](../features/008-permanent-loss-filter/spec.md) | +| 009 | Portfolio Evolution | [`spec/features/009-portfolio-evolution/spec.md`](../features/009-portfolio-evolution/spec.md) | +| 010 | Sell Watch | [`spec/features/010-sell-watch/spec.md`](../features/010-sell-watch/spec.md) | +| 011 | Universe Construction | [`spec/features/011-universe-construction/spec.md`](../features/011-universe-construction/spec.md) | +| 012 | Unstructured Financial Data | [`spec/features/012-unstructured-financial-data/spec.md`](../features/012-unstructured-financial-data/spec.md) | + +## Informal backlog + +See [`../backlog/backlog.md`](../backlog/backlog.md). diff --git a/spec/constitution/tech-stack.md b/spec/constitution/tech-stack.md new file mode 100644 index 0000000..670a1e4 --- /dev/null +++ b/spec/constitution/tech-stack.md @@ -0,0 +1,103 @@ +# Tech stack + +Technologies, infrastructure, and runtime conventions for SmartWealthAI. +## Infrastructure (AWS, cheapest-first) + + +| Area | Decision | +| --- | --- | +| Storage | S3 (raw + curated parquet) + DuckDB as the local query engine. No Athena bill. | +| Experiment tracking | MLflow from day one. Tracking server on a small EC2 (SQLite backend) with `s3://` as the artifact root. | +| Dashboard | Streamlit. | +| Email alerts | AWS SES (sporadic emails, very cheap). | +| Compute / runtime | ECS Fargate Spot tasks (or AWS Batch on Fargate Spot), whichever is cheaper for the daily run. Triggered by Prefect. | +| Orchestration | Prefect Core (self-hosted on the same EC2 as MLflow, or via Prefect Cloud free tier). | +| Scheduling | EventBridge cron triggers the Prefect deployment once per day. | +| CI/CD | GitHub Actions builds and pushes Docker images to ECR. | +| Observability | CloudWatch Logs + Prefect UI for the MVP. Per-module metrics added if/when needed. | + + +## MLOps stack (cost-aware) + +User proposed: GitHub Actions + MLflow + Prefect + Kubernetes. + +For a daily run over the S&P 500 historical universe, Kubernetes is overkill and expensive. Recommended cost-aware mapping: + +| User goal | MVP-cheap option | Growth path | +| --- | --- | --- | +| Orchestration | Prefect Core running on a small EC2 (or Prefect Cloud free tier) | Prefect on EKS | +| Execution | AWS Batch or ECS Fargate spot tasks triggered by Prefect, or a small EC2 with cron + Docker | EKS with autoscaling | +| Experiment tracking | MLflow with SQLite + S3 artifact root, on the same EC2 | MLflow on RDS + EC2 / Fargate | +| CI/CD | GitHub Actions building Docker images and pushing to ECR | Same | +| Scheduling | EventBridge cron triggering a Prefect deployment | Same | +| Secrets | GitHub Secrets in CI; AWS Secrets Manager at runtime | Same | +| Observability | CloudWatch Logs + Prefect UI | CloudWatch Logs + Prefect Cloud + Grafana | + +This still showcases MLOps competence (CI/CD, container build, orchestrator, experiment tracking, secrets, observability) without paying for EKS in the MVP. + + +## MLflow + +User question: "are immutable snapshots like artifacts? What if we used MLflow?" + +Yes. MLflow is a natural fit here, and it covers three needs at once: + +- **Runs**: each pipeline execution (daily, plus every backtest) is logged as an MLflow run. Parameters (universe definition, rebalance frequency, weighting scheme, thresholds, git commit SHA) are logged via `mlflow.log_param`. Metrics (Sharpe, drawdown, CAGR, hit rate, alpha, number of exclusions, count of sell signals) are logged via `mlflow.log_metric`. +- **Artifacts**: the curated input slice (or its hash), the watchlist, the model portfolio, the backtest report, and the markdown / HTML dashboard snapshot are logged as artifacts. The artifact store points at S3 with versioning and / or object lock so a past run is byte-for-byte recoverable. +- **Model registry (optional, future)**: when the scoring model evolves beyond the Greenblatt placeholder, MLflow's model registry can promote a candidate from `staging` to `production` and tie that decision back to a backtest run. + +Practical MVP setup: + +- MLflow tracking server: a tiny EC2 (or AWS Fargate task on demand) with SQLite or RDS Postgres as the backend store. For the absolute cheapest setup, an MLflow tracking server is not strictly required: `mlflow.start_run(...)` with `file://` or `s3://` as the artifact root works for a single user. +- Artifact root: `s3://smartwealthai-mlflow-artifacts/`. +- Each run is tagged with the commit SHA and pipeline name, which makes the "immutable snapshot" effectively the MLflow run id. + +Treating this as nice-to-have for the MVP is fine: we can start by writing snapshots straight to S3 with predictable paths, and slot in MLflow once the pipeline stabilizes. + + +## Runtime (local development) + +- **Python:** 3.11+ (`requires-python = "^3.11"` in `pyproject.toml`) +- **Dependencies:** Poetry + +```bash +poetry env use python3.11 +poetry install +poetry run pytest +``` + +### MLflow (demo pipeline runs) + +`mlflow-skinny` is a Poetry dependency. No local MLflow UI server is required for pipeline logging. + +```bash +export MLFLOW_TRACKING_URI="file://$(pwd)/mlruns" +export MLFLOW_ALLOW_FILE_STORE=true +``` + +### Secrets + +- **Local dev:** `.env` (gitignored) for `SIMFIN_API_KEY`; devcontainer loads via `.devcontainer/install-env-hook.sh` +- **CI:** GitHub Actions secrets +- **Cloud runtime:** AWS Secrets Manager + +### Data lake (technical) + +- S3 raw + curated parquet zones; DuckDB as analytical engine +- Provider connectors: SimFin (demo), SEC EDGAR (phase 2 spike frozen) +- Cache: `yfinance` responses on S3 under `s3://smartwealthai-cache/yfinance/` (phase 2) + +## CI/CD and tooling PRDs + +| PRD | Path | +| --- | --- | +| Devcontainer | [`spec/prds/devcontainer/prd.md`](../prds/devcontainer/prd.md) | +| CI/CD | [`spec/prds/ci-cd/ci-cd-prd.md`](../prds/ci-cd/ci-cd-prd.md) | +| Phase 2 coordination | [`spec/prds/phase2/prd.md`](../prds/phase2/prd.md) | + +## Operator guides + +| Guide | Path | +| --- | --- | +| SimFin download (demo) | [`spec/guides/download-simfin.md`](../guides/download-simfin.md) | +| SEC fundamentals (frozen spike) | [`spec/guides/download-fundamentals.md`](../guides/download-fundamentals.md) | diff --git a/docs/mvp/features/backtesting.md b/spec/features/001-backtesting/spec.md similarity index 100% rename from docs/mvp/features/backtesting.md rename to spec/features/001-backtesting/spec.md diff --git a/docs/mvp/features/broker-execution.md b/spec/features/002-broker-execution/spec.md similarity index 100% rename from docs/mvp/features/broker-execution.md rename to spec/features/002-broker-execution/spec.md diff --git a/docs/mvp/features/cheap-stocks.md b/spec/features/003-cheap-stocks/spec.md similarity index 100% rename from docs/mvp/features/cheap-stocks.md rename to spec/features/003-cheap-stocks/spec.md diff --git a/docs/mvp/features/corroborative-signals.md b/spec/features/004-corroborative-signals/spec.md similarity index 100% rename from docs/mvp/features/corroborative-signals.md rename to spec/features/004-corroborative-signals/spec.md diff --git a/docs/mvp/features/dashboard-reporting.md b/spec/features/005-dashboard-reporting/spec.md similarity index 100% rename from docs/mvp/features/dashboard-reporting.md rename to spec/features/005-dashboard-reporting/spec.md diff --git a/docs/mvp/features/etl-data-lake.md b/spec/features/006-etl-data-lake/spec.md similarity index 98% rename from docs/mvp/features/etl-data-lake.md rename to spec/features/006-etl-data-lake/spec.md index 58a8c8e..ba93021 100644 --- a/docs/mvp/features/etl-data-lake.md +++ b/spec/features/006-etl-data-lake/spec.md @@ -164,7 +164,7 @@ Phase 2 yfinance cache semantics: - SimFin bulk connector implemented in `src/smartwealthai/download_simfin.py`, `simfin_client.py`, and `lake_paths.simfin_bulk_path`. Operator guide: - [`docs/mvp/guides/download-simfin.md`](../guides/download-simfin.md). + [`spec/guides/download-simfin.md`](../../guides/download-simfin.md). - SimFin fundamentals normalizer in `src/smartwealthai/simfin_normalizer.py` and `normalize_simfin.py` CLI; mapping at `config/fundamentals/simfin_mapping_v1.yaml`. Hermetic tests in `tests/test_simfin_normalizer.py` with fixtures under @@ -218,7 +218,7 @@ poetry run score-universe --run-date 2026-06-19 ## SimFin bulk connector (demo) Downloads US fundamentals via the `simfin` Python package into `raw/simfin/`. -Operator guide: [`docs/mvp/guides/download-simfin.md`](../guides/download-simfin.md). +Operator guide: [`spec/guides/download-simfin.md`](../../guides/download-simfin.md). ### CLI @@ -244,7 +244,7 @@ poetry run download-simfin --refresh-days 7 --force - [x] Per-dataset failures recorded in run summary; batch continues when possible. - [x] Hermetic tests cover path building, skip/force logic, and mocked download. - [x] Bulk ZIP extraction validates member paths (zip-slip guard); does not use simfin `load_*` extractall path. -- [x] Operator steps in [`download-simfin.md`](../guides/download-simfin.md). +- [x] Operator steps in [`download-simfin.md`](../../guides/download-simfin.md). ## SimFin normalizer (demo) @@ -361,7 +361,7 @@ notebook exploration and cross-checks — **not** for the production normalizer. No `submissions` ingest, no curated parquet normalizer in this slice (normalizer: [#52](https://github.com/JLaborda/SmartWealthAI/issues/52)), and no S3 upload. -**Operator guide:** [`docs/mvp/guides/download-fundamentals.md`](../guides/download-fundamentals.md) +**Operator guide:** [`spec/guides/download-fundamentals.md`](../../guides/download-fundamentals.md) ### Scope diff --git a/docs/mvp/features/high-quality-stocks.md b/spec/features/007-high-quality-stocks/spec.md similarity index 100% rename from docs/mvp/features/high-quality-stocks.md rename to spec/features/007-high-quality-stocks/spec.md diff --git a/docs/mvp/features/permanent-loss-filter.md b/spec/features/008-permanent-loss-filter/spec.md similarity index 100% rename from docs/mvp/features/permanent-loss-filter.md rename to spec/features/008-permanent-loss-filter/spec.md diff --git a/docs/mvp/features/portfolio-evolution.md b/spec/features/009-portfolio-evolution/spec.md similarity index 96% rename from docs/mvp/features/portfolio-evolution.md rename to spec/features/009-portfolio-evolution/spec.md index 27cf29a..0181335 100644 --- a/docs/mvp/features/portfolio-evolution.md +++ b/spec/features/009-portfolio-evolution/spec.md @@ -12,7 +12,7 @@ The user must be able to answer the questions "have my own decisions been worth ## MVP scope - Consume the consolidated personal-operations CSV (already in EUR) as the single source of truth for the personal portfolio. -- Use the closed schema documented in `docs/mvp/architecture/architecture.md` (Date, Symbol, Type, Volume, Price, Value, Commission, Currency). +- Use the closed schema documented in `spec/constitution/mission.md` (Date, Symbol, Type, Volume, Price, Value, Commission, Currency). - Reconstruct daily NAV in EUR using adjusted prices from the curated price store. - Apply the broker-to-yfinance ticker mapping (`data/reference/ticker_mapping.csv`). - Treat bankrupt or delisted holdings systematically: write a closing price of zero on the delisting date instead of dropping the position. @@ -113,5 +113,5 @@ flowchart TD - The CSV schema is the contract; any silent change in the cleaning pipeline (`src/preprocessing/cleaning_operations.py`) can break the portfolio evolution module. The CI should include a schema test. - Ticker remapping in code paths can drift from the CSV file in `data/reference/`. The mapping must be loaded from the file only. - Bankrupt-ticker handling can mask data errors as real losses. The pipeline must log a clear warning when a ticker disappears, and require an explicit "delisted_on" entry in `data/reference/delistings.csv` before zeroing the price. -- Free yfinance prices can be wrong or missing for older tickers (especially European listings). The fallback chain in `etl-data-lake` must cover this. +- Free yfinance prices can be wrong or missing for older tickers (especially European listings). The fallback chain in [`../006-etl-data-lake/spec.md`](../006-etl-data-lake/spec.md) must cover this. - Paper-trading the model assumes execution at the close of the rebalance date. This is optimistic; future iterations should model open-next-day execution and a basic spread cost. diff --git a/docs/mvp/features/sell-watch.md b/spec/features/010-sell-watch/spec.md similarity index 100% rename from docs/mvp/features/sell-watch.md rename to spec/features/010-sell-watch/spec.md diff --git a/docs/mvp/features/universe-construction.md b/spec/features/011-universe-construction/spec.md similarity index 100% rename from docs/mvp/features/universe-construction.md rename to spec/features/011-universe-construction/spec.md diff --git a/docs/mvp/features/unstructured-financial-data.md b/spec/features/012-unstructured-financial-data/spec.md similarity index 100% rename from docs/mvp/features/unstructured-financial-data.md rename to spec/features/012-unstructured-financial-data/spec.md diff --git a/docs/mvp/guides/download-fundamentals.md b/spec/guides/download-fundamentals.md similarity index 95% rename from docs/mvp/guides/download-fundamentals.md rename to spec/guides/download-fundamentals.md index 3aa2cee..76f44ae 100644 --- a/docs/mvp/guides/download-fundamentals.md +++ b/spec/guides/download-fundamentals.md @@ -1,11 +1,11 @@ # Guide: Download fundamentals (local spike — frozen) -> **Status:** This guide documents the **frozen SEC ETL spike** ([ADR-0001](../../adr/0001-simfin-fundamentals-mvp.md)). The June 30 demo pipeline uses **SimFin** instead — see [`demo-slice.md`](../demo-slice.md) and [`etl-data-lake.md`](../features/etl-data-lake.md). Do not delete this spike; it resumes in phase 2. +> **Status:** This guide documents the **frozen SEC ETL spike** ([ADR-0001](../../adr/0001-simfin-fundamentals-mvp.md)). The June 30 demo pipeline uses **SimFin** instead — see [`roadmap.md`](../roadmap.md) and [`etl-data-lake.md`](../../006-etl-data-lake/spec.md). Do not delete this spike; it resumes in phase 2. Operator guide for the first ETL vertical slice: download raw SEC `companyfacts` and standardized annual statements from `edgartools` for a parameterized universe. -**Canonical spec:** [`../features/etl-data-lake.md`](../features/etl-data-lake.md) (section +**Canonical spec:** [`../../006-etl-data-lake/spec.md`](../../006-etl-data-lake/spec.md) (section *Fundamentals download spike*). ## Prerequisites diff --git a/docs/mvp/guides/download-simfin.md b/spec/guides/download-simfin.md similarity index 97% rename from docs/mvp/guides/download-simfin.md rename to spec/guides/download-simfin.md index d94cff1..d3c5362 100644 --- a/docs/mvp/guides/download-simfin.md +++ b/spec/guides/download-simfin.md @@ -1,6 +1,6 @@ # Download SimFin bulk fundamentals and prices (demo) -Operator guide for the **SimFin bulk connector** on the June 30 demo path. Canonical spec: [`etl-data-lake.md`](../features/etl-data-lake.md). +Operator guide for the **SimFin bulk connector** on the June 30 demo path. Canonical spec: [`etl-data-lake.md`](../../006-etl-data-lake/spec.md). ## Prerequisites diff --git a/spec/meta/feature-spec-template.md b/spec/meta/feature-spec-template.md new file mode 100644 index 0000000..77f954e --- /dev/null +++ b/spec/meta/feature-spec-template.md @@ -0,0 +1,108 @@ +# Feature spec template + +Copy this file when creating a new feature. Do **not** edit this template in place. + +## Folder layout + +```text +spec/features/00N-slug/ + spec.md ← required (this template) + plan.md ← add when implementation starts (in_progress) + tasks.md ← add when implementation starts (checklist) +``` + +### Stable ID (`00N`) + +- Assign the **next unused three-digit prefix** when the spec file is first created (chronological ID, not priority). +- Check [`../constitution/roadmap.md`](../constitution/roadmap.md) for existing IDs. +- **Never renumber** an existing folder. Priority changes go only in `roadmap.md`. +- Use a lowercase **slug** (`kebab-case`) describing the module. + +### Workflow + +1. Copy sections below into `spec/features/00N-slug/spec.md`. +2. Align with [`../constitution/mission.md`](../constitution/mission.md) and related feature specs. +3. Add the feature to [`../constitution/roadmap.md`](../constitution/roadmap.md) (priority / phase). +4. Update [`../../CONTEXT.md`](../../CONTEXT.md) if new domain terms are introduced (`/grill-with-docs`). +5. Open **one GitHub issue** per feature when moving to implementation — see [`github-issues.md`](github-issues.md). + +--- + +# Feature: + +## Implementation status + + + + +## Delivery + + + + +**Phase:** + +## Objective + + + +## In scope + + + +## Out of scope + + + +## Inputs + +| Input | Source | Notes | +| --- | --- | --- | +| | | | + +## Outputs + +| Output | Path / target | +| --- | --- | +| | | + +## Mermaid diagram + + + +```mermaid +flowchart TD + A["Input"] --> B["Step"] + B --> C["Output"] +``` + +## Expected flow + + + +1. … +2. … + +## Acceptance criteria + + + +- … + +## Open questions + + + +- … + +## Risks + + + +- … + +## Related specs + + + +- [`../NNN-other-feature/spec.md`](../NNN-other-feature/spec.md) — … diff --git a/spec/meta/github-issues.md b/spec/meta/github-issues.md new file mode 100644 index 0000000..29799a0 --- /dev/null +++ b/spec/meta/github-issues.md @@ -0,0 +1,50 @@ +# GitHub Issues workflow + +Git specs under `spec/` are canonical. **GitHub Issues** track execution. + +## One issue per feature + +When a feature moves to implementation: + +1. Open **one GitHub issue** for the feature (e.g. `Feature: Cheap stocks — EY rank`). +2. Add label `ready-for-agent` when the spec is complete and work can start. +3. Create `plan.md` and `tasks.md` under `spec/features/00N-slug/` when implementation begins. + +## Issue body template + +```markdown +## Spec +spec/features/003-cheap-stocks/spec.md + +## Plan +spec/features/003-cheap-stocks/plan.md (when exists) + +## Tasks +See spec/features/003-cheap-stocks/tasks.md +``` + +Copy acceptance criteria from `spec.md` into the issue description or link to the spec path. + +## Sync discipline + +1. Change MVP decisions in the Git spec first (`spec.md`, or `constitution/` for cross-cutting rules). +2. Create or update the GitHub issue. +3. On completion: update `spec.md` (implementation status, acceptance criteria), check off `tasks.md`, then close the issue. + +## `tasks.md` format + +Markdown checklist in the feature folder. Example: + +```markdown +# Tasks: Cheap stocks + +- [ ] Implement EY from curated fundamentals +- [ ] Cross-sectional EY rank +- [ ] Wire into combined rank +``` + +`tasks.md` is the versioned checklist; the GitHub issue is the tracking unit on the project board. + +## Issue tracker + +Repository: `JLaborda/SmartWealthAI` via `gh` CLI. See [`.cursor/rules/issue-tracker.md`](../../.cursor/rules/issue-tracker.md). diff --git a/docs/mvp/prds/ci-cd/ci-cd-prd.md b/spec/prds/ci-cd/ci-cd-prd.md similarity index 97% rename from docs/mvp/prds/ci-cd/ci-cd-prd.md rename to spec/prds/ci-cd/ci-cd-prd.md index 5951ba3..210f82b 100644 --- a/docs/mvp/prds/ci-cd/ci-cd-prd.md +++ b/spec/prds/ci-cd/ci-cd-prd.md @@ -1,7 +1,7 @@ # PRD: CI/CD and MLOps Infrastructure (Phase 0) **Status:** Ready for implementation -**Canonical architecture:** `docs/mvp/architecture/architecture.md` +**Canonical architecture:** `spec/constitution/mission.md` **Related specs:** ETL + data lake, permanent loss filter, backtesting, sell-watch (pipeline vertical slice) --- @@ -224,8 +224,8 @@ Makefile targets wrap Poetry commands so CI and humans share entrypoints: instal ## Further Notes -- This PRD captures decisions from the CI/CD design session (grill-me). It should be reflected in a future feature spec under `docs/mvp/features/` (e.g. `cicd-infrastructure.md`) and cross-linked from `docs/mvp/architecture/architecture.md` when implementation starts — per spec-driven workflow, the feature spec becomes canonical for acceptance criteria and implementation status. +- This PRD captures decisions from the CI/CD design session (grill-me). It should be reflected in a future feature spec under `spec/features/` (e.g. `cicd-infrastructure.md`) and cross-linked from `spec/constitution/mission.md` when implementation starts — per spec-driven workflow, the feature spec becomes canonical for acceptance criteria and implementation status. - The core product vertical slice remains: ingest → permanent loss filter → quality (ROC) → cheapness (EY) → combined rank → model portfolio (~30 names) → backtest → sell-watch. CI/CD serves that slice, not the reverse. - Cost awareness: dev integration and ECS tasks should use minimal resource sizes and Spot where acceptable; align with the architecture soft budget (~low single-digit USD/month for control plane before storage growth). -- When implementation begins, update `docs/README.md` to index this PRD under an MVP PRDs section alongside the devcontainer PRD. -- GitHub issue creation with label `ready-for-agent` is recommended for tracking vertical implementation slices (`to-issues`), but this document is the saved PRD artifact at `docs/mvp/prds/ci-cd/ci-cd-prd.md` as requested. +- When implementation begins, update `spec/README.md` to index this PRD under an MVP PRDs section alongside the devcontainer PRD. +- GitHub issue creation with label `ready-for-agent` is recommended for tracking vertical implementation slices (`to-issues`), but this document is the saved PRD artifact at `spec/prds/ci-cd/ci-cd-prd.md` as requested. diff --git a/docs/mvp/prds/devcontainer/prd.md b/spec/prds/devcontainer/prd.md similarity index 95% rename from docs/mvp/prds/devcontainer/prd.md rename to spec/prds/devcontainer/prd.md index 8b8ee9b..2490227 100644 --- a/docs/mvp/prds/devcontainer/prd.md +++ b/spec/prds/devcontainer/prd.md @@ -119,8 +119,8 @@ This is an infrastructure/tooling PRD, not a feature module. There is no applica ## Further Notes - The devcontainer config is fully version-controlled under `.devcontainer/` and evolves with the project. Any team member (or the developer on a new machine) gets the same environment by opening the repo. -- The architecture doc (`docs/mvp/architecture/architecture.md`) references GitHub Actions + ECR for Docker image builds. The devcontainer's Docker-in-Docker feature allows local testing of those images before pushing. -- This PRD does not create a feature spec under `docs/mvp/features/` because the devcontainer is developer tooling, not an MVP feature module. It is tracked as a standalone PRD. +- The architecture doc (`spec/constitution/mission.md`) references GitHub Actions + ECR for Docker image builds. The devcontainer's Docker-in-Docker feature allows local testing of those images before pushing. +- This PRD does not create a feature spec under `spec/features/` because the devcontainer is developer tooling, not an MVP feature module. It is tracked as a standalone PRD. ## Files to Create or Modify diff --git a/docs/mvp/prds/phase2/prd.md b/spec/prds/phase2/prd.md similarity index 97% rename from docs/mvp/prds/phase2/prd.md rename to spec/prds/phase2/prd.md index b03e11b..c7232e9 100644 --- a/docs/mvp/prds/phase2/prd.md +++ b/spec/prds/phase2/prd.md @@ -1,10 +1,10 @@ # PRD: MVP Phase 2 — Quantitative Value, Cloud, Backtest, Sell-Watch **Status:** Ready for implementation -**Canonical architecture:** `docs/mvp/architecture/architecture.md` -**Prior delivery:** June 30 demo slice (`docs/mvp/demo-slice.md`, ADR-0002) +**Canonical architecture:** `spec/constitution/mission.md` +**Prior delivery:** June 30 demo slice (`spec/constitution/roadmap.md`, ADR-0002) **Related specs:** ETL + data lake, permanent loss filter, backtesting, sell-watch, universe construction, dashboard reporting -**Related PRDs:** CI/CD (`docs/mvp/prds/ci-cd/ci-cd-prd.md`) +**Related PRDs:** CI/CD (`spec/prds/ci-cd/ci-cd-prd.md`) **Capacity assumption:** Solo developer, ~10–15 hours per week **Estimated calendar:** Phase 2a ~13–16 weeks; Phase 2b ~8–12 weeks (~5–7 months total) @@ -117,7 +117,7 @@ Magic Formula remains the strategy's **benchmark comparator** for Sharpe pass/fa 49. As a developer, I want a new feature spec `quantitative-value.md` as the canonical QV module before implementation, so that spec-driven workflow is preserved. 50. As a developer, I want CONTEXT.md updated when QV terms (quality, cheap, funnel rank) are resolved, so that agents and humans share one vocabulary. -51. As a developer, I want `demo-slice.md` "After the demo" ordering updated to reflect QV-first production, so that docs do not contradict this PRD. +51. As a developer, I want `roadmap.md` "After the demo" ordering updated to reflect QV-first production, so that docs do not contradict this PRD. ## Implementation Decisions @@ -311,9 +311,9 @@ When implementation starts, resolve: ### Relationship to other documents -- This PRD does **not** replace per-module feature specs; it coordinates them. Each module keeps acceptance criteria in `docs/mvp/features/`. +- This PRD does **not** replace per-module feature specs; it coordinates them. Each module keeps acceptance criteria in `spec/features/`. - Implementers should read ADR-0001 (SimFin fundamentals), ADR-0002 (demo scope cut / no Zipline), and the CI/CD PRD before cloud work. -- After Phase 2a ships, update `demo-slice.md` "After the demo" ordering and `architecture.md` functional flow to state QV as production scoring. +- After Phase 2a ships, update `roadmap.md` "After the demo" ordering and `mission.md` functional flow to state QV as production scoring. ### Suggested GitHub issue title diff --git a/src/smartwealthai/download_fundamentals.py b/src/smartwealthai/download_fundamentals.py index b027cd9..a3e489c 100644 --- a/src/smartwealthai/download_fundamentals.py +++ b/src/smartwealthai/download_fundamentals.py @@ -1,7 +1,7 @@ """CLI to download raw SEC companyfacts and edgartools statements for a universe. Orchestrates the fundamentals download spike documented in -``docs/mvp/features/etl-data-lake.md``. For each ``(ticker, cik)`` in the universe: +``spec/features/006-etl-data-lake/spec.md``. For each ``(ticker, cik)`` in the universe: 1. SEC REST — verbatim ``companyfacts`` JSON. 2. edgartools — annual income, balance, and cash-flow statements as parquet. @@ -11,7 +11,7 @@ export SEC_IDENTITY="Your Name your@email.com" poetry run download-fundamentals --universe dow30 -See ``docs/mvp/guides/download-fundamentals.md`` for the full operator guide. +See ``spec/guides/download-fundamentals.md`` for the full operator guide. """ from __future__ import annotations @@ -181,7 +181,7 @@ def run_download( @click.command( context_settings={"help_option_names": ["-h", "--help"]}, - epilog="Guide: docs/mvp/guides/download-fundamentals.md", + epilog="Guide: spec/guides/download-fundamentals.md", ) @click.option("--universe", help="Universe preset name (e.g. dow30).") @click.option( diff --git a/src/smartwealthai/download_simfin.py b/src/smartwealthai/download_simfin.py index 96d065d..c4541b6 100644 --- a/src/smartwealthai/download_simfin.py +++ b/src/smartwealthai/download_simfin.py @@ -1,7 +1,7 @@ """CLI to download SimFin bulk US fundamentals into the raw data lake. Orchestrates the demo SimFin connector documented in -``docs/mvp/features/etl-data-lake.md``. Downloads ``companies``, ``industries``, +``spec/features/006-etl-data-lake/spec.md``. Downloads ``companies``, ``industries``, ``income`` (TTM), ``balance`` (quarterly), ``cashflow`` (TTM), and ``shareprices`` (latest) for ``market=us``. @@ -10,7 +10,7 @@ export SIMFIN_API_KEY="" poetry run download-simfin -See ``docs/mvp/guides/download-simfin.md`` for the operator guide. +See ``spec/guides/download-simfin.md`` for the operator guide. """ from __future__ import annotations diff --git a/src/smartwealthai/edgartools_client.py b/src/smartwealthai/edgartools_client.py index c2ed9ff..b968ce6 100644 --- a/src/smartwealthai/edgartools_client.py +++ b/src/smartwealthai/edgartools_client.py @@ -7,7 +7,7 @@ Output parquets preserve the edgartools dataframe shape (``concept``, ``label``, ``section``, ``FY 20xx`` columns). -Operator guide: ``docs/mvp/guides/download-fundamentals.md``. +Operator guide: ``spec/guides/download-fundamentals.md``. """ from __future__ import annotations diff --git a/src/smartwealthai/lake_paths.py b/src/smartwealthai/lake_paths.py index c5064fa..f96e650 100644 --- a/src/smartwealthai/lake_paths.py +++ b/src/smartwealthai/lake_paths.py @@ -1,6 +1,6 @@ """Path builders for the local data lake raw zone. -Paths follow the layout documented in ``docs/mvp/features/etl-data-lake.md`` (Fundamentals +Paths follow the layout documented in ``spec/features/006-etl-data-lake/spec.md`` (Fundamentals download spike) so local runs can move to S3 without renaming partitions. Example:: diff --git a/src/smartwealthai/normalize_simfin.py b/src/smartwealthai/normalize_simfin.py index 4190e3e..0143004 100644 --- a/src/smartwealthai/normalize_simfin.py +++ b/src/smartwealthai/normalize_simfin.py @@ -54,7 +54,7 @@ def resolve_normalize_tickers( if ticker_set is None: msg = ( "Pass --universe-run-date (after build-universe) or --ticker to limit scope. " - "See docs/mvp/guides/download-simfin.md." + "See spec/guides/download-simfin.md." ) raise click.ClickException(msg) return ticker_set diff --git a/src/smartwealthai/sec_client.py b/src/smartwealthai/sec_client.py index d2bb8c6..52c773c 100644 --- a/src/smartwealthai/sec_client.py +++ b/src/smartwealthai/sec_client.py @@ -6,7 +6,7 @@ - Request throttling (~8 req/s). - Retries with exponential backoff on transient failures. -Operator guide: ``docs/mvp/guides/download-fundamentals.md``. +Operator guide: ``spec/guides/download-fundamentals.md``. """ from __future__ import annotations diff --git a/tests/test_download_fundamentals.py b/tests/test_download_fundamentals.py index 6c75b52..d748eb5 100644 --- a/tests/test_download_fundamentals.py +++ b/tests/test_download_fundamentals.py @@ -2,7 +2,7 @@ Covers universe loading, raw-zone path layout, and skip/force cache semantics. Network integration tests are intentionally excluded from PR CI; see -``docs/mvp/guides/download-fundamentals.md``. +``spec/guides/download-fundamentals.md``. """ from __future__ import annotations