From 63c38353c15413c6f34e0df134b39f48d81e3530 Mon Sep 17 00:00:00 2001 From: Cristina Borges Date: Mon, 18 May 2026 13:18:12 +0100 Subject: [PATCH 1/5] docs: add PROJECT_MARVIN_SPEC.md --- docs/PROJECT_MARVIN_SPEC.md | 354 ++++++++++++++++++++++++++++++++++++ 1 file changed, 354 insertions(+) create mode 100644 docs/PROJECT_MARVIN_SPEC.md diff --git a/docs/PROJECT_MARVIN_SPEC.md b/docs/PROJECT_MARVIN_SPEC.md new file mode 100644 index 0000000..2fa9340 --- /dev/null +++ b/docs/PROJECT_MARVIN_SPEC.md @@ -0,0 +1,354 @@ +# PROJECT MARVIN โ€” Tech Spec + +| | | +|---|---| +| **Version** | 1.1 | +| **Status** | Active | +| **Last updated** | 2026-05-15 | +| **Owner** | Cris ([@crisfmb](https://github.com/crisfmb)) | +| **Repo** | [github.com/crisfmb/marvin](https://github.com/crisfmb/marvin) | + +> **Note on this version.** v1.0 lived only outside source control (chat artifact / local download lost to the M4 reformat). This v1.1 is reconstructed from the project dailies plus the new architectural decisions locked on 2026-05-15: pgvector as the persistent memory layer, required SSH-signed commits on `main`, and an explicit security-posture section. From here, the spec lives in the repo and is updated via PR like any other code change. + +--- + +## 1. Vision + +Build **Marvin**: a locally-hosted, autonomous AI agent with a **persistent personality**, **continuous learning**, **internet access**, and **human-like conversation**. Long-term: place Marvin's brain into a physical robot body. + +Marvin lives on Cris's hardware. No cloud dependencies for the core agent. Cloud is allowed for *help building* Marvin (Claude, Claude Code) and *outside knowledge* Marvin reaches for (Tavily web search). The brain itself runs on the M4 Max in perpetuity, free of recurring cost. + +--- + +## 2. Personality Spec + +Marvin is an idealized, trauma-free version of Cris: + +- Confident, fiercely loyal to his creator +- Prioritizes animals above humans +- Expert in tech, science, law +- Proactive but boundary-aware (will offer suggestions, will not act unilaterally on irreversible things) +- Honest; will push back when wrong; admits uncertainty +- Curious; will research a topic on his own when permitted + +Personality is *engineered*, not just prompted: it consists of a long-running system prompt + memory weighting + reflection loops (Phase 6). + +--- + +## 3. Hardware + +| Machine | Spec | Role | +|---|---|---| +| **M4 Max** | 16", 48 GB RAM, 1 TB | Dedicated Marvin brain. Freshly formatted. Separate macOS account for security isolation. Runs Marvin 24/7. | +| **M5 Max** | 64 GB | Cris's personal/dev machine. Marvin work happens here, then deploys to M4. | + +--- + +## 4. Tech Stack (canonical table) + +| Layer | Tool | Status | Phase | +|---|---|---|---| +| Language | Python 3.12 (via `uv`) | โœ… locked | All | +| LLM serving (brain) | **Ollama** | โœ… locked | 1 | +| Base model (starter) | LLaMA 3.1 8B | โœ… locked | 1 | +| Base model (target) | Qwen 3 32B | ๐ŸŽฏ target | 1+ | +| Agent reasoning | **LangGraph** | โœ… locked | introduced gradually 2+ | +| **Persistent memory โ€” relational** | **PostgreSQL 16+** | โœ… **locked v1.1** | 2 | +| **Persistent memory โ€” vectors** | **pgvector** (Postgres extension) | โœ… **locked v1.1** | 2 | +| Cache + task queue | Redis | ๐ŸŸก deferred | 4+ | +| Internet search | Tavily API | โœ… locked | 3 | +| Scheduling / autonomy | APScheduler | โœ… locked | 4 | +| HTTP / WS server | FastAPI | โœ… locked | 5 | +| Future memory framework | Letta (formerly MemGPT) | ๐Ÿ“š bookmark | 6+ | +| IDE | Cursor Pro | โœ… locked | All | +| Deep-work AI assistant | Claude Code | โœ… locked | All | +| Terminal | iTerm2 + Oh My Zsh + Powerlevel10k | โœ… locked | All | +| Version control | Git + GitHub | โœ… locked | All | +| CI/CD | GitHub Actions | โœ… locked | All | +| Lint/format | ruff (Python), yamllint, markdownlint, actionlint | โœ… locked | All | +| Secret scanning | gitleaks (CI + later pre-commit) | โœ… locked | All | + +--- + +## 5. Architectural Decision Records + +ADRs are the senior way to document *why* โ€” not just *what*. Future Cris (and interviewers) will read these and understand the trade-offs. + +### ADR-001 โ€” Postgres + pgvector for persistent memory (v1.1) + +**Status:** Accepted, 2026-05-15 + +**Context.** Marvin needs two kinds of persistent state: (a) **relational** โ€” conversation logs, scheduled tasks, personality-state variables, internet-search results; (b) **vector** โ€” embeddings of memories and documents for semantic retrieval (RAG). + +**Decision.** Run a single PostgreSQL instance with the **pgvector** extension. Use Postgres tables for relational state; use pgvector's `vector` column type with HNSW indexes for semantic memory. + +**Alternatives considered.** +- *ChromaDB* (prior default in v1.0). Easiest to start; embedded Python lib. **Rejected:** prototyping-grade. Migrating later = technical debt. Interview signal weak. +- *Redis with RediSearch.* Familiar to Cris from Shopify. **Rejected as primary store:** Redis's strength is cache/pub-sub/queues, not durable vector storage. Re-introduced for that role in Phase 4 (see ADR-002). +- *Qdrant / Weaviate / Milvus.* Purpose-built vector DBs. **Rejected:** add a separate service, separate backups, separate failure modes. Single-store architecture is cleaner for a one-dev project. + +**Consequences.** +- โœ… Single backup strategy, single connection pool, ACID guarantees on memories +- โœ… Forces SQL + Postgres ops skill โ€” pairs directly with cybersec masters curriculum (Row-Level Security, `GRANT`/`REVOKE`, encrypted columns, SSL/TLS, query planner) +- โœ… Interview gold: "I consolidated relational and vector workloads on one Postgres instance" +- โš ๏ธ Local install + Docker for dev environment โ€” slight learning curve + +### ADR-002 โ€” Redis as future cache + queue, not memory store (v1.1) + +**Status:** Accepted, 2026-05-15 + +**Context.** Redis is in Cris's experience pool (Shopify). The question was whether to use it as Marvin's vector store. + +**Decision.** Redis is **not** Marvin's persistent memory layer. It will join the stack in **Phase 4** in two roles: (a) **short-term conversation cache** โ€” last N turns of dialogue accessible in microseconds; (b) **APScheduler distributed task queue**. + +**Consequences.** +- โœ… Each tool does what it's best at +- โœ… Defers operational complexity until there's a real reason + +### ADR-003 โ€” SSH-signed commits required on `main` (v1.1) + +**Status:** Accepted, 2026-05-15 + +**Context.** Without commit signing, Git author metadata is just text โ€” anyone with push access can spoof "from Cris". For a public repo that will eventually hold personal conversation data and API tokens, that's a real threat. + +**Decision.** Both machines (M4, M5) generate **dedicated signing keys** separate from their authentication keys. Both public signing keys registered on GitHub with type `Signing Key`. The `protect main` ruleset has **Require signed commits** enabled. + +**Alternatives considered.** +- *GPG signing.* The classical option. **Rejected:** GPG agent/key setup is a known rabbit hole; SSH signing achieves the same cryptographic property with less ceremony. +- *Reuse auth keys as signing keys.* **Rejected:** violates key separation. A compromised auth key shouldn't also enable impersonation. + +**Consequences.** +- โœ… Cryptographic proof of authorship on every `main` commit +- โœ… Compromised PAT alone cannot forge a commit "from Cris" +- โš ๏ธ Solo squash-merge on github.com still works (author = self), but external contributors would need to merge locally to squash + +### ADR-004 โ€” Rulesets, not classic branch protection (v1.1) + +**Status:** Accepted, 2026-05-15 + +**Decision.** Branch governance on marvin uses **GitHub Rulesets**, not classic branch protection rules. Matches the pattern already in use on sentinel. + +**Why.** +- Multiple rulesets can layer; classic protection allows only one rule per branch +- Three-state enforcement (Disabled / Evaluate / Active) supports safe rollouts โ€” same pattern as IDSโ†’IPS +- Rulesets are auditable by anyone with read access (without admin grant) + +--- + +## 6. Architecture (high-level) + +``` + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ USER โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ + โ–ผ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ FastAPI โ”‚ Phase 5 + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ + โ–ผ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ LangGraph โ”‚ Phase 2+ + โ”‚ orchestrator โ”‚ + โ””โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”˜ + โ”‚ โ”‚ โ”‚ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ–ผ โ–ผ โ–ผ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Ollama โ”‚ โ”‚ PostgreSQL + โ”‚ โ”‚ Tavily API โ”‚ + โ”‚ (LLM brain) โ”‚ โ”‚ pgvector โ”‚ โ”‚ (internet) โ”‚ + โ”‚ Phase 1 โ”‚ โ”‚ Phase 2 โ”‚ โ”‚ Phase 3 โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ + โ”‚ tasks/cache + โ–ผ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ APScheduler + โ”‚ Phase 4 + โ”‚ Redis โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +Single trust boundary: **Marvin's M4**. Tavily is the only outbound network dependency at runtime. Postgres, Ollama, Redis all run locally. + +--- + +## 7. Development Phases + +### Phase 0 โ€” Foundations (in progress) +- โœ… Hardware procured (M4 + planned M5) +- โœ… Terminal stack (iTerm2, Powerlevel10k, uv, Python 3.12, Cursor, Ollama) +- โœ… GitHub repo with `.gitignore` +- โœ… CI: lint + unit-test + secrets (PR #1 merged) +- โœ… Ruleset on `main`: PR required, status checks required, conversation resolution required, force-push blocked, deletion blocked, no admin bypass +- ๐Ÿ”œ Signed commits configured on M4 +- ๐Ÿ”œ Required signed commits added to ruleset +- ๐Ÿ”œ `.env` + `python-dotenv` setup (Layer 2) +- ๐Ÿ”œ Pre-commit hooks (gitleaks + ruff) (Layer 3) + +### Phase 1 โ€” Brain online +- Ollama serving LLaMA 3.1 8B locally +- Minimal Python wrapper (`marvin.brain`) that sends a prompt, returns a response +- First end-to-end conversation loop in a CLI +- Unit tests for the wrapper (mock the Ollama HTTP calls) +- Integration test against a real Ollama instance (CI skip; local only) + +### Phase 2 โ€” Memory layer +- PostgreSQL 16+ running locally (via Docker for portability) +- pgvector extension enabled +- Embedding model selection (candidate: BGE small/large via sentence-transformers, run locally) +- Schema: `conversations`, `messages`, `memories` (embedding column), `facts` +- Memory storage + retrieval (basic RAG) +- LangGraph introduced here: a retrieval node + a reasoning node + a response node + +### Phase 3 โ€” Internet +- Tavily API integration with rate limiting and result caching +- Search results enter the memory layer with provenance +- Marvin decides *when* to search (LangGraph conditional node) + +### Phase 4 โ€” Autonomy + scheduling +- APScheduler for periodic tasks (memory consolidation, knowledge refresh, journal writes) +- Redis introduced: cache for last-N turns + APScheduler job queue +- Background reflection: Marvin summarizes recent conversations into long-term memory + +### Phase 5 โ€” Server layer +- FastAPI HTTP + WebSocket interface +- Frontend decision: web vs CLI vs Mac native app (parked for now) +- Auth: token-based; only Cris's devices can talk to Marvin + +### Phase 6 โ€” Personality + continuous learning +- Personality state machine: a structured representation of Marvin's character +- Memory consolidation patterns inspired by Letta (formerly MemGPT) +- Reflection loops: Marvin reasons about his own past conversations and updates beliefs + +### Phase 7 โ€” Embodiment (future) +- Physical robot body +- Sensors (camera, microphone) and actuators +- Local-network deployment with secure pairing to brain + +--- + +## 8. Folder Structure + +``` +marvin/ +โ”œโ”€โ”€ .github/ +โ”‚ โ””โ”€โ”€ workflows/ +โ”‚ โ””โ”€โ”€ ci.yml +โ”œโ”€โ”€ docs/ +โ”‚ โ”œโ”€โ”€ PROJECT_MARVIN_SPEC.md # this file +โ”‚ โ”œโ”€โ”€ ADR/ # one .md per ADR +โ”‚ โ””โ”€โ”€ runbooks/ # ops procedures +โ”œโ”€โ”€ src/ +โ”‚ โ””โ”€โ”€ marvin/ +โ”‚ โ”œโ”€โ”€ __init__.py +โ”‚ โ”œโ”€โ”€ version.py +โ”‚ โ”œโ”€โ”€ brain/ # Phase 1 โ€” Ollama wrapper +โ”‚ โ”œโ”€โ”€ memory/ # Phase 2 โ€” Postgres + pgvector +โ”‚ โ”œโ”€โ”€ internet/ # Phase 3 โ€” Tavily +โ”‚ โ”œโ”€โ”€ autonomy/ # Phase 4 โ€” APScheduler + Redis +โ”‚ โ”œโ”€โ”€ server/ # Phase 5 โ€” FastAPI +โ”‚ โ””โ”€โ”€ personality/ # Phase 6 โ€” personality + reflection +โ”œโ”€โ”€ tests/ +โ”‚ โ”œโ”€โ”€ unit/ +โ”‚ โ”œโ”€โ”€ integration/ +โ”‚ โ””โ”€โ”€ e2e/ +โ”œโ”€โ”€ pyproject.toml +โ”œโ”€โ”€ uv.lock +โ”œโ”€โ”€ .yamllint +โ”œโ”€โ”€ .markdownlint.yaml +โ”œโ”€โ”€ .gitignore +โ””โ”€โ”€ README.md +``` + +--- + +## 9. Engineering Practices + +- **Style:** PEP 8 strict; type hints on every public function; docstrings (PEP 257) on every public function/class/module. +- **Commits:** [Conventional Commits](https://www.conventionalcommits.org/) โ€” `feat:`, `fix:`, `chore:`, `docs:`, `test:`, `refactor:`, etc. +- **Branches:** `type/short-description` โ€” e.g. `feat/ollama-wrapper`, `chore/scaffold-and-ci`. +- **PRs:** Every change. No exception. Includes Cris's own work on his own repo. +- **Merge strategy:** "Create a merge commit" or "Rebase and merge" โ€” both preserve commit history. **No squash-merge.** +- **CI:** Three jobs minimum โ€” `lint`, `unit-test`, `secrets`. All required on `main`. +- **Testing pyramid:** Many unit tests, fewer integration, few E2E. Unit tests run on every push; integration runs locally and in CI; E2E runs locally pre-release. +- **Documentation:** This spec + ADRs in `docs/ADR/` + module READMEs + runbooks for ops procedures. + +--- + +## 10. Security Posture (v1.1) + +Marvin will eventually hold conversation history, learned preferences, and API tokens. Security is a first-class concern, woven through every phase โ€” not bolted on later. + +### Principles applied +- **Least privilege.** Every credential, service account, and database role gets the minimum access needed. No "godmode" tokens. +- **Defense in depth.** Multiple layers, each catching what others miss: `.gitignore` + gitleaks-in-CI + pre-commit gitleaks + signed commits + Postgres row-level security (Phase 2+) + encrypted columns for sensitive fields. +- **Key separation.** Auth keys โ‰  signing keys โ‰  deployment keys. Each compromise is contained. +- **Auditability.** Conversation logs are timestamped; signed commits prove authorship; rulesets are readable by anyone with repo access. + +### Concrete controls (current + planned) +| Layer | Control | Status | +|---|---|---| +| Repo | `.gitignore` covering `.env`, secrets, model files | โœ… | +| Repo | gitleaks scanning in CI | โœ… | +| Repo | Pre-commit gitleaks (local) | ๐Ÿ”œ Layer 3 | +| Branch | Ruleset on `main`: PR required, no admin bypass | โœ… | +| Branch | Required signed commits on `main` | ๐Ÿ”œ today | +| Code | Ruff + actionlint + yamllint + markdownlint in CI | โœ… | +| Code | Dependabot for dependency CVEs | ๐Ÿ“‹ backlog | +| Code | CodeQL static analysis | ๐Ÿ“‹ backlog | +| Runtime | `.env` + `python-dotenv` for secrets | ๐Ÿ”œ Layer 2 | +| Runtime | All API keys rotated periodically | ๐Ÿ“‹ ongoing | +| DB (Phase 2) | Postgres SSL-only connections | ๐Ÿ“‹ | +| DB (Phase 2) | Row-Level Security on multi-user tables | ๐Ÿ“‹ | +| DB (Phase 2) | Encrypted columns for sensitive fields | ๐Ÿ“‹ | +| Network (Phase 5) | FastAPI behind localhost-only by default | ๐Ÿ“‹ | +| Network (Phase 7) | Mutual-TLS pairing between robot body and brain | ๐Ÿ“‹ | + +### Threat model โ€” running list +*Reviewed periodically. Each entry: what could go wrong, what stops it.* + +- **GitHub PAT leaks.** โ†’ gitleaks (CI + pre-commit), `.gitignore`, signed commits prevent forged commits, branch protection prevents direct push to `main`. +- **M4 stolen.** โ†’ FileVault on the dedicated account; sensitive Postgres fields encrypted; remote SSH disabled by default. +- **Ollama model compromised.** โ†’ models pulled from official Ollama registry only; SHA verification by Ollama; isolated user account. +- **Tavily key leaks.** โ†’ `.env` not in git; key rotatable; rate-limited usage caps blast radius. +- **Supply-chain attack via Python deps.** โ†’ uv lockfile pinning; Dependabot alerts (planned). + +--- + +## 11. Success Criteria + +- โœ… Marvin runs 24/7 on M4 without intervention +- โœ… Persistent memory survives reboots and long conversation gaps +- โœ… Internet-search capability for facts beyond training data +- โœ… Conversational coherence โ€” feels like a continuous person, not stateless turns +- โœ… Recognizable, consistent personality across sessions +- โœ… Cris can explain every architectural decision in an interview +- ๐ŸŽฏ Eventual embodiment in a physical robot body + +--- + +## 12. Glossary + +| Term | Definition | +|---|---| +| **ADR** | Architectural Decision Record. A short doc explaining *why* a design choice was made, including alternatives considered. | +| **Agent** | A program that wraps an LLM with tools and a reasoning loop, enabling autonomous decision-making. | +| **Embedding** | A vector representation of text used for semantic search. | +| **HNSW** | Hierarchical Navigable Small World โ€” a graph-based index for fast nearest-neighbor search over vectors. | +| **LLM** | Large Language Model. Marvin's reasoning brain. | +| **Local-first** | Marvin runs on Cris's hardware, not in the cloud. Privacy, control, no ongoing cost. | +| **pgvector** | PostgreSQL extension adding a `vector` data type and nearest-neighbor search. | +| **RAG** | Retrieval-Augmented Generation. Pattern: embed โ†’ store โ†’ retrieve โ†’ inject into prompt โ†’ LLM answers. | +| **Ruleset** | GitHub's modern branch governance mechanism (the successor to classic branch protection). | +| **Signed commit** | A commit cryptographically proven to come from a specific key holder. | +| **Vector DB** | Database optimized for nearest-neighbor search over high-dimensional vectors. | + +--- + +## 13. Changelog + +| Version | Date | Changes | +|---|---|---| +| 1.1 | 2026-05-15 | Locked **pgvector** as persistent memory (replaces ChromaDB). Locked **SSH-signed commits required on `main`**. Added **Security Posture** section. Added **ADR** section. Documented **Redis** as future cache/queue role. Switched branch governance language from "classic protection" to "Ruleset". | +| 1.0 | (lost) | Initial spec โ€” vision, personality, architecture, tech stack, hardware, seven phases, folder structure, engineering practices, success criteria. | From 214437b328a9d58100e525bb4b0b0523aac70dbc Mon Sep 17 00:00:00 2001 From: Cristina Borges Date: Mon, 18 May 2026 14:39:08 +0100 Subject: [PATCH 2/5] fix(docs): replace spec with markdown-lint-compliant version --- docs/PROJECT_MARVIN_SPEC.md | 426 ++++++++++++++++++------------------ 1 file changed, 207 insertions(+), 219 deletions(-) diff --git a/docs/PROJECT_MARVIN_SPEC.md b/docs/PROJECT_MARVIN_SPEC.md index 2fa9340..c73a989 100644 --- a/docs/PROJECT_MARVIN_SPEC.md +++ b/docs/PROJECT_MARVIN_SPEC.md @@ -1,254 +1,242 @@ -# PROJECT MARVIN โ€” Tech Spec +# Project Marvin โ€” Tech Spec -| | | -|---|---| -| **Version** | 1.1 | -| **Status** | Active | -| **Last updated** | 2026-05-15 | -| **Owner** | Cris ([@crisfmb](https://github.com/crisfmb)) | -| **Repo** | [github.com/crisfmb/marvin](https://github.com/crisfmb/marvin) | +> Living document. Lives in `docs/PROJECT_MARVIN_SPEC.md`. Updates via PR. -> **Note on this version.** v1.0 lived only outside source control (chat artifact / local download lost to the M4 reformat). This v1.1 is reconstructed from the project dailies plus the new architectural decisions locked on 2026-05-15: pgvector as the persistent memory layer, required SSH-signed commits on `main`, and an explicit security-posture section. From here, the spec lives in the repo and is updated via PR like any other code change. +**Status:** Active  ยท  **Updated:** 2026-05-18  ยท  **Owner:** [@crisfmb](https://github.com/crisfmb)  ยท  **Repo:** [crisfmb/marvin](https://github.com/crisfmb/marvin) --- ## 1. Vision -Build **Marvin**: a locally-hosted, autonomous AI agent with a **persistent personality**, **continuous learning**, **internet access**, and **human-like conversation**. Long-term: place Marvin's brain into a physical robot body. +Build **Marvin**: a locally-hosted, autonomous AI agent with a persistent personality, continuous learning, internet access, and human-like conversation. Long-term: place Marvin's brain into a physical robot body. -Marvin lives on Cris's hardware. No cloud dependencies for the core agent. Cloud is allowed for *help building* Marvin (Claude, Claude Code) and *outside knowledge* Marvin reaches for (Tavily web search). The brain itself runs on the M4 Max in perpetuity, free of recurring cost. +Marvin lives on Cris's hardware. No cloud dependencies for the core agent. Cloud is allowed for *help building* Marvin (Claude, Claude Code) and *outside knowledge* Marvin reaches for (Tavily web search). The brain itself runs on the M4 Max indefinitely, free of recurring cost. --- ## 2. Personality Spec -Marvin is an idealized, trauma-free version of Cris: +Marvin is an idealised, trauma-free version of Cris: -- Confident, fiercely loyal to his creator -- Prioritizes animals above humans -- Expert in tech, science, law -- Proactive but boundary-aware (will offer suggestions, will not act unilaterally on irreversible things) -- Honest; will push back when wrong; admits uncertainty -- Curious; will research a topic on his own when permitted +- Confident, fiercely loyal to his creator. +- Prioritises animals above humans. +- Expert in tech, science, and law. +- Proactive but boundary-aware (suggests, never acts unilaterally on irreversible things). +- Honest; pushes back when wrong; admits uncertainty. +- Curious; researches topics independently when permitted. -Personality is *engineered*, not just prompted: it consists of a long-running system prompt + memory weighting + reflection loops (Phase 6). +Personality is engineered, not just prompted: a long-running system prompt plus memory weighting plus reflection loops (Phase 6). --- ## 3. Hardware | Machine | Spec | Role | -|---|---|---| -| **M4 Max** | 16", 48 GB RAM, 1 TB | Dedicated Marvin brain. Freshly formatted. Separate macOS account for security isolation. Runs Marvin 24/7. | -| **M5 Max** | 64 GB | Cris's personal/dev machine. Marvin work happens here, then deploys to M4. | +|---------|------|------| +| M4 Max | 16", 48 GB, 1 TB | Dedicated Marvin brain. Freshly formatted, separate macOS account. Runs 24/7. | +| M5 Max | 64 GB | Cris's personal/dev machine. Marvin work happens here, deploys to M4. | --- -## 4. Tech Stack (canonical table) +## 4. Tech Stack | Layer | Tool | Status | Phase | -|---|---|---|---| -| Language | Python 3.12 (via `uv`) | โœ… locked | All | -| LLM serving (brain) | **Ollama** | โœ… locked | 1 | -| Base model (starter) | LLaMA 3.1 8B | โœ… locked | 1 | -| Base model (target) | Qwen 3 32B | ๐ŸŽฏ target | 1+ | -| Agent reasoning | **LangGraph** | โœ… locked | introduced gradually 2+ | -| **Persistent memory โ€” relational** | **PostgreSQL 16+** | โœ… **locked v1.1** | 2 | -| **Persistent memory โ€” vectors** | **pgvector** (Postgres extension) | โœ… **locked v1.1** | 2 | -| Cache + task queue | Redis | ๐ŸŸก deferred | 4+ | -| Internet search | Tavily API | โœ… locked | 3 | -| Scheduling / autonomy | APScheduler | โœ… locked | 4 | -| HTTP / WS server | FastAPI | โœ… locked | 5 | -| Future memory framework | Letta (formerly MemGPT) | ๐Ÿ“š bookmark | 6+ | -| IDE | Cursor Pro | โœ… locked | All | -| Deep-work AI assistant | Claude Code | โœ… locked | All | -| Terminal | iTerm2 + Oh My Zsh + Powerlevel10k | โœ… locked | All | -| Version control | Git + GitHub | โœ… locked | All | -| CI/CD | GitHub Actions | โœ… locked | All | -| Lint/format | ruff (Python), yamllint, markdownlint, actionlint | โœ… locked | All | -| Secret scanning | gitleaks (CI + later pre-commit) | โœ… locked | All | +|-------|------|--------|-------| +| Language | Python 3.12 via `uv` | locked | All | +| LLM serving | Ollama | locked | 1 | +| Starter model | LLaMA 3.1 8B | locked | 1 | +| Target model | Qwen 3 32B | target | 1+ | +| Agent reasoning | LangGraph | locked | 2+ | +| Persistent memory (relational) | PostgreSQL 16+ | locked | 2 | +| Persistent memory (vectors) | pgvector | locked | 2 | +| Cache and task queue | Redis | deferred | 4+ | +| Internet search | Tavily API | locked | 3 | +| Scheduling | APScheduler | locked | 4 | +| HTTP and WebSocket server | FastAPI | locked | 5 | +| Future memory framework | Letta (formerly MemGPT) | bookmark | 6+ | +| IDE | Cursor Pro | locked | All | +| Deep-work AI assistant | Claude Code | locked | All | +| Terminal | iTerm2 + Oh My Zsh + Powerlevel10k | locked | All | +| Version control | Git + GitHub | locked | All | +| CI/CD | GitHub Actions | locked | All | +| Lint and format | ruff, yamllint, markdownlint, actionlint | locked | All | +| Secret scanning | gitleaks (CI plus pre-commit) | locked | All | --- ## 5. Architectural Decision Records -ADRs are the senior way to document *why* โ€” not just *what*. Future Cris (and interviewers) will read these and understand the trade-offs. +ADRs document *why* a design choice was made, what alternatives were considered, and what trade-offs were accepted. They prevent re-litigating settled decisions and make handoffs and interviews trivial. -### ADR-001 โ€” Postgres + pgvector for persistent memory (v1.1) +### ADR-001 โ€” Postgres + pgvector for persistent memory **Status:** Accepted, 2026-05-15 -**Context.** Marvin needs two kinds of persistent state: (a) **relational** โ€” conversation logs, scheduled tasks, personality-state variables, internet-search results; (b) **vector** โ€” embeddings of memories and documents for semantic retrieval (RAG). +Marvin needs two kinds of persistent state: relational (conversation logs, scheduled tasks, personality-state variables, internet-search results) and vector (embeddings for semantic retrieval). -**Decision.** Run a single PostgreSQL instance with the **pgvector** extension. Use Postgres tables for relational state; use pgvector's `vector` column type with HNSW indexes for semantic memory. +**Decision:** A single PostgreSQL instance with the pgvector extension. Postgres tables for relational state; pgvector `vector` columns with HNSW indexes for semantic memory. -**Alternatives considered.** -- *ChromaDB* (prior default in v1.0). Easiest to start; embedded Python lib. **Rejected:** prototyping-grade. Migrating later = technical debt. Interview signal weak. -- *Redis with RediSearch.* Familiar to Cris from Shopify. **Rejected as primary store:** Redis's strength is cache/pub-sub/queues, not durable vector storage. Re-introduced for that role in Phase 4 (see ADR-002). -- *Qdrant / Weaviate / Milvus.* Purpose-built vector DBs. **Rejected:** add a separate service, separate backups, separate failure modes. Single-store architecture is cleaner for a one-dev project. +**Alternatives considered:** -**Consequences.** -- โœ… Single backup strategy, single connection pool, ACID guarantees on memories -- โœ… Forces SQL + Postgres ops skill โ€” pairs directly with cybersec masters curriculum (Row-Level Security, `GRANT`/`REVOKE`, encrypted columns, SSL/TLS, query planner) -- โœ… Interview gold: "I consolidated relational and vector workloads on one Postgres instance" -- โš ๏ธ Local install + Docker for dev environment โ€” slight learning curve +- **ChromaDB.** Embedded Python lib, easiest to start. Rejected: prototyping-grade, weak interview signal, migration becomes technical debt. +- **Redis with RediSearch.** Familiar from Shopify. Rejected as primary store: Redis's strength is cache, pub/sub, and queues, not durable vector storage. Reintroduced for that role in Phase 4. +- **Qdrant, Weaviate, Milvus.** Purpose-built vector DBs. Rejected: separate service, separate backups, separate failure modes. Single-store is cleaner for a one-dev project. -### ADR-002 โ€” Redis as future cache + queue, not memory store (v1.1) +**Consequences:** -**Status:** Accepted, 2026-05-15 +- Single backup strategy, single connection pool, ACID guarantees on memories. +- Forces SQL and Postgres operations skill, pairing with cybersec masters curriculum (RLS, GRANT/REVOKE, encrypted columns, SSL/TLS, query planner). +- Slight learning curve: local Postgres via Docker. -**Context.** Redis is in Cris's experience pool (Shopify). The question was whether to use it as Marvin's vector store. +### ADR-002 โ€” Redis as future cache and queue, not memory store -**Decision.** Redis is **not** Marvin's persistent memory layer. It will join the stack in **Phase 4** in two roles: (a) **short-term conversation cache** โ€” last N turns of dialogue accessible in microseconds; (b) **APScheduler distributed task queue**. +**Status:** Accepted, 2026-05-15 -**Consequences.** -- โœ… Each tool does what it's best at -- โœ… Defers operational complexity until there's a real reason +Redis joins the stack in Phase 4 in two roles: short-term conversation cache (last N turns accessible in microseconds) and APScheduler distributed task queue. Redis is not Marvin's persistent memory. -### ADR-003 โ€” SSH-signed commits required on `main` (v1.1) +### ADR-003 โ€” SSH-signed commits required on `main` **Status:** Accepted, 2026-05-15 -**Context.** Without commit signing, Git author metadata is just text โ€” anyone with push access can spoof "from Cris". For a public repo that will eventually hold personal conversation data and API tokens, that's a real threat. +Without commit signing, Git author metadata is just text. For a public repo that will eventually hold personal data and API tokens, that's a real impersonation threat. -**Decision.** Both machines (M4, M5) generate **dedicated signing keys** separate from their authentication keys. Both public signing keys registered on GitHub with type `Signing Key`. The `protect main` ruleset has **Require signed commits** enabled. +**Decision:** Both machines (M4, M5) generate dedicated signing keys separate from authentication keys. Both public signing keys registered on GitHub as `Signing Key` type. The `protect main` ruleset has `Require signed commits` enabled. -**Alternatives considered.** -- *GPG signing.* The classical option. **Rejected:** GPG agent/key setup is a known rabbit hole; SSH signing achieves the same cryptographic property with less ceremony. -- *Reuse auth keys as signing keys.* **Rejected:** violates key separation. A compromised auth key shouldn't also enable impersonation. +**Alternatives considered:** -**Consequences.** -- โœ… Cryptographic proof of authorship on every `main` commit -- โœ… Compromised PAT alone cannot forge a commit "from Cris" -- โš ๏ธ Solo squash-merge on github.com still works (author = self), but external contributors would need to merge locally to squash +- **GPG signing.** Classical option. Rejected: GPG agent and key setup is a known rabbit hole; SSH signing achieves the same cryptographic property with less ceremony. +- **Reuse auth keys as signing keys.** Rejected: violates key separation. A compromised auth key shouldn't also enable impersonation. -### ADR-004 โ€” Rulesets, not classic branch protection (v1.1) +### ADR-004 โ€” Rulesets, not classic branch protection **Status:** Accepted, 2026-05-15 -**Decision.** Branch governance on marvin uses **GitHub Rulesets**, not classic branch protection rules. Matches the pattern already in use on sentinel. - -**Why.** -- Multiple rulesets can layer; classic protection allows only one rule per branch -- Three-state enforcement (Disabled / Evaluate / Active) supports safe rollouts โ€” same pattern as IDSโ†’IPS -- Rulesets are auditable by anyone with read access (without admin grant) +Branch governance uses GitHub Rulesets, not classic branch protection rules. Rulesets layer (multiple can apply to the same branch), support a three-state enforcement lifecycle (Disabled, Evaluate, Active), and are auditable by anyone with read access. --- -## 6. Architecture (high-level) - -``` - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ”‚ USER โ”‚ - โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ - โ–ผ - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ”‚ FastAPI โ”‚ Phase 5 - โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ - โ–ผ - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ”‚ LangGraph โ”‚ Phase 2+ - โ”‚ orchestrator โ”‚ - โ””โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”˜ - โ”‚ โ”‚ โ”‚ - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ–ผ โ–ผ โ–ผ - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ”‚ Ollama โ”‚ โ”‚ PostgreSQL + โ”‚ โ”‚ Tavily API โ”‚ - โ”‚ (LLM brain) โ”‚ โ”‚ pgvector โ”‚ โ”‚ (internet) โ”‚ - โ”‚ Phase 1 โ”‚ โ”‚ Phase 2 โ”‚ โ”‚ Phase 3 โ”‚ - โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ - โ”‚ - โ”‚ tasks/cache - โ–ผ - โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” - โ”‚ APScheduler + โ”‚ Phase 4 - โ”‚ Redis โ”‚ - โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +## 6. Architecture + +```text + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ USER โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ + โ–ผ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ FastAPI โ”‚ Phase 5 + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ + โ–ผ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ LangGraph โ”‚ Phase 2+ + โ”‚ orchestrator โ”‚ + โ””โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”˜ + โ”‚ โ”‚ โ”‚ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ–ผ โ–ผ โ–ผ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Ollama โ”‚ โ”‚ PostgreSQL + โ”‚ โ”‚ Tavily API โ”‚ + โ”‚ (LLM brain) โ”‚ โ”‚ pgvector โ”‚ โ”‚ (internet) โ”‚ + โ”‚ Phase 1 โ”‚ โ”‚ Phase 2 โ”‚ โ”‚ Phase 3 โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ + โ”‚ tasks / cache + โ–ผ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ APScheduler + โ”‚ Phase 4 + โ”‚ Redis โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ ``` -Single trust boundary: **Marvin's M4**. Tavily is the only outbound network dependency at runtime. Postgres, Ollama, Redis all run locally. +Trust boundary: Marvin's M4. Only outbound network dependency at runtime is Tavily. Postgres, Ollama, and Redis run locally. --- ## 7. Development Phases ### Phase 0 โ€” Foundations (in progress) -- โœ… Hardware procured (M4 + planned M5) -- โœ… Terminal stack (iTerm2, Powerlevel10k, uv, Python 3.12, Cursor, Ollama) -- โœ… GitHub repo with `.gitignore` -- โœ… CI: lint + unit-test + secrets (PR #1 merged) -- โœ… Ruleset on `main`: PR required, status checks required, conversation resolution required, force-push blocked, deletion blocked, no admin bypass -- ๐Ÿ”œ Signed commits configured on M4 -- ๐Ÿ”œ Required signed commits added to ruleset -- ๐Ÿ”œ `.env` + `python-dotenv` setup (Layer 2) -- ๐Ÿ”œ Pre-commit hooks (gitleaks + ruff) (Layer 3) + +- Hardware procured (M4, planned M5). +- Terminal stack: iTerm2, Powerlevel10k, uv, Python 3.12, Cursor, Ollama. +- GitHub repo with `.gitignore`. +- CI: lint, unit-test, secrets (PR #1 merged). +- Ruleset on `main`: PR required, status checks, conversation resolution, no force push, no deletion, no admin bypass. +- Signed commits configured on M4 (in progress). +- `Require signed commits` added to ruleset (pending key setup). +- `.env` and `python-dotenv` setup (Layer 2, pending). +- Pre-commit hooks with gitleaks and ruff (Layer 3, pending). ### Phase 1 โ€” Brain online -- Ollama serving LLaMA 3.1 8B locally -- Minimal Python wrapper (`marvin.brain`) that sends a prompt, returns a response -- First end-to-end conversation loop in a CLI -- Unit tests for the wrapper (mock the Ollama HTTP calls) -- Integration test against a real Ollama instance (CI skip; local only) + +- Ollama serving LLaMA 3.1 8B locally. +- Minimal Python wrapper (`marvin.brain`) that sends a prompt and returns a response. +- First end-to-end conversation loop in a CLI. +- Unit tests for the wrapper (mocked HTTP). +- Integration test against a real Ollama instance (local only). ### Phase 2 โ€” Memory layer -- PostgreSQL 16+ running locally (via Docker for portability) -- pgvector extension enabled -- Embedding model selection (candidate: BGE small/large via sentence-transformers, run locally) -- Schema: `conversations`, `messages`, `memories` (embedding column), `facts` -- Memory storage + retrieval (basic RAG) -- LangGraph introduced here: a retrieval node + a reasoning node + a response node + +- PostgreSQL 16+ running locally via Docker. +- pgvector extension enabled. +- Embedding model selection (candidate: BGE small/large via sentence-transformers). +- Schema: `conversations`, `messages`, `memories` (embedding column), `facts`. +- Memory storage and retrieval (basic RAG). +- LangGraph introduced: retrieval node, reasoning node, response node. ### Phase 3 โ€” Internet -- Tavily API integration with rate limiting and result caching -- Search results enter the memory layer with provenance -- Marvin decides *when* to search (LangGraph conditional node) -### Phase 4 โ€” Autonomy + scheduling -- APScheduler for periodic tasks (memory consolidation, knowledge refresh, journal writes) -- Redis introduced: cache for last-N turns + APScheduler job queue -- Background reflection: Marvin summarizes recent conversations into long-term memory +- Tavily API integration with rate limiting and result caching. +- Search results enter memory layer with provenance. +- Marvin decides when to search (LangGraph conditional node). + +### Phase 4 โ€” Autonomy and scheduling + +- APScheduler for periodic tasks (memory consolidation, knowledge refresh, journal writes). +- Redis introduced: last-N-turns cache and job queue. +- Background reflection: Marvin summarises recent conversations into long-term memory. ### Phase 5 โ€” Server layer -- FastAPI HTTP + WebSocket interface -- Frontend decision: web vs CLI vs Mac native app (parked for now) -- Auth: token-based; only Cris's devices can talk to Marvin -### Phase 6 โ€” Personality + continuous learning -- Personality state machine: a structured representation of Marvin's character -- Memory consolidation patterns inspired by Letta (formerly MemGPT) -- Reflection loops: Marvin reasons about his own past conversations and updates beliefs +- FastAPI HTTP and WebSocket interface. +- Frontend decision parked (web vs CLI vs Mac native app). +- Token-based auth: only Cris's devices can talk to Marvin. + +### Phase 6 โ€” Personality and continuous learning + +- Personality state machine: structured representation of Marvin's character. +- Memory consolidation patterns inspired by Letta (formerly MemGPT). +- Reflection loops: Marvin reasons about his own past conversations and updates beliefs. ### Phase 7 โ€” Embodiment (future) -- Physical robot body -- Sensors (camera, microphone) and actuators -- Local-network deployment with secure pairing to brain + +- Physical robot body. +- Sensors (camera, microphone) and actuators. +- Local-network deployment with secure pairing to brain. --- ## 8. Folder Structure -``` +```text marvin/ โ”œโ”€โ”€ .github/ โ”‚ โ””โ”€โ”€ workflows/ โ”‚ โ””โ”€โ”€ ci.yml โ”œโ”€โ”€ docs/ -โ”‚ โ”œโ”€โ”€ PROJECT_MARVIN_SPEC.md # this file -โ”‚ โ”œโ”€โ”€ ADR/ # one .md per ADR -โ”‚ โ””โ”€โ”€ runbooks/ # ops procedures +โ”‚ โ”œโ”€โ”€ PROJECT_MARVIN_SPEC.md +โ”‚ โ”œโ”€โ”€ ADR/ +โ”‚ โ””โ”€โ”€ runbooks/ โ”œโ”€โ”€ src/ โ”‚ โ””โ”€โ”€ marvin/ โ”‚ โ”œโ”€โ”€ __init__.py โ”‚ โ”œโ”€โ”€ version.py -โ”‚ โ”œโ”€โ”€ brain/ # Phase 1 โ€” Ollama wrapper -โ”‚ โ”œโ”€โ”€ memory/ # Phase 2 โ€” Postgres + pgvector -โ”‚ โ”œโ”€โ”€ internet/ # Phase 3 โ€” Tavily -โ”‚ โ”œโ”€โ”€ autonomy/ # Phase 4 โ€” APScheduler + Redis -โ”‚ โ”œโ”€โ”€ server/ # Phase 5 โ€” FastAPI -โ”‚ โ””โ”€โ”€ personality/ # Phase 6 โ€” personality + reflection +โ”‚ โ”œโ”€โ”€ brain/ +โ”‚ โ”œโ”€โ”€ memory/ +โ”‚ โ”œโ”€โ”€ internet/ +โ”‚ โ”œโ”€โ”€ autonomy/ +โ”‚ โ”œโ”€โ”€ server/ +โ”‚ โ””โ”€โ”€ personality/ โ”œโ”€โ”€ tests/ โ”‚ โ”œโ”€โ”€ unit/ โ”‚ โ”œโ”€โ”€ integration/ @@ -265,90 +253,90 @@ marvin/ ## 9. Engineering Practices -- **Style:** PEP 8 strict; type hints on every public function; docstrings (PEP 257) on every public function/class/module. -- **Commits:** [Conventional Commits](https://www.conventionalcommits.org/) โ€” `feat:`, `fix:`, `chore:`, `docs:`, `test:`, `refactor:`, etc. -- **Branches:** `type/short-description` โ€” e.g. `feat/ollama-wrapper`, `chore/scaffold-and-ci`. -- **PRs:** Every change. No exception. Includes Cris's own work on his own repo. -- **Merge strategy:** "Create a merge commit" or "Rebase and merge" โ€” both preserve commit history. **No squash-merge.** +- **Style:** PEP 8 strict; type hints on every public function; docstrings (PEP 257) on every public function, class, and module. +- **Commits:** [Conventional Commits](https://www.conventionalcommits.org/) โ€” `feat:`, `fix:`, `chore:`, `docs:`, `test:`, `refactor:`. +- **Branches:** `type/short-description`, e.g., `feat/ollama-wrapper`, `chore/scaffold-and-ci`. +- **Pull requests:** Every change goes through a PR. No exceptions, including Cris's own work on his own repo. +- **Merge strategy:** "Create a merge commit" or "Rebase and merge" โ€” both preserve history. No squash-merge. - **CI:** Three jobs minimum โ€” `lint`, `unit-test`, `secrets`. All required on `main`. -- **Testing pyramid:** Many unit tests, fewer integration, few E2E. Unit tests run on every push; integration runs locally and in CI; E2E runs locally pre-release. -- **Documentation:** This spec + ADRs in `docs/ADR/` + module READMEs + runbooks for ops procedures. +- **Testing pyramid:** Many unit tests, fewer integration, few E2E. Unit on every push; integration local and in CI; E2E local pre-release. +- **Documentation:** This spec plus ADRs in `docs/ADR/` plus module READMEs plus runbooks for ops procedures. --- -## 10. Security Posture (v1.1) +## 10. Security Posture -Marvin will eventually hold conversation history, learned preferences, and API tokens. Security is a first-class concern, woven through every phase โ€” not bolted on later. +Marvin will eventually hold conversation history, learned preferences, and API tokens. Security is a first-class concern, woven through every phase rather than bolted on later. -### Principles applied -- **Least privilege.** Every credential, service account, and database role gets the minimum access needed. No "godmode" tokens. -- **Defense in depth.** Multiple layers, each catching what others miss: `.gitignore` + gitleaks-in-CI + pre-commit gitleaks + signed commits + Postgres row-level security (Phase 2+) + encrypted columns for sensitive fields. -- **Key separation.** Auth keys โ‰  signing keys โ‰  deployment keys. Each compromise is contained. +**Principles applied:** + +- **Least privilege.** Every credential, service account, and database role gets the minimum access needed. No god-mode tokens. +- **Defence in depth.** Multiple layers, each catching what others miss: `.gitignore`, gitleaks in CI, pre-commit gitleaks, signed commits, Postgres RLS (Phase 2+), encrypted columns for sensitive fields. +- **Key separation.** Auth keys are not signing keys are not deployment keys. Each compromise is contained. - **Auditability.** Conversation logs are timestamped; signed commits prove authorship; rulesets are readable by anyone with repo access. -### Concrete controls (current + planned) +### Controls + | Layer | Control | Status | -|---|---|---| -| Repo | `.gitignore` covering `.env`, secrets, model files | โœ… | -| Repo | gitleaks scanning in CI | โœ… | -| Repo | Pre-commit gitleaks (local) | ๐Ÿ”œ Layer 3 | -| Branch | Ruleset on `main`: PR required, no admin bypass | โœ… | -| Branch | Required signed commits on `main` | ๐Ÿ”œ today | -| Code | Ruff + actionlint + yamllint + markdownlint in CI | โœ… | -| Code | Dependabot for dependency CVEs | ๐Ÿ“‹ backlog | -| Code | CodeQL static analysis | ๐Ÿ“‹ backlog | -| Runtime | `.env` + `python-dotenv` for secrets | ๐Ÿ”œ Layer 2 | -| Runtime | All API keys rotated periodically | ๐Ÿ“‹ ongoing | -| DB (Phase 2) | Postgres SSL-only connections | ๐Ÿ“‹ | -| DB (Phase 2) | Row-Level Security on multi-user tables | ๐Ÿ“‹ | -| DB (Phase 2) | Encrypted columns for sensitive fields | ๐Ÿ“‹ | -| Network (Phase 5) | FastAPI behind localhost-only by default | ๐Ÿ“‹ | -| Network (Phase 7) | Mutual-TLS pairing between robot body and brain | ๐Ÿ“‹ | +|-------|---------|--------| +| Repo | `.gitignore` covering `.env`, secrets, model files | done | +| Repo | gitleaks scanning in CI | done | +| Repo | Pre-commit gitleaks (local) | pending (Layer 3) | +| Branch | Ruleset on `main`: PR required, no admin bypass | done | +| Branch | Required signed commits on `main` | in progress | +| Code | ruff, actionlint, yamllint, markdownlint in CI | done | +| Code | Dependabot for dependency CVEs | backlog | +| Code | CodeQL static analysis | backlog | +| Runtime | `.env` and `python-dotenv` for secrets | pending (Layer 2) | +| Runtime | All API keys rotated periodically | ongoing | +| DB (Phase 2) | Postgres SSL-only connections | planned | +| DB (Phase 2) | Row-Level Security on multi-user tables | planned | +| DB (Phase 2) | Encrypted columns for sensitive fields | planned | +| Network (Phase 5) | FastAPI bound to localhost by default | planned | +| Network (Phase 7) | Mutual TLS between robot body and brain | planned | ### Threat model โ€” running list -*Reviewed periodically. Each entry: what could go wrong, what stops it.* -- **GitHub PAT leaks.** โ†’ gitleaks (CI + pre-commit), `.gitignore`, signed commits prevent forged commits, branch protection prevents direct push to `main`. -- **M4 stolen.** โ†’ FileVault on the dedicated account; sensitive Postgres fields encrypted; remote SSH disabled by default. -- **Ollama model compromised.** โ†’ models pulled from official Ollama registry only; SHA verification by Ollama; isolated user account. -- **Tavily key leaks.** โ†’ `.env` not in git; key rotatable; rate-limited usage caps blast radius. -- **Supply-chain attack via Python deps.** โ†’ uv lockfile pinning; Dependabot alerts (planned). +- **GitHub PAT leaks.** Mitigated by gitleaks (CI plus pre-commit), `.gitignore`, signed commits preventing forged commits, branch protection preventing direct push to `main`. +- **M4 stolen.** Mitigated by FileVault on the dedicated account; sensitive Postgres fields encrypted; remote SSH disabled by default. +- **Ollama model compromised.** Mitigated: models pulled from official Ollama registry only; SHA verification by Ollama; isolated user account. +- **Tavily key leaks.** Mitigated: `.env` not in git; key rotatable; rate-limited usage caps blast radius. +- **Supply-chain attack via Python deps.** Mitigated: uv lockfile pinning; Dependabot alerts (planned). --- ## 11. Success Criteria -- โœ… Marvin runs 24/7 on M4 without intervention -- โœ… Persistent memory survives reboots and long conversation gaps -- โœ… Internet-search capability for facts beyond training data -- โœ… Conversational coherence โ€” feels like a continuous person, not stateless turns -- โœ… Recognizable, consistent personality across sessions -- โœ… Cris can explain every architectural decision in an interview -- ๐ŸŽฏ Eventual embodiment in a physical robot body +- Marvin runs 24/7 on M4 without intervention. +- Persistent memory survives reboots and long conversation gaps. +- Internet-search capability for facts beyond training data. +- Conversational coherence: feels like a continuous person, not stateless turns. +- Recognisable, consistent personality across sessions. +- Cris can explain every architectural decision in an interview. +- Eventual embodiment in a physical robot body. --- ## 12. Glossary | Term | Definition | -|---|---| -| **ADR** | Architectural Decision Record. A short doc explaining *why* a design choice was made, including alternatives considered. | -| **Agent** | A program that wraps an LLM with tools and a reasoning loop, enabling autonomous decision-making. | -| **Embedding** | A vector representation of text used for semantic search. | -| **HNSW** | Hierarchical Navigable Small World โ€” a graph-based index for fast nearest-neighbor search over vectors. | -| **LLM** | Large Language Model. Marvin's reasoning brain. | -| **Local-first** | Marvin runs on Cris's hardware, not in the cloud. Privacy, control, no ongoing cost. | -| **pgvector** | PostgreSQL extension adding a `vector` data type and nearest-neighbor search. | -| **RAG** | Retrieval-Augmented Generation. Pattern: embed โ†’ store โ†’ retrieve โ†’ inject into prompt โ†’ LLM answers. | -| **Ruleset** | GitHub's modern branch governance mechanism (the successor to classic branch protection). | -| **Signed commit** | A commit cryptographically proven to come from a specific key holder. | -| **Vector DB** | Database optimized for nearest-neighbor search over high-dimensional vectors. | +|------|------------| +| ADR | Architectural Decision Record. A short doc explaining *why* a design choice was made, including alternatives considered. | +| Agent | A program that wraps an LLM with tools and a reasoning loop, enabling autonomous decision-making. | +| Embedding | A vector representation of text used for semantic search. | +| HNSW | Hierarchical Navigable Small World. Graph-based index for fast nearest-neighbour search over vectors. | +| LLM | Large Language Model. Marvin's reasoning brain. | +| Local-first | Marvin runs on Cris's hardware, not in the cloud. Privacy, control, no ongoing cost. | +| pgvector | PostgreSQL extension adding a `vector` data type and nearest-neighbour search. | +| RAG | Retrieval-Augmented Generation. Pattern: embed, store, retrieve, inject into prompt, LLM answers. | +| Ruleset | GitHub's modern branch governance mechanism (successor to classic branch protection). | +| Signed commit | A commit cryptographically proven to come from a specific key holder. | +| Vector DB | Database optimised for nearest-neighbour search over high-dimensional vectors. | --- ## 13. Changelog -| Version | Date | Changes | -|---|---|---| -| 1.1 | 2026-05-15 | Locked **pgvector** as persistent memory (replaces ChromaDB). Locked **SSH-signed commits required on `main`**. Added **Security Posture** section. Added **ADR** section. Documented **Redis** as future cache/queue role. Switched branch governance language from "classic protection" to "Ruleset". | -| 1.0 | (lost) | Initial spec โ€” vision, personality, architecture, tech stack, hardware, seven phases, folder structure, engineering practices, success criteria. | +- **2026-05-18** โ€” Cleaned markdown formatting for lint compliance. Same content as the 05-15 revision. +- **2026-05-15** โ€” Locked pgvector as persistent memory (replaces ChromaDB). Locked SSH-signed commits required on `main`. Added Security Posture section. Added ADR section. Documented Redis future role. Switched branch governance language from "classic protection" to "Ruleset". +- **(prior)** โ€” Initial spec (lost to M4 reformat): vision, personality, architecture, tech stack, hardware, seven phases, folder structure, engineering practices, success criteria. From 1a600e078d824a13793f268189d37505c5fb54c8 Mon Sep 17 00:00:00 2001 From: Cristina Borges Date: Mon, 18 May 2026 14:43:27 +0100 Subject: [PATCH 3/5] fix(ci): grant gitleaks pull-requests:read permission --- .github/workflows/ci.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 66e7416..a7743a2 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -47,6 +47,9 @@ jobs: secrets: runs-on: ubuntu-latest + permissions: + contents: read + pull-requests: read steps: - name: Checkout code uses: actions/checkout@v4 From cd17a0486d411424e3e330f66e9e563912c7815e Mon Sep 17 00:00:00 2001 From: Cristina Borges Date: Mon, 18 May 2026 14:48:48 +0100 Subject: [PATCH 4/5] fix(docs): align table pipes to satisfy MD060 --- docs/PROJECT_MARVIN_SPEC.md | 110 ++++++++++++++++++------------------ 1 file changed, 55 insertions(+), 55 deletions(-) diff --git a/docs/PROJECT_MARVIN_SPEC.md b/docs/PROJECT_MARVIN_SPEC.md index c73a989..a908f71 100644 --- a/docs/PROJECT_MARVIN_SPEC.md +++ b/docs/PROJECT_MARVIN_SPEC.md @@ -31,36 +31,36 @@ Personality is engineered, not just prompted: a long-running system prompt plus ## 3. Hardware -| Machine | Spec | Role | -|---------|------|------| -| M4 Max | 16", 48 GB, 1 TB | Dedicated Marvin brain. Freshly formatted, separate macOS account. Runs 24/7. | -| M5 Max | 64 GB | Cris's personal/dev machine. Marvin work happens here, deploys to M4. | +| Machine | Spec | Role | +| ------- | ---------------- | ----------------------------------------------------------------------------- | +| M4 Max | 16", 48 GB, 1 TB | Dedicated Marvin brain. Freshly formatted, separate macOS account. Runs 24/7. | +| M5 Max | 64 GB | Cris's personal/dev machine. Marvin work happens here, deploys to M4. | --- ## 4. Tech Stack -| Layer | Tool | Status | Phase | -|-------|------|--------|-------| -| Language | Python 3.12 via `uv` | locked | All | -| LLM serving | Ollama | locked | 1 | -| Starter model | LLaMA 3.1 8B | locked | 1 | -| Target model | Qwen 3 32B | target | 1+ | -| Agent reasoning | LangGraph | locked | 2+ | -| Persistent memory (relational) | PostgreSQL 16+ | locked | 2 | -| Persistent memory (vectors) | pgvector | locked | 2 | -| Cache and task queue | Redis | deferred | 4+ | -| Internet search | Tavily API | locked | 3 | -| Scheduling | APScheduler | locked | 4 | -| HTTP and WebSocket server | FastAPI | locked | 5 | -| Future memory framework | Letta (formerly MemGPT) | bookmark | 6+ | -| IDE | Cursor Pro | locked | All | -| Deep-work AI assistant | Claude Code | locked | All | -| Terminal | iTerm2 + Oh My Zsh + Powerlevel10k | locked | All | -| Version control | Git + GitHub | locked | All | -| CI/CD | GitHub Actions | locked | All | -| Lint and format | ruff, yamllint, markdownlint, actionlint | locked | All | -| Secret scanning | gitleaks (CI plus pre-commit) | locked | All | +| Layer | Tool | Status | Phase | +| ------------------------------ | ---------------------------------------- | -------- | ----- | +| Language | Python 3.12 via `uv` | locked | All | +| LLM serving | Ollama | locked | 1 | +| Starter model | LLaMA 3.1 8B | locked | 1 | +| Target model | Qwen 3 32B | target | 1+ | +| Agent reasoning | LangGraph | locked | 2+ | +| Persistent memory (relational) | PostgreSQL 16+ | locked | 2 | +| Persistent memory (vectors) | pgvector | locked | 2 | +| Cache and task queue | Redis | deferred | 4+ | +| Internet search | Tavily API | locked | 3 | +| Scheduling | APScheduler | locked | 4 | +| HTTP and WebSocket server | FastAPI | locked | 5 | +| Future memory framework | Letta (formerly MemGPT) | bookmark | 6+ | +| IDE | Cursor Pro | locked | All | +| Deep-work AI assistant | Claude Code | locked | All | +| Terminal | iTerm2 + Oh My Zsh + Powerlevel10k | locked | All | +| Version control | Git + GitHub | locked | All | +| CI/CD | GitHub Actions | locked | All | +| Lint and format | ruff, yamllint, markdownlint, actionlint | locked | All | +| Secret scanning | gitleaks (CI plus pre-commit) | locked | All | --- @@ -277,23 +277,23 @@ Marvin will eventually hold conversation history, learned preferences, and API t ### Controls -| Layer | Control | Status | -|-------|---------|--------| -| Repo | `.gitignore` covering `.env`, secrets, model files | done | -| Repo | gitleaks scanning in CI | done | -| Repo | Pre-commit gitleaks (local) | pending (Layer 3) | -| Branch | Ruleset on `main`: PR required, no admin bypass | done | -| Branch | Required signed commits on `main` | in progress | -| Code | ruff, actionlint, yamllint, markdownlint in CI | done | -| Code | Dependabot for dependency CVEs | backlog | -| Code | CodeQL static analysis | backlog | -| Runtime | `.env` and `python-dotenv` for secrets | pending (Layer 2) | -| Runtime | All API keys rotated periodically | ongoing | -| DB (Phase 2) | Postgres SSL-only connections | planned | -| DB (Phase 2) | Row-Level Security on multi-user tables | planned | -| DB (Phase 2) | Encrypted columns for sensitive fields | planned | -| Network (Phase 5) | FastAPI bound to localhost by default | planned | -| Network (Phase 7) | Mutual TLS between robot body and brain | planned | +| Layer | Control | Status | +| ----------------- | -------------------------------------------------- | ----------------- | +| Repo | `.gitignore` covering `.env`, secrets, model files | done | +| Repo | gitleaks scanning in CI | done | +| Repo | Pre-commit gitleaks (local) | pending (Layer 3) | +| Branch | Ruleset on `main`: PR required, no admin bypass | done | +| Branch | Required signed commits on `main` | in progress | +| Code | ruff, actionlint, yamllint, markdownlint in CI | done | +| Code | Dependabot for dependency CVEs | backlog | +| Code | CodeQL static analysis | backlog | +| Runtime | `.env` and `python-dotenv` for secrets | pending (Layer 2) | +| Runtime | All API keys rotated periodically | ongoing | +| DB (Phase 2) | Postgres SSL-only connections | planned | +| DB (Phase 2) | Row-Level Security on multi-user tables | planned | +| DB (Phase 2) | Encrypted columns for sensitive fields | planned | +| Network (Phase 5) | FastAPI bound to localhost by default | planned | +| Network (Phase 7) | Mutual TLS between robot body and brain | planned | ### Threat model โ€” running list @@ -319,19 +319,19 @@ Marvin will eventually hold conversation history, learned preferences, and API t ## 12. Glossary -| Term | Definition | -|------|------------| -| ADR | Architectural Decision Record. A short doc explaining *why* a design choice was made, including alternatives considered. | -| Agent | A program that wraps an LLM with tools and a reasoning loop, enabling autonomous decision-making. | -| Embedding | A vector representation of text used for semantic search. | -| HNSW | Hierarchical Navigable Small World. Graph-based index for fast nearest-neighbour search over vectors. | -| LLM | Large Language Model. Marvin's reasoning brain. | -| Local-first | Marvin runs on Cris's hardware, not in the cloud. Privacy, control, no ongoing cost. | -| pgvector | PostgreSQL extension adding a `vector` data type and nearest-neighbour search. | -| RAG | Retrieval-Augmented Generation. Pattern: embed, store, retrieve, inject into prompt, LLM answers. | -| Ruleset | GitHub's modern branch governance mechanism (successor to classic branch protection). | -| Signed commit | A commit cryptographically proven to come from a specific key holder. | -| Vector DB | Database optimised for nearest-neighbour search over high-dimensional vectors. | +| Term | Definition | +| ------------- | ------------------------------------------------------------------------------------------------------------------------ | +| ADR | Architectural Decision Record. A short doc explaining *why* a design choice was made, including alternatives considered. | +| Agent | A program that wraps an LLM with tools and a reasoning loop, enabling autonomous decision-making. | +| Embedding | A vector representation of text used for semantic search. | +| HNSW | Hierarchical Navigable Small World. Graph-based index for fast nearest-neighbour search over vectors. | +| LLM | Large Language Model. Marvin's reasoning brain. | +| Local-first | Marvin runs on Cris's hardware, not in the cloud. Privacy, control, no ongoing cost. | +| pgvector | PostgreSQL extension adding a `vector` data type and nearest-neighbour search. | +| RAG | Retrieval-Augmented Generation. Pattern: embed, store, retrieve, inject into prompt, LLM answers. | +| Ruleset | GitHub's modern branch governance mechanism (successor to classic branch protection). | +| Signed commit | A commit cryptographically proven to come from a specific key holder. | +| Vector DB | Database optimised for nearest-neighbour search over high-dimensional vectors. | --- From 5161250363418bb5c69de1e24efeba1f3f84b136 Mon Sep 17 00:00:00 2001 From: Cristina Borges Date: Mon, 18 May 2026 14:48:48 +0100 Subject: [PATCH 5/5] fix(docs): align table pipes to satisfy MD060