From 4f97b4b11776955b9a857f5aa3a36bc93537e391 Mon Sep 17 00:00:00 2001 From: devlux76 Date: Sat, 14 Mar 2026 01:15:58 -0600 Subject: [PATCH 1/5] docs: migrate DESIGN.md into wiki landing + add wiki draft pages --- .github/ISSUE_TEMPLATE/config.yml | 4 +- .github/copilot-instructions.md | 5 +- DESIGN.md | 920 +----------------------------- README.md | 3 +- docs/development.md | 9 +- docs/wiki-draft/Architecture.md | 30 + docs/wiki-draft/Consolidation.md | 14 + docs/wiki-draft/Home.md | 38 ++ docs/wiki-draft/Ingestion.md | 16 + docs/wiki-draft/Math-Appendix.md | 37 ++ docs/wiki-draft/Performance.md | 28 + docs/wiki-draft/Retrieval.md | 29 + docs/wiki-draft/Security.md | 16 + docs/wiki-draft/Storage.md | 17 + docs/wiki-draft/Terminology.md | 23 + 15 files changed, 283 insertions(+), 906 deletions(-) create mode 100644 docs/wiki-draft/Architecture.md create mode 100644 docs/wiki-draft/Consolidation.md create mode 100644 docs/wiki-draft/Home.md create mode 100644 docs/wiki-draft/Ingestion.md create mode 100644 docs/wiki-draft/Math-Appendix.md create mode 100644 docs/wiki-draft/Performance.md create mode 100644 docs/wiki-draft/Retrieval.md create mode 100644 docs/wiki-draft/Security.md create mode 100644 docs/wiki-draft/Storage.md create mode 100644 docs/wiki-draft/Terminology.md diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml index ab8c2fc..8b22eda 100644 --- a/.github/ISSUE_TEMPLATE/config.yml +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -1,5 +1,5 @@ blank_issues_enabled: true contact_links: - name: "📖 Architecture Reference" - url: https://github.com/devlux76/cortex/blob/main/DESIGN.md - about: Review DESIGN.md before proposing architectural changes. + url: https://github.com/devlux76/cortex/wiki + about: Review the architecture wiki before proposing architectural changes. diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 9f08291..f6c56be 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -14,11 +14,12 @@ The engine models three biological brain regions: | File | Purpose | |---|---| | `README.md` | Product vision and quick start | -| `DESIGN.md` | Complete architecture specification and design principles | +| **CORTEX Wiki** | Canonical architecture specification and design principles | +| `DESIGN.md` | Repo landing page / TOC into the wiki | | `PLAN.md` | Module-by-module implementation status and development phases | | `TODO.md` | Prioritized actionable tasks to ship v1.0 | -Keep `DESIGN.md` synchronized with the real code state after every implementation pass. +Keep the wiki and `DESIGN.md` synchronized with the real code state after every implementation pass. ## Project Management diff --git a/DESIGN.md b/DESIGN.md index 7e49365..94da581 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -1,913 +1,39 @@ -# CORTEX Design Specification +# CORTEX Design (Landing Page) -## Executive Summary +This file is intentionally short. The **full architecture specification and design rationale** lives in the **CORTEX GitHub Wiki**, which is the canonical source for detailed design content. -CORTEX (**C**lustered **O**ntic **R**outing **T**hrough **E**ntangled e**X**changes) is a neurobiologically inspired, fully on-device episodic memory engine for autonomous agents. It runs 100% in the browser with no servers, no cloud, and no telemetry. All memory stays local and private. - -## Product Surface Reminder (App vs Library) - -The same core engine serves two product surfaces with different user expectations: - -### 1) Standalone App (Browser Extension) - -The standalone product should feel like a clean, fast, personalized search engine over the internet the user has actually seen. - -UX intent: -- Passive capture of visited pages to build a private recall index -- Search-first interface with fast response and minimal visual noise -- Recovery from vague recollection ("rabbit-hole memory"), not only exact keyword matching -- Lightweight metrics display (informative but secondary to search) - -Model-mode requirement for app UX: -- **Nomic mode**: multimodal retrieval (text + images in shared latent space) -- **Gemma mode**: high-precision text retrieval (no image embedding) -- UI must make this capability boundary explicit so users understand when image recall is available - -### 2) Library Surface (Embeddable) - -The library remains headless and integration-first: ingest, retrieve, consolidate, and route memory for external tools without prescribing browser-extension UX patterns. - -Design implication: -- Keep engine interfaces surface-agnostic -- Keep app-shell concerns (extension permissions, search controls, metrics presentation) outside core memory contracts - -## Core Architecture - -### Three Living Regions - -CORTEX models three biological brain regions working in concert: - -#### 1. Hippocampus — Fast Associative Encoding -The rapid-write system that turns raw experience into structured memory scaffolding. - -**Responsibilities:** -- Embed new observations with Matryoshka-capable models -- Perform lightning-fast WebGPU multi-prototype lookups -- Build hierarchical prototypes (Pages → Books → Volumes → Shelves) -- Create probabilistic Hebbian edges -- Store raw vectors in append-only OPFS file - -**Performance Target:** Single-page persist + fast neighbor update under 50ms on WebGPU hardware - -#### 2. Cortex — Intelligent Routing & Coherence -Returns self-consistent, coherent context chains rather than bag-of-vectors. Critically, Cortex **constructs Metroids** — structured dialectical search probes — to explore knowledge epistemically rather than merely confirming existing beliefs. - -**Responsibilities:** -- Construct Metroids (dialectical probes `{ m1, m2, c }`) for each query topic -- Perform Matryoshka dimensional unwinding to discover antithesis medoids -- Perform parallel WebGPU "scoops" across the active universe (sub-millisecond) -- Pull relevant sub-graphs from IndexedDB -- Trace closed-loop paths through Hebbian connections -- Return only coherent context chains -- Detect knowledge gaps when antithesis discovery fails within dimensional constraints -- Broadcast P2P curiosity requests when a knowledge gap is detected - -**Performance Target:** Shelf→page seed ranking under 20ms; coherence path solve under 10ms for <30 node subgraphs - -#### 3. Daydreamer — The Default Mode Network -Idle background consolidation that prevents catastrophic forgetting. - -**Responsibilities:** -- Strengthen important connections (Long-Term Potentiation) -- Gently decay and prune weak edges (Long-Term Depression) -- Recompute medoids and centroids -- Replay recent experiences -- Keep memory universe coherent and alive - -**Performance Target:** Opportunistic, interruptible, no foreground blocking - -## Conceptual Constructs: Medoid, Centroid, and Metroid - -Three separate mathematical constructs are central to CORTEX. They must never be conflated. - -| Concept | Meaning | -|---------|---------| -| **Medoid** | An actual memory node selected as the statistical representative of a cluster. A medoid is always an existing page in the graph. | -| **Centroid** | A mathematical average of vectors — a computed geometric point, never a stored memory node. | -| **Metroid** | A structured dialectical search probe: `{ m1, m2, c }`. Constructed at query time. Never stored as a graph edge or persistent entity. | - -> **Critical invariant:** These three constructs are entirely distinct. The sparse semantic neighbor graph that connects pages for subgraph expansion is **not** a Metroid. A Metroid is built from medoids, but a medoid is not a Metroid. +✅ **Wiki (primary design doc):** https://github.com/devlux76/cortex/wiki --- -### The Metroid - -A Metroid is a structured search primitive for epistemically balanced exploration of a topic. - -The name captures a key architectural insight: what looks like an obstacle to progress — a medoid representing conceptual opposition — is not an enemy. The centroid computed from that opposition can be **held as a stable, frozen platform**, turning semantic divergence into a navigable step toward a goal. Every Metroid construction converts the antithesis (m2) into the anchor for the frozen centroid (c), which then provides structural support for deeper exploration. - -``` -Metroid = { m1, m2, c } -``` +## How to use this page -Where: -- **m1** — thesis medoid: found via medoid search from the query vector. A medoid (not a centroid) is always an existing memory node — it keeps the search on the correct conceptual road. -- **m2** — antithesis medoid: the medoid of the cosine-opposite set — not merely the nearest semantically-opposing node, but the **most coherent existing memory node in the direction of maximal divergence** from m1. Like m1, m2 is always an actual memory node, never a computed phantom position. -- **c** — centroid: the synthetic center of mass between m1 and m2, computed **once** and **frozen** as a stable platform. - `c` is a "Kansas space" position — typically empty; no real node lives at the centroid. - Its value is as a neutral vantage point: from `c`, distances to both poles and all - candidates can be measured without anchoring bias toward either m1 or m2. - -**Philosophical foundation:** Centroids (means) provide gravitational pull toward the midpoint. Medoids (medians) keep the search on the right road by anchoring to actual existing nodes. Neither alone guarantees epistemic honesty. The Metroid loop combines them: the medoid ensures the search never drifts to a phantom position; the frozen centroid ensures all subsequent evaluation is unbiased between the poles. - -The Metroid is constructed at query time by the `MetroidBuilder`. It is **not** a persistent graph structure. It is a transient epistemological instrument. +- **If you’re writing an issue or PR:** start with the wiki page that best matches your change. +- **If you’re reviewing a design change:** use this page as a table of contents to find the right wiki section quickly. --- -### MetroidBuilder Algorithm - -One full Metroid step is a **thesis → freeze → antithesis → synthesis** cycle: - -1. **Thesis — Select m1** — From the query vector `q`, perform a medoid search to find `m1`: the - median representative of the most relevant cluster. A medoid is always an existing memory node, - ensuring the search stays on the correct conceptual road. Centroids (means) provide - gravitational pull; medoids (medians) provide the road. - -2. **Freeze** — Lock the first `n` protected Matryoshka dimensions in place. These dimensions - encode invariant semantic context (domain, language register, topic class). Locking them - preserves early decisions as fixed structure — preventing the search from drifting into - vocabulary that shares surface-level patterns but belongs to a different conceptual domain. - -3. **Antithesis — Find m2** — On the remaining free (unfrozen) dimensions: - - Compute the **cosine-opposite score** for every candidate medoid: score each candidate as - `-cosine_similarity(candidate_free_dims, m1_free_dims)`. The highest-scoring candidates are - farthest from m1 in the free dimensions — representing maximal conceptual divergence. - - Find the **medoid of that cosine-opposite set** (the top-scoring candidates). This is `m2`. - - `m2` is the medoid of the top-scoring candidates — not the result of a direct vector - negation. The medoid operation selects the most coherent existing memory node in the - direction of maximal divergence. The medoid operation ensures `m2` is always - an actual memory node. - -4. **Synthesis — Freeze the centroid** — Compute `c` as the center of mass between m1 and m2 - and immediately **freeze it**. `c` is computed once per Metroid construction and never - recalculated: - - Protected dimensions (index < `matryoshkaProtectedDim`): copy directly from m1. These - dimensions are invariant; averaging them would dilute the domain anchor. - - Free dimensions (index >= `matryoshkaProtectedDim`): element-wise average of m1 and m2 — - `c[i] = (m1[i] + m2[i]) / 2`. - - `c` is a "Kansas space" position — typically empty; no real node lives at the centroid. - Its value is as a neutral vantage point: from `c`, distances to both poles and all - candidates can be measured without anchoring bias toward either m1 or m2. - -5. **Evaluate subsequent candidates against the frozen centroid** — All further medoids - (`m3`, `m4`, ...) found during Matryoshka unwinding are evaluated relative to this frozen `c`: - - Near `c`: synthesis territory — balanced between both poles. - - Much closer to m1 than to `c`: thesis-supporting. - - Much closer to m2 than to `c`: antithesis-supporting. - - Far from `c`, m1, and m2 simultaneously: third conceptual region — signal for further - unwinding or a knowledge gap. - The centroid is a platform. Opposition has been frozen into a stepping stone. - -6. **Unwind Matryoshka layers** — Progressively free deeper embedding dimensions and repeat from - step 3. Each unwinding broadens the antithesis search space. Subsequent antithesis candidates - are still evaluated relative to the original frozen `c` — it is never recomputed. - -7. **Stop at the protected dimension** — The protected lower dimensions are never unwound. Once - the Matryoshka unwind has reached the protected floor, no further antithesis search is possible. - If no satisfactory `m2` was found at any layer, set `knowledge_gap = true` and broadcast a - curiosity query (see Knowledge Gap Detection). - -**Why protect dimensions?** - -Without dimensional protection, high-dimensional similarity in unrelated vocabulary can dominate -the search. Upper Matryoshka dimensions encode fine-grained distinctions that may closely match -surface-level word patterns regardless of topic. Protected lower dimensions encode domain context -(e.g., "food/cooking") that anchors the search. Without this anchor, a query about pizza toppings -could accumulate similarity mass toward adhesive-related terms — because words describing how -things stick together are statistically present in both culinary and industrial glue contexts. -The protected dimensions ensure the culinary domain context is never overridden by this incidental -high-dimensional similarity. - ---- - -### Matryoshka Dimensional Unwinding - -CORTEX uses Matryoshka Representation Learning (MRL) models that pack semantic information into nested dimensional layers: - -- **Protected layer** (lower dimensions): invariant context — domain, topic class, language. Never searched for antithesis. -- **Exploration layers** (upper dimensions): fine-grained semantic distinctions. Progressively unwound during antithesis search. - -At each unwinding step: -1. The protected dimension boundary shifts one layer outward. -2. The antithesis search space expands into the newly freed dimensions. -3. A new `m2` candidate is found via cosine-opposite medoid search in the expanded space. -4. The new candidate is evaluated relative to the **frozen** `c` (computed in the first synthesis - step and never recalculated). If it is close enough to `c`, the step is accepted; otherwise - the search continues unwinding or declares a knowledge gap. - -This produces progressively wider dialectical exploration while maintaining semantic coherence. -The frozen centroid ensures that each expansion step is measured against a stable platform rather -than a shifting target. The search terminates either when the protected dimension floor is reached -or when a satisfactory `m2` is found. - ---- - -### Dialectical Search - -Every Metroid-driven query explores three zones, with all scoring anchored at the centroid `c`: - -| Zone | Pole | Meaning | -|------|------|---------| -| Thesis zone | closer to m1 than to c | Supporting ideas, corroborating evidence | -| Antithesis zone | closer to m2 than to c | Opposing ideas, counterevidence, alternative perspectives | -| Synthesis zone | near c, equidistant from m1 and m2 | Conceptually balanced territory between both poles | - -**Scoring from the centroid vantage point:** candidates are ranked by their distance to `c`. A candidate significantly closer to m1 than to `c` is thesis-supporting; significantly closer to m2 is antithesis-supporting; near `c` is synthesis-zone content. Candidates far from all three (`c`, m1, m2) indicate a third conceptual region — either an undiscovered knowledge area or a signal to unwind another Matryoshka layer. Scoring from m1 or m2 instead of `c` would anchor all results toward one pole, introducing confirmation bias. - -This three-zone exploration prevents **confirmation bias**: a system that only retrieves nearest neighbors to m1 returns documents that confirm the query's premise. By also exploring m2 and c, CORTEX surfaces contradictions, alternatives, and knowledge gaps. - -The dialectical structure is the core reason CORTEX is described as an _epistemic_ memory system, not a vector retrieval engine. - ---- - -### Knowledge Gap Detection - -If at any stage of MetroidBuilder execution no suitable antithesis medoid `m2` can be found within the constrained search space: - -``` -knowledge_gap = true -``` - -This means CORTEX does not possess sufficient knowledge to provide an epistemically balanced answer. The correct response is to acknowledge the gap rather than fill it with ungrounded content. - -**Response to a knowledge gap:** - -1. Return a `KnowledgeGap` result indicating the topic, the deepest dimensional layer reached, and the search constraints that failed. -2. Emit a P2P curiosity request containing the incomplete Metroid. - ---- - -### P2P Curiosity Requests - -When a knowledge gap is detected, CORTEX broadcasts the incomplete Metroid as a curiosity probe to connected peers: - -``` -CuriosityProbe = { - m1, - partialMetroid, - queryContext, - knowledgeBoundary, - mimeType, - modelUrn -} -``` - -Where: -- **m1** — the thesis medoid (the topic for which antithesis was not found) -- **partialMetroid** — the incomplete Metroid at the boundary of local knowledge -- **queryContext** — the original query embedding, used for scoring by the responding peer -- **knowledgeBoundary** — the Matryoshka dimensional layer at which antithesis search failed -- **mimeType** — the MIME type of the embedded content (e.g. `text/plain`, `image/jpeg`). Required so receiving peers can validate commensurability of their graph sections. -- **modelUrn** — a URN identifying the specific embedding model and version used to produce the vectors (e.g. `urn:model:onnx-community/embeddinggemma-300m-ONNX:v1`). Peers **must** reject probes whose `modelUrn` does not match a model they can compare against. Accepting graph fragments embedded by a different model would produce incommensurable similarity scores at the dimensional boundaries where the models' Matryoshka layers overlap. - -> **Why `mimeType` and `modelUrn` are required:** -> Embedding models project content into incompatible latent spaces. A fragment embedded with `nomic-embed-text-v1.5` (matryoshkaProtectedDim=64) cannot be meaningfully compared against a fragment embedded with `embeddinggemma-300m` (matryoshkaProtectedDim=128). Without explicit model and content-type identity on the probe, a peer could return graph sections that appear similar by cosine score but are semantically incommensurable — introducing hallucination-equivalent errors at the knowledge boundary. - -Peers receiving this probe: - -1. Verify `mimeType` and `modelUrn` match a supported local model. -2. Search their own memory graphs for medoids that could serve as `m2` using the same embedding space. -3. If found, respond with the relevant graph fragment (subject to eligibility filtering; see Smart Sharing Guardrails). -4. The originating node integrates the received fragment and may retry MetroidBuilder. - -This mechanism enables **distributed learning without hallucination**: the system discovers knowledge through structured peer exchange rather than generating plausible-sounding but ungrounded content. - ---- - - - -### Motivation - -#### The Hollow Sphere: Why High Dimensions Break Everything - -To perform a vector search, you are essentially defining a boundary in a high-dimensional space and creating a hypersphere. Intuitively that should just be a big bag of vectors. - -The mathematical formula for the volume of this hypersphere (assuming an even number of dimensions `n` and a radius of 1) is: - -``` - π^(n/2) -Vₙ = ──────── - (n/2)! -``` - -This equation reveals a mind-bending geometric quirk: as dimensionality increases, the interior volume of the hypersphere decreases catastrophically. The inside of the sphere hollows out, and almost all of its volume gets pushed to an infinitesimally thin shell right at the boundary. - -By the time we hit 64 dimensions, the total volume has collapsed to approximately `3.08 × 10⁻²⁰`. To give you an idea of how incomprehensibly small that number is, if that were a physical measurement in metres, it would be roughly 50,000 times smaller than a single proton. If it were Joules, it would be a fraction of the energy of a single photon of infrared light. - -What this means for vector search is that we are searching inside a shape that effectively has no inside. Just like the physics of a black hole governed by the holographic principle, all the meaningful information is encoded exclusively on or near the surface. - -This geometric reality is what makes the CORTEX architecture so memory-efficient. Because the interior is a vast, empty void, we do not need to load the entire vector space into WebGPU memory. By using a hierarchical prototype structure to skip the void and navigate directly to the relevant "surface shell," our algorithm mirrors the Williams 2025 spacetime tradeoff: - -``` -S = O(√(t · log t)) -``` - -This theorem proves that the computational memory (`S`) required to evaluate a search tree can be tightly constrained relative to the search time (`t`). More importantly, `t` is treated as just another orthogonal dimension, like frames in a filmstrip or slices in a block. - -By leveraging the hollowing-out of high-dimensional hyperspheres, CORTEX aggressively discards irrelevant vectors, keeping its active memory footprint strictly bounded, echoing this theoretical computational limit — which, interestingly, was itself inspired by the holographic principle in physics. - -#### The Geometric Root: Curse of Dimensionality (Formal Treatment) - -CORTEX operates on high-dimensional Matryoshka embeddings. In `n`-dimensional Euclidean space the volume of the unit ball is: - -``` -Vₙ = π^(n/2) / Γ(n/2 + 1) -``` - -where Γ is the Gamma function. For even `n = 2m` this reduces to `πᵐ / m!`; for odd `n` the half-integer Gamma applies. In either case, as `n` grows, `Vₙ` collapses toward zero exponentially fast — by Stirling's approximation, `Vₙ ≈ (2πe/n)^(n/2) · (πn)^(−1/2)`. At `n = 768` this gives `log₁₀(V₇₆₈) ≈ −636`: the unit ball is effectively empty. This is the geometric driver of the **curse of dimensionality**: pairwise distances concentrate (everything looks equally far away), interiors vanish (rejection sampling and kernel methods fail), and any linear or polynomial scaling law blows up. Naïve nearest-neighbor search, flat clustering, fixed-K neighbor graphs, and uniform fan-out become either useless or unboundedly expensive as the corpus scales. - -Every structural decision in CORTEX — protected Matryoshka layers, hierarchical medoids, the Metroid antithesis hunt, dimensional unwinding, Williams-derived index sizes — is a direct geometric counter-measure to this collapse. - -#### The Fix: Williams 2025 Sublinear Bound - -CORTEX applies the Williams 2025 result — S = O(√(t log t)) — as a universal sublinear growth law everywhere the system trades space against time: the resident hotpath index, per-tier hierarchy quotas, per-community graph budgets, semantic neighbor degree limits, and Daydreamer maintenance batch sizing. This single principle ensures the system stays efficient as the memory graph scales from hundreds to millions of nodes. - -Concretely: where a naïve system would grow capacity linearly (O(t)) or even quadratically (O(t²) for pairwise operations), CORTEX caps every space-or-time budget at O(√(t log t)). This is the mathematically precise bound that keeps the engine on-device forever, regardless of corpus size. - -### Graph Mass Definition - -``` -t = |V| + |E| = total pages + (Hebbian edges + semantic neighbor edges) -``` - -This is the canonical measure of graph complexity used in all capacity formulas. - -### Resident Hotpath Capacity - -``` -H(t) = ⌈c · √(t · log₂(1 + t))⌉ -``` - -`c` is an empirically tuned constant (default in `core/HotpathPolicy.ts`; not a theorem output). H(t) defines the maximum number of entities resident in the in-memory hotpath index across all tiers. - -**Growth properties (required by tests):** -- H(t) is monotonically non-decreasing as t grows -- H(t) grows sublinearly relative to t (confirmed by benchmark at 1K, 10K, 100K, 1M) - -**Key distinction — graph mass (t) vs embedding dimension (n):** The graph mass `t = |V| + |E|` grows without bound as the corpus scales (potentially to millions). The embedding dimension `n` (e.g. 768 for embeddinggemma-300m) is a fixed property of the ML model. These are entirely separate quantities. The scaling constant `c` in H(t) controls how aggressively the hotpath compresses relative to graph mass — it has no relationship to embedding dimensionality. At the default `c = 0.5`: - -| Graph Mass (t) | H(t) | Compression Ratio | -|----------------|-------|-------------------| -| 100 | 13 | 13.0% | -| 1,000 | 50 | 5.0% | -| 10,000 | 183 | 1.8% | -| 100,000 | 645 | 0.65% | -| 1,000,000 | 2,233 | 0.22% | - -Setting `c` much above 1.0 defeats the sublinear bound: e.g. `c = 9.0` yields H(100) = 233 — larger than the graph itself. The purpose of `c = 0.5` is aggressive compression: even a million-entity graph keeps only ~2,200 entries resident. - -### Three-Zone Memory Model - -| Zone | Resident? | Storage | Typical Lookup Cost | -|------|-----------|---------|---------------------| -| **HOT** | Yes — in resident index, capacity H(t) | RAM | Sub-millisecond | -| **WARM** | No — indexed, not resident | IndexedDB | Single-digit milliseconds | -| **COLD** | No — raw bytes only, no index entry | OPFS | Tens of milliseconds | - -All data is retained locally across all three zones. Zones control lookup **cost**, not data **lifetime**. The runtime continuously promotes and evicts entries between HOT and WARM based on salience. - -### Node Salience - -Each page `v` carries a node-level salience score that drives promotion into and eviction from the hotpath: - -``` -σ(v) = α · H_in(v) + β · R(v) + γ · Q(v) -``` - -| Component | Meaning | -|-----------|---------| -| `H_in(v)` | Sum of incident Hebbian edge weights | -| `R(v)` | Recency score — exponential decay from `createdAt` / `lastQueryAt` | -| `Q(v)` | Query-hit count for the node | -| α, β, γ | Tunable weights summing to 1.0 (defaults: 0.5 / 0.3 / 0.2) | - -Salience requires lightweight per-page activity metadata (`queryHitCount`, `lastQueryAt`) stored in the `page_activity` IndexedDB object store. - -### Hierarchical Tier Quotas - -H(t) is partitioned across the four-tier hierarchy so no single tier can monopolise the resident index: - -| Tier | Default Quota | Purpose | -|------|--------------|---------| -| Shelf | q_s = 10% | Routing prototypes | -| Volume | q_v = 20% | Cluster prototypes | -| Book | q_b = 20% | Book medoids | -| Page | q_p = 50% | Individual page representatives | - -**Constraint:** q_s + q_v + q_b + q_p = 1.0 - -Within each tier, entries are ranked by salience; the highest-salience representatives are admitted up to the tier budget. Shelf, Volume, and Book representatives are selected by the medoid statistic within their cluster, then ranked by salience for admission. - -### Graph-Community Coverage Quotas +## Quick start (where to read) -Within each tier's budget, slots are allocated proportionally across detected graph communities to prevent a single dense topic from consuming all capacity. The allocation uses the **largest-remainder method** to guarantee the quotas sum exactly to `tier_budget`: - -1. Compute the ideal fractional share for each community: - ``` - share(Cᵢ) = tier_budget · nᵢ / N - ``` -2. Floor each share to get a base allocation: - ``` - base(Cᵢ) = ⌊share(Cᵢ)⌋ - ``` -3. Distribute the remaining `tier_budget − Σ base(Cᵢ)` slots one-by-one to the communities with the largest fractional remainders (`share(Cᵢ) − base(Cᵢ)`), breaking ties by community size (larger community wins). -4. Communities that receive a base of 0 and are not selected in step 3 are **excluded** from this tier (no slot). This is intentional: sparse communities are not promoted until they grow. - -The resulting quotas sum to exactly `tier_budget` regardless of the number or sizes of communities, even when there are more communities than `tier_budget`. - -where `nᵢ` is the number of pages in community Cᵢ and N is the total page count. Community detection runs via lightweight label propagation on the semantic neighbor graph during Daydreamer idle passes. - -This **dual constraint** — tier quota × community quota — ensures both vertical coverage across hierarchy levels and horizontal coverage across topics. - -### Promotion and Eviction Lifecycle - -**Bootstrap phase** (while resident count < H(t)): admit the highest-salience candidate not yet resident. - -**Steady-state phase**: promote a new or updated node only if its salience exceeds the weakest resident in the same tier and community bucket. On promotion, evict the weakest; break ties by recency. - -**Trigger points:** -- On ingest — newly ingested pages become candidates -- On query hit — `queryHitCount` increases; salience is recomputed; promotion sweep runs -- On Daydreamer pass — after LTP/LTD, recompute salience for affected nodes; run promotion sweep - -### Sublinear Fanout Bounds - -Maximum children per hierarchy node also respect Williams-derived limits to prevent unbounded fan-out: - -``` -Max volumes per shelf = O(√(|volumes| · log |volumes|)) -Max books per volume = O(√(|books_in_volume| · log |books_in_volume|)) -``` - -When exceeded, `HierarchyBuilder` or `ClusterStability` triggers a split. - -### Dynamic Subgraph Expansion Bounds - -The fixed `<30 node` subgraph target is replaced by dynamic formulas that shrink gracefully as the graph grows: - -``` -t_eff = max(t, 2) -- bootstrap floor (see below) -maxSubgraphSize = min(30, ⌊√(t_eff · log₂(1+t_eff)) / log₂(t_eff)⌋) -maxHops = max(1, ⌈log₂(log₂(1 + t_eff))⌉) -perHopBranching = max(1, ⌊maxSubgraphSize ^ (1 / maxHops)⌋) -``` - -**Domain and bootstrap floor.** The raw formulas are undefined when t ≤ 1 (`log₂(1) = 0` → division by zero; `log₂(t) < 0` for t < 1). The effective-mass floor `t_eff = max(t, 2)` eliminates these edge cases. At cold-start (t < 2) the formulas evaluate conservatively to `maxSubgraphSize = 1, maxHops = 1, perHopBranching = 1`, which is safe and correct — a single-node subgraph is the only valid result when fewer than two nodes exist. As the corpus grows past the floor the clamp becomes inactive (`t_eff = t` for all t ≥ 2), so large-corpus dynamics are completely unaffected. The explicit `max(1, …)` guards on `maxHops` and `perHopBranching` provide a secondary safety net against rounding to zero on very small but valid inputs. - -This keeps subgraph expansion cost sublinear in graph mass at scale while remaining well-behaved during cold-start and for tiny corpora. - -### Policy Source of Truth - -All hotpath constants — `c`, `α`, `β`, `γ`, `q_s`, `q_v`, `q_b`, `q_p` — live in `core/HotpathPolicy.ts` as a frozen default policy object. These are **policy-derived constants** (not model-derived) and are kept strictly separate from `core/ModelDefaults.ts`. A companion guard (or an extension to `guard:model-derived`) is planned to prevent these constants from being hardcoded elsewhere; until that guard is in place, discipline is enforced by convention. +| Topic | Wiki page | +|---|---| +| Architecture overview | [Architecture Overview](https://github.com/devlux76/cortex/wiki/Architecture-Overview) | +| Retrieval & Metroid algorithm | [Retrieval & Metroid Algorithm](https://github.com/devlux76/cortex/wiki/Retrieval-&-Metroid-Algorithm) | +| Ingestion (Hippocampus) | [Ingestion (Hippocampus)](https://github.com/devlux76/cortex/wiki/Ingestion-(Hippocampus)) | +| Consolidation (Daydreamer) | [Consolidation (Daydreamer)](https://github.com/devlux76/cortex/wiki/Consolidation-(Daydreamer)) | +| Storage architecture | [Storage Architecture](https://github.com/devlux76/cortex/wiki/Storage-Architecture) | +| Performance model & constraints | [Performance Model & Constraints](https://github.com/devlux76/cortex/wiki/Performance-Model-&-Constraints) | +| Security & trust | [Security & Trust](https://github.com/devlux76/cortex/wiki/Security-&-Trust) | +| Terminology & numerics | [Terminology + Numerics](https://github.com/devlux76/cortex/wiki/Terminology-+-Numerics) | +| Math appendix | [Math Appendix](https://github.com/devlux76/cortex/wiki/Math-Appendix) | --- -### Entity Hierarchy - -``` -Shelf (coarsest) - └─ Volume (cluster of books) - └─ Book (ordered page sequence) - └─ Page (atomic content chunk + embedding) -``` - -### Core Entities - -#### Page -Immutable content chunk with embedding and metadata. - -```typescript -interface Page { - pageId: Hash; // SHA-256(content) - content: string; // bounded by chunk policy - embeddingOffset: number; // byte offset into vector file - embeddingDim: number; // from ModelProfile - contentHash: Hash; // SHA-256(content) - vectorHash: Hash; // SHA-256(vector bytes) - prevPageId?: Hash; // linked-list for sequential content - nextPageId?: Hash; - creatorPubKey: PublicKey; - signature: Signature; - createdAt: string; // ISO timestamp -} -``` - -#### Book -Ordered sequence of pages from a **single ingest call** with a representative medoid. -One `ingestText()` call always produces exactly one Book — the entire ingested document. -A collection of Books forms a Volume; a collection of Volumes forms a Shelf. -Books are identified by `SHA-256(sorted pageIds)` so their identity is content-addressed. - -```typescript -interface Book { - bookId: Hash; // SHA-256(pageIds) or Merkle root - pageIds: Hash[]; - medoidPageId: Hash; // representative via medoid statistic - meta: BookMetadata; // title, sourceUri, tags, extra -} -``` - -#### Volume -Cluster of books with multiple prototypes. - -```typescript -interface Volume { - volumeId: Hash; - bookIds: Hash[]; - prototypeOffsets: number[]; // byte offsets into vector file - prototypeDim: number; // runtime policy dimension - variance: number; -} -``` - -#### Shelf -Top-level routing structure with coarse prototypes. - -```typescript -interface Shelf { - shelfId: Hash; - volumeIds: Hash[]; - routingPrototypeOffsets: number[]; - routingDim: number; -} -``` - -### Graph Structures - -#### Hebbian Edge -Weighted connection that strengthens/decays over time. - -```typescript -interface Edge { - fromPageId: Hash; - toPageId: Hash; - weight: number; // 0.0 to 1.0 - lastUpdatedAt: string; -} -``` - -#### Semantic Neighbor (Proximity Edge) -Sparse radius-graph edge connecting pages with high cosine similarity. Used for subgraph expansion during retrieval. - -**Critical distinction — two edge types, two roles:** - -| Edge type | Storage | Role | -|-----------|---------|------| -| `SemanticNeighbor` | `neighbor_graph` IDB store | Neighbor discovery during ingest; Bayesian belief updates | -| Hebbian edge (`Edge`) | `edges_hebbian` IDB store | TSP tour traversal distance; LTP/LTD strengthening/decay | - -`SemanticNeighbor.cosineSimilarity` drives: -- Which pages become neighbors during `FastNeighborInsert` (Williams-cutoff distance, not a fixed K) -- Bayesian belief score updates for the retrieved page set - -Hebbian `Edge.weight` drives: -- The distance metric used by `OpenTSPSolver` when ordering pages into a coherent narrative path -- Strength of connection for LTP/LTD during Daydreamer consolidation - -These two edge types must **never** be conflated or substituted for one another. - -```typescript -interface SemanticNeighbor { - neighborPageId: Hash; - cosineSimilarity: number; - distance: number; // 1 - cosineSimilarity; used for subgraph edge weight -} -``` - -#### Semantic Neighbor Subgraph -Induced subgraph for BFS-based coherence path expansion. - -```typescript -interface SemanticNeighborSubgraph { - nodes: Hash[]; - // distance: 1 - cosineSimilarity; used for BFS expansion candidate selection. - // OpenTSPSolver uses Hebbian edge weights (from edges_hebbian) as the tour - // traversal distance to determine how far to walk — not these cosine distances. - edges: { from: Hash; to: Hash; distance: number }[]; -} -``` - -### Hotpath Entities - -#### PageActivity -Lightweight per-page activity metadata maintained alongside each Page. Drives salience computation and community assignment. - -```typescript -interface PageActivity { - pageId: Hash; - queryHitCount: number; // incremented on each query hit - lastQueryAt: string; // ISO timestamp of most recent query hit - communityId?: string; // set by Daydreamer label propagation -} -``` - -#### HotpathEntry -The shared record type for HOT membership. Used in two complementary roles: - -1. **Live RAM index** — the active resident set (size ≤ H(t)) that every query scans first. -2. **IndexedDB persistence** — the `hotpath_index` store holds a periodic snapshot of the live index so that HOT membership and salience values survive a page reload or machine reboot. On startup, `HotpathEntry` rows are loaded from IndexedDB to reconstruct the RAM index without requiring a full corpus replay. - -The Daydreamer worker owns the write path to `hotpath_index`; it checkpoints the live index whenever it runs its maintenance cycle (LTP/LTD pass), making the persisted snapshot no more than one cycle stale. - -```typescript -interface HotpathEntry { - entityId: Hash; // pageId, bookId, volumeId, or shelfId - tier: 'shelf' | 'volume' | 'book' | 'page'; - salience: number; // σ value at last computation - communityId?: string; // community this entry counts against -} -``` - -## Storage Architecture - -### Vector Storage (OPFS) -Append-only binary file storing raw IEEE-754 float32 vectors. - -**Key Properties:** -- Mixed-dimension support (full embeddings + compressed prototypes) -- Byte-offset addressing -- Zero-copy reads -- Persistent across sessions - -**File:** `cortex-vectors.bin` - -### Metadata Storage (IndexedDB) -Structured entity storage with automatic reverse indexes. - -**Object Stores:** -- `pages`, `books`, `volumes`, `shelves` -- `edges_hebbian` (Hebbian weights) -- `neighbor_graph` (sparse semantic neighbor graph) -- `flags` (dirty-volume recalc markers) -- `page_to_book`, `book_to_volume`, `volume_to_shelf` (reverse indexes) -- `hotpath_index` (periodic HOT-membership checkpoint, keyed by `entityId`; loaded on startup to reconstruct the RAM resident index; written by Daydreamer each maintenance cycle) -- `page_activity` (per-page activity metadata for salience computation) - -## Retrieval Design - -### Cortex Query Path - -1. **Embed Query** — Generate query embedding -2. **Select m1** — Score resident medoids (HOT shelf/volume/book prototypes) to identify the topic medoid -3. **Build Metroid** — `MetroidBuilder` constructs `{ m1, m2, c }` via Matryoshka dimensional unwinding; if `m2` cannot be found, set `knowledge_gap = true` and emit a curiosity probe -4. **Score Resident Hierarchy** — Score query (anchored at centroid `c`) against HOT shelf prototypes in H(t) resident index -5. **Score Resident Volumes** — Score against HOT volume prototypes within top-ranked shelves -6. **Score Resident Books** — Score against HOT book medoids within top-ranked volumes -7. **Score Resident Pages** — Score against HOT page representatives within top-ranked books; explore thesis zone (m1), antithesis zone (m2), and synthesis zone (c) -8. **Spill to Warm/Cold** — If resident coverage is insufficient, expand lookup to WARM (IndexedDB) and COLD (OPFS) tiers -9. **Expand Subgraph** — BFS through semantic neighbor graph using dynamic Williams-derived bounds (see below) -10. **Solve Coherent Path** — Open TSP with dummy-node heuristic -11. **Return Result** — Ordered memory chain + provenance metadata (including whether a knowledge gap was detected) - -Steps 2–3 are the dialectical heart of CORTEX. Steps 4–7 are the Williams-bound-controlled resident-first scoring cascade. - -**Query Cost Meter:** The query path counts vector operations. If the cumulative cost exceeds a Williams-derived budget, the query early-stops and returns the best result found so far. - -### Coherence via Open TSP -Rather than returning nearest neighbors by similarity, Cortex traces a coherent path through the induced subgraph using a dummy-node open TSP strategy. This produces a natural "narrative flow" through related memories. - -### Key Constraints -- Steps 4–7 operate on the resident hotpath (H(t) entries), not the full corpus -- Metroid construction (step 3) is a prerequisite for dialectically balanced exploration; if it fails, a knowledge gap is declared -- Subgraph expansion (step 9) uses the **semantic neighbor graph** and dynamic Williams-derived bounds, not a fixed node cap: - - `maxSubgraphSize = min(30, ⌊√(t · log₂(1+t)) / log₂(t)⌋)` - - `maxHops = ⌈log₂(log₂(1 + t))⌉` - - `perHopBranching = ⌊maxSubgraphSize ^ (1/maxHops)⌋` -- Deterministic under same input for reproducibility -- Query cost is metered; early-stop prevents unbounded latency - -## Ingestion Design - -### Hippocampus Ingest Path - -1. **Chunk Text** — Split into pages respecting token budgets from ModelProfile -2. **Generate Embeddings** — Batch embed with selected provider -3. **Persist Vectors** — Append to OPFS vector file -4. **Persist Pages** — Write page metadata to IndexedDB; initialise `PageActivity` record -5. **Create Ingest Book** — Build exactly one Book for the entire ingest: compute the medoid page (minimum total cosine distance to all other pages in the document), derive `bookId = SHA-256(sorted pageIds)`, persist. Hotpath admission for the book runs via `SalienceEngine`. Volumes and Shelves are assembled lazily by the Daydreamer from accumulated Books. -6. **Fast Semantic Neighbor Insert** — Update semantic neighbor graph incrementally; bounded degree via `HotpathPolicy`; check new pages for hotpath admission -7. **Mark Dirty** — Flag volumes for full recalc by Daydreamer - -**Incremental Strategy (fast and lightweight):** -Ingest must remain fast and lightweight. At ingest time only two classes of edges are created: -- **Document-order adjacency** — Forward and reverse `SemanticNeighbor` edges between each consecutive page pair within the book slice, inserted unconditionally (document-adjacent chunks are always related). This uses a pre-built `Map` for O(1) lookups; no O(n²) index scans. -- **Proximity edges** — Additional `SemanticNeighbor` edges to nearby pages already in the corpus, bounded by cosine-distance cutoff and `maxDegree` eviction. - -Full cross-edge reconnection is intentionally deferred: Daydreamer walks the graph during idle passes to build additional edges — connections we never noticed at ingest time — and strengthens or prunes them via LTP/LTD. This keeps ingest cost sublinear while converging to a well-connected graph over time. - -**IndexedDB Schema Upgrade Strategy:** -The schema upgrade path intentionally drops and recreates object stores rather than migrating data. This keeps upgrade code minimal and avoids cruft until the data model stabilises. The neighbor graph is rebuilt from scratch after any ingest replay. - -## Consolidation Design - -### Daydreamer Responsibilities - -**LTP/LTD (Hebbian Updates):** -- Strengthen edges traversed during successful queries -- Decay unused edges toward zero -- Prune edges below threshold, keeping semantic neighbor degree within Williams-derived bounds -- After LTP/LTD: recompute σ(v) for all nodes whose incident edges changed; run promotion/eviction sweep via `SalienceEngine` - -**Prototype Recomputation:** -- Recompute volume/shelf medoids and centroids -- Update prototype vectors in vector file -- After recomputation: recompute salience for affected representative entries; run tier-quota promotion/eviction for volume and shelf tiers - -**Full Neighbor Graph Recalc:** -- For dirty volumes, recompute all pairwise similarities -- Bound batch size: process at most O(√(t log t)) pairwise comparisons per idle cycle -- Prioritise dirtiest volumes first -- Rebuild bounded neighbor lists; degree limit derived from `HotpathPolicy` -- Clear dirty flags; recompute salience for affected nodes; run promotion sweep - -**Community Detection:** -- Run lightweight label propagation on the semantic neighbor graph during idle passes -- Store community labels in `PageActivity.communityId` -- Rerun when dirty-volume flags indicate meaningful structural change -- Empty communities release their slots; new communities receive at least one slot - -**Experience Replay:** -- Simulate queries over recent memories -- Reinforce important connection patterns - -**Cluster Stability:** -- Detect unstable clusters (high variance, imbalanced size, Williams fanout violation) -- Trigger split/merge when thresholds exceeded -- Run community detection after structural changes - -**Scheduling Invariant:** The Daydreamer's reindexing frequency must track the rate of graph mass growth. If `t` grows faster than the background loop can reconcile the semantic neighbor graph, the Williams-derived degree bounds fall out of sync with actual graph state. The idle scheduler (`daydreamer/IdleScheduler.ts`) enforces this by gating recalc on dirty-volume flags — volumes are flagged at ingest time and processed in priority order during idle cycles, ensuring structural consistency converges even during high-velocity ingestion bursts. The Williams-bounded batch size (O(√(t log t)) pairwise comparisons per cycle) guarantees each maintenance pass is lightweight, while the dirty-flag mechanism guarantees no ingested data is permanently orphaned from the index. - -## Security & Trust - -### Cryptographic Integrity - -**Every Page Includes:** -- `contentHash`: SHA-256 of content text -- `vectorHash`: SHA-256 of embedding bytes -- `signature`: Ed25519 or equivalent signature -- `creatorPubKey`: Public key of creator - -**Verification Points:** -- On page creation (sign) -- On page retrieval (verify) -- On peer payload import (reject invalid) - -**Isolation:** -Keep cryptographic service separate from routing/storage concerns. All hashing/signing operations go through dedicated `core/crypto` module. - -## Performance Model - -### Backend Fallback Chain - -**Transformers.js Path:** -1. `webnn` (hardware ML accelerator) -2. `webgpu` (compute shaders) -3. `wasm` (guaranteed baseline) - -**Explicit ORT Path:** -1. `webnn` -2. `webgpu` -3. `webgl` (fragment shader fallback) -4. `wasm` - -**Selection Strategy:** -- Capability check (API availability) -- Benchmark race (small batch inference) -- Telemetry-informed hint (cached winner from previous session) - -### Performance Budgets - -| Operation | Target | Hardware Assumption | -|-----------|--------|---------------------| -| Ingest single page | <50ms | WebGPU-class | -| Query seed ranking (resident) | <20ms | H(t) resident index | -| Coherence path solve | <10ms | Dynamic subgraph (≤30 nodes) | -| Daydreamer work | Interruptible | No blocking | -| Hotpath promotion/eviction | <5ms | Per trigger point | - -**Graceful Degradation:** -All operations must complete on WASM fallback, albeit slower. The resident hotpath index reduces query latency proportionally to H(t) coverage of the working set. - -## Non-Negotiable Constraints - -1. **No Cloud Dependency** — Core operation 100% on-device -2. **Fast Local Retrieval** — Must work on degraded hardware -3. **Persistent Local State** — Survive browser restart with integrity checks -4. **Idle Consolidation** — Background quality improvements, not expensive write-time computation -5. **Sublinear Growth** — The resident hotpath index must never exceed H(t); all space-time tradeoff subsystems must target O(√(t log t)) scaling -6. **Privacy-Safe Sharing** — Shared payloads must pass eligibility filtering so identity/PII-bearing nodes are not exported - -## System Boundaries - -### In Scope -- On-device ingest, query, consolidation, persistence -- Multi-backend vector compute (`webgpu`, `webgl`, `webnn`, `wasm`) -- Signed graph entities with hash verification -- Sparse semantic neighbor graph for coherence routing -- Smart interest sharing: opt-in signed subgraph exchange over P2P with pre-share eligibility filtering - -### Out of Scope -- Full production-grade distributed consensus -- Cross-device key escrow or account systems -- Large-scale multi-tenant synchronization services -- Raw, unfiltered whole-graph export - -### Smart Sharing Guardrails - -Smart sharing is a core capability. The exchange path must: - -- Share only user-opted, public-interest graph sections (topic slices), not identity-bearing personal traces -- Run an eligibility classifier pass before export to block PII/person-specific leakage -- Preserve signatures and provenance on shared nodes so recipients can verify authenticity -- Keep transport peer-to-peer and on-device, with no central telemetry dependency - -## Terminology - -**Medoid** (mathematical term): The existing memory node selected as the statistical representative of a cluster. Selected by minimising the sum of distances to all other nodes in the cluster. Used throughout algorithmic descriptions and internal implementation comments. - -**Centroid** (mathematical term): In MetroidBuilder, the centroid `c` is a full-dimensional vector -where protected dimensions are copied from m1 (domain invariant) and unfrozen dimensions are the -element-wise average of m1 and m2. `c` is a synthetic "Kansas space" position — a center of mass -where nothing in the memory graph typically exists. Its value is as a neutral vantage point: -scoring candidates by distance to `c` gives equal weight to both poles. A candidate closer to m1 -is thesis-supporting; closer to m2 is antithesis-supporting; near `c` is genuinely balanced. - -**Metroid** (CORTEX architectural term): A structured dialectical search primitive constructed at -query time: `{ m1, m2, c }`. m1 is the thesis medoid (found via medoid search from query vector q); -m2 is the antithesis medoid (the medoid of the cosine-opposite set in the free dimensions — not -merely a semantically-opposing node, but the most coherent representative of maximal divergence); -c is the centroid (protected dims from m1; free dims averaged), computed **once and frozen** as a -stable evaluation platform. All subsequent candidates in the Matryoshka unwind are evaluated -relative to this frozen c. **A Metroid is never stored as a persistent graph structure.** It is an -ephemeral instrument used by the CORTEX retrieval subsystem. - -**MetroidBuilder**: The CORTEX module responsible for constructing a Metroid for a given query via -Matryoshka dimensional unwinding. Runs the thesis→freeze→antithesis→synthesis loop: m1 via medoid -search; m2 via cosine-opposite medoid; c computed once and frozen; subsequent candidates evaluated -relative to frozen c. Planned module: `cortex/MetroidBuilder.ts`. - -**Semantic neighbor graph** (also: proximity graph, neighbor graph): The sparse radius-graph of cosine-similarity edges between pages, used for subgraph expansion during retrieval. This is **not** the same as a Metroid. The edges connect pages with high cosine similarity and are used for BFS expansion. - -**Hotpath**: The in-memory resident index of H(t) entries spanning all four hierarchy tiers. The hotpath is the first lookup target for every query; misses spill to WARM/COLD storage. HOT membership and salience are checkpointed to the `hotpath_index` IndexedDB store by Daydreamer each maintenance cycle, allowing the RAM index to be restored after a page reload or machine reboot without full corpus replay. - -**Williams Bound**: The theoretical result S = O(√(t log t)) from Williams 2025, applied here as a universal sublinear growth law for all space-time tradeoff subsystems in CORTEX. The bound is the constructive answer to the curse of dimensionality: in `n`-dimensional space the unit-ball volume collapses as `π^(n/2) / Γ(n/2 + 1)`, making linear-scale data structures infeasible. The Williams sublinear bound keeps every budget — hotpath capacity, hierarchy fanout, neighbor degree, maintenance batch size — proportional to √(t log t) rather than t, ensuring on-device viability at any corpus scale. - -**Graph mass (t)**: t = |V| + |E| = total pages plus all edges (Hebbian + semantic neighbor). The canonical input to all capacity and bound formulas. - -**Salience (σ)**: Node-level score combining Hebbian edge weight, recency, and query-hit frequency. Drives admission to and eviction from the hotpath. - -**Three-zone model**: HOT (resident), WARM (IndexedDB-indexed), COLD (OPFS bytes only). All zones retain data locally; zones differ only in lookup cost. - -**Community**: A topically coherent subgraph identified by label propagation on the semantic neighbor graph. Community quotas prevent any single topic from monopolising the hotpath. - -**Knowledge gap**: A state where MetroidBuilder cannot find a valid antithesis medoid `m2` within dimensional constraints. Triggers a P2P curiosity request. - -**Curiosity probe**: A P2P broadcast containing an incomplete Metroid (`{ m1, partialMetroid, knowledgeBoundary }`) sent when a knowledge gap is detected. Peers respond with graph fragments that may enable antithesis discovery. - -## Model-Derived Numerics - -**Critical Rule:** All numeric values derived from ML model architecture (embedding dimensions, context lengths, thresholds, and Matryoshka sub-dimension boundaries) must **never** be hardcoded as magic numbers. - -**Source of Truth:** -- `core/ModelProfile.ts` — Interface definition (includes `matryoshkaProtectedDim`) -- `core/ModelDefaults.ts` — Default derivation from seed values -- `core/BuiltInModelProfiles.ts` — Concrete model registrations (includes per-model `matryoshkaProtectedDim`) -- `core/ModelProfileResolver.ts` — Runtime resolution - -**Model-specific `matryoshkaProtectedDim` values (must be sourced from `BuiltInModelProfiles.ts`):** - -| Model | `matryoshkaProtectedDim` | Notes | -|-------|--------------------------|-------| -| `onnx-community/embeddinggemma-300m-ONNX` | 128 | Smallest supported Matryoshka sub-dimension | -| `nomic-ai/nomic-embed-text-v1.5` | 64 | To be added when nomic provider is wired | - -**Enforcement:** `npm run guard:model-derived` scans for violations before CI merge. The guard checks for `matryoshkaProtectedDim` in addition to the standard embedding dimension and context length fields. - -## Policy-Derived Constants - -A parallel class of constants governs the Williams Bound hotpath architecture. These are **not** model-derived (they do not depend on ML architecture); they are empirically tuned policy values. - -**Source of Truth:** `core/HotpathPolicy.ts` — frozen default policy object - -| Constant | Default | Meaning | -|----------|---------|---------| -| `c` | 0.5 | Scaling factor in H(t) formula | -| `α` | 0.5 | Salience weight for Hebbian connectivity | -| `β` | 0.3 | Salience weight for recency | -| `γ` | 0.2 | Salience weight for query-hit frequency | -| `q_s` | 0.10 | Shelf tier quota fraction | -| `q_v` | 0.20 | Volume tier quota fraction | -| `q_b` | 0.20 | Book tier quota fraction | -| `q_p` | 0.50 | Page tier quota fraction | +## Quick glossary (for fast reference) -**Enforcement:** Policy constants must not be hardcoded outside `core/HotpathPolicy.ts`. A companion guard or ESLint rule prevents silent duplication. +- **Medoid:** an actual memory node used as a cluster representative. +- **Centroid:** a computed average vector (not stored as a node). +- **Metroid:** a transient `{ m1, m2, c }` structure used in retrieval. -## Future Directions +> Note: Most detailed definitions and rationale are in the wiki pages above. -- **Federated Sharing Optimization** — Better peer-ranking, deduplication, and prioritization for high-signal interest updates -- **Advanced Consolidation** — More sophisticated LTP/LTD policies -- **Query Reranking** — Optional second-pass reranking for quality -- **Adaptive Chunking** — Context-aware page boundary detection -- **Multi-Modal Support** — Image/audio embeddings alongside text -- **CRDT-based Merge** — Conflict-free replicated data structures for multi-device sync -- **Empirical Calibration of c** — Instrument real workloads to tune the Williams Bound scaling constant across diverse corpus profiles diff --git a/README.md b/README.md index 1dbcc92..4e03187 100644 --- a/README.md +++ b/README.md @@ -113,7 +113,8 @@ bun run dev:harness # start the browser runtime harness at http://127.0.0.1:4173 | Document | Purpose | |---|---| -| [`DESIGN.md`](DESIGN.md) | Architecture specification and core design principles | +| [CORTEX Wiki](https://github.com/devlux76/cortex/wiki) | Canonical design documentation (architecture, algorithms, and math). | +| [`DESIGN.md`](DESIGN.md) | Static repo landing page / TOC into the wiki | | [`PLAN.md`](PLAN.md) | Module-by-module implementation status and development phases | | [`TODO.md`](TODO.md) | Prioritized actionable tasks to ship v1.0 | | [`docs/api.md`](docs/api.md) | API reference for developers integrating with CORTEX | diff --git a/docs/development.md b/docs/development.md index e53d0aa..a08372e 100644 --- a/docs/development.md +++ b/docs/development.md @@ -170,10 +170,11 @@ This command will fail the build if it detects numeric literals that are likely At the end of every implementation pass, update documents in this order: -1. **`DESIGN.md`** — update if architecture changes. -2. **`README.md`** — confirm the project description still reflects reality. -3. **`docs/api.md`** — update if new public APIs are added or existing ones change. -4. **GitHub Issues** — close completed tasks, create new ones as needed via `gh` CLI or the web UI. +1. **`DESIGN.md`** — update if the design landing/TOC changes. +2. **Wiki** — update the relevant wiki page(s) for any architecture or algorithm changes. +3. **`README.md`** — confirm the project description still reflects reality. +4. **`docs/api.md`** — update if new public APIs are added or existing ones change. +5. **GitHub Issues** — close completed tasks, create new ones as needed via `gh` CLI or the web UI. > Numeric examples in design docs are illustrative unless explicitly sourced from model metadata. diff --git a/docs/wiki-draft/Architecture.md b/docs/wiki-draft/Architecture.md new file mode 100644 index 0000000..590b6d2 --- /dev/null +++ b/docs/wiki-draft/Architecture.md @@ -0,0 +1,30 @@ +# Architecture Overview + +This page describes the high-level architecture of CORTEX and the major subsystems. + +## The Three Living Regions + +CORTEX models three biological brain regions working in concert: + +- **Hippocampus** — Fast associative encoding and incremental prototype construction. +- **Cortex** — Intelligent routing, dialectical retrieval, and coherence. +- **Daydreamer** — Background consolidation and maintenance. + +Each region is responsible for a distinct phase of the memory lifecycle. Together they form a pipeline from ingestion through retrieval. + +## Core Concepts + +### Medoid vs. Centroid vs. Metroid + +- **Medoid** — An actual memory node (page) selected as the representative of a cluster. +- **Centroid** — A computed geometric average (never stored as a real node). +- **Metroid** — A transient, structured dialectical search probe (`{ m1, m2, c }`) used at query time. + +### How the subsystems interact + +1. **Ingestion:** Hippocampus embeds content and creates/update prototypes. +2. **Retrieval:** Cortex constructs Metroids and performs dialectical search for coherent context. +3. **Consolidation:** Daydreamer updates prototypes, prunes edges, and maintains stability. + + +> For the full algorithmic detail, see **Retrieval & Metroid Algorithm**. diff --git a/docs/wiki-draft/Consolidation.md b/docs/wiki-draft/Consolidation.md new file mode 100644 index 0000000..f3ed08f --- /dev/null +++ b/docs/wiki-draft/Consolidation.md @@ -0,0 +1,14 @@ +# Consolidation (Daydreamer) + +This page covers background consolidation and maintenance mechanisms. + +## Daydreamer Responsibilities + +- **Long-term potentiation (LTP):** strengthen important connections. +- **Long-term depression (LTD):** decay and prune weak edges. +- **Medoid/centroid recomputation:** keep prototypes coherent as the graph evolves. +- **Experience replay:** rehearse recent data in background when idle. + +## Stability & Throttling + +The Daydreamer is designed to run opportunistically without blocking foreground query performance. Its work is throttled and batch-sized according to the current memory graph complexity. diff --git a/docs/wiki-draft/Home.md b/docs/wiki-draft/Home.md new file mode 100644 index 0000000..fc8af8d --- /dev/null +++ b/docs/wiki-draft/Home.md @@ -0,0 +1,38 @@ +# CORTEX Design Wiki + +This wiki is the **canonical home for CORTEX architecture and design documentation**. + +> The repository includes `DESIGN.md` as a small **TOC/landing page** that points here. Use this wiki for all deep dives, algorithm descriptions, and mathematical reasoning. + +## Where to start + +- **If you’re writing an issue or PR that affects the architecture:** start with **Architecture Overview**. +- **If you need to understand retrieval behavior:** see **Retrieval & Metroid Algorithm**. +- **If you’re changing ingestion or indexing:** see **Ingestion (Hippocampus)**. +- **If you’re optimizing or debugging consolidation:** see **Consolidation (Daydreamer)**. +- **If you’re changing storage formats or persistence:** see **Storage Architecture**. +- **If you’re tuning performance budgets:** see **Performance Model & Constraints**. +- **If you need definitions or constant values:** see **Terminology + Numerics**. +- **If you need the math behind the design:** see **Math Appendix**. + +--- + +## Wiki Pages (Table of Contents) + +- [Architecture Overview](Architecture.md) +- [Retrieval & Metroid Algorithm](Retrieval.md) +- [Ingestion (Hippocampus)](Ingestion.md) +- [Consolidation (Daydreamer)](Consolidation.md) +- [Storage Architecture](Storage.md) +- [Performance Model & Constraints](Performance.md) +- [Security & Trust](Security.md) +- [Terminology + Numerics](Terminology.md) +- [Math Appendix](Math-Appendix.md) + +--- + +## How to use this wiki + +- **Edit on GitHub**: Edit the markdown directly in the wiki repo (`cortex.wiki`) and push. +- **Linking from issues/PRs**: Link to the relevant wiki page (e.g., `https://github.com/devlux76/cortex/wiki/Retrieval-&-Metroid-Algorithm`). +- **Keeping it up to date**: When the code changes, update the relevant wiki page; include the wiki link in the PR description. diff --git a/docs/wiki-draft/Ingestion.md b/docs/wiki-draft/Ingestion.md new file mode 100644 index 0000000..3c8c859 --- /dev/null +++ b/docs/wiki-draft/Ingestion.md @@ -0,0 +1,16 @@ +# Ingestion (Hippocampus) + +This page describes how CORTEX ingests new observations and integrates them into memory. + +## Ingest Path + +1. **Chunking/Parsing** — Raw inputs are segmented into pages/blocks. +2. **Embedding** — Each chunk is embedded using a Matryoshka-capable model. +3. **Fast Neighbor Insert** — New vectors are connected into the semantic neighbor graph. +4. **Hierarchical Prototypes** — Pages are organized into Books, Volumes, and Shelves. + +## Hierarchy & Promotion + +CORTEX manages a hierarchical prototype structure to keep hot (frequently-accessed) concepts in memory while relying on disk-backed storage for the long tail. + +> For implementation details, see the code in `hippocampus/` and the `HierachyBuilder` design notes. diff --git a/docs/wiki-draft/Math-Appendix.md b/docs/wiki-draft/Math-Appendix.md new file mode 100644 index 0000000..9b3ff53 --- /dev/null +++ b/docs/wiki-draft/Math-Appendix.md @@ -0,0 +1,37 @@ +# Math Appendix + +This appendix contains the mathematical background that motivates several of CORTEX’s key design decisions. + +## Curse of Dimensionality + +In high-dimensional spaces, the volume of a unit ball collapses rapidly. For even dimension `n = 2m`: + +``` +V_n = π^m / m! +``` + +Stirling’s approximation shows this shrinks exponentially with `n`, meaning nearly all the volume is concentrated near the surface. + +## Hypersphere Volume and the Hollow Sphere + +CORTEX leverages this “hollow sphere” phenomenon: in high dimensions, the interior of a ball is essentially empty, so nearest-neighbor search can focus on the surface shell. + +## Williams 2025 Sublinear Bound + +CORTEX applies the result: + +``` +S = O(√(t · log t)) +``` + +to bound space requirements (hotpath capacity, fanout limits, maintenance budgets) in a way that maintains on-device performance. + +## Why This Matters + +These mathematical observations drive several design decisions in CORTEX: + +- Matryoshka dimension protection (to prevent domain drift) +- Sublinear fanout quotas (to avoid explosion in edge counts) +- The Metroid dialectical search pattern (to avoid confirmation bias in high-D retrieval) + +> For full details, see the source code and the other wiki pages. diff --git a/docs/wiki-draft/Performance.md b/docs/wiki-draft/Performance.md new file mode 100644 index 0000000..5002036 --- /dev/null +++ b/docs/wiki-draft/Performance.md @@ -0,0 +1,28 @@ +# Performance Model & Constraints + +This page explains the performance budget model and the key formulas that keep CORTEX sublinear. + +## Williams Sublinear Bound + +CORTEX uses the Williams 2025 result: + +> **S = O(√(t · log t))** + +This bound is applied to multiple budgets (hotpath index size, hierarchy fanout, neighbor degrees, maintenance batch sizes) to ensure the system stays efficient as the graph grows. + +## Hotpath Capacity + +The resident hotpath index is capped to a sublinear growth function, often expressed as: + +``` +H(t) = ⌈c · √(t · log₂(1 + t))⌉ +``` + +## Budgeting & Fanout Limits + +The same sublinear law is used for: +- Hierarchy fanout limits +- Semantic neighbor degree caps +- Daydreamer maintenance batch sizing + +> See the code in `core/HotpathPolicy.ts` and `hippocampus/HierarchyBuilder.ts` for the concrete implementations. diff --git a/docs/wiki-draft/Retrieval.md b/docs/wiki-draft/Retrieval.md new file mode 100644 index 0000000..046de6a --- /dev/null +++ b/docs/wiki-draft/Retrieval.md @@ -0,0 +1,29 @@ +# Retrieval & Metroid Algorithm + +This page explains the retrieval pipeline and the Metroid-based dialectical search mechanism. + +## Metroid Overview + +A **Metroid** is a structured search primitive: it contains a thesis (`m1`), an antithesis (`m2`), and a frozen centroid (`c`). + +- `m1` is the medoid closest to the query. +- `m2` is an opposite medoid found via cosine-opposite medoid search. +- `c` is the frozen centroid between `m1` and `m2`. + +## Dialectical Search Zones + +From the centroid `c`, the system classifies candidates into three zones: + +- **Thesis zone:** closer to `m1` than to `c`. +- **Antithesis zone:** closer to `m2` than to `c`. +- **Synthesis zone:** near `c`, balanced between both poles. + +## Matryoshka Dimensional Unwinding + +CORTEX uses Matryoshka embeddings with protected dimensions (lower dimensions that anchor domain context). The retrieval algorithm progressively frees dimensions to explore antithesis candidates while keeping the centroid frozen. + +## Knowledge Gap Detection + +When no suitable `m2` can be found within constraints, the system flags a **knowledge gap** and may broadcast a P2P curiosity request. + +> See the **Math Appendix** for the geometric intuition behind why this approach is necessary in high-dimensional spaces. diff --git a/docs/wiki-draft/Security.md b/docs/wiki-draft/Security.md new file mode 100644 index 0000000..603b4a5 --- /dev/null +++ b/docs/wiki-draft/Security.md @@ -0,0 +1,16 @@ +# Security & Trust + +This page covers trust assumptions, cryptographic integrity, and smart sharing guardrails. + +## Cryptographic Integrity + +CORTEX uses cryptographic signing to ensure that stored vectors and metadata cannot be tampered with. + +## Smart Sharing Guardrails + +When sharing memory fragments over P2P, CORTEX enforces: +- MIME type validation +- Model URN compatibility checks +- Eligibility filtering (to avoid leaking sensitive or irrelevant data) + +> See `sharing/` for the implementation details of peer exchange and eligibility classification. diff --git a/docs/wiki-draft/Storage.md b/docs/wiki-draft/Storage.md new file mode 100644 index 0000000..f8c407e --- /dev/null +++ b/docs/wiki-draft/Storage.md @@ -0,0 +1,17 @@ +# Storage Architecture + +This page describes how CORTEX stores vectors and metadata in the browser. + +## Vector Storage (OPFS) + +- Vectors are stored in an append-only OPFS file for fast writes. +- The OPFS backend is designed to be zero-copy for WebGPU and WebGL consumption. + +## Metadata Storage (IndexedDB) + +- Metadata (nodes, edges, schema) is stored in IndexedDB. +- IndexedDB is used for fast subgraph retrieval and persistence across sessions. + +## Maintenance & Corruption Resistance + +CORTEX includes mechanisms to verify cryptographic integrity of stored data and recover from partial failures. diff --git a/docs/wiki-draft/Terminology.md b/docs/wiki-draft/Terminology.md new file mode 100644 index 0000000..43df90c --- /dev/null +++ b/docs/wiki-draft/Terminology.md @@ -0,0 +1,23 @@ +# Terminology & Numerics + +This page collects key terms, model-derived numeric constants, and policy-derived constants. + +## Key Terms + +- **Medoid** — an existing memory node selected as a cluster representative. +- **Centroid** — a computed average vector (not necessarily a real node). +- **Metroid** — a transient search construct `{ m1, m2, c }` used during retrieval. +- **Hotpath** — the in-memory resident index of active nodes. + +## Model-Derived Numerics + +These values are derived from the embedding model profile (not hardcoded): +- Embedding dimensionality +- Matryoshka protected dimension boundary +- Query context length limits + +## Policy-Derived Constants + +Policy constants (e.g. fanout caps, quota ratios) are defined in the code and kept in sync with the design. + +> For the authoritative source of policy constants, see `core/Policy.ts` and `core/HotpathPolicy.ts`. From f2f55f044e3b2ddef9fa3ae4b05557b97e948fde Mon Sep 17 00:00:00 2001 From: "S. Dale Morrey" <86517969+devlux76@users.noreply.github.com> Date: Sat, 14 Mar 2026 01:20:38 -0600 Subject: [PATCH 2/5] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/wiki-draft/Terminology.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/wiki-draft/Terminology.md b/docs/wiki-draft/Terminology.md index 43df90c..53f116d 100644 --- a/docs/wiki-draft/Terminology.md +++ b/docs/wiki-draft/Terminology.md @@ -20,4 +20,4 @@ These values are derived from the embedding model profile (not hardcoded): Policy constants (e.g. fanout caps, quota ratios) are defined in the code and kept in sync with the design. -> For the authoritative source of policy constants, see `core/Policy.ts` and `core/HotpathPolicy.ts`. +> For the authoritative source of policy constants, see `core/HotpathPolicy.ts`. From 57165bf820cae880dc57ce8a30ae94a674db2a60 Mon Sep 17 00:00:00 2001 From: "S. Dale Morrey" <86517969+devlux76@users.noreply.github.com> Date: Sat, 14 Mar 2026 01:21:39 -0600 Subject: [PATCH 3/5] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/wiki-draft/Ingestion.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/wiki-draft/Ingestion.md b/docs/wiki-draft/Ingestion.md index 3c8c859..3bf7eeb 100644 --- a/docs/wiki-draft/Ingestion.md +++ b/docs/wiki-draft/Ingestion.md @@ -13,4 +13,4 @@ This page describes how CORTEX ingests new observations and integrates them into CORTEX manages a hierarchical prototype structure to keep hot (frequently-accessed) concepts in memory while relying on disk-backed storage for the long tail. -> For implementation details, see the code in `hippocampus/` and the `HierachyBuilder` design notes. +> For implementation details, see the code in `hippocampus/` and the `HierarchyBuilder` design notes. From 7f7493e260d915557b678b35bca431adf34617fa Mon Sep 17 00:00:00 2001 From: "S. Dale Morrey" <86517969+devlux76@users.noreply.github.com> Date: Sat, 14 Mar 2026 01:22:20 -0600 Subject: [PATCH 4/5] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/wiki-draft/Storage.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/wiki-draft/Storage.md b/docs/wiki-draft/Storage.md index f8c407e..9971cb3 100644 --- a/docs/wiki-draft/Storage.md +++ b/docs/wiki-draft/Storage.md @@ -14,4 +14,6 @@ This page describes how CORTEX stores vectors and metadata in the browser. ## Maintenance & Corruption Resistance -CORTEX includes mechanisms to verify cryptographic integrity of stored data and recover from partial failures. +Today, CORTEX relies on the browser’s OPFS and IndexedDB durability guarantees, with limited, optional integrity checks (for example, content-hash verification on incoming peer fragments). + +Planned: a broader integrity verification and corruption-detection/recovery flow for OPFS/IndexedDB-backed data, including cryptographic integrity validation of stored payloads and automated remediation of partial failures. From 30e6b8d03978f773d827e4333d5519551fb43219 Mon Sep 17 00:00:00 2001 From: "S. Dale Morrey" <86517969+devlux76@users.noreply.github.com> Date: Sat, 14 Mar 2026 01:24:13 -0600 Subject: [PATCH 5/5] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/wiki-draft/Security.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/wiki-draft/Security.md b/docs/wiki-draft/Security.md index 603b4a5..8c2c4e0 100644 --- a/docs/wiki-draft/Security.md +++ b/docs/wiki-draft/Security.md @@ -4,7 +4,7 @@ This page covers trust assumptions, cryptographic integrity, and smart sharing g ## Cryptographic Integrity -CORTEX uses cryptographic signing to ensure that stored vectors and metadata cannot be tampered with. +CORTEX supports cryptographic signing (and optional verification) for stored vectors and metadata to help detect tampering and integrity issues. ## Smart Sharing Guardrails