[P1] feat(telemetry): Implement Kalman-Based Drift Detection

.github/workflows/telemetry_gate.yml

@@ -19,21 +19,19 @@ jobs:
         with:
           go-version: '1.26'
 
-      - name: Build telemetry_replay
+      - name: Build phoenix-runtime
         run: |
-          cd 14_experiments/telemetry_replay
-          go build -o telemetry_replay main.go
+          cd phoenix_os
+          go build -o ../build/phoenix-runtime main.go
 
-      - name: Run Telemetry Gate
+      - name: Run Deterministic Replay Gate
         run: |
-          ./14_experiments/telemetry_replay/telemetry_replay \
-            --file 09_telemetry/replay_events_large.jsonl \
-            --rate 1000 \
-            --out results/telemetry_gate
-
-      - name: Upload Artifacts
-        if: always()
-        uses: actions/upload-artifact@v4
-        with:
-          name: telemetry-gate-results
-          path: results/telemetry_gate/
+          HASH1=$(./build/phoenix-runtime 2>&1 | grep 'Deterministic output hash' | awk '{print $NF}')
+          HASH2=$(./build/phoenix-runtime 2>&1 | grep 'Deterministic output hash' | awk '{print $NF}')
+          echo "Run 1 hash: $HASH1"
+          echo "Run 2 hash: $HASH2"
+          if [ "$HASH1" != "$HASH2" ]; then
+            echo "DETERMINISM VIOLATION: Hashes do not match!"
+            exit 1
+          fi
+          echo "✓ Replay determinism verified: $HASH1"

00_program_management/roadmap/main_roadmap.md

@@ -1,59 +1,101 @@
-# Phoenix Master Architecture & Roadmap (5-7 Year Plan)
-
-## Core Principle
-`Physics + Mathematics + Security + Operating Systems = Phoenix`
-
-## Master System Stack (GTOS Model)
-- **L7:** Autonomous Security Layer (Multi-Agent Swarm)
-- **L6:** Phoenix Sentinel Engine (Thermodynamics & Disorder)
-- **L5:** Game Theory & Control Dynamics (Nash, Stackelberg, VCG)
-- **L4:** Graph Intelligence (Incident DAGs)
-- **L3:** Mathematical Telemetry (Entropy, Signals)
-- **L2:** Kernel Telemetry Runtime (eBPF)
-- **L1:** Hardware / Hypervisor
-
-## Strict Execution Order (Do Not Deviate)
-1. **Math Foundations:** (Information Theory, Entropy, KL Divergence) -> *Currently Executing*
-2. **Telemetry:** (eBPF, Math Filters, State Engine)
-3. **Graphs:** (DAGs, Shortest Path, Community Detection)
-4. **Control:** (PID, Adaptive Response)
-5. **Physics:** (Disorder, Thermodynamics, Dynamics)
-6. **Game Theory (GTOS):** (Nash, VCG, Stackelberg, Shapley, ESS)
-7. **Kernel:** (Physics Hooks, Adaptive Scheduler)
-8. **Autonomous OS:** (AI, Multi-agent execution)
-
-## Phase A: Mathematical Core (Months 1-6)
-- **A1:** Information Theory (Phoenix Monitor, KL Detector)
-- **A2:** Graph Mathematics (Incident DAGs)
-- **A3:** Optimization Theory (Minimizing containment cost)
-
-## Phase B: Physics Layer (Months 6-12)
-- **B1:** Statistical Physics (Security Disorder Index - SDI)
-- **B2:** Dynamical Systems (Trajectory drift)
-- **B3:** Signal Processing (FFT, Wavelets for chaos detection)
-
-## Phase C: Game Theory Layer (GTOS Implementation) (Months 12-20)
-- **G1:** Resource Allocation Games (Nash Scheduling)
-- **G2:** Mechanism Design (VCG Resource Allocation)
-- **G3:** Zero-Sum Security Games (Minimax Defense)
-- **G4:** Stackelberg Defense (Adaptive Policies)
-- **G5:** Cooperative Games (Shapley Value Accounting)
-- **G6:** Evolutionary Security (Stable Detection)
-- **G7:** Auction-based QoS (Vickrey Routing)
-- **G8:** Multi-Agent Swarm Integration
-
-## Kernel Roadmap (Year 1-4)
-- **Year 1:** Resource Games, VCG Allocator, Fairness Metrics.
-- **Year 2:** Security Games, Stackelberg Defense, Bayesian Inference.
-- **Year 3:** Shapley Cache, Auction QoS, ESS Defense.
-- **Year 4:** Game Scheduler, Adaptive Kernel, Autonomous Policies.
-
-## Phase D: Control Layer (Months 20-25)
-- **D1:** PID Containment (Throttle -> Isolate -> Snapshot)
-
-## Phase E: Kernel Integration (Months 25-37)
-- **E1:** Physics/Game-aware Scheduler
-- **E2:** Energy/Entropy tracking memory allocation
-
-## Phase F: Autonomous Phoenix (Months 37+)
-- Full LLM/Graph reasoning runtime.
+# PhoenixOS: Master Execution Roadmap
+
+## Project Definition
+**PhoenixOS** is a **Deterministic Cybernetic Security Runtime** running on Linux. It focuses on mathematically reproducible replay, bounded control, and telemetry-first validation.
+
+## Core Axioms
+1. **Determinism is sacred.**
+2. **Replay is authoritative.**
+3. **AI is advisory.**
+4. **Control must remain bounded.**
+5. **Telemetry correctness > AI sophistication.**
+6. **Never scale instability.**
+
+---
+
+## Phase A: Core Product (Stages 0–4)
+*Focus: Deterministic single-node cybernetic security runtime.*
+
+### Stage 0: Foundation Stabilization
+* **Goal:** Architectural coherence.
+* **Deliverables:** RFC system, repository structure, FSM model, `DETERMINISM.md`.
+* **Exit Criteria:** No major architectural contradictions remain.
+
+### Stage 1: Deterministic Replay Runtime (ACTIVE)
+* **Goal:** Mathematically reproducible single-node replay.
+* **Systems:** Replay engine, Event bus, Ledger, FSM, TCS, Drift engine.
+* **Features:** Logical time, Canonical serialization, Stable hashing, Bounded queues.
+* **Exit Criteria:** Replay hashes are byte-for-byte identical across runs.
+
+### Stage 2: Real Telemetry Runtime
+* **Goal:** Replace simulated data with real Linux eBPF/XDP telemetry.
+* **Build:** Phoenix Guard, Kernel adapters, Telemetry collectors.
+* **Exit Criteria:** Stable single-node telemetry replay.
+
+### Stage 3: Immutable Runtime Image
+* **Goal:** Boot Phoenix as a minimal immutable environment.
+* **Build:** LinuxKit appliance, Minimal kernel, Immutable initrd, PID1 Warden.
+* **Exit Criteria:** Phoenix runtime boots as a standalone immutable appliance.
+
+### Stage 4: Constrained Actuation
+* **Goal:** Safe, bounded autonomous response.
+* **Systems:** Warden FSM, Actuation budgets, Cooldowns, Rate limits.
+* **Allowed Actions:** Log, Throttle, Socket block, Process isolate.
+* **Exit Criteria:** No state oscillation under adversarial stress.
+
+---
+
+## Phase B: Advanced Platform (Stages 5–8)
+*Focus: Distributed, forensic-grade enterprise observability.*
+
+### Stage 5: Evidence & Replay Infrastructure
+* **Goal:** Forensic-grade replay DAGs and cross-session provenance.
+* **Features:** Causal linking, Event provenance, Hash chains.
+* **Exit Criteria:** Replay becomes the authoritative system truth.
+
+### Stage 6: Graph Runtime
+* **Goal:** Transform telemetry into temporal causality structures.
+* **Systems:** Process DAG, Node graph, Attack chains.
+* **Exit Criteria:** Stable attack graph reconstruction.
+
+### Stage 7: Distributed Telemetry
+* **Goal:** Multi-node observability using vector clocks.
+* **Features:** Event replication, Global replay, Distributed evidence.
+* **Exit Criteria:** Cross-node deterministic replay succeeds.
+
+### Stage 8: Cloud Runtime
+* **Goal:** Cluster orchestration and immutable deployment fleet.
+* **Build:** Scheduler, Policy distribution, Image rollout.
+* **Exit Criteria:** Phoenix nodes are manageable as a fleet.
+
+---
+
+## Phase C: Cybernetic Research (Stages 9–12)
+*Focus: Advanced reasoning, state estimation, and automated strategic defense.*
+
+### Stage 9: Advisory AI Layer
+* **Goal:** Explainable AI correlation and ranking (Advisory only).
+* **Allowed Tasks:** Summaries, Policy suggestions, Replay explanation.
+* **Exit Criteria:** AI outputs are strictly bounded and explainable.
+
+### Stage 10: Physics + Control Theory
+* **Goal:** Formal system-state estimation.
+* **Concepts:** Entropy, Variance (Welford's), Kalman filters, Signal processing.
+* **Exit Criteria:** Physics layer measurably improves detection.
+
+### Stage 11: Game Theory Runtime
+* **Goal:** Strategic bounded defense planning.
+* **Build:** Resource allocation games, Stackelberg defense, Decoy placement.
+* **Exit Criteria:** Algorithmic defense decisions outperform heuristics.
+
+### Stage 12: Cybernetic Runtime
+* **Goal:** Full closed-loop bounded defense system.
+* **Properties:** Observable, Bounded, Reversible, Deterministic.
+* **Exit Criteria:** Stable and autonomous under adversarial stress.
+
+---
+
+## Phase D: Optional Evolution (Stage 13)
+### Stage 13: Research OS
+* **Focus:** Custom schedulers, hardware isolation (SGX/SEV), microkernel experiments.
+* **Status:** Pure OS research, not required for core project success.

02_docs/decisions/ADR-002-swarm-restructuring.md

@@ -0,0 +1,23 @@
+# ADR-002: Swarm Logic Restructuring
+
+## Status
+
+Proposed.
+
+## Context
+
+The `phoenix_os/agents/internal/swarm` package contained mixed concerns, causing package collisions and potential circular dependencies between the Warden (actuation) and the Arbiter (reasoning).
+
+## Decision
+
+Restructure `phoenix_os/agents/internal/swarm` into sub-packages to encapsulate agent logic and governance.
+
+- Move entry point/agent logic to `phoenix_os/agents/internal/swarm/agent`.
+- Move governance logic to `phoenix_os/agents/internal/swarm/governance`.
+
+## Consequences
+
+- Resolves package collisions.
+- Eliminates circular dependencies between Warden and Arbiter.
+- Improves code maintainability and modularity.
+- Aligns with the Phoenix Matrix separation of concerns.

02_docs/rfc/RFC-010_arbiter_warden_interface.md

@@ -0,0 +1,45 @@
+# RFC-010: Arbiter-Warden Control Interface
+
+**Status:** Draft (Revised)
+**Review Date:** 2026-05-22
+**Author:** Documentation Evolution Agent
+**Target Subsystem:** L5/L5.5 Control Loop
+
+## 1. Objective
+Define a rigid, state-aware, and failure-safe interface between the L5.5 Strategic Policy Arbiter and the L5 Actuation Warden. The system must adhere to the principle: **Every autonomous action must be observable, explainable, reversible, and bounded.**
+
+## 2. Interface Formalization
+The interface shall be modeled as a restricted Finite State Machine with mandatory policy constraints:
+
+```text
+Arbiter (Goal) -> Policy Constraint Layer -> Warden (Action)
+```
+
+### 2.1 Actuation Classes (Risk Tiers)
+All actions must be categorized:
+*   **CLASS 0 (Observe):** Data gathering only.
+*   **CLASS 1 (Log/Alert):** Informational only.
+*   **CLASS 2 (Throttle):** Resource constraints (e.g., process limit).
+*   **CLASS 3 (Local Isolate):** Local process/container isolation.
+*   **CLASS 4 (Cluster Isolate):** Distributed node isolation (Requires consensus).
+*   **CLASS 5 (Kernel Emergency):** Direct kernel intervention (Requires high-confidence quorum).
+
+### 2.2 Safety Guardrails & Telemetry Confidence
+*   **Telemetry Confidence Score (TCS):** Arbiter actions are gated by a calculated TCS (based on packet loss, drift, queue pressure, etc.). High-risk classes (3+) require TCS > threshold.
+*   **Actuation Budgeting:** Enforce hard limits on CPU, Network, Isolation, and AI inference budgets consumed by security actions to prevent self-DoS.
+*   **Simulation-before-Execution:** High-risk actions (Class 4+) must pass through a shadow simulation to predict impact before real-world execution.
+
+## 3. Failure Modes, Recovery & Distributed Coordination
+*   **Distributed Consensus:** Cross-node actions (Class 4+) require leader-validated quorum to prevent fragmentation and split-brain scenarios.
+*   **Recovery Semantics:** Reintegration requires validated evidence of safety (e.g., threat cleared, state stable) before moving back to `SAFE` state.
+*   **Communication Loss:** Fail-closed (maintain current state).
+
+## 4. Implementation Sequencing
+1.  Define Protocol Buffer definition (Class + Action + Constraints).
+2.  Implement Safety Layer (Validation logic + TCS check).
+3.  Implement Budgeting and Rate-limiting.
+4.  Integrate with Warden FSM (starting with SAFE -> WATCH -> SUSPICIOUS).
+
+## 5. Risks
+- Undefined failure modes leading to cluster-wide self-DoS.
+- Latency added to the Fast-Path by safety checks.

02_docs/rfc/RFC-011_telemetry_confidence.md

@@ -0,0 +1,21 @@
+# RFC-011: Telemetry Confidence Model (TCS)
+
+**Status:** Draft
+**Goal:** Define a mathematical model to quantify the fidelity of incoming telemetry streams, providing a "trust threshold" for autonomous control logic.
+
+## 1. Mathematical Model
+The Telemetry Confidence Score ($T_C$) is defined as:
+$$ T_C = w_1(1 - P_{loss}) + w_2(1 - J_{norm}) + w_3(E_{sig}) $$
+
+Weights:
+*   $w_1 = 0.4$ (Event Loss Sensitivity)
+*   $w_2 = 0.3$ (Jitter Sensitivity)
+*   $w_3 = 0.3$ (Structural Entropy)
+
+## 2. Thresholds
+*   **$T_C \ge 0.85$:** System `TRUSTED`. Autonomous actions authorized.
+*   **$0.60 \le T_C < 0.85$:** System `DEGRADED`. Arbiter restricted to Class 0-2 actions only.
+*   **$T_C < 0.60$:** System `UNTRUSTED`. Arbiter enters fail-safe mode; human intervention required.
+
+## 3. Implementation
+The TCS engine calculates these metrics over a sliding window of the last $N$ telemetry events.

02_docs/rfc/roadmap_control_loop_pipeline.md

@@ -0,0 +1,14 @@
+# Roadmap: Control Loop RFC Pipeline
+
+This pipeline establishes the necessary specifications for building a robust, cybernetic control system for PhoenixOS, prioritized by the Telemetry-First Core Executable Strategy.
+
+| RFC ID | Title | Objective | Phase |
+| :--- | :--- | :--- | :--- |
+| RFC-011 | Telemetry Confidence Model | Define TCS metrics and fidelity requirements. | 1 |
+| RFC-015 | Evidence Integrity Model | Specify SHA-256 ledger interactions. | 1 |
+| RFC-014 | Replay & Simulation Runtime | Detail shadow simulation architecture. | 1 |
+| RFC-010 | Arbiter-Warden Control Interface | Formalize safe actuation and control constraints. | 2 |
+| RFC-012 | Warden FSM Specification | Detail SAFE -> WATCH -> SUSPICIOUS FSM states. | 2 |
+| RFC-016 | Actuation Budget System | Define hard limits for safety-critical budget management. | 2 |
+| RFC-013 | Distributed Isolation Protocol | Define consensus and quorum for cluster actions. | 3 |
+| RFC-017 | Recovery & Reintegration Semantics | Specify trust-rebuilding and quarantine exit protocols. | 3 |

06_ai/agents/Arbiter_SI_Research.md

@@ -0,0 +1,23 @@
+# Research: Importance Scoring (SI) for Phoenix Arbiter
+
+## Objective
+Develop a mathematical framework to dynamically calculate the Importance Score ($S_I$) to differentiate between routine system noise and adversarial lateral movement.
+
+## Proposed Formula Framework: The Multiplicative Model
+$$ S_I = \min\left(1.0, \left( C_N \times G_S \right) \cdot e^{k \cdot F_A} \right) $$
+
+- **Baseline Threat ($C_N \times G_S$):** The inherent risk is the severity of the event multiplied by the value of the node.
+- **Compounding Factor ($e^{k \cdot F_A}$):** $F_A$ (Historical Frequency) acts as an exponential growth modifier, where $k$ is a tuning constant derived from the Phoenix Sentinel's thermodynamic entropy state.
+
+## Variable Mapping Table
+
+| Variable | Definition | Source in Phoenix OS |
+| --- | --- | --- |
+| **$C_N$** | Node Criticality (0.1 to 1.0) | Static YAML config injected via Cloud Control Plane. |
+| **$G_S$** | Cgroup Severity (0.1 to 1.0) | eBPF cgroup/connect4 hook payload + PID context. |
+| **$F_A$** | Historical Frequency (0.0 to 1.0) | Phoenix Monitor in-memory time-series window (e.g., last 60s). |
+
+## Next Steps
+1. Map out data sources for $C_N, F_A, G_S$ within the existing Telemetry/Trace framework.
+2. Define the normalization functions for each variable (0.0 to 1.0).
+3. Draft a simulation plan to test the $S_I$ sensitivity to synthetic attack patterns vs. normal operation.

ALGORITHM_MAP.md

@@ -0,0 +1,41 @@
+# PhoenixOS: Algorithm & Model Map
+
+This document maps the project's evolution stages to specific algorithms and mathematical models.
+
+## 1. Determinism & Evidence (Stages 1 & 5)
+*   **Canonical Serialization:** Sorted key ordering and stable float representation (e.g., Canonical JSON) to ensure identical hashes for identical data.
+*   **Merkle DAGs:** Structured evidence ledger allowing parallel event branching with a single root hash for system state verification.
+*   **Lamport Logical Clocks:** Causal event ordering replacing non-deterministic wall-clock time in replay traces.
+
+## 2. Signal Processing & Physics (Stage 10)
+*   **Welford’s Online Algorithm:** $O(1)$ memory, numerically stable running variance and standard deviation calculation for telemetry streams.
+*   **EWMA (Exponentially Weighted Moving Average):** Bounded, predictable smoothing of noisy telemetry signals.
+*   **Shannon Entropy:** $H(X) = -\sum P(x) \log_2 P(x)$ to detect packed malware or encrypted exfiltration.
+*   **Kalman Filters:** Real-time system-state estimation to isolate signals from background OS noise and detect telemetry spoofing.
+
+## 3. Constrained Control & Actuation (Stages 4 & 12)
+*   **PID Control (Proportional-Integral-Derivative):** Smooth, non-oscillatory throttling of system resources based on anomaly magnitude and persistence.
+*   **Token/Leaky Bucket:** Strict mathematical bounds on data egress and process execution rates during suspicious states.
+
+## 4. Graph & Causality Runtime (Stage 6)
+*   **Temporal Graph Traversal:** Time-respecting pathfinding ($t_1 < t_2$) to reconstruct valid attack chains.
+*   **Personalized PageRank:** Propagating risk scores through the process lineage and network DAG to identify blast radius.
+
+## 5. Strategic Defense & Game Theory (Stage 11)
+*   **Bayesian Inference:** Updating attacker profiles and intent probabilities as new telemetry evidence is replayed.
+*   **Stackelberg Security Games:** Solving for optimal defensive allocations (probes, decoys, isolation) against a rational, observing adversary.
+
+## Stage-Algorithm Alignment Matrix
+
+| Stage | Key Algorithms / Models |
+| :--- | :--- |
+| **1: Det. Replay** | Lamport Clocks, Canonical Serialization, Stable Hashing, Bounded FIFO |
+| **2: Real Telemetry** | eBPF-Ringbuf, EWMA Rate-Estimators, Little’s Law Queue Design |
+| **4: Actuation** | PID Control, Token Bucket, Warden FSM |
+| **5: Evidence** | Hash Chains, Merkle-DAGs, LTL Runtime Verification |
+| **6: Graphs** | DAG Traversal, BFS/DFS, PageRank-Centrality, Cycle Detection |
+| **7: Distributed** | Vector Clocks, CRDTs, Gossip/Consensus-lite |
+| **9: Advisory AI** | Isolation Forest, Autoencoders (Anomaly), Graph Embeddings |
+| **10: Physics** | Kalman Filters, Shannon Entropy, Welford's Algorithm |
+| **11: Game Theory** | Stackelberg Games, Bayesian Games, LP/IP Optimization |
+| **12: Cybernetic** | Closed-loop Bounded Policy, SMT-based Model Checking |

CHANGELOG_RESEARCH.md

@@ -0,0 +1,10 @@
+# CHANGELOG_RESEARCH
+
+This log tracks evolution and research synthesis across the documentation repository.
+
+## 2026-05-22
+- **Analyzed Files:** `GEMINI.md`, `README.md`, `PHOENIX_TASKS.md`, `docs/meta/repository_document_map.md`, `docs/meta/knowledge_graph.md`.
+- **Conflicts Detected:** None in core foundational files.
+- **New Concepts:** Mapped the 7-layer stack, Security Disorder Index (SDI), and established the knowledge graph structure.
+- **Architectural Evolution:** Initial audit of foundation layer complete.
+- **Unresolved Issues:** Arbiter/Warden interface clarification needed.