Phase 10: Polish, Documentation & Demo

CHANGELOG.md

@@ -0,0 +1,43 @@
+# Changelog
+
+## Control Plane — initial build
+
+Adds the `clawforge-controlplane` crate: the enterprise/government control plane
+for governing, securing, observing, auditing, and operating AI agents and MCP
+servers. Built in ten phases, each merged via its own pull request.
+
+### Added
+
+- **Foundation** — `ControlPlaneConfig`, shared vocabularies (`RiskLevel`,
+  `DataAccessLevel`, `LifecycleStatus`), unified `ControlPlaneError`, structured
+  logging macros.
+- **Agent Registry** — CRUD, validation, and an enforced lifecycle state
+  machine (no `Draft → Active` without approval).
+- **Governance Engine** — approval workflow with human gate, mandatory decision
+  reasons, and append-only change history.
+- **Observability** — append-only execution events with on-demand per-agent and
+  fleet-wide metric summaries.
+- **Security Gateway** — pre-execution checks (agent state, tool, MCP, model,
+  data access, capabilities, budget, human approval), risk scoring, and a
+  blocked-execution log.
+- **MCP Governance** — registry with approval, health, and usage tracking.
+- **Agent Marketplace** — verified, reusable templates with verification and
+  compliance badges; install into the registry.
+- **Enterprise Integrations** — governed connectors (DBs, SSO, GIS, ITSM, …)
+  with credential *references* (never secrets) and risk classification.
+- **Government Compliance Pack** — PII classification, retention, approval
+  chains, audit evidence, investigation holds, and compliance reporting
+  (UAE PDPL-aware).
+- **Docs & demo** — per-domain docs, Mermaid diagrams, government and enterprise
+  use cases, installation and developer guides, roadmap, limitations, security
+  disclaimer, and a runnable end-to-end example.
+
+### Tests
+
+`cargo test -p clawforge-controlplane` — **82 passing, 0 failing**.
+
+### Notes
+
+The control plane is a self-contained library crate. It is not yet wired into
+the live runtime or exposed over HTTP; see [docs/roadmap.md](docs/roadmap.md)
+and [docs/limitations.md](docs/limitations.md).

README.md

@@ -34,6 +34,31 @@ agents safely at scale.
 See [docs/product-positioning.md](docs/product-positioning.md) and
 [docs/architecture.md](docs/architecture.md) for the full picture.
 
+## The Control Plane
+
+The `clawforge-controlplane` crate adds the layers an organisation needs to run
+agents safely at scale. Each is a self-contained, SQLite-backed, fully tested
+domain module:
+
+| Capability | What it does | Docs |
+|------------|--------------|------|
+| **Agent Registry** | Single source of truth for every agent (owner, tools, MCP, data access, risk, lifecycle) | [registry.md](docs/registry.md) |
+| **Governance Engine** | Human approval workflow with department ownership, change history, and audit | [governance.md](docs/governance.md) |
+| **Observability** | Execution events → task/cost/latency/failure/risk metrics, per-agent and fleet-wide | [observability.md](docs/observability.md) |
+| **Security Gateway** | Pre-execution checks on every action (tool/MCP/model/data/budget/approval) + risk score | [security-gateway.md](docs/security-gateway.md) |
+| **MCP Governance** | Registry, approval, health, and usage tracking for MCP servers | [mcp-governance.md](docs/mcp-governance.md) |
+| **Agent Marketplace** | Verified, reusable internal agent templates with compliance badges | [marketplace.md](docs/marketplace.md) |
+| **Enterprise Integrations** | Governed connectors (DBs, SSO, GIS, ITSM) — credentials referenced, never stored | [enterprise-integrations.md](docs/enterprise-integrations.md) |
+| **Government Compliance** | PII classification, retention, approval chains, audit evidence, reporting (UAE PDPL-aware) | [government-compliance.md](docs/government-compliance.md) |
+
+See [docs/architecture.md](docs/architecture.md) for how these fit together, and
+[docs/use-cases.md](docs/use-cases.md) for end-to-end government and enterprise
+walkthroughs.
+
+> **ClawForge is an enterprise-grade AI agent control plane for governing,
+> securing, monitoring, auditing, and operating AI agents and MCP servers across
+> government and enterprise environments.**
+
 ## Install & Quick Start
 
 Requires **Rust ≥ 1.80** and **Node ≥ 20** (for frontend UI).

backend/controlplane/examples/demo.rs

@@ -0,0 +1,130 @@
+//! End-to-end control-plane demo.
+//!
+//! Run with:
+//!
+//! ```bash
+//! cargo run -p clawforge-controlplane --example demo
+//! ```
+//!
+//! Walks a single agent through the whole control plane: publish from the
+//! marketplace, register an MCP server, govern the agent through approval,
+//! check an action at the security gateway, record observability, and produce
+//! a compliance report — all in-memory.
+
+use clawforge_controlplane::compliance::{
+    ApprovalChain, CompliancePolicy, ComplianceReport, PiiClassification,
+};
+use clawforge_controlplane::constants::{DataAccessLevel, LifecycleStatus, RiskLevel};
+use clawforge_controlplane::gateway::{ActionRequest, SecurityGateway, SecurityPolicy};
+use clawforge_controlplane::governance::{ApprovalKind, GovernanceEngine, NewApprovalRequest};
+use clawforge_controlplane::marketplace::Marketplace;
+use clawforge_controlplane::mcp::{McpRegistry, McpTool, NewMcpServer, TransportType};
+use clawforge_controlplane::observability::{NewExecutionEvent, ObservabilityStore};
+use clawforge_controlplane::registry::AgentRegistry;
+
+fn main() -> anyhow::Result<()> {
+    println!("== ClawForge Control Plane demo ==\n");
+
+    // Stores (in-memory for the demo).
+    let registry = AgentRegistry::in_memory()?;
+    let governance = GovernanceEngine::in_memory()?;
+    let mcp = McpRegistry::in_memory()?;
+    let marketplace = Marketplace::in_memory()?;
+    let obs = ObservabilityStore::in_memory()?;
+
+    // 1. Marketplace → install a verified template into the registry.
+    let listings = clawforge_controlplane::marketplace::seed::seed(&marketplace)?;
+    let listing = &listings[0];
+    println!(
+        "1. Marketplace listing '{}' trusted={}",
+        listing.name,
+        listing.is_trusted()
+    );
+    let agent = marketplace.install(
+        &listing.id,
+        &registry,
+        "Permit Bot A",
+        "team-a",
+        "Licensing",
+    )?;
+    println!(
+        "   installed agent {} (status {:?})",
+        agent.name, agent.status
+    );
+
+    // 2. MCP governance → register + approve the server the agent needs.
+    let server = mcp.register(NewMcpServer {
+        name: "records-mcp".into(),
+        description: "Resident records".into(),
+        owner: "data-platform".into(),
+        endpoint: "https://mcp.internal/records".into(),
+        transport: TransportType::Http,
+        tools_exposed: vec![McpTool {
+            name: "lookup".into(),
+            description: "read records".into(),
+            permissions: vec!["read".into()],
+        }],
+        permissions_required: vec!["read".into()],
+        risk_level: RiskLevel::High,
+    })?;
+    mcp.approve(&server.id)?;
+    println!("2. MCP server '{}' approved", server.name);
+
+    // 3. Governance → approve the agent, then move it to Active.
+    let req = governance.submit(NewApprovalRequest {
+        kind: ApprovalKind::Agent,
+        subject_id: agent.id.clone(),
+        subject_name: agent.name.clone(),
+        requested_by: "team-a".into(),
+        department: "Licensing".into(),
+        risk_level: agent.risk_level,
+        justification: "Permit triage".into(),
+    })?;
+    governance.approve(&req.id, "ciso", "meets data-access policy")?;
+    registry.set_status(&agent.id, LifecycleStatus::PendingApproval)?;
+    let agent = registry.set_status(&agent.id, LifecycleStatus::Active)?;
+    println!("3. Governance approved; agent now {:?}", agent.status);
+
+    // 4. Security gateway → check an action before execution.
+    let gateway = SecurityGateway::new(SecurityPolicy::permissive());
+    let mut action = ActionRequest::for_agent(agent.clone());
+    action.tool = Some("search".into());
+    action.mcp_server = Some("records-mcp".into());
+    action.data_access_level = DataAccessLevel::Internal;
+    action.estimated_cost = 0.02;
+    let decision = gateway.evaluate(&action);
+    println!("4. Gateway decision: {}", decision.summary());
+
+    // 5. Observability → record the execution.
+    obs.log_event(NewExecutionEvent::task(
+        &agent.id,
+        decision.allowed,
+        120,
+        0.02,
+    ))?;
+    let metrics = obs.summary(Some(&agent.id))?;
+    println!(
+        "5. Observability: {} task(s), success rate {:.0}%",
+        metrics.task_count,
+        metrics.success_rate() * 100.0
+    );
+
+    // 6. Compliance → classify and report.
+    let mut policy = CompliancePolicy::pdpl(&agent.id);
+    policy.pii_classification = PiiClassification::Pii;
+    policy.data_retention_days = 365;
+    let chain = ApprovalChain::from_roles(&agent.id, &["data-owner", "dpo"]);
+    let report = ComplianceReport::generate(&policy, &[], Some(&chain));
+    println!(
+        "6. Compliance: framework {}, compliant={} ({} finding(s))",
+        report.framework,
+        report.is_compliant(),
+        report.findings.len()
+    );
+    for f in &report.findings {
+        println!("     - {f}");
+    }
+
+    println!("\n== demo complete ==");
+    Ok(())
+}

backend/controlplane/src/compliance/model.rs

@@ -181,7 +181,9 @@ impl CompliancePolicy {
     /// Indefinite retention (`0`) and active investigation holds are never past
     /// due (a legal hold overrides routine deletion).
     pub fn is_past_retention(&self, age_days: u32) -> bool {
-        !self.investigation_mode && self.data_retention_days != 0 && age_days > self.data_retention_days
+        !self.investigation_mode
+            && self.data_retention_days != 0
+            && age_days > self.data_retention_days
     }
 
     /// A baseline UAE PDPL policy for a subject.

backend/controlplane/src/compliance/report.rs

@@ -8,7 +8,9 @@
 use chrono::Utc;
 use serde::{Deserialize, Serialize};
 
-use super::model::{ApprovalChain, AuditEvidence, CompliancePolicy, ExportControl, PiiClassification};
+use super::model::{
+    ApprovalChain, AuditEvidence, CompliancePolicy, ExportControl, PiiClassification,
+};
 
 /// A point-in-time compliance assessment for a single subject.
 #[derive(Debug, Clone, Serialize, Deserialize)]
@@ -53,7 +55,10 @@ impl ComplianceReport {
         if !approval_complete {
             findings.push("approval chain is incomplete".into());
         }
-        if matches!(policy.export_control, ExportControl::Prohibited) && signed == 0 && !evidence.is_empty() {
+        if matches!(policy.export_control, ExportControl::Prohibited)
+            && signed == 0
+            && !evidence.is_empty()
+        {
             findings.push("export-prohibited data lacks signed evidence".into());
         }
 
@@ -102,7 +107,11 @@ impl DepartmentComplianceSummary {
         let under_investigation = reports.iter().filter(|r| r.investigation_mode).count();
         let mut findings: Vec<String> = reports
             .iter()
-            .flat_map(|r| r.findings.iter().map(|f| format!("{}: {}", r.subject_id, f)))
+            .flat_map(|r| {
+                r.findings
+                    .iter()
+                    .map(|f| format!("{}: {}", r.subject_id, f))
+            })
             .collect();
         findings.sort();
         DepartmentComplianceSummary {
@@ -143,7 +152,10 @@ mod tests {
         policy.pii_classification = PiiClassification::Pii;
         let report = ComplianceReport::generate(&policy, &[], None);
         assert!(!report.is_compliant());
-        assert!(report.findings.iter().any(|f| f.contains("no audit evidence")));
+        assert!(report
+            .findings
+            .iter()
+            .any(|f| f.contains("no audit evidence")));
     }
 
     #[test]

backend/controlplane/src/config.rs

@@ -63,7 +63,10 @@ impl ControlPlaneConfig {
 }
 
 fn parse_bool(value: &str) -> bool {
-    matches!(value.trim().to_ascii_lowercase().as_str(), "1" | "true" | "yes" | "on")
+    matches!(
+        value.trim().to_ascii_lowercase().as_str(),
+        "1" | "true" | "yes" | "on"
+    )
 }
 
 #[cfg(test)]

backend/controlplane/src/error.rs

@@ -40,7 +40,10 @@ pub enum ControlPlaneError {
 impl ControlPlaneError {
     /// Build a [`ControlPlaneError::NotFound`] for the given entity and id.
     pub fn not_found(entity: &'static str, id: impl Into<String>) -> Self {
-        ControlPlaneError::NotFound { entity, id: id.into() }
+        ControlPlaneError::NotFound {
+            entity,
+            id: id.into(),
+        }
     }
 
     /// Build a validation error.

backend/controlplane/src/gateway/decision.rs

@@ -47,7 +47,11 @@ impl SecurityDecision {
     /// One-line, human-readable verdict suitable for logs and API responses.
     pub fn summary(&self) -> String {
         if self.allowed {
-            format!("ALLOW (risk: {}, score: {})", self.risk_band(), self.risk_score)
+            format!(
+                "ALLOW (risk: {}, score: {})",
+                self.risk_band(),
+                self.risk_score
+            )
         } else {
             format!(
                 "DENY (risk: {}, score: {}) — {}",

backend/controlplane/src/gateway/engine.rs

@@ -59,7 +59,9 @@ impl SecurityGateway {
     fn check_mcp(&self, req: &ActionRequest, denials: &mut Vec<String>) {
         if let Some(server) = &req.mcp_server {
             if !req.agent.mcp_servers_allowed.iter().any(|s| s == server) {
-                denials.push(format!("MCP server '{server}' is not allowed for this agent"));
+                denials.push(format!(
+                    "MCP server '{server}' is not allowed for this agent"
+                ));
             }
         }
     }
@@ -263,7 +265,10 @@ mod tests {
         req.model = Some("gpt-4".into());
         let d = gw.evaluate(&req);
         assert!(!d.allowed);
-        assert!(d.denials.iter().any(|r| r.contains("MCP server 'rogue-mcp'")));
+        assert!(d
+            .denials
+            .iter()
+            .any(|r| r.contains("MCP server 'rogue-mcp'")));
         assert!(d.denials.iter().any(|r| r.contains("model 'gpt-4'")));
     }
 
@@ -274,7 +279,10 @@ mod tests {
         req.data_access_level = DataAccessLevel::Restricted; // agent clearance is Internal
         let d = gw.evaluate(&req);
         assert!(!d.allowed);
-        assert!(d.denials.iter().any(|r| r.contains("exceeds agent clearance")));
+        assert!(d
+            .denials
+            .iter()
+            .any(|r| r.contains("exceeds agent clearance")));
     }
 
     #[test]
@@ -306,7 +314,10 @@ mod tests {
         let mut req = ActionRequest::for_agent(a);
         req.tool = Some("search".into());
         let d = gw.evaluate(&req);
-        assert!(d.denials.iter().any(|r| r.contains("human approval required")));
+        assert!(d
+            .denials
+            .iter()
+            .any(|r| r.contains("human approval required")));
     }
 
     #[test]

backend/controlplane/src/gateway/log.rs

@@ -46,19 +46,27 @@ impl BlockedExecutionLog {
     pub fn open(path: &str) -> Result<Self> {
         let conn = Connection::open(path)?;
         conn.execute_batch(&format!("PRAGMA journal_mode=WAL;{SCHEMA}"))?;
-        Ok(Self { conn: Mutex::new(conn) })
+        Ok(Self {
+            conn: Mutex::new(conn),
+        })
     }
 
     /// Open an ephemeral in-memory log (used by tests).
     pub fn in_memory() -> Result<Self> {
         let conn = Connection::open_in_memory()?;
         conn.execute_batch(SCHEMA)?;
-        Ok(Self { conn: Mutex::new(conn) })
+        Ok(Self {
+            conn: Mutex::new(conn),
+        })
     }
 
     /// Record a denied decision. No-op (returns `None`) if the decision was
     /// actually allowed, so callers can pass every decision unconditionally.
-    pub fn record(&self, agent_id: &str, decision: &SecurityDecision) -> Result<Option<BlockedExecution>> {
+    pub fn record(
+        &self,
+        agent_id: &str,
+        decision: &SecurityDecision,
+    ) -> Result<Option<BlockedExecution>> {
         if decision.allowed {
             return Ok(None);
         }
@@ -73,7 +81,13 @@ impl BlockedExecutionLog {
         conn.execute(
             "INSERT INTO blocked_executions (id, agent_id, reasons, risk_score, at)
              VALUES (?1,?2,?3,?4,?5)",
-            params![entry.id, entry.agent_id, entry.reasons, entry.risk_score, entry.at],
+            params![
+                entry.id,
+                entry.agent_id,
+                entry.reasons,
+                entry.risk_score,
+                entry.at
+            ],
         )?;
         cp_blocked!("gateway.blocked", agent_id = %entry.agent_id, reasons = %entry.reasons);
         Ok(Some(entry))