diff --git a/API_REFERENCE.md b/API_REFERENCE.md index bcf354d..ba5af61 100644 --- a/API_REFERENCE.md +++ b/API_REFERENCE.md @@ -2,12 +2,12 @@ The UDIE API is an authoritative HTTP contract exposed by the NestJS backend on `/api/v1`. -## ๐ŸŒ Base URL +## Base URL - **Local Development**: `http://localhost:3000/api/v1` -- **Prefix Discovery**: Client-side logic in `APIClient.swift` dynamically probes `/api/v1` and `/api` to handle version drift. +- **Prefix Discovery**: `APIClient.swift` dynamically probes `/api/v1` and `/api` to handle version drift. -## ๐Ÿฅ Health & Readiness +## Health & Readiness ### `GET /health/live` Confirms the process is running. Returns `200 OK`. @@ -20,7 +20,7 @@ Confirms the system is ready to serve queries. Reflects: --- -## ๐Ÿ“ Events +## Events ### `GET /events` Returns disruptions within a bounding box. @@ -33,7 +33,7 @@ Returns disruptions within a bounding box. --- -## ๐ŸŽ Route Risk +## Route Risk ### `POST /risk` Computes the risk score for a polyline. @@ -50,13 +50,13 @@ Computes the risk score for a polyline. ``` **Response Fields:** -- `riskScore`: Saturated score $[0, 1)$. -- `riskLevel`: Categorical rating (LOW, MEDIUM, HIGH, CRITICAL). +- `riskScore`: Saturated score in `[0, 1)`. +- `riskLevel`: Categorical rating (`LOW`, `MEDIUM`, `HIGH`, `CRITICAL`). - `explanation`: Human-readable summary of contributing disruptions. --- -## ๐Ÿ“Š Dashboard & Intelligence +## Dashboard & Intelligence ### `GET /city-dashboard` Aggregated metrics (hotspots, event counts, trend gradients) for city-level visualization. @@ -66,8 +66,8 @@ Cell-level intelligence including historical weight and current decay state. --- -## โš™๏ธ Constraints +## Constraints - **Bounded Queries**: Geographic queries are limited to a maximum bounding box size (default 0.5 degยฒ). - **Contract-Valid Parameters**: Required query parameters must be provided; empty/default values result in `400 Bad Request`. -- **Deterministic Latency**: Evaluation endpoints are optimized for sub-5ms responses via in-memory grid lookups. +- **Deterministic Latency**: Evaluation endpoints target sub-5ms responses via in-memory grid lookups. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index e2d0de6..77ca3a2 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,17 +1,17 @@ # Architecture Reference -UDIE follows a **Log-Derived Pattern**. The system distinguishes strictly between the immutable source of truth and versioned projections. +UDIE follows a log-derived pattern. The system distinguishes strictly between the immutable source of truth and versioned projections. -## ๐Ÿ— System Philosophy +## System Philosophy -- **Events Log = Authoritative State**: Every derived grid or view can be replayed from the event log. -- **Bounded Request Logic**: Request-time logic is O(route\_cells), never O(historical\_events). -- **Spatial Sovereignty**: The backend owns all intelligence; clients are thin observers. +- **Events log = authoritative state.** Every derived grid or view can be replayed from the event log. +- **Bounded request logic.** Request-time logic is O(route\_cells), never O(historical\_events). +- **Spatial sovereignty.** The backend owns all intelligence; clients are thin observers. -## ๐Ÿงฑ Core Subsystems +## Core Subsystems ### 1. Ingestion Layer -Normalizes signals from REST, WebSocket, and Kafka. Signals are validated against geographic bounds and appended to the `events_log`. +Normalizes signals from REST and WebSocket transports. Signals are validated against geographic bounds and appended to the `events_log`. ### 2. Spatial Compute Engine Uses Uber H3 resolution 9 as the primary indexing unit. Aggregates event weights into decayed risk scores across a materialized `risk_cells` surface. @@ -21,19 +21,16 @@ Uses Uber H3 resolution 9 as the primary indexing unit. Aggregates event weights - **Lifecycle Worker**: Manages temporal decay and event expiration. - **Snapshot Worker**: Captures periodic global grid states for analysis. -## ๐Ÿšฆ Architecture Decisions (ADRs) +## Architecture Decision Records -- **ADR-001**: Event-Sourced Spatial Compute as the sole source of truth. -- **ADR-002**: H3 Standard for discretized spatial addressing. -- **ADR-003**: Precomputed Risk Surface to ensure sub-millisecond hot-path lookups. -- **ADR-004**: Thin Clients to centralize intelligence authority. -- **ADR-005**: Environment-Aware Base URL resolution for simulator vs physical device stability. +- **ADR-001**: Event-sourced spatial compute as the sole source of truth. +- **ADR-002**: H3 standard for discretized spatial addressing. +- **ADR-003**: Precomputed risk surface to ensure sub-millisecond hot-path lookups. +- **ADR-004**: Thin clients to centralize intelligence authority. +- **ADR-005**: Environment-aware base URL resolution for simulator vs physical device stability. -## โš–๏ธ Architectural Laws +## Architectural Constraints -All components must adhere to the **Laws of UDIE**, including: -- **Law of Deterministic Rebuild**: $\text{state} = f(\text{logs})$. -- **Law of Hot Path Isolation**: Evaluation cost $\perp$ event count. -- **Law of Derived-Status Purity**: No manual mutation of materialized tables. - -Refer to [SYSTEM_DESIGN.md](file:///Users/fallofpheonix/Project/UDIE/SYSTEM_DESIGN.md) for detailed mathematical and theoretical foundations. +- **Deterministic rebuild**: `state = f(logs)`. All derived state must be reproducible from the event log alone. +- **Hot path isolation**: Evaluation cost must be independent of total event volume. +- **Derived-state purity**: No manual mutation of materialized tables. diff --git a/BACKEND_ARCHITECTURE.md b/BACKEND_ARCHITECTURE.md deleted file mode 100644 index 87e8246..0000000 --- a/BACKEND_ARCHITECTURE.md +++ /dev/null @@ -1,38 +0,0 @@ -# Backend Architecture - -The backend substrate is the authoritative source of intelligence for UDIE, handling spatial compute, event persistence, and prediction. - -## ๐Ÿ— Subsystems - -### 1. Spatial Substrate (NestJS) -- **Location**: `/engine-backend` -- **Domain Modules**: - - `ingestion`: Signal capture, normalization, and deduplication. - - `events`: Management of the immutable `events_log`. - - `risk`: Implementation of spatial kernels and risk aggregation. - - `spatial`: PostGIS and H3 indexing services. -- **Database**: PostgreSQL with PostGIS, Partitioned by `observed_at` and `region_id`. - -### 2. Prediction Engine (Python) -- **Location**: `/udie_backend_py` -- **Role**: Handles heavy-lift spatial utilities, forecasting kernels, and T+X risk projections. -- **Stack**: FastAPI, GeoPandas, H3-Py. - -## โš™๏ธ Operational Flow - -1. **Ingestion**: Raw disruptions enter via REST/WS. -2. **Persistence**: Events are committed to the immutable `events_log`. -3. **Projection**: Background workers materialize risk into `risk_cells` and Redis caches. -4. **Evaluation**: API handlers query the materialized hot-path for sub-millisecond route risk analysis. - -## โš–๏ธ Backend Laws - -- **Law of Deterministic Rebuild**: All state must be reproducible from the event log. -- **Law of Hot-Path Isolation**: Evaluation cost must be independent of total event volume. -- **Law of Memory Residency**: Hot-path evaluation data must reside in RAM (Redis). - -## ๐Ÿ›  Observability & Health - -- **Heartbeat Service**: Monitors worker lag and materialization freshness. -- **Architecture Audit Service**: Continuously verifies schema integrity and query plan safety. -- **Diagnostics**: Specialized endpoints (`/diagnostics/architecture`) surface data-plane degradations. diff --git a/CODEBASE_STRUCTURE.md b/CODEBASE_STRUCTURE.md deleted file mode 100644 index 59475c3..0000000 --- a/CODEBASE_STRUCTURE.md +++ /dev/null @@ -1,30 +0,0 @@ -# Codebase Structure - -The UDIE repository is organized to decouple spatial compute, client interfaces, and system contracts. - -## ๐ŸŒฒ Technical Hierarchy - -- **`engine-backend/`** (NestJS): The core spatial substrate. - - `src/modules/ingestion`: Signal capture and normalization. - - `src/modules/risk`: Field weight and kernel evaluation. - - `src/modules/spatial`: H3 indexing and PostGIS geometry. - - `src/database`: Schema definitions, migrations, and spatial views. -- **`udie_backend_py/`**: Python-based spatial utilities and prediction kernels. -- **`UDIE/`** (Swift): Native iOS Client featuring map intelligence and sync state management. -- **`udie_mobile/`** (Flutter): Cross-platform mobile client for Android/iOS. -- **`dashboard-admin/`**: Web-based interface for city operations and monitoring. -- **`infra/`**: Global orchestration, Docker configurations, and monitoring (Prometheus/Grafana). -- **`docs/`**: (DEPRECATED - Replaced by canonical root documentation). - -## โš–๏ธ Ownership Rules - -1. **Backend Authority**: All spatial risk logic resides in the backend. Clients must never reimplement risk kernels. -2. **State Isolation**: UI state (syncing, connectivity) must reside in core state machines, not within views. -3. **Module Decoupling**: Core modules must not depend on feature-level implementations. -4. **Contract-First**: A feature is only complete when its API contract and diagnostic endpoints are verified. - -## ๐Ÿšซ Forbidden Patterns - -- **Absolute Paths**: Machine-specific paths in scripts or configs are prohibited. -- **Raw SQL Scans**: All requests must query derived surfaces/repos, never the raw event log. -- **Unverified Connectivity**: Clients must distinguish between Transport, Contract, and Data-Plane failures. diff --git a/DATASETS.md b/DATASETS.md deleted file mode 100644 index 0383fdd..0000000 --- a/DATASETS.md +++ /dev/null @@ -1,31 +0,0 @@ -# Datasets & Data Schema - -UDIE operates on an **Event-Sourced Substrate**. There are no static datasets in the conventional sense; the state is a projection of high-volume event logs. - -## ๐Ÿงฑ Authoritative Schema (`events_log`) - -All system intelligence is derived from the immutable `events_log`: - -| Field | Type | Description | -| :--- | :--- | :--- | -| `id` | `UUID` | Unique event identifier. | -| `coordinate` | `Geography` | WGS84 Point. | -| `h3_index` | `BIGINT` | H3 resolution 9 cell index. | -| `observed_at` | `Timestamp` | Time of occurrence. | -| `weight` | `Float` | Initial impact weight. | - -## โšก Materialized Projections - -- **`risk_cells`**: Aggregated, decayed risk field indexed by H3. -- **`regional_geo_events_v`**: Spatially partitioned view for viewport-bounded event fetching. -- **`risk_snapshots`**: Periodic captures of the global cell state for historical analysis. - -## ๐Ÿงช Simulation Data - -Simulation data is stored in `simulation_events` to prevent contamination of production metrics. This data is ingested via the `/events` endpoint when the `ENABLE_SIMULATION` flag is active. - -## ๐Ÿ“ Data Contracts (STI) - -The **Standard Tool Interface (STI)** defines how agents and external sensors submit signals. -- **Ingestion Contract**: Requires `lat`, `lng`, `type`, and `initial_weight`. -- **Validation**: Signals are rejected if they fall outside bounded operational zones (e.g., city limits). diff --git a/DATA_PIPELINE.md b/DATA_PIPELINE.md deleted file mode 100644 index f7caeb6..0000000 --- a/DATA_PIPELINE.md +++ /dev/null @@ -1,29 +0,0 @@ -# Data Pipeline - -The UDIE Data Pipeline is a stream-processing architecture optimized for real-time spatial telemetry. - -## ๐ŸŒŠ Pipeline Overview - -1. **Ingestion Layer**: REST API and WebSocket listeners capture raw disruption signals. -2. **Normalization**: Signals are validated against JSON schemas and mapped to H3 resolution 9 cells. -3. **Authoritative Log**: Normalized events are appended to the immutable `events_log`. -4. **Projection Workers**: Async workers detect log appends and refresh the `risk_cells` surface using spatial diffusion kernels. -5. **Lifecycle Maintenance**: Temporal decay and signal expiration are applied periodically to maintain field freshness. -6. **Hot-Path Delivery**: Live cell scores are streamed via WebSockets or exposed via $O(1)$ API lookups from Redis. - -## โš™๏ธ Ingestion Components - -- **Validation Engine**: Enforces geographic bounds (Law of Bounded Input). -- **Deduplicator**: Prevents duplicate ingestion of sensor spikes using Bloom filters. -- **Weighting Filter**: Assigns Severity and Confidence scores based on source reliability. - -## ๐Ÿ“Š Processing Kernels - -- **Spatial Diffusion**: Influence of a disruption decays according to a distance-weighted kernel (Law of Spatial Locality). -- **Temporal Decay**: Signal strength decays exponentially in the absence of reinforcement (Law of Temporal Dissipation). -- **Saturation**: Final risk values are normalized to $[0, 1)$ to prevent scale-out. - -## โฑ Performance Invariants - -- **End-to-End Latency**: < 5 seconds from sensor ingestion to UI render. -- **Aggregation Throughput**: Millions of spatial updates processed per hour via optimized PG/PostGIS kernels. diff --git a/FRONTEND_ARCHITECTURE.md b/FRONTEND_ARCHITECTURE.md deleted file mode 100644 index f90669e..0000000 --- a/FRONTEND_ARCHITECTURE.md +++ /dev/null @@ -1,33 +0,0 @@ -# Frontend Architecture - -UDIE features a multi-platform operational interface designed for real-time spatial visibility and thin-client execution. - -## ๐Ÿ“ฑ Mobile Clients - -### 1. Native iOS (Swift/SwiftUI) -- **Repo Location**: `/UDIE` -- **Core Principle**: Centralized state management via `MapViewModel`. -- **Sync Engine**: Implements the `BackendSyncState` machine (`connecting`, `syncing`, `synced`, `error`). -- **Resilience**: Features dynamic API prefix resolution and LAN IP fallback for physical device testing. - -### 2. Cross-Platform Mobile (Flutter) -- **Repo Location**: `/udie_mobile` -- **Features**: Radius-based event fetching, area news filtering (construction/safety/VIP), and route risk evaluation. -- **Routing**: Clean separation between UI layers and the `APIClient` substrate. - -## ๐Ÿ“Š Admin Dashboard (React/NextJS) -- **Repo Location**: `/dashboard-admin` -- **Focus**: City-level operations, snapshot inspection, and system health monitoring. -- **Visuals**: Leverages H3-indexed heatmaps and real-time WebSocket streams for disruption visualization. - -## ๐ŸŽจ Interface System Rules - -- **Map-First Experience**: Actionable spatial intelligence must be the primary view. -- **Explicit State**: Connectivity, freshness, and sync status must always be visible and never implied. -- **Design Consistency**: Severity, confidence, and risk colors must remain consistent across all platforms. - -## ๐Ÿšซ Anti-Patterns - -- **Local Intelligence**: Frontend must never calculate risk scores; it is strictly an observer. -- **Deep Navigation**: UI hierarchy must remain shallow to support high-pressure operational use. -- **Raw-Event Bloat**: Clients should only fetch viewport-bounded data to maintain performance. diff --git a/INSTALLATION.md b/INSTALLATION.md index 3fc68df..05f5fbd 100644 --- a/INSTALLATION.md +++ b/INSTALLATION.md @@ -2,40 +2,39 @@ Follow these steps to set up the UDIE ecosystem for local development and verification. -## ๐Ÿ“ฆ Prerequisites +## Prerequisites -- **Docker & Docker Compose** -- **Node.js v18+ & npm** -- **Python 3.10+** (pyenv recommended) -- **Flutter SDK** (for mobile) -- **PostgreSQL / PostGIS** (if running outside Docker) +- Docker & Docker Compose +- Node.js v18+ & npm +- Python 3.10+ (pyenv recommended) +- Flutter SDK (for mobile) +- PostgreSQL / PostGIS (if running outside Docker) -## ๐Ÿš€ 1. Backend Substrate (NestJS) +## 1. Backend (NestJS) ```bash cd engine-backend npm install -# Configure .env (see CONFIGURATION.md) +cp .env.example .env # edit as needed npm run db:migrate npm run dev ``` -## ๐Ÿ 2. Spatial Metrics Backend (Python) +## 2. Spatial Metrics Backend (Python) ```bash cd udie_backend_py python -m venv .venv source .venv/bin/activate pip install -r requirements.txt -python main.py +uvicorn app.main:app --reload ``` -## ๐Ÿ“ฑ 3. Mobile Clients +## 3. Mobile Clients ### iOS (Swift) ```bash -cd UDIE -open UDIE.xcodeproj +open UDIE/UDIE.xcodeproj # Select Simulator/Device and Run ``` @@ -46,19 +45,15 @@ flutter pub get flutter run ``` -## ๐Ÿ“Š 4. Admin Dashboard +## 4. Admin Dashboard -```bash -cd dashboard-admin -npm install -npm run dev -``` +Open `dashboard-admin/index.html` directly in a browser or serve with any static file server. -## ๐Ÿณ 5. Full Stack (Docker) +## 5. Full Stack (Docker) ```bash cd infra -docker-compose up --build +docker compose up --build ``` -Refer to [DEPLOYMENT_GUIDE.md](file:///Users/fallofpheonix/Project/UDIE/DEPLOYMENT_GUIDE.md) for production environment setup. +See [CONFIGURATION.md](./CONFIGURATION.md) for environment variable reference. diff --git a/MODEL_ARCHITECTURE.md b/MODEL_ARCHITECTURE.md deleted file mode 100644 index 371cac2..0000000 --- a/MODEL_ARCHITECTURE.md +++ /dev/null @@ -1,34 +0,0 @@ -# Model Architecture - -UDIE uses a **Deterministic Spatial Model** to represent urban risks. It avoids stochastic black-box models in favor of verifiable spatiotemporal fields. - -## ๐Ÿ“ Spatial Intelligence Model (SIM) - -The core model treats disruption as a scalar field $R(s, t)$ defined over the H3 grid. - -### 1. Discretization -Geography is discretized using the H3 Hierarchical Hexagonal Indexing system. -- **Resolution 9**: Primary cell size for risk evaluation. -- **Resolution 6**: Primary cell size for geographic sharding and partitioning. - -### 2. Risk Evaluation Function -The risk at any point is the sum of weighted influences from proximate events: -$$R_{total} = \sum_{i \in \text{Events}} w_i \cdot K(d_i) \cdot e^{-(t-t_i)/\tau}$$ -Where $K(d)$ is the spatial kernel and $\tau$ is the temporal decay constant. - -### 3. Saturation Kernel -To ensure system stability, raw risk is saturated: -$$R_{saturated} = 1 - e^{-R_{total} / k}$$ - -## ๐Ÿ”ฎ Predictive Forecasting - -Future risk ($T+X$) is projected by analyzing: -- **Diffusion Vectors**: How disruption energy is currently spreading between adjacent cells. -- **Historical Recurrence**: High-score periods observed in saved `risk_snapshots`. -- **Trend Gradients**: Recent growth or decay rates in cell weights. - -## โš–๏ธ Stability Requirements - -- **Convergence**: Replay from the same event log must always yield the same risk surface. -- **Consistency**: Adjacent partitions (Res 6) must maintain continuous fields across boundaries. -- **Verifiability**: Every risk score must be explainable via the contributing events and decay parameters. diff --git a/PERFORMANCE_AND_SCALING.md b/PERFORMANCE_AND_SCALING.md deleted file mode 100644 index ed4ce7e..0000000 --- a/PERFORMANCE_AND_SCALING.md +++ /dev/null @@ -1,29 +0,0 @@ -# Performance & Scaling - -UDIE is optimized for real-time spatial evaluation at nationwide scale with deterministic latency. - -## ๐ŸŽ Performance Targets (SLA) - -| Operation | P99 Target | Law Enforced | Mechanism | -| :--- | :--- | :--- | :--- | -| **Cell Risk Lookup** | < 0.1 ms | Law 8 | Redis in-memory lookup. | -| **Route Evaluation** | < 5.0 ms | Law 2 | Sampled route-cell lookup. | -| **Grid Materialization** | < 500 ms | Law 1 | Async projection workers. | -| **State Rebuild** | < 30 s | Law 1 | PG/PostGIS optimized replay. | - -## ๐Ÿ“ˆ Scaling Strategies - -### 1. Horizontal Spatial Sharding -The system scales by partitioning the globe into **H3 Resolution 6** shards. Each shard is an independent compute unit, allowing linear horizontal scaling without global lock contention. - -### 2. Execution Isolation -- **Read Shards**: Dedicated to high-concurrency API evaluation using mirrored in-memory caches. -- **Write Shards**: Dedicated to ingestion normalization and authoritative log commits. - -### 3. Query Optimization -Request-time logic is strictly constrained to prevent "Request-time aggregation" (Law of Hot Path Isolation). Complex spatial joins are performed during materialization, not during API requests. - -## ๐Ÿ“‰ Known Bottlenecks - -- **High-Density Saturation**: Extremely dense event clusters can increase materialization lag; mitigated by nonlinear saturation functions. -- **Cold Boot Latency**: Large regions require log replay to prime in-memory caches upon node startup. diff --git a/README.md b/README.md index 094ce77..9021542 100644 --- a/README.md +++ b/README.md @@ -1,29 +1,104 @@ -# UDIE: Universal Disruption Intelligence Engine +# UDIE โ€” Urban Disruption Intelligence Engine -UDIE is a spatial intelligence system designed to convert volatile, multi-source urban disruption signals into a stable operational risk view. It leverages H3 spatial indexing, event-sourced persistence, and deterministic materialization to provide high-performance route risk evaluation and city-level intelligence. +UDIE converts raw, multi-source urban disruption signals into a stable, materialized spatial risk surface. Rather than treating events as simple map annotations, it accumulates them as weighted energy in an H3 hexagonal grid, subject to spatial diffusion and temporal decay. The result is a backend that answers "what is the current risk on this route?" in under 5ms, regardless of how many raw events exist. -## ๐Ÿš€ Quick Links +## Architecture -- [PROJECT_OVERVIEW.md](./PROJECT_OVERVIEW.md) - Goals, use cases, and requirements. -- [ARCHITECTURE.md](./ARCHITECTURE.md) - System philosophy and core subsystems. -- [INSTALLATION.md](./INSTALLATION.md) - Setup guides for backend, mobile, and dashboard. -- [DEVELOPER_GUIDE.md](./DEVELOPER_GUIDE.md) - Engineering playbook and diagnostic protocols. -- [API_REFERENCE.md](./API_REFERENCE.md) - Backend HTTP contract. +The system is log-derived: the `events_log` table is the only authoritative state. Every other table (`risk_cells`, projections, snapshots) is a deterministic function of that log. This means the entire risk surface can be rebuilt from scratch by replaying eventsโ€”useful for schema migrations, disaster recovery, and test reproducibility. -## ๐ŸŒฒ Core Subsystems +**Components:** -1. **Ingestion Substrate**: Normalizes and appends raw signals to the authoritative event log. -2. **Spatial Compute**: Performs H3-indexed aggregation and risk field evaluation. -3. **Projections & Workers**: Materializes derived states (risk cells, hotspots) for high-performance querying. -4. **Operational Interface**: Thin mobile and web clients for real-time visualization. +| Directory | Role | +|---|---| +| `engine-backend/` | NestJS API. Ingestion, spatial compute, routing, simulation. | +| `udie_backend_py/` | FastAPI service. Aggregates open government data (NDMA). | +| `UDIE/` | Native iOS app (SwiftUI). Real-time disruption map. | +| `udie_mobile/` | Flutter app (cross-platform). Radius-based risk view. | +| `dashboard-admin/` | Static web dashboard. City ops and system health. | +| `infra/` | Docker Compose, Kubernetes, Prometheus/Grafana. | -## ๐Ÿ›  Tech Stack +**Key design decisions:** -- **Backend**: NestJS (TypeScript), Python (Spatial Utils). -- **Persistence**: PostgreSQL + PostGIS (Authoritative), Redis (Hot-path caching). -- **Mobile**: Swift (iOS Native), Flutter (Cross-platform). -- **Indexing**: H3 (Uber's Hexagonal Hierarchical Spatial Index). +- **No ORM.** Direct `pg` queries against a known schema. Avoids the impedance mismatch when working with PostGIS and H3 types. +- **Role-based nodes.** A single binary supports `INGESTION`, `MATERIALIZATION`, or `EVALUATION` roles via `NODE_ROLE` env var. Scales each concern independently. +- **Precomputed risk surface.** Route evaluation reads from Redis-cached `risk_cells`, not the event log. Evaluation cost is O(route cells), not O(events). +- **H3 resolution 9.** ~174mยฒ cellsโ€”fine enough for intersection-level accuracy, coarse enough for efficient neighbor operations. -## โš–๏ธ License +## Setup -MIT ยฉ 2026 **UDIE Engineering Group**. "Stability is the foundation of intelligence." +**Prerequisites:** Docker, Node.js 18+, Python 3.10+ + +### Backend (NestJS) + +```bash +cd engine-backend +npm install +cp .env.example .env # edit DATABASE_URL, REDIS_URL as needed +npm run db:migrate +npm run dev +``` + +### Python spatial service + +```bash +cd udie_backend_py +python -m venv .venv && source .venv/bin/activate +pip install -r requirements.txt +uvicorn app.main:app --reload +``` + +### Full stack (Docker) + +```bash +cd infra +docker compose up --build +``` + +This brings up Postgres (with PostGIS), Redis, both backend services, Prometheus, and Grafana. + +### Mobile + +```bash +# iOS +open UDIE/UDIE.xcodeproj # build and run in Xcode + +# Flutter +cd udie_mobile && flutter pub get && flutter run +``` + +> **Physical device note:** iOS simulators can reach `localhost`. Physical devices cannot. Set `UDIE_BACKEND_URL` in your Xcode scheme to your host's LAN IP (e.g. `http://192.168.1.x:3000`). + +## Key endpoints + +See [API_REFERENCE.md](./API_REFERENCE.md) for the full contract. + +``` +GET /api/v1/health/ready โ€” readiness (DB + risk surface freshness) +GET /api/v1/events โ€” events within a bounding box +POST /api/v1/risk โ€” route risk score for a coordinate polyline +GET /api/v1/city-dashboard โ€” aggregated city-level metrics +GET /api/v1/cell-insight โ€” per-cell risk history and decay state +``` + +## Running tests + +```bash +cd engine-backend +npm test # all tests +npm run test:risk # risk kernel only +npm run verify:architecture # enforces no-ORM, append-only log, and other invariants +``` + +## Rebuilding the risk surface + +If `risk_cells` is empty or stale (common after a fresh migration), trigger a replay: + +```bash +npm run validate:rebuild +``` + +The readiness endpoint (`GET /health/ready`) reflects surface freshness and will show `degraded` until materialization completes. + +## License + +MIT ยฉ 2026 UDIE Engineering Group diff --git a/SYSTEM_DESIGN.md b/SYSTEM_DESIGN.md deleted file mode 100644 index c29b5cd..0000000 --- a/SYSTEM_DESIGN.md +++ /dev/null @@ -1,32 +0,0 @@ -# System Design & Theoretical Model - -UDIE represents urban disruption as a spatiotemporal scalar field over a discretized hexagonal grid. - -## ๐Ÿง  Core Theoretical Model - -### 1. Spatial Discretization -Geography is partitioned into H3 cells (typically resolution 9). This converts continuous geographic coordinates into discrete, addressable units, enabling $O(1)$ lookups and consistent adjacency operations. - -### 2. Risk Field Dynamics -- **Signal Injection**: Occurs when events are appended to the log. -- **Distance Decay**: Influence of an event follows a kernel function; proximate cells receive higher weight. -- **Temporal Decay**: Risk values decay exponentially over time: $R(t) = R_0 \cdot e^{-t/\tau}$. -- **Saturation**: To keep risk between $[0, 1)$, we apply $R_{norm} = 1 - e^{-R_{raw}/k}$. - -## ๐Ÿ— Agent Runtime Architecture - -UDIE employs a conceptual Agent System for internal diagnostics and forecasting: - -- **Diagnostic Agents**: Audit system invariants, query plan safety, and data-plane health. -- **Forecasting Agents**: Compute T+X risk projections using historical snapshots. -- **Runtime Execution**: Agents are treated as stateless compute workers operating within a deterministic execution graph (DAG). - -## ๐Ÿ”’ Security & Governance - -- **Law of Simulation Isolation**: Simulation traffic is physically or logically separated from production ingestion. -- **Execution Jails**: Forecasting and diagnostic logic run in sandboxed adapters to prevent side-channel mutations. -- **Signal Invariants**: MITM or signal spam is mitigated through multi-source reinforcement and credibility decay. - -## ๐Ÿ“Š Modeling Boundaries - -Current kernels prioritize operational speed over complex urban morphology (e.g., building-level occlusion). Stability and rebuildability are the primary engineering constraints. diff --git a/TRAINING_PIPELINE.md b/TRAINING_PIPELINE.md deleted file mode 100644 index c6409c3..0000000 --- a/TRAINING_PIPELINE.md +++ /dev/null @@ -1,26 +0,0 @@ -# Training Pipeline - -UDIE primarily uses **Deterministic Spatial Intelligence** rather than weights trained via backpropagation. However, the system supports a pipeline for optimizing model parameters (kernels and decay constants). - -## ๐Ÿ“Š Learning Objectives - -The "Training" process in UDIE refers to the empirical optimization of: -1. **Spatial Decays ($k$)**: Calibrating how far disruption influence spreads. -2. **Temporal Constants ($\tau$)**: Tuning how quickly signals disappear in different city contexts. -3. **Source Weights ($w$)**: Optimizing initial event importance based on historical outcome correlation. - -## ๐Ÿ”„ Parameter Optimization Loop - -1. **Snapshot Extraction**: Load historical `risk_snapshots` and corresponding `events_log` segments from S3 (Cold Storage). -2. **Validation Simulation**: Replay logs against a range of candidate parameters. -3. **Loss Evaluation**: Measure the divergence between projected risk and actual reported outcomes (e.g., traffic velocity drops, emergency calls). -4. **Parameter Injection**: Validated parameters are injected into the [CONFIGURATION.md](file:///Users/fallofpheonix/Project/UDIE/CONFIGURATION.md) of production workers. - -## ๐Ÿ›  Model Validation Tools - -- **`scripts/ops/grid-rebuild.sh`**: Replays event logs to verify deterministic convergence. -- **`ArchitectureAuditService`**: Continuously monitors if runtime risk surfaces deviate from theoretical expectations. - -## ๐Ÿ”ฎ Future RL/ML Integration - -Future versions of UDIE may incorporate Reinforcement Learning (RL) agents to dynamically adjust kernels based on real-time city state (e.g., weather or time-of-day), but the current pipeline remains focused on **observable deterministic parameter tuning**. diff --git a/engine-backend/package.json b/engine-backend/package.json index dc9976e..d820618 100644 --- a/engine-backend/package.json +++ b/engine-backend/package.json @@ -6,7 +6,6 @@ "scripts": { "start": "nest start", "dev": "nest start --watch", - "start:dev": "nest start --watch", "build": "nest build", "lint": "eslint \"src/**/*.ts\"", "test": "npm run build && node --test test/*.test.cjs", diff --git a/engine-backend/src/app.module.ts b/engine-backend/src/app.module.ts index c0b4ebe..7cb60e7 100644 --- a/engine-backend/src/app.module.ts +++ b/engine-backend/src/app.module.ts @@ -20,7 +20,6 @@ import { DiagnosticsModule } from './modules/diagnostics/diagnostics.module'; import { SimulationModule } from './modules/simulation/simulation.module'; import { MetricsModule } from './metrics/metrics.module'; -// Role definition defaults const NODE_ROLE = process.env.NODE_ROLE || 'ALL'; const roles = NODE_ROLE.toUpperCase().split(','); const isAll = roles.includes('ALL'); diff --git a/engine-backend/src/intelligence/intelligence.controller.ts b/engine-backend/src/intelligence/intelligence.controller.ts index 24e837f..8c5691f 100644 --- a/engine-backend/src/intelligence/intelligence.controller.ts +++ b/engine-backend/src/intelligence/intelligence.controller.ts @@ -1,5 +1,5 @@ import { Controller, Get, Query } from '@nestjs/common'; -import { IntelligenceService } from './IntelligenceService'; +import { IntelligenceService } from './intelligence.service'; class IntelligenceQueryDto { regionId?: string; diff --git a/engine-backend/src/intelligence/intelligence.module.ts b/engine-backend/src/intelligence/intelligence.module.ts index 82a8dd0..b822f32 100644 --- a/engine-backend/src/intelligence/intelligence.module.ts +++ b/engine-backend/src/intelligence/intelligence.module.ts @@ -1,6 +1,6 @@ import { Module } from '@nestjs/common'; import { DatabaseModule } from '../database/database.module'; -import { IntelligenceService } from './IntelligenceService'; +import { IntelligenceService } from './intelligence.service'; import { IntelligenceController } from './intelligence.controller'; import { IntelligenceWorker } from './intelligence.worker'; import { RiskModule } from '../modules/risk/risk.module'; diff --git a/engine-backend/src/intelligence/IntelligenceRules.ts b/engine-backend/src/intelligence/intelligence.rules.ts similarity index 98% rename from engine-backend/src/intelligence/IntelligenceRules.ts rename to engine-backend/src/intelligence/intelligence.rules.ts index 933992e..1f39585 100644 --- a/engine-backend/src/intelligence/IntelligenceRules.ts +++ b/engine-backend/src/intelligence/intelligence.rules.ts @@ -1,4 +1,4 @@ -import { IntelligenceInsight, IntelligenceRuleConfig, InsightSeverity } from './IntelligenceTypes'; +import { IntelligenceInsight, IntelligenceRuleConfig, InsightSeverity } from './intelligence.types'; export function toSeverity(weight: number, hotspotThreshold: number): InsightSeverity { if (weight >= hotspotThreshold * 2) return 'HIGH'; diff --git a/engine-backend/src/intelligence/IntelligenceService.ts b/engine-backend/src/intelligence/intelligence.service.ts similarity index 97% rename from engine-backend/src/intelligence/IntelligenceService.ts rename to engine-backend/src/intelligence/intelligence.service.ts index 3f5c0ee..0e13c60 100644 --- a/engine-backend/src/intelligence/IntelligenceService.ts +++ b/engine-backend/src/intelligence/intelligence.service.ts @@ -7,8 +7,8 @@ import { IntelligenceInsight, IntelligenceQuery, IntelligenceRuleConfig, -} from './IntelligenceTypes'; -import { hotspotInsight, recurringInsight, spikeInsight } from './IntelligenceRules'; +} from './intelligence.types'; +import { hotspotInsight, recurringInsight, spikeInsight } from './intelligence.rules'; @Injectable() export class IntelligenceService { @@ -21,8 +21,8 @@ export class IntelligenceService { ) { } /** - * Orchestrates the pattern detection loop. - * Law: Intelligence is asynchronous and does not block ingestion. + * Runs the full pattern analysis loop over all active H3 cells. + * Runs asynchronously to avoid blocking ingestion throughput. */ async runAnalysis(): Promise { this.logger.log('[INTEL] Starting nationwide pattern analysis...'); diff --git a/engine-backend/src/intelligence/IntelligenceTypes.ts b/engine-backend/src/intelligence/intelligence.types.ts similarity index 100% rename from engine-backend/src/intelligence/IntelligenceTypes.ts rename to engine-backend/src/intelligence/intelligence.types.ts diff --git a/engine-backend/src/intelligence/intelligence.worker.ts b/engine-backend/src/intelligence/intelligence.worker.ts index f66f54e..75465b3 100644 --- a/engine-backend/src/intelligence/intelligence.worker.ts +++ b/engine-backend/src/intelligence/intelligence.worker.ts @@ -1,6 +1,6 @@ import { Injectable, Logger } from '@nestjs/common'; import { Cron } from '@nestjs/schedule'; -import { IntelligenceService } from './IntelligenceService'; +import { IntelligenceService } from './intelligence.service'; import { DatabaseService } from '../database/database.service'; import { AnomalyDetectionService } from './anomaly-detection.service';