Stream market: payment channels, access control, and datastream pub/sub

Dockerfile

@@ -4,12 +4,29 @@ FROM python:3.10-slim
 # System dependencies
 RUN apt-get update && \
     apt-get install -y \
+        ca-certificates \
         build-essential \
         cmake \
-        libleveldb-dev && \
+        git \
+        libflatbuffers-dev \
+        libleveldb-dev \
+        liblmdb-dev \
+        libsecp256k1-dev \
+        libssl-dev \
+        libzstd-dev \
+        zlib1g-dev && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 
+# Build a glibc-compatible strfry binary for the embedded relay runtime.
+RUN git clone --depth 1 https://github.com/hoytech/strfry.git /tmp/strfry && \
+    cd /tmp/strfry && \
+    git submodule update --init && \
+    make setup-golpe && \
+    make -j"$(nproc)" && \
+    cp /tmp/strfry/strfry /usr/local/bin/strfry && \
+    rm -rf /tmp/strfry
+
 # Create directory structure
 RUN mkdir -p /Satori/Lib /Satori/Engine /Satori/Neuron /Satori/Neuron/satorineuron/web
 

Dockerfile.dev

@@ -4,12 +4,29 @@ FROM python:3.10-slim
 # System dependencies
 RUN apt-get update && \
     apt-get install -y \
+        ca-certificates \
         build-essential \
         cmake \
-        libleveldb-dev && \
+        git \
+        libflatbuffers-dev \
+        libleveldb-dev \
+        liblmdb-dev \
+        libsecp256k1-dev \
+        libssl-dev \
+        libzstd-dev \
+        zlib1g-dev && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 
+# Build a glibc-compatible strfry binary for the embedded relay runtime.
+RUN git clone --depth 1 https://github.com/hoytech/strfry.git /tmp/strfry && \
+    cd /tmp/strfry && \
+    git submodule update --init && \
+    make setup-golpe && \
+    make -j"$(nproc)" && \
+    cp /tmp/strfry/strfry /usr/local/bin/strfry && \
+    rm -rf /tmp/strfry
+
 # Create directory structure
 RUN mkdir -p /Satori/Lib /Satori/Engine /Satori/Neuron
 

README.md

@@ -2,156 +2,207 @@
 
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-A simplified, lightweight version of the Satori neuron system with a CLI interface (similar to Claude Code).
+Satori Lite is a streamlined distribution of the Satori Network neuron, designed
+to run a complete prediction node with a minimal footprint. It bundles a
+forecasting engine, an Evrmore wallet, a stream relay, and a local web
+interface into a single deployable unit suitable for individual operators,
+research deployments, and integration testing.
 
-## Structure
+The project is composed of two repositories that are intended to be used
+together:
+
+- [satori-lite](https://github.com/SatoriNetwork/satori-lite) — the neuron, the
+  AI engine, the embedded relay tooling, and the operator-facing web and CLI
+  interfaces.
+- [satorilib](https://github.com/SatoriNetwork/satorilib) — the shared Python
+  library providing wallets, server clients, datastream concepts, transport
+  adapters, and the Nostr-based publish/subscribe stack used by the neuron.
+
+## Overview
+
+A Satori neuron consumes public datastreams, trains models against them, and
+publishes its own predictions back to the network as new datastreams. Satori
+Lite preserves this end-to-end behavior while removing infrastructure that is
+not required for an individual node, producing a build that is easier to
+operate, audit, and extend.
+
+The neuron is organized as three cooperating components:
+
+- **neuron-lite** — process supervisor, configuration, wallet management,
+  scheduling, scoring, and the operator interfaces (CLI and web UI).
+- **engine-lite** — the forecasting engine responsible for feature
+  engineering, model training, evaluation, and prediction publication.
+- **lib-lite / satorilib** — shared primitives for wallets, identity, stream
+  data structures, server communication, persistence, and transport.
+
+## Repository Layout
 
 ```
-neuron/
-├── lib-lite/          # Minimal satorilib with only essential features
-├── neuron-lite/       # Lightweight neuron implementation
-├── engine-lite/       # AI engine (to be added)
-└── requirements.txt   # Python dependencies
+satori-lite/
+  neuron-lite/         Neuron runtime, CLI, scheduling, wallet, scoring
+  engine-lite/         Forecasting engine, adapters, storage, model code
+  lib-lite/            Vendored subset of satorilib used by the neuron
+  web/                 Operator web interface (Flask application)
+  docs/                Architecture, guides, and implementation notes
+  migrations/          Schema migrations for local persistence
+  tests/               Unit, integration, and performance test suites
+  Dockerfile           Container build for the full neuron
+  docker-compose.local.yml
+  requirements.txt
 ```
 
-## Features
+The companion `satorilib` repository provides the wider library surface,
+including additional transports, the Nostr datastream stack
+(`satorilib.satori_nostr`), payment-channel primitives, and modules that are
+not required by the lightweight neuron build but are available for advanced
+deployments.
 
-### lib-lite
-Minimal version of satorilib containing only:
-- **Central Server Communication**: Checkin, balances, stream management
-- **Wallet Support**: Evrmore blockchain wallet and identity
-- **Stream Data Structures**: StreamId, Stream, StreamPairs, StreamOverview
-- **Utilities**: Disk operations, IP utilities, async threading
-- **Optional Centrifugo**: Real-time messaging support (can be enabled/disabled)
+## Key Components
 
-### Excluded Features
-To keep the system simple and lightweight, the following are NOT included:
-- P2P networking
-- Data relay engines
-- IPFS integration
-- Complex data management systems
+### Neuron (`neuron-lite`)
 
-## Installation
+- Startup orchestration via a directed acyclic graph of initialization steps.
+- Wallet creation, unlocking, and vault management for Evrmore identities.
+- Local stream relay management and publication scheduling.
+- Scoring and reward tracking against the central server.
+- A Flask-based web interface served on port `24601`.
+- An interactive terminal CLI for wallet, vault, and node operations.
 
-1. **Install lib-lite package:**
-   ```bash
-   cd lib-lite
-   pip install -e .
-   ```
+### Engine (`engine-lite`)
 
-2. **Install with optional features:**
-   ```bash
-   # With Centrifugo support
-   pip install -e ".[centrifugo]"
+- Model training and continuous retraining for assigned datastreams.
+- Pluggable adapters for data acquisition and storage backends.
+- Prediction emission as first-class datastreams consumable by other neurons.
 
-   # With telemetry support
-   pip install -e ".[telemetry]"
+### Shared Library (`satorilib`)
 
-   # With all optional features
-   pip install -e ".[centrifugo,telemetry]"
-   ```
+- **Wallet**: Evrmore wallet, identity derivation, and signing primitives.
+- **Server**: Authenticated client for the Satori central server (checkin,
+  balances, stream and subscription management).
+- **Concepts**: Canonical data structures for streams, observations, and
+  related domain objects.
+- **Transports**: Nostr-based datastream pub/sub
+  (`satorilib.satori_nostr`), Centrifugo client, websockets, and additional
+  experimental transports.
+- **Persistence**: SQLite helpers, on-disk caching, and file utilities.
+- **Asynchronous**: Thread and task helpers used across the neuron.
 
-3. **Or install from requirements.txt:**
-   ```bash
-   pip install -r requirements.txt
-   ```
+## Installation
 
-## Usage
+### From source
 
-The neuron-lite will have a CLI interface similar to Claude Code for easy interaction and monitoring.
+Clone both repositories side by side and install them into the same Python
+environment:
 
-## Testing
+```bash
+git clone https://github.com/SatoriNetwork/satorilib.git
+git clone https://github.com/SatoriNetwork/satori-lite.git
 
-Satori-lite has a comprehensive test suite with **155 tests** covering all functionality:
+pip install -e ./satorilib
+pip install -r ./satori-lite/requirements.txt
+```
 
-### Test Organization
+Python 3.10 or newer is recommended. A C toolchain and the development
+headers for `libsecp256k1`, `libleveldb`, `liblmdb`, and `libssl` are
+required to build the cryptography and storage dependencies on Linux.
 
-```
-tests/
-├── unit/                  (27 tests - fast, mocked)
-│   ├── test_auth_unit.py
-│   └── test_client_unit.py
-├── integration/           (85 tests - requires server)
-│   ├── test_health.py
-│   ├── test_auth.py
-│   ├── test_predictions.py
-│   ├── test_peer.py
-│   ├── test_balance.py
-│   ├── test_lending.py
-│   ├── test_pool.py
-│   ├── test_workflows.py
-│   ├── test_data_persistence.py    # NEW: Phase 6
-│   ├── test_challenge_lifecycle.py # NEW: Phase 6
-│   └── test_user_journeys.py       # NEW: Phase 6
-└── performance/           (37 tests - load & benchmarks)
-    ├── test_performance.py
-    ├── test_load.py
-    └── test_edge_cases.py
-```
+### Using Docker
 
-### Running Tests
+A reference container build is provided. The build expects `satorilib` to be
+supplied as an external build context so the library can be installed into the
+image without publishing it as a package:
 
-**Start the server first** (required for integration and performance tests):
 ```bash
-export DATABASE_URL="sqlite:///:memory:"
-uvicorn src.main:app --host 0.0.0.0 --port 8000 &
+docker build \
+  --build-context satorilib=https://github.com/SatoriNetwork/satorilib.git \
+  -t satori-lite:local \
+  https://github.com/SatoriNetwork/satori-lite.git
 ```
 
-**Run all tests:**
+A Compose file (`docker-compose.local.yml`) is included for local development
+and exposes the web interface on port `24601`.
+
+## Running the Neuron
+
+After installation, start the neuron from the `neuron-lite` directory:
+
 ```bash
-cd /app/neuron
-pytest tests/ -v
-# 155 tests pass in ~23 seconds
+python neuron-lite/start.py
 ```
 
-**Run specific test categories:**
+The process performs wallet initialization, registers with the central
+server, starts the embedded relay, launches the engine, and serves the
+operator web interface at `http://localhost:24601`.
+
+The interactive CLI can be used for wallet inspection, vault management, and
+operational commands:
+
 ```bash
-# Unit tests only (no server required)
-pytest tests/unit/ -v                    # 27 tests in ~2 seconds
+python neuron-lite/cli.py
+```
+
+## Configuration
+
+Configuration is layered and resolved in the following order:
 
-# Integration tests (requires server)
-pytest tests/integration/ -v              # 85 tests
+1. Built-in defaults shipped with the neuron.
+2. YAML configuration files under `neuron-lite/config/`.
+3. Environment variables.
+4. Command-line flags supplied to `start.py`.
 
-# Performance tests (requires server)
-pytest tests/performance/ -v              # 37 tests
+The minimum configuration required for a working node is the central server
+endpoint and a wallet location. Additional settings control the engine,
+relay, scoring policy, and optional transports. See `docs/guides/` for
+operator-focused documentation and `docs/architecture/` for design
+references.
 
-# By marker
-pytest -m unit                           # All unit tests
-pytest -m integration                    # All integration tests
-pytest -m slow                           # Slow tests (performance)
+## Testing
+
+The repository ships with a layered test suite covering unit, integration,
+and performance scenarios.
+
+```bash
+pytest tests/                  # full suite
+pytest tests/unit/             # fast, isolated tests
+pytest tests/integration/      # require a running local server
+pytest tests/performance/      # load and benchmark tests
 ```
 
-**Run with coverage:**
+Markers are available for selective runs (`-m unit`, `-m integration`,
+`-m slow`). Coverage can be measured against the bundled library subset:
+
 ```bash
-pytest tests/ --cov=lib-lite/satorilib/server --cov-report=term-missing
+pytest tests/ --cov=lib-lite/satorilib --cov-report=term-missing
 ```
 
-### Test Coverage
+Integration and performance tests expect the local API to be reachable on
+`http://localhost:8000`. See `tests/README` and `docs/plans/` for detailed
+instructions.
 
-- ✅ **100% API endpoint coverage** (all 15 endpoints)
-- ✅ **Complete workflows** (authentication, predictions, observations)
-- ✅ **Performance benchmarks** (response times, throughput)
-- ✅ **Load testing** (concurrent operations, burst traffic)
-- ✅ **Edge cases** (boundary conditions, data validation)
+## Documentation
 
-### Performance Benchmarks
+In-tree documentation is organized under `docs/`:
 
-All benchmarks validated with automated tests:
+- `docs/architecture/` — system design and component interactions.
+- `docs/engine/` — forecasting engine internals and training pipeline.
+- `docs/guides/` — operator and developer guides.
+- `docs/implementation/` — implementation notes for specific subsystems.
+- `docs/plans/` — design and test plans.
 
-- Health endpoint: < 100ms average
-- Challenge generation: < 150ms average
-- Observation retrieval: < 200ms average
-- Concurrent operations: > 95% success rate
-- Throughput: > 20 requests/second
+## Contributing
 
-For detailed test documentation, see: [Test Plan Documentation](/app/docs/plans/)
+Contributions are welcome. Before submitting a pull request, please:
 
-## Configuration
+1. Open an issue describing the proposed change.
+2. Ensure the relevant test suites pass locally.
+3. Follow the existing module layout and avoid introducing dependencies that
+   would expand the lightweight footprint of the neuron.
+
+For changes that affect shared primitives (wallets, server clients, stream
+concepts, transports), please open the corresponding pull request against the
+`satorilib` repository.
 
-The system focuses on:
-1. Central server communication (checkin, authentication)
-2. Wallet management (Evrmore)
-3. Stream subscriptions and publications
-4. Basic data flow
+## License
 
-All unnecessary complexity has been removed to create a clean, maintainable codebase.
+Satori Lite is released under the MIT License. See `LICENSE` for the full
+text.

docker-compose.local.yml

@@ -29,6 +29,8 @@ services:
       - ${SATORI_DOCKER_SOCKET:-/var/run/docker.sock}:/var/run/docker.sock
     ports:
       - "24601:24601"
+      - "7777:7777"
+      - "7171:7171"
     networks:
       - default
     command: python /Satori/Neuron/start.py
@@ -62,6 +64,8 @@ services:
       - ./tests:/Satori/tests
     ports:
       - "24601:24601"
+      - "7777:7777"
+      - "7171:7171"
     networks:
       - default
     command: python /Satori/Neuron/start.py