Skip to content

Technical-Architecture-Overview.md

Latest commit

 

History

History
250 lines (190 loc) · 14.9 KB

Technical-Architecture-Overview.md

File metadata and controls

250 lines (190 loc) · 14.9 KB
Raw

CryptoWildfire Technical Architecture Document

Project Overview

This comprehensive technical document outlines the detailed architecture, components, development process, and implementation guidelines for the CryptoWildfire (CryptoEdge) platform - a professional-grade cryptocurrency analysis system for personal use.

Table of Contents

  1. System Architecture Overview
  2. Data Collection Layer
  3. Serverless Data Pipeline
  4. Database Layer
  5. AI/Analytics Layer
  6. Signal Aggregation Engine
  7. Frontend User Interface
  8. Alert System
  9. Development Roadmap
  10. Deployment Strategy
  11. Maintenance & Operations

This document will be implemented across multiple files, each covering specific sections in detail:

1. System Architecture Overview

CryptoWildfire follows a modern, event-driven, multi-layered architecture optimized for real-time data processing, analysis, and visualization.

1.1 Architectural Principles

  • Modular Design: Each component is independent, allowing for isolated development and testing
  • Event-Driven: Real-time data flows through the system as events
  • Cloud-Native: Leverages serverless and managed services to minimize operational overhead
  • Cost-Efficient: Optimized for personal use with predictable, manageable costs
  • Scalable: Can handle increasing data volume without architectural changes

1.2 Technology Stack Summary

Layer Core Technologies Purpose
Data Collection Python, API clients (src/collectors) Gather data from multiple sources
Backend API Python, FastAPI (src/api) Serve data, handle requests
Database Supabase PostgreSQL Store time-series and relational data
Backend ML Python, Pandas, Scikit-learn (src/ml) Feature engineering, model training (potential Vertex AI integration)
Signal Engine Python (src/engine/aggregator.py) Combine signals into actionable insights
Frontend React, Vite, TypeScript, Tailwind CSS (frontend/) Visualize data and present insights
Frontend ML TypeScript (frontend/src/ml) Prediction engines, ensemble, validity, monitoring
Alerts (Planned) Telegram Bot API, Email Deliver timely notifications

1.3 System Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                       DATA COLLECTION LAYER                          │
├───────────────┬───────────────┬─────────────────┬───────────────────┤
│ Market Data   │ On-Chain Data │ DEX/Liquidity   │ Social/Sentiment  │
│ CoinGecko     │ Public RPC    │ Dexscreener     │ Reddit API        │
│ CryptoCompare │ Endpoints     │ Uniswap Graph   │ Market Sentiment  │
└───────┬───────┴───────┬───────┴────────┬────────┴────────┬──────────┘
        │               │                │                 │
        ▼               ▼                ▼                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                     BACKEND API (FastAPI)                            │
└────────────────────────────────────┬────────────────────────────────┘
                                      │ (API Calls)
                                      ▼
┌─────────────────────────────────────────────────────────────────────┐
│                     DATABASE LAYER (Supabase)                        │
├───────────────────────────┬─────────────────────────────────────────┤
│ PostgreSQL (Time-series)  │            Storage Buckets               │
└───────────────┬───────────┴──────────────────┬─────────────────────┘
                │ (Data for Backend)           │
                ▼                              │
┌─────────────────────────────────────────────────────────────────────┐
│                  BACKEND PROCESSING & ML LAYER                       │
├───────────────┬───────────────┬─────────────────┬───────────────────┤
│ Signal Engine │ Backend ML    │ Feature         │ Sentiment         │
│ (Aggregator)  │ (Training)    │ Engineering     │ Analysis (Hybrid) │
└───────────────┴───────────────┴─────────────────┴───────────────────┘
        │ (Processed Data / Signals via API)
        ▼
┌─────────────────────────────────────────────────────────────────────┐
│                  FRONTEND LAYER (React + Vite + TS)                  │
├───────────────────────────┬─────────────────────────────────────────┤
│  React UI Components      │      Frontend ML Engine & Ensemble       │
│  (Dashboard, Charts)      │      (Predictions, Validity)             │
└───────────────┬───────────┴──────────────────┬─────────────────────┘
                │                              │
                ▼                              ▼
┌─────────────────────────────────────────────────────────────────────┐
│                           User Interface                             │
├───────────────────────────┬─────────────────────────────────────────┤
│  Web Browser Display      │      Alert System (Planned)              │
└───────────────────────────┴─────────────────────────────────────────┘

1.4 Key Design Decisions

  • Use of Supabase: Provides a PostgreSQL database with real-time capabilities, plus authentication and storage, reducing integration complexity
  • Backend API: Using FastAPI for a robust Python backend.
  • Frontend Framework: Modern React stack (Vite, TypeScript, Tailwind) for a responsive and interactive UI.
  • Frontend ML Layer: Implementing prediction engines and ensemble logic directly in the frontend for real-time analysis presentation.
  • ML Model Design: Potential for backend training (Vertex AI or local) feeding into frontend prediction logic.
  • Hybrid Sentiment Analysis: Combining pre-analyzed external sentiment with in-house analysis for cost optimization.
  • Alert System Design: (Planned) Multi-channel approach ensures timely delivery of critical signals.

1.5 Integration Points

The system incorporates 15+ integration points with external services including:

  • Cryptocurrency exchange APIs (Binance, etc.)
  • Market data providers (CoinGecko, etc.)
  • Blockchain data sources (Public RPC, etc.)
  • Social media platforms (Reddit, etc.)
  • DEX aggregators (Dexscreener, 1inch)
  • AI inference services (OpenRouter, Gemini via src/ai)
  • Notification services (Planned: Telegram, Email)

5. AI/Analytics Layer

The AI/Analytics layer has been significantly enhanced to provide a self-improving machine learning system that continuously optimizes trading signals based on historical performance.

5.1 Machine Learning Architecture

5.1.1 Core ML Components

Component Technology/Implementation Purpose
Feature Engineering Python, Pandas Transform raw market and sentiment data into ML-ready features
Model Training Google Vertex AI / Local ML Models Train tabular classification models to predict price movements
Signal Optimizer Python Adjust signal weights based on feature importance from trained models
Training Scheduler Python Manage periodic retraining and continuous learning cycles

5.1.2 Vertex AI Integration

The system leverages Google Cloud's Vertex AI platform for scalable machine learning:

  • AutoML Tabular Models: Trains classification models to predict price direction
  • Model Deployment: Deploys trained models to endpoints for real-time prediction
  • Managed Training: Handles resource-intensive training in the cloud
  • Feature Importance: Extracts insights to optimize signal weighting

5.1.3 Continuous Learning Process

  1. Data Collection: Historical market, sentiment, and social data is collected and stored
  2. Feature Creation: Raw data is transformed into time-series features with price movement targets
  3. Model Training: Models are trained to predict future price movements
  4. Evaluation: Model performance is evaluated on test data
  5. Signal Optimization: Feature importance is extracted to optimize signal weights
  6. Deployment: Models are deployed for real-time prediction
  7. Feedback Loop: New market data and actual price movements are captured for future training

5.1.4 Fallback Mechanisms

The system includes graceful degradation if cloud services are unavailable:

  • Automatic fallback to local model training if Vertex AI is unavailable
  • Cached model predictions to handle temporary service disruptions
  • Default signal weights when ML optimization isn't possible

5.2 Sentiment Analysis System

5.2.1 Hybrid Approach

The system employs a cost-optimized hybrid approach to sentiment analysis:

  1. Pre-analyzed External Sources (primary):

    • Fear & Greed Index (market-wide sentiment)
    • CryptoCompare Social Stats (pre-analyzed social metrics)
    • TradingView Technical Indicators (web-scraped technical signals)

  2. In-house Analysis (selective):

    • Reddit content analyzed using DeepSeek AI
    • Only processes content not already pre-analyzed elsewhere

This hybrid approach follows the project's cost optimization strategy by leveraging free external resources where analysis has already been done, and only using AI computation for content not already analyzed elsewhere. This significantly reduces computational costs while actually improving signal quality through diverse data sources.

5.2.2 Sentiment Integration

Sentiment signals are:

  • Stored in structured tables (social_mentions, market_sentiment)
  • Normalized to a consistent scale (-1.0 to 1.0)
  • Weighted based on source reliability and signal strength
  • Combined with price and volume signals in the aggregator
  • Continuously optimized by the ML system based on predictive power

5.3. Signal Weight Distribution

The default distribution of signal weights follows industry best practices and internal testing:

  • Price momentum (30%)
  • Reddit sentiment (15%)
  • External sentiment sources (30% combined)
  • Volume and technical indicators (25% combined)

As the ML system gathers more data, these weights are progressively adjusted based on feature importance, with the ML influence scaling from 0% (no data) to 100% (sufficient historical data).

7. Frontend User Interface

7.1 Dashboard Implementation

The frontend is a modern React application built with Vite, TypeScript, and Tailwind CSS (frontend/ directory). It features:

  • Asset overview with latest prices and market data
  • Trading signals with ML confidence indicators
  • Market sentiment visualization (Fear & Greed Index, social metrics)
  • Social data monitoring and sentiment analysis
  • A sophisticated ML layer (frontend/src/ml) handling real-time prediction generation using multiple engines (High Frequency, Medium Term, Macro) and an ensemble model.
  • Timeframe-specific prediction validity and confidence display.
  • Interactive charts using libraries like TradingView or custom D3 implementations.
  • Component-based architecture for maintainability (frontend/src/components).
  • API service (frontend/src/services/api.ts) for communicating with the backend.

7.2 User Experience

The dashboard is organized into tabs for better navigation:

  • Assets & Signals: Current prices, trading signals and market data
  • Market Sentiment: Visualization of sentiment indicators from external sources
  • Social Data: Reddit mentions and sentiment analysis
  • ML Insights: Model predictions, confidence scores, and signal weight distribution

7.3 Data Visualization

The dashboard includes:

  • Tabular data for precise information
  • Gauge charts for sentiment visualization
  • Pie charts for sentiment and prediction distribution
  • Bar charts for weight distribution and feature importance

8. Alert System

8.1 Planned Implementation

The alert system will be implemented in a future phase with:

  • Telegram Bot integration for real-time alerts
  • Email notifications for summary reports
  • Web push notifications for dashboard users

8.2 Alert Types

Alerts will be generated for:

  • Strong trading signals (configurable threshold)
  • Significant sentiment shifts
  • Technical indicator pattern completion
  • ML confidence thresholds