Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
567 lines (458 loc) · 16.2 KB

ARCHITECTURE.md

File metadata and controls

567 lines (458 loc) · 16.2 KB

Vectorizer Sync - Technical Architecture

Overview

Vectorizer Sync is a cross-platform desktop application built with Electron and SQLite that enables users to manage and synchronize project directories with Vectorizer workspaces (local or remote). The application provides a graphical interface for selecting project directories, exporting workspace configurations, and synchronizing documents to HiveHub Cloud when enabled.

Technology Stack

  • Framework: Electron (cross-platform desktop application)
  • Database: SQLite (local data storage)
  • Language: TypeScript
  • UI Framework: React (based on project configuration)
  • Platform Support: Windows, macOS, Linux

System Architecture

High-Level Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Electron Main Process                     │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐       │
│  │   Database   │  │  File System │  │   Sync      │       │
│  │   Manager    │  │   Watcher    │  │   Engine    │       │
│  └──────────────┘  └──────────────┘  └──────────────┘       │
│         │                │                   │              │
│         └────────────────┴───────────────────┘              │
│                            │                                 │
│                    ┌───────▼────────┐                        │
│                    │  HiveHub API  │                        │
│                    │   Client      │                        │
│                    └───────────────┘                        │
└─────────────────────────────────────────────────────────────┘
                            │
                            │ IPC
                            │
┌─────────────────────────────────────────────────────────────┐
│                  Electron Renderer Process                    │
│  ┌──────────────────────────────────────────────────────┐   │
│  │              React UI Components                      │   │
│  │  - Project Selection                                 │   │
│  │  - Workspace Configuration                           │   │
│  │  - Sync Status                                       │   │
│  │  - Notifications                                     │   │
│  └──────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘

Core Components

1. Database Manager

Purpose: Manages all application data stored in SQLite database.

Location: User's home directory

  • Windows: C:\Users\<username>\vectorizer-sync\database.db
  • macOS: ~/vectorizer-sync/database.db
  • Linux: ~/vectorizer-sync/database.db

Responsibilities:

  • Store project configurations
  • Store workspace settings
  • Store sync status and history
  • Store user preferences
  • Store notification settings

2. File System Watcher

Purpose: Monitors selected project directories for file changes.

Responsibilities:

  • Watch for file additions
  • Watch for file modifications
  • Watch for file deletions
  • Trigger sync operations when changes detected
  • Filter files based on exclusion rules

3. Sync Engine

Purpose: Handles synchronization between local directories and Vectorizer workspaces.

Responsibilities:

  • Export workspace.yml configuration files
  • Upload files to HiveHub Cloud (when enabled)
  • Update files when changes detected
  • Handle sync conflicts
  • Respect file size and type restrictions
  • Monitor and enforce plan limits

4. HiveHub API Client

Purpose: Communicates with HiveHub Cloud services.

Responsibilities:

  • Authentication (future implementation)
  • Upload files to cloud workspace
  • Check plan limits and quotas
  • Receive notifications from HiveHub
  • Handle API errors and rate limiting

5. Notification System

Purpose: Manages and displays notifications to users.

Responsibilities:

  • Display internal notifications (sync status, errors)
  • Display HiveHub notifications (quota limits, alerts)
  • Store notification history
  • Provide notification preferences

Data Models

Project Configuration

interface Project {
  id: string;
  name: string;
  path: string;
  workspaceType: 'local' | 'remote';
  workspacePath?: string; // For local workspaces
  cloudWorkspaceId?: string; // For remote workspaces
  syncEnabled: boolean;
  lastSyncAt?: Date;
  createdAt: Date;
  updatedAt: Date;
}

Workspace Configuration

interface WorkspaceConfig {
  projectId: string;
  workspaceYml: string; // YAML content
  exportedAt: Date;
  version: number;
}

Sync History

interface SyncHistory {
  id: string;
  projectId: string;
  type: 'export' | 'upload' | 'update' | 'delete';
  status: 'success' | 'failed' | 'partial';
  filesCount: number;
  filesProcessed: number;
  errors?: string[];
  startedAt: Date;
  completedAt?: Date;
}

File Metadata

interface FileMetadata {
  id: string;
  projectId: string;
  relativePath: string;
  absolutePath: string;
  size: number;
  hash: string; // For change detection
  lastModified: Date;
  syncedAt?: Date;
  syncStatus: 'pending' | 'synced' | 'failed' | 'excluded';
  exclusionReason?: string;
}

Notification

interface Notification {
  id: string;
  type: 'internal' | 'hivehub';
  severity: 'info' | 'warning' | 'error';
  title: string;
  message: string;
  read: boolean;
  actionUrl?: string;
  createdAt: Date;
}

User Settings

interface UserSettings {
  id: string;
  syncEnabled: boolean;
  autoSyncEnabled: boolean;
  syncInterval?: number; // in minutes
  maxFileSize: number; // in bytes (default: 100KB)
  excludedPatterns: string[]; // glob patterns
  notificationPreferences: {
    syncStatus: boolean;
    errors: boolean;
    quotaWarnings: boolean;
    hivehubAlerts: boolean;
  };
  hivehubAccount?: {
    email?: string;
    planType?: string;
    quotaLimit?: number;
    quotaUsed?: number;
  };
}

Database Schema

Tables

projects

CREATE TABLE projects (
  id TEXT PRIMARY KEY,
  name TEXT NOT NULL,
  path TEXT NOT NULL UNIQUE,
  workspace_type TEXT NOT NULL CHECK(workspace_type IN ('local', 'remote')),
  workspace_path TEXT,
  cloud_workspace_id TEXT,
  sync_enabled INTEGER NOT NULL DEFAULT 0,
  last_sync_at DATETIME,
  created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
  updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);

workspace_configs

CREATE TABLE workspace_configs (
  id TEXT PRIMARY KEY,
  project_id TEXT NOT NULL,
  workspace_yml TEXT NOT NULL,
  exported_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
  version INTEGER NOT NULL DEFAULT 1,
  FOREIGN KEY (project_id) REFERENCES projects(id) ON DELETE CASCADE
);

sync_history

CREATE TABLE sync_history (
  id TEXT PRIMARY KEY,
  project_id TEXT NOT NULL,
  type TEXT NOT NULL CHECK(type IN ('export', 'upload', 'update', 'delete')),
  status TEXT NOT NULL CHECK(status IN ('success', 'failed', 'partial')),
  files_count INTEGER NOT NULL DEFAULT 0,
  files_processed INTEGER NOT NULL DEFAULT 0,
  errors TEXT, -- JSON array
  started_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
  completed_at DATETIME,
  FOREIGN KEY (project_id) REFERENCES projects(id) ON DELETE CASCADE
);

file_metadata

CREATE TABLE file_metadata (
  id TEXT PRIMARY KEY,
  project_id TEXT NOT NULL,
  relative_path TEXT NOT NULL,
  absolute_path TEXT NOT NULL,
  size INTEGER NOT NULL,
  hash TEXT NOT NULL,
  last_modified DATETIME NOT NULL,
  synced_at DATETIME,
  sync_status TEXT NOT NULL DEFAULT 'pending' CHECK(sync_status IN ('pending', 'synced', 'failed', 'excluded')),
  exclusion_reason TEXT,
  FOREIGN KEY (project_id) REFERENCES projects(id) ON DELETE CASCADE,
  UNIQUE(project_id, relative_path)
);

notifications

CREATE TABLE notifications (
  id TEXT PRIMARY KEY,
  type TEXT NOT NULL CHECK(type IN ('internal', 'hivehub')),
  severity TEXT NOT NULL CHECK(severity IN ('info', 'warning', 'error')),
  title TEXT NOT NULL,
  message TEXT NOT NULL,
  read INTEGER NOT NULL DEFAULT 0,
  action_url TEXT,
  created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);

user_settings

CREATE TABLE user_settings (
  id TEXT PRIMARY KEY DEFAULT 'default',
  sync_enabled INTEGER NOT NULL DEFAULT 1,
  auto_sync_enabled INTEGER NOT NULL DEFAULT 0,
  sync_interval INTEGER, -- in minutes
  max_file_size INTEGER NOT NULL DEFAULT 102400, -- 100KB in bytes
  excluded_patterns TEXT, -- JSON array
  notification_preferences TEXT, -- JSON object
  hivehub_account TEXT, -- JSON object
  updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP
);

Indexes

CREATE INDEX idx_projects_path ON projects(path);
CREATE INDEX idx_projects_sync_enabled ON projects(sync_enabled);
CREATE INDEX idx_workspace_configs_project_id ON workspace_configs(project_id);
CREATE INDEX idx_sync_history_project_id ON sync_history(project_id);
CREATE INDEX idx_sync_history_started_at ON sync_history(started_at);
CREATE INDEX idx_file_metadata_project_id ON file_metadata(project_id);
CREATE INDEX idx_file_metadata_sync_status ON file_metadata(sync_status);
CREATE INDEX idx_file_metadata_hash ON file_metadata(hash);
CREATE INDEX idx_notifications_read ON notifications(read);
CREATE INDEX idx_notifications_created_at ON notifications(created_at);

File Filtering Rules

Excluded File Patterns

The system MUST exclude the following:

  1. Module directories: node_modules/, vendor/, packages/, etc.
  2. Build artifacts: dist/, build/, out/, .next/, etc.
  3. Large files: Files larger than 100KB (configurable)
  4. Hidden files: Files starting with . (except .gitignore, .env.example, etc.)
  5. Binary files: Common binary extensions (.exe, .dll, .so, .dylib, etc.)
  6. Temporary files: .tmp, .temp, .cache, etc.

Exclusion Logic

interface FileFilter {
  isExcluded(filePath: string, fileSize: number): {
    excluded: boolean;
    reason?: string;
  };
}

Workspace.yml Format

The application MUST export workspace configurations in the Vectorizer workspace.yml format:

global_settings:
  file_watcher:
    enabled: true
    auto_discovery: true
    enable_auto_update: true
    exclude_patterns: []
    hot_reload: true
    watch_paths:
      - <workspace-root-path>

projects:
  - name: <project-name>
    path: <project-path>
    description: <project-description>
    collections:
      - name: <collection-name>
        description: <collection-description>
        include_patterns:
          - <glob-pattern>
        exclude_patterns:
          - <glob-pattern>

For detailed format specification, see docs/WORKSPACE_FORMAT.md.

HiveHub Cloud Integration

Authentication (Future)

Status: Documented only, not implemented

Planned Implementation:

  • OAuth 2.0 authentication flow
  • Token storage and refresh
  • Account management UI
  • Session management

API Endpoints (Planned)

  • POST /api/auth/login - User authentication
  • GET /api/workspaces - List user workspaces
  • POST /api/workspaces/:id/files - Upload files
  • PUT /api/workspaces/:id/files/:fileId - Update file
  • DELETE /api/workspaces/:id/files/:fileId - Delete file
  • GET /api/account/quota - Get quota information
  • GET /api/notifications - Get notifications

Plan Limits

The system MUST:

  • Check quota before uploading files
  • Display quota usage to user
  • Block uploads when quota exceeded
  • Show notification when approaching limit
  • Provide link to upgrade plan

Notification System

Notification Types

  1. Internal Notifications:

    • Sync started/completed
    • Sync errors
    • File exclusion warnings
    • Configuration changes

  2. HiveHub Notifications:

    • Quota limit reached
    • Quota warning (80%, 90%)
    • Plan upgrade available
    • Service maintenance
    • Security alerts

Notification Display

  • System tray notifications (OS-level)
  • In-app notification center
  • Badge count on notification icon
  • Sound alerts (configurable)

Configuration Import/Export

Import Format

The system MUST support importing configurations from:

  • JSON files
  • Previous Vectorizer Sync installations
  • Workspace.yml files (read-only)

Export Format

The system MUST export:

  • Project configurations
  • Workspace settings
  • User preferences
  • Sync history (optional)

Platform-Specific Considerations

Windows

  • Database location: C:\Users\<username>\vectorizer-sync\
  • File path handling: Use backslashes, handle UNC paths
  • Permissions: Handle UAC prompts for file access

macOS

  • Database location: ~/vectorizer-sync/
  • File path handling: Use forward slashes
  • Permissions: Handle macOS privacy permissions (Files and Folders access)

Linux

  • Database location: ~/vectorizer-sync/
  • File path handling: Use forward slashes
  • Permissions: Handle file system permissions

Security Considerations

  1. Data Storage:

    • SQLite database encryption (optional)
    • Secure storage of API tokens (if implemented)
    • No storage of passwords (OAuth only)

  2. File Access:

    • Validate file paths to prevent directory traversal
    • Check file permissions before operations
    • Handle symlinks appropriately

  3. Network Security:

    • Use HTTPS for all API communications
    • Validate SSL certificates
    • Implement request timeout and retry logic

Performance Considerations

  1. File Watching:

    • Use efficient file system watchers (chokidar)
    • Debounce file change events
    • Batch file operations

  2. Database Operations:

    • Use transactions for bulk operations
    • Implement connection pooling
    • Use prepared statements
    • Create appropriate indexes

  3. Sync Operations:

    • Implement queue for sync operations
    • Limit concurrent uploads
    • Use file hashing for change detection
    • Implement incremental sync

Error Handling

  1. File System Errors:

    • Handle permission denied errors
    • Handle file not found errors
    • Handle disk full errors
    • Log all errors with context

  2. Network Errors:

    • Handle connection timeouts
    • Handle rate limiting
    • Implement exponential backoff
    • Retry failed operations

  3. Database Errors:

    • Handle database locked errors
    • Handle corruption errors
    • Implement database backup/restore

Logging

The system MUST log:

  • All sync operations (start, progress, completion, errors)
  • File system operations
  • API requests and responses
  • User actions
  • Errors with stack traces

Log location: User's home directory in vectorizer-sync/logs/

Testing Strategy

  1. Unit Tests:

    • Database operations
    • File filtering logic
    • Workspace.yml generation
    • Configuration management

  2. Integration Tests:

    • File system watcher
    • Sync engine
    • API client (mocked)

  3. E2E Tests:

    • Full sync workflow
    • UI interactions
    • Cross-platform compatibility

Deployment

Build Process

  • Electron Builder for packaging
  • Code signing for Windows and macOS
  • Auto-updater integration (future)

Distribution

  • Windows: Installer (.exe) and portable version
  • macOS: DMG and App Store (future)
  • Linux: AppImage, DEB, and RPM packages