diff --git a/AGENTS.md b/AGENTS.md index 6f2e677..0fb52ec 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,5 +1,5 @@ # AGENTS.md -# Enterprise RAG System (v1) — Production-Ready Architecture +# MultiDoc-RAG System (v1) - Production-Ready Architecture ## Context First @@ -38,2714 +38,20 @@ Ordering rules: If the user asks to implement one task directly, still read the dependent earlier task docs when needed so the work stays aligned with the intended sequence. -**Version**: 1.0 Final -**Last Updated**: 2024-02-13 -**Status**: Locked for Implementation +## Naming Note ---- +This repository and its documentation should use the product name `MultiDoc-RAG`. -## Table of Contents -1. [Project Structure](#1-project-structure) -2. [Docker Setup](#2-docker-setup) -3. [Core Principles](#3-core-principles) -4. [Platform Stack](#4-platform-stack) -5. [Workspace Model](#5-workspace-model) -6. [Hard Limits & Constraints](#6-hard-limits--constraints) -7. [Token Budget System](#7-token-budget-system) -8. [Chunking & Retrieval Strategy](#8-chunking--retrieval-strategy) -9. [Database Schema](#9-database-schema) -10. [Upload Architecture](#10-upload-architecture) -11. [Ingestion Pipeline](#11-ingestion-pipeline) -12. [Query System (RAG)](#12-query-system-rag) -13. [API Endpoints](#13-api-endpoints) -14. [Worker Configuration](#14-worker-configuration) -15. [Error Handling & Retries](#15-error-handling--retries) -16. [Monitoring & Health Checks](#16-monitoring--health-checks) -17. [Security & Rate Limiting](#17-security--rate-limiting) -18. [Maintenance & Operations](#18-maintenance--operations) +## Repository Layout ---- - -## 1) Project Structure - -### Repository Layout -``` -enterprise-rag/ -├── docker-compose.yml -├── docker-compose.dev.yml -├── .env.example -├── .gitignore -├── README.md -├── AGENTS.md -│ -├── server/ # FastAPI Backend -│ ├── Dockerfile -│ ├── requirements.txt -│ ├── .dockerignore -│ ├── pyproject.toml -│ ├── pytest.ini -│ │ -│ ├── app/ -│ │ ├── __init__.py -│ │ ├── main.py # FastAPI app entry point -│ │ ├── config.py # Settings & environment -│ │ │ -│ │ ├── api/ # API routes -│ │ │ ├── __init__.py -│ │ │ ├── deps.py # Dependencies (auth, db) -│ │ │ ├── workspaces.py -│ │ │ ├── documents.py -│ │ │ ├── query.py -│ │ │ └── usage.py -│ │ │ -│ │ ├── core/ # Core business logic -│ │ │ ├── __init__.py -│ │ │ ├── auth.py # JWT validation -│ │ │ ├── token_budget.py # Token reservation logic -│ │ │ ├── chunking.py # Text chunking -│ │ │ ├── embeddings.py # OpenAI embeddings -│ │ │ └── retrieval.py # Vector search -│ │ │ -│ │ ├── db/ # Database layer -│ │ │ ├── __init__.py -│ │ │ ├── session.py # DB connection -│ │ │ ├── models.py # SQLAlchemy models -│ │ │ └── repositories/ -│ │ │ ├── workspace.py -│ │ │ ├── document.py -│ │ │ ├── chunk.py -│ │ │ └── usage.py -│ │ │ -│ │ ├── schemas/ # Pydantic schemas -│ │ │ ├── __init__.py -│ │ │ ├── workspace.py -│ │ │ ├── document.py -│ │ │ ├── query.py -│ │ │ └── usage.py -│ │ │ -│ │ ├── storage/ # Supabase Storage -│ │ │ ├── __init__.py -│ │ │ └── client.py -│ │ │ -│ │ └── utils/ -│ │ ├── __init__.py -│ │ ├── rate_limit.py -│ │ └── logging.py -│ │ -│ ├── migrations/ # Alembic migrations -│ │ ├── env.py -│ │ └── versions/ -│ │ -│ └── tests/ -│ ├── __init__.py -│ ├── conftest.py -│ ├── test_api/ -│ ├── test_core/ -│ └── test_db/ -│ -├── worker/ # RQ Workers -│ ├── Dockerfile -│ ├── requirements.txt -│ ├── .dockerignore -│ │ -│ ├── jobs/ -│ │ ├── __init__.py -│ │ ├── ingest_extract.py # PDF extraction job -│ │ ├── ingest_index.py # Embedding generation job -│ │ └── maintenance.py # Cleanup jobs -│ │ -│ ├── shared/ # Shared with server -│ │ ├── __init__.py -│ │ ├── config.py -│ │ ├── db/ # Same models as server -│ │ └── core/ # Same logic as server -│ │ -│ ├── worker.py # Worker entry point -│ └── tests/ -│ -├── client/ # React Frontend -│ ├── Dockerfile -│ ├── .dockerignore -│ ├── package.json -│ ├── package-lock.json -│ ├── vite.config.ts -│ ├── tsconfig.json -│ ├── index.html -│ │ -│ ├── public/ -│ │ └── assets/ -│ │ -│ ├── src/ -│ │ ├── main.tsx -│ │ ├── App.tsx -│ │ ├── vite-env.d.ts -│ │ │ -│ │ ├── components/ -│ │ │ ├── layout/ -│ │ │ │ ├── Header.tsx -│ │ │ │ ├── Sidebar.tsx -│ │ │ │ └── Layout.tsx -│ │ │ │ -│ │ │ ├── documents/ -│ │ │ │ ├── DocumentList.tsx -│ │ │ │ ├── DocumentUpload.tsx -│ │ │ │ ├── DocumentStatus.tsx -│ │ │ │ └── DocumentCard.tsx -│ │ │ │ -│ │ │ ├── query/ -│ │ │ │ ├── QueryInput.tsx -│ │ │ │ ├── QueryResults.tsx -│ │ │ │ ├── Citation.tsx -│ │ │ │ └── DocumentSelector.tsx -│ │ │ │ -│ │ │ └── usage/ -│ │ │ ├── TokenMeter.tsx -│ │ │ ├── UsageChart.tsx -│ │ │ └── UsageBreakdown.tsx -│ │ │ -│ │ ├── pages/ -│ │ │ ├── Dashboard.tsx -│ │ │ ├── Documents.tsx -│ │ │ ├── Query.tsx -│ │ │ ├── Usage.tsx -│ │ │ └── Login.tsx -│ │ │ -│ │ ├── hooks/ -│ │ │ ├── useAuth.ts -│ │ │ ├── useDocuments.ts -│ │ │ ├── useQuery.ts -│ │ │ └── useUsage.ts -│ │ │ -│ │ ├── lib/ -│ │ │ ├── api.ts # API client -│ │ │ ├── supabase.ts # Supabase client -│ │ │ └── utils.ts -│ │ │ -│ │ ├── types/ -│ │ │ ├── api.ts -│ │ │ ├── document.ts -│ │ │ └── workspace.ts -│ │ │ -│ │ └── styles/ -│ │ └── globals.css -│ │ -│ └── tests/ -│ -├── scripts/ # Utility scripts -│ ├── setup-db.sh -│ ├── seed-data.py -│ ├── backup.sh -│ └── deploy.sh -│ -└── infrastructure/ # Optional: IaC - ├── terraform/ - └── k8s/ -``` - -### Key Design Decisions - -#### Monorepo vs Multi-repo -**Choice**: Monorepo (all in one repository) - -**Rationale**: -- Simplified versioning (single source of truth) -- Easier code sharing between server/worker -- Atomic commits across frontend/backend -- Simplified CI/CD - -#### Server-Worker Code Sharing -``` -worker/shared/ → symlink to server/app/ -``` -Workers import from shared codebase to avoid duplication of: -- Database models -- Configuration -- Core business logic (chunking, embeddings) - -#### Environment-Specific Configs -- `.env.example` - Template for all env vars -- `.env.local` - Development (gitignored) -- `.env.production` - Production secrets (not in repo) - ---- - -## 2) Docker Setup - -### Docker Compose Architecture - -```yaml -# docker-compose.yml -version: '3.8' - -services: - # PostgreSQL (local dev only - production uses Supabase) - postgres: - image: pgvector/pgvector:pg16 - container_name: rag-postgres - environment: - POSTGRES_DB: enterprise_rag - POSTGRES_USER: postgres - POSTGRES_PASSWORD: postgres - volumes: - - postgres_data:/var/lib/postgresql/data - - ./scripts/init-db.sql:/docker-entrypoint-initdb.d/init.sql - ports: - - "5432:5432" - healthcheck: - test: ["CMD-SHELL", "pg_isready -U postgres"] - interval: 5s - timeout: 5s - retries: 5 - - # Redis (queue backend + cache) - redis: - image: redis:7-alpine - container_name: rag-redis - command: redis-server --appendonly yes - volumes: - - redis_data:/data - ports: - - "6379:6379" - healthcheck: - test: ["CMD", "redis-cli", "ping"] - interval: 5s - timeout: 3s - retries: 5 - - # FastAPI Server - server: - build: - context: ./server - dockerfile: Dockerfile - container_name: rag-server - environment: - - DATABASE_URL=postgresql://postgres:postgres@postgres:5432/enterprise_rag - - REDIS_URL=redis://redis:6379/0 - - SUPABASE_URL=${SUPABASE_URL} - - SUPABASE_KEY=${SUPABASE_KEY} - - SUPABASE_JWT_SECRET=${SUPABASE_JWT_SECRET} - - OPENAI_API_KEY=${OPENAI_API_KEY} - - ENVIRONMENT=development - volumes: - - ./server:/app - - /app/__pycache__ - ports: - - "8000:8000" - depends_on: - postgres: - condition: service_healthy - redis: - condition: service_healthy - command: uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload - - # RQ Workers (Extract) - worker-extract: - build: - context: ./worker - dockerfile: Dockerfile - container_name: rag-worker-extract - environment: - - DATABASE_URL=postgresql://postgres:postgres@postgres:5432/enterprise_rag - - REDIS_URL=redis://redis:6379/0 - - SUPABASE_URL=${SUPABASE_URL} - - SUPABASE_KEY=${SUPABASE_KEY} - - OPENAI_API_KEY=${OPENAI_API_KEY} - - QUEUE_NAME=ingest_extract - - WORKER_COUNT=5 - volumes: - - ./worker:/app - - ./server/app:/app/shared # Share code - depends_on: - - redis - - postgres - deploy: - replicas: 5 - command: python worker.py ingest_extract - - # RQ Workers (Index) - worker-index: - build: - context: ./worker - dockerfile: Dockerfile - container_name: rag-worker-index - environment: - - DATABASE_URL=postgresql://postgres:postgres@postgres:5432/enterprise_rag - - REDIS_URL=redis://redis:6379/0 - - SUPABASE_URL=${SUPABASE_URL} - - SUPABASE_KEY=${SUPABASE_KEY} - - OPENAI_API_KEY=${OPENAI_API_KEY} - - QUEUE_NAME=ingest_index - - WORKER_COUNT=3 - volumes: - - ./worker:/app - - ./server/app:/app/shared - depends_on: - - redis - - postgres - deploy: - replicas: 3 - command: python worker.py ingest_index - - # RQ Dashboard (monitoring) - rq-dashboard: - image: eoranged/rq-dashboard - container_name: rag-rq-dashboard - environment: - - RQ_DASHBOARD_REDIS_URL=redis://redis:6379/0 - ports: - - "9181:9181" - depends_on: - - redis - - # React Client - client: - build: - context: ./client - dockerfile: Dockerfile - target: development - container_name: rag-client - environment: - - VITE_API_URL=http://localhost:8000 - - VITE_SUPABASE_URL=${SUPABASE_URL} - - VITE_SUPABASE_ANON_KEY=${SUPABASE_ANON_KEY} - volumes: - - ./client:/app - - /app/node_modules - ports: - - "5173:5173" - command: npm run dev -- --host 0.0.0.0 - -volumes: - postgres_data: - redis_data: -``` - -### Server Dockerfile - -```dockerfile -# server/Dockerfile -FROM python:3.11-slim - -WORKDIR /app - -# Install system dependencies -RUN apt-get update && apt-get install -y \ - gcc \ - g++ \ - libpq-dev \ - && rm -rf /var/lib/apt/lists/* - -# Install Python dependencies -COPY requirements.txt . -RUN pip install --no-cache-dir -r requirements.txt - -# Copy application code -COPY . . - -# Create non-root user -RUN useradd -m -u 1000 appuser && chown -R appuser:appuser /app -USER appuser - -# Expose port -EXPOSE 8000 - -# Health check -HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \ - CMD python -c "import requests; requests.get('http://localhost:8000/health')" - -# Default command (can be overridden) -CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"] -``` - -### Server requirements.txt - -```txt -# server/requirements.txt - -# FastAPI -fastapi==0.109.0 -uvicorn[standard]==0.27.0 -pydantic==2.5.3 -pydantic-settings==2.1.0 - -# Database -sqlalchemy==2.0.25 -psycopg2-binary==2.9.9 -alembic==1.13.1 -pgvector==0.2.4 - -# Redis & Queue -redis==5.0.1 -rq==1.16.0 - -# Supabase -supabase==2.3.2 -storage3==0.7.4 - -# OpenAI -openai==1.10.0 - -# PDF Processing -unstructured==0.12.0 -pdf2image==1.17.0 -pillow==10.2.0 - -# Auth -python-jose[cryptography]==3.3.0 -python-multipart==0.0.6 - -# Text Processing -tiktoken==0.5.2 - -# Utilities -python-dotenv==1.0.0 -httpx==0.26.0 -tenacity==8.2.3 - -# Monitoring -prometheus-client==0.19.0 - -# Development -pytest==7.4.4 -pytest-asyncio==0.23.3 -pytest-cov==4.1.0 -black==24.1.1 -ruff==0.1.14 -mypy==1.8.0 -``` - -### Worker Dockerfile - -```dockerfile -# worker/Dockerfile -FROM python:3.11-slim - -WORKDIR /app - -# Install system dependencies (same as server) -RUN apt-get update && apt-get install -y \ - gcc \ - g++ \ - libpq-dev \ - poppler-utils \ - && rm -rf /var/lib/apt/lists/* - -# Install Python dependencies -COPY requirements.txt . -RUN pip install --no-cache-dir -r requirements.txt - -# Copy worker code -COPY . . - -# Create non-root user -RUN useradd -m -u 1000 worker && chown -R worker:worker /app -USER worker - -# Default command (queue name passed as arg) -CMD ["python", "worker.py"] -``` - -### Worker requirements.txt - -```txt -# worker/requirements.txt - -# Same as server/requirements.txt (workers share dependencies) -# Alternatively, use shared requirements.txt at root --r ../server/requirements.txt -``` - -### Client Dockerfile - -```dockerfile -# client/Dockerfile - -# Multi-stage build -FROM node:20-alpine AS base - -WORKDIR /app - -COPY package*.json ./ - -# Development stage -FROM base AS development -RUN npm install -COPY . . -EXPOSE 5173 -CMD ["npm", "run", "dev", "--", "--host", "0.0.0.0"] - -# Build stage -FROM base AS build -RUN npm ci -COPY . . -RUN npm run build - -# Production stage -FROM nginx:alpine AS production -COPY --from=build /app/dist /usr/share/nginx/html -COPY nginx.conf /etc/nginx/conf.d/default.conf -EXPOSE 80 -CMD ["nginx", "-g", "daemon off;"] -``` - -### Client package.json - -```json -{ - "name": "enterprise-rag-client", - "version": "1.0.0", - "type": "module", - "scripts": { - "dev": "vite", - "build": "tsc && vite build", - "preview": "vite preview", - "lint": "eslint . --ext ts,tsx", - "test": "vitest" - }, - "dependencies": { - "react": "^18.2.0", - "react-dom": "^18.2.0", - "react-router-dom": "^6.21.3", - "@supabase/supabase-js": "^2.39.3", - "@tanstack/react-query": "^5.17.19", - "axios": "^1.6.5", - "zustand": "^4.5.0", - "lucide-react": "^0.309.0", - "clsx": "^2.1.0", - "tailwindcss": "^3.4.1" - }, - "devDependencies": { - "@types/react": "^18.2.48", - "@types/react-dom": "^18.2.18", - "@vitejs/plugin-react": "^4.2.1", - "typescript": "^5.3.3", - "vite": "^5.0.11", - "vitest": "^1.2.0", - "eslint": "^8.56.0", - "autoprefixer": "^10.4.17", - "postcss": "^8.4.33" - } -} -``` - -### Environment Variables - -```bash -# .env.example - -# ===== Supabase (Required) ===== -SUPABASE_URL=https://your-project.supabase.co -SUPABASE_KEY=your-service-role-key -SUPABASE_ANON_KEY=your-anon-key -SUPABASE_JWT_SECRET=your-jwt-secret - -# ===== OpenAI (Required) ===== -OPENAI_API_KEY=sk-... - -# ===== Database (Local Dev) ===== -# Production uses Supabase Postgres -DATABASE_URL=postgresql://postgres:postgres@localhost:5432/enterprise_rag - -# ===== Redis (Required) ===== -REDIS_URL=redis://localhost:6379/0 - -# ===== Application Config ===== -ENVIRONMENT=development -LOG_LEVEL=INFO -API_HOST=0.0.0.0 -API_PORT=8000 - -# ===== Rate Limiting ===== -RATE_LIMIT_ENABLED=true - -# ===== Token Budget ===== -DAILY_TOKEN_LIMIT=100000 - -# ===== Client (React) ===== -VITE_API_URL=http://localhost:8000 -VITE_SUPABASE_URL=https://your-project.supabase.co -VITE_SUPABASE_ANON_KEY=your-anon-key -``` - -### Development Commands - -```bash -# Start all services -docker-compose up - -# Start specific services -docker-compose up server client - -# Rebuild after code changes -docker-compose up --build - -# View logs -docker-compose logs -f server -docker-compose logs -f worker-extract - -# Run migrations -docker-compose exec server alembic upgrade head - -# Access server shell -docker-compose exec server bash - -# Stop all services -docker-compose down - -# Clean everything (including volumes) -docker-compose down -v -``` - -### Production Deployment (docker-compose.prod.yml) - -```yaml -# docker-compose.prod.yml -version: '3.8' - -services: - server: - build: - context: ./server - dockerfile: Dockerfile - environment: - - DATABASE_URL=${DATABASE_URL} # Supabase Postgres - - REDIS_URL=${REDIS_URL} # Managed Redis - - ENVIRONMENT=production - deploy: - replicas: 3 - resources: - limits: - cpus: '2' - memory: 4G - restart: always - - worker-extract: - build: - context: ./worker - environment: - - DATABASE_URL=${DATABASE_URL} - - REDIS_URL=${REDIS_URL} - - QUEUE_NAME=ingest_extract - deploy: - replicas: 5 - resources: - limits: - cpus: '1' - memory: 2G - restart: always - - worker-index: - build: - context: ./worker - environment: - - DATABASE_URL=${DATABASE_URL} - - REDIS_URL=${REDIS_URL} - - QUEUE_NAME=ingest_index - deploy: - replicas: 3 - resources: - limits: - cpus: '1' - memory: 2G - restart: always - - client: - build: - context: ./client - target: production - ports: - - "80:80" - - "443:443" - volumes: - - ./ssl:/etc/nginx/ssl:ro - restart: always -``` - -### CI/CD Pipeline (.github/workflows/deploy.yml) - -```yaml -name: Deploy - -on: - push: - branches: [main] - -jobs: - test: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v3 - - - name: Run server tests - run: | - cd server - docker build -t server-test . - docker run server-test pytest - - - name: Run client tests - run: | - cd client - npm ci - npm test - - deploy: - needs: test - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v3 - - - name: Deploy to production - run: | - docker-compose -f docker-compose.prod.yml build - docker-compose -f docker-compose.prod.yml up -d -``` - ---- - -## 3) Core Principles - -### Design Philosophy -- **Strict grounded RAG**: LLM answers ONLY from retrieved context -- **Progressive availability**: Each document becomes searchable immediately after indexing -- **Cost control**: Hard daily token budget with atomic enforcement -- **Workspace isolation**: Every operation scoped to `workspace_id` -- **Fail-safe defaults**: Reject on uncertainty, never truncate silently - -### Out of Scope (v1) -- ❌ OCR for scanned PDFs -- ❌ Document summaries -- ❌ Batch orchestrator tables -- ❌ Multi-user workspaces (no roles/invites) -- ❌ Custom embedding models - ---- - -## 4) Platform Stack - -### Supabase (Managed Backend) -- **PostgreSQL** with pgvector extension -- **Storage** for PDF blobs -- **Auth** for JWT-based authentication - -### Application Layer -- **FastAPI** (API server) -- **Redis** (queue backend + cache) -- **RQ** (Redis Queue for workers) - -### AI Services -- **OpenAI Embeddings**: `text-embedding-3-small` (1536 dimensions) -- **OpenAI LLM**: `gpt-4o-mini` - ---- - -## 5) Workspace Model - -### Authentication -- Users authenticate via **Supabase Auth** (JWT tokens) -- JWT includes `user_id` from `auth.users.id` - -### Workspace Ownership -- Any authenticated user can create a workspace -- **Single-user workspace** in v1 (creator = owner) -- All data queries MUST filter by `workspace_id` - -### Workspace Limits -- 1 workspace per user in v1 (enforced at creation) -- Can be extended in v2 with workspace switching - ---- - -## 6) Hard Limits & Constraints - -### Document Limits (Per Workspace) -| Limit | Value | Enforcement | -|-------|-------|-------------| -| Max PDFs | 100 | Reject upload-prepare if exceeded | -| Max pages per PDF | 10 | Reject on upload-complete validation | -| Max file size | 20 MB | Reject on upload-prepare and re-validate on complete | - -### Query Limits (Per Request) -| Limit | Value | Enforcement | -|-------|-------|-------------| -| Max documents per query | 10 | Reject if `document_ids` array > 10 | -| Max total pages in query | 50 | Sum pages across selected docs, reject if > 50 | -| Max query text length | 500 chars | Reject with validation error | - -### Rate Limits (Per Workspace) -| Operation | Limit | Window | -|-----------|-------|--------| -| Queries | 100 requests | 1 minute | -| Upload-prepare | 10 requests | 1 minute | -| Upload-complete | 20 requests | 1 minute | - -### Content Requirements -- **Text-based PDFs only** (no OCR in v1) -- UI must communicate: "Scanned documents not supported" -- Extraction failures → `status = failed` - ---- - -## 7) Token Budget System - -### Daily Limit -- **100,000 tokens per workspace per UTC day** -- Resets at `00:00:00 UTC` - -### Token Accounting -Budget includes ALL token usage: -1. **Embedding tokens** (ingestion + query) -2. **LLM input tokens** (context + prompt) -3. **LLM output tokens** (generated answer) - -### Token Estimation Rules - -#### Query Embedding -```python -estimated_tokens = (len(query_text) / 4) * 1.3 # Conservative estimate -``` - -#### LLM Input -```python -estimated_input = ( - sum(chunk.token_count for chunk in retrieved_chunks) + - PROMPT_TEMPLATE_TOKENS + # ~200 tokens for system prompt - (len(query_text) / 4) -) -``` - -#### LLM Output -```python -MAX_OUTPUT_TOKENS = 2000 # Hard cap configured in API call -``` - -#### Total Reservation -```python -total_reservation = ( - estimated_query_embedding + - estimated_input + - MAX_OUTPUT_TOKENS -) -``` - -### Reservation Model (LLM Calls) - -**Before LLM call:** -```sql --- 1. Acquire row lock and check budget -BEGIN; -SELECT tokens_used, tokens_reserved -FROM workspace_daily_usage -WHERE workspace_id = ? AND date = CURRENT_DATE -FOR UPDATE; - --- 2. Check if reservation fits -IF (tokens_used + tokens_reserved + estimated_total) > 100000 THEN - ROLLBACK; - RETURN error("Budget exceeded"); -END IF; - --- 3. Reserve tokens -UPDATE workspace_daily_usage -SET tokens_reserved = tokens_reserved + estimated_total, - updated_at = NOW() -WHERE workspace_id = ? AND date = CURRENT_DATE; -COMMIT; -``` - -**After LLM call:** -```sql --- 4. Release reservation and deduct actual usage -BEGIN; -UPDATE workspace_daily_usage -SET tokens_reserved = tokens_reserved - estimated_total, - tokens_used = tokens_used + actual_total, - updated_at = NOW() -WHERE workspace_id = ? AND date = CURRENT_DATE; -COMMIT; -``` - -### Pre-check Model (Embedding Calls) - -**Before document ingestion:** -```python -# Estimate total embedding tokens for all chunks -estimated_embedding_tokens = sum( - (len(chunk.content) / 4) * 1.1 - for chunk in document_chunks -) - -# Check budget atomically -result = db.execute(""" - SELECT (tokens_used + tokens_reserved + %s) <= 100000 as fits - FROM workspace_daily_usage - WHERE workspace_id = %s AND date = CURRENT_DATE -""", [estimated_embedding_tokens, workspace_id]) - -if not result.fits: - # Reject entire document ingestion - update_document_status(doc_id, 'failed', - 'Insufficient token budget for embeddings') - return -``` - -**After embeddings:** -```sql --- Deduct actual usage immediately -UPDATE workspace_daily_usage -SET tokens_used = tokens_used + actual_embedding_tokens, - updated_at = NOW() -WHERE workspace_id = ? AND date = CURRENT_DATE; -``` - -### Reserved Token Cleanup (Critical) - -**Problem**: Server crashes or timeouts leave tokens reserved forever - -**Solution**: Background job runs every 5 minutes -```python -# Job: cleanup_stale_reservations() -# Runs: every 5 minutes - -STALE_THRESHOLD = 10 minutes - -reserved_entries = db.execute(""" - SELECT workspace_id, date, tokens_reserved - FROM workspace_daily_usage - WHERE tokens_reserved > 0 - AND updated_at < NOW() - INTERVAL '10 minutes' -""") - -for entry in reserved_entries: - db.execute(""" - UPDATE workspace_daily_usage - SET tokens_reserved = 0, - updated_at = NOW() - WHERE workspace_id = %s AND date = %s - """, [entry.workspace_id, entry.date]) - - # Log for investigation - logger.warning(f"Released stale reservation: {entry}") -``` - -### Budget Exceeded Response -```json -{ - "error": { - "code": "BUDGET_EXCEEDED", - "message": "Daily token limit reached for this workspace", - "details": { - "used": 98500, - "reserved": 1500, - "limit": 100000, - "remaining": 0, - "resets_at": "2024-02-14T00:00:00Z" - } - } -} -``` - ---- - -## 8) Chunking & Retrieval Strategy - -### Document Storage -- Store extracted text per page in `document_pages` table -- Preserve original page structure for citations -- Full page text available for user context - -### Chunking Rules (Page-Based) - -**Constraints:** -- Chunks **NEVER cross page boundaries** -- Each page may produce 1+ chunks if content is long -- Target chunk size: 400-600 tokens (adjustable) - -**Overlap Strategy:** -```python -# For chunks within the same page: -CHUNK_SIZE = 500 tokens -OVERLAP = 100 tokens - -# Example: Page with 1200 tokens -# Chunk 1: tokens 0-500 -# Chunk 2: tokens 400-900 (100 token overlap with chunk 1) -# Chunk 3: tokens 800-1200 (100 token overlap with chunk 2) -``` - -**Metadata Stored Per Chunk:** -- `page_start`: Page number where chunk starts -- `page_end`: Page number where chunk ends (same as page_start in v1) -- `chunk_index`: Sequential index within document -- `content_hash`: SHA256 for idempotency - -### Retrieval Strategy - -**Vector Search:** -```sql --- Retrieve top 5 chunks using pgvector -SELECT c.id, c.content, c.page_start, c.page_end, c.document_id, - ce.embedding <=> query_embedding AS similarity -FROM chunks c -JOIN chunk_embeddings ce ON ce.chunk_id = c.id -WHERE c.workspace_id = ? - AND c.document_id = ANY(selected_document_ids) -ORDER BY ce.embedding <=> query_embedding -LIMIT 5; -``` - -**HNSW Index Configuration:** -```sql -CREATE INDEX idx_chunk_embeddings_vector -ON chunk_embeddings -USING hnsw (embedding vector_cosine_ops) -WITH (m = 16, ef_construction = 64); - --- Runtime setting for queries: -SET hnsw.ef_search = 40; -``` - -**Context Assembly:** -1. Retrieve top 5 chunks -2. For each chunk, fetch full page text from `document_pages` -3. Present to LLM: chunk snippet + full page context -4. Display to user: answer + citations with page numbers - ---- - -## 9) Database Schema - -> **Critical**: All tables include `workspace_id` for isolation -> All queries MUST filter by `workspace_id` - -### Table: `workspaces` -```sql -CREATE TABLE workspaces ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - name TEXT NOT NULL, - owner_id UUID NOT NULL REFERENCES auth.users(id), - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() -); - -CREATE INDEX idx_workspaces_owner ON workspaces(owner_id); -``` - -### Table: `documents` -```sql -CREATE TABLE documents ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - workspace_id UUID NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE, - filename TEXT NOT NULL, - file_size_bytes BIGINT NOT NULL, - page_count INT, - file_hash_sha256 TEXT NOT NULL, - storage_path TEXT NOT NULL, - status TEXT NOT NULL DEFAULT 'pending_upload', - error_message TEXT, - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), - updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), - - CONSTRAINT chk_file_size CHECK (file_size_bytes > 0 AND file_size_bytes <= 20971520), - CONSTRAINT chk_page_count CHECK (page_count IS NULL OR (page_count > 0 AND page_count <= 10)), - CONSTRAINT chk_status CHECK (status IN ('pending_upload', 'uploaded', 'indexing', 'ready', 'failed')) -); - -CREATE UNIQUE INDEX idx_documents_workspace_hash - ON documents(workspace_id, file_hash_sha256); -CREATE INDEX idx_documents_workspace ON documents(workspace_id); -CREATE INDEX idx_documents_workspace_status ON documents(workspace_id, status); -``` - -**Status Flow:** -``` -pending_upload → uploaded → indexing → ready - ↓ - failed -``` - -### Table: `document_pages` -```sql -CREATE TABLE document_pages ( - id BIGSERIAL PRIMARY KEY, - workspace_id UUID NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE, - document_id UUID NOT NULL REFERENCES documents(id) ON DELETE CASCADE, - page_number INT NOT NULL, - content TEXT NOT NULL, - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), - - CONSTRAINT chk_page_number CHECK (page_number > 0) -); - -CREATE UNIQUE INDEX idx_document_pages_doc_page - ON document_pages(document_id, page_number); -CREATE INDEX idx_document_pages_workspace ON document_pages(workspace_id); -CREATE INDEX idx_document_pages_document ON document_pages(document_id); -``` - -### Table: `chunks` -```sql -CREATE TABLE chunks ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - workspace_id UUID NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE, - document_id UUID NOT NULL REFERENCES documents(id) ON DELETE CASCADE, - page_start INT NOT NULL, - page_end INT NOT NULL, - chunk_index INT NOT NULL, - content TEXT NOT NULL, - content_hash TEXT NOT NULL, - token_count INT NOT NULL, - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), - - CONSTRAINT chk_page_range CHECK (page_start > 0 AND page_end >= page_start), - CONSTRAINT chk_chunk_index CHECK (chunk_index >= 0), - CONSTRAINT chk_token_count CHECK (token_count > 0) -); - -CREATE UNIQUE INDEX idx_chunks_doc_index - ON chunks(document_id, chunk_index); -CREATE INDEX idx_chunks_workspace ON chunks(workspace_id); -CREATE INDEX idx_chunks_document ON chunks(document_id); -CREATE INDEX idx_chunks_content_hash ON chunks(content_hash); -``` - -### Table: `chunk_embeddings` -```sql -CREATE TABLE chunk_embeddings ( - chunk_id UUID PRIMARY KEY REFERENCES chunks(id) ON DELETE CASCADE, - workspace_id UUID NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE, - document_id UUID NOT NULL REFERENCES documents(id) ON DELETE CASCADE, - embedding vector(1536) NOT NULL, - embedding_model TEXT NOT NULL DEFAULT 'text-embedding-3-small', - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() -); - -CREATE INDEX idx_chunk_embeddings_workspace ON chunk_embeddings(workspace_id); -CREATE INDEX idx_chunk_embeddings_document ON chunk_embeddings(document_id); - --- HNSW index for vector similarity search -CREATE INDEX idx_chunk_embeddings_vector - ON chunk_embeddings - USING hnsw (embedding vector_cosine_ops) - WITH (m = 16, ef_construction = 64); -``` - -### Table: `workspace_daily_usage` -```sql -CREATE TABLE workspace_daily_usage ( - workspace_id UUID NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE, - date DATE NOT NULL, - tokens_used BIGINT NOT NULL DEFAULT 0, - tokens_reserved BIGINT NOT NULL DEFAULT 0, - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), - updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), - - PRIMARY KEY (workspace_id, date), - CONSTRAINT chk_tokens_non_negative CHECK (tokens_used >= 0 AND tokens_reserved >= 0) -); - -CREATE INDEX idx_workspace_daily_usage_date ON workspace_daily_usage(date); -``` - -### Table: `query_logs` -```sql -CREATE TABLE query_logs ( - id UUID PRIMARY KEY DEFAULT gen_random_uuid(), - workspace_id UUID NOT NULL REFERENCES workspaces(id) ON DELETE CASCADE, - user_id UUID NOT NULL, - query_text TEXT NOT NULL, - documents_searched UUID[] NOT NULL, - retrieved_chunk_ids UUID[] NOT NULL, - chunk_scores FLOAT[] NOT NULL, - answer_text TEXT, - error_message TEXT, - - retrieval_latency_ms INT, - llm_latency_ms INT, - total_latency_ms INT NOT NULL, - - embedding_tokens_used INT NOT NULL, - llm_input_tokens INT, - llm_output_tokens INT, - total_tokens_used INT NOT NULL, - - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() -); - -CREATE INDEX idx_query_logs_workspace ON query_logs(workspace_id); -CREATE INDEX idx_query_logs_user ON query_logs(user_id); -CREATE INDEX idx_query_logs_created ON query_logs(created_at DESC); -``` - ---- - -## 10) Upload Architecture - -**Goal**: Support bulk uploads (up to 100 PDFs) without API timeouts or payload size issues. - -### Flow Overview -``` -1. Client → POST /documents/upload-prepare -2. Server → Creates placeholders, returns signed URLs -3. Client → Uploads PDFs directly to Supabase Storage (parallel) -4. Client → POST /documents/upload-complete -5. Server → Validates, enqueues per-document jobs -``` - -### Endpoint 1: Prepare Upload - -**Request:** -```http -POST /documents/upload-prepare -Authorization: Bearer {jwt_token} -Content-Type: application/json - -{ - "files": [ - {"filename": "policy.pdf", "size_bytes": 1234567}, - {"filename": "manual.pdf", "size_bytes": 987654} - ] -} -``` - -**Server Logic:** -1. Extract `workspace_id` from JWT -2. Validate workspace document count + new files <= 100 -3. Validate each `size_bytes` <= 20 MB -4. For each file: - - Create `documents` row with `status = 'pending_upload'` - - Generate signed upload URL (Supabase Storage, 1 hour expiry) - - Use storage path: `workspaces/{workspace_id}/{document_id}.pdf` - -**Response:** -```json -{ - "uploads": [ - { - "document_id": "550e8400-e29b-41d4-a716-446655440000", - "filename": "policy.pdf", - "upload_url": "https://your-project.supabase.co/storage/v1/object/...", - "storage_path": "workspaces/{workspace_id}/{document_id}.pdf", - "expires_at": "2024-02-13T15:30:00Z" - } - ] -} -``` - -**Error Cases:** -```json -// Exceeds workspace limit -{ - "error": { - "code": "WORKSPACE_LIMIT_EXCEEDED", - "message": "Cannot upload 20 files. Workspace limit is 100 documents (currently 85).", - "details": { - "current_count": 85, - "requested": 20, - "limit": 100 - } - } -} - -// File too large -{ - "error": { - "code": "FILE_TOO_LARGE", - "message": "File 'large.pdf' exceeds 20MB limit", - "details": { - "filename": "large.pdf", - "size_bytes": 25000000, - "limit_bytes": 20971520 - } - } -} -``` - -### Client Upload Phase - -**Client responsibility:** -```javascript -// Upload files in parallel using signed URLs -const uploadPromises = prepareResponse.uploads.map(upload => - fetch(upload.upload_url, { - method: 'PUT', - body: pdfFile, - headers: {'Content-Type': 'application/pdf'} - }) -); - -await Promise.all(uploadPromises); -``` - -### Endpoint 2: Complete Upload - -**Request:** -```http -POST /documents/upload-complete -Authorization: Bearer {jwt_token} -Content-Type: application/json - -{ - "documents": [ - { - "document_id": "550e8400-e29b-41d4-a716-446655440000", - "filename": "policy.pdf", - "file_size_bytes": 1234567, - "page_count": 8, - "file_hash_sha256": "a3c5e8...", - "storage_path": "workspaces/{workspace_id}/{document_id}.pdf" - } - ] -} -``` - -**Server Logic (Per Document):** -```python -for doc_data in request.documents: - # 1. Idempotency check - atomic status transition - result = db.execute(""" - UPDATE documents - SET status = 'uploaded', - filename = %s, - file_size_bytes = %s, - page_count = %s, - file_hash_sha256 = %s, - storage_path = %s, - updated_at = NOW() - WHERE id = %s - AND workspace_id = %s - AND status = 'pending_upload' - RETURNING id - """, [doc_data.filename, doc_data.file_size_bytes, ...]) - - if result.rowcount == 0: - # Already processed or invalid state - skip - failed.append({ - "document_id": doc_data.document_id, - "reason": "Already processed or invalid state" - }) - continue - - # 2. Re-validate constraints - if doc_data.file_size_bytes > 20_971_520: - update_status(doc_id, 'failed', 'File exceeds 20MB') - failed.append(...) - continue - - if doc_data.page_count > 10: - update_status(doc_id, 'failed', 'Exceeds 10 page limit') - failed.append(...) - continue - - # 3. Check deduplication (workspace-scoped) - duplicate = db.execute(""" - SELECT id, filename FROM documents - WHERE workspace_id = %s - AND file_hash_sha256 = %s - AND id != %s - AND status = 'ready' - """, [workspace_id, doc_data.file_hash_sha256, doc_id]) - - if duplicate: - # Point to existing document, delete placeholder - update_status(doc_id, 'failed', - f'Duplicate of existing document: {duplicate.filename}') - failed.append(...) - continue - - # 4. Enqueue extraction job - queue.enqueue('ingest_extract', document_id=doc_id) - successful.append({ - "document_id": doc_id, - "status": "uploaded" - }) -``` - -**Response (Partial Success Support):** -```json -{ - "successful": [ - { - "document_id": "550e8400-e29b-41d4-a716-446655440000", - "status": "uploaded", - "message": "Queued for processing" - } - ], - "failed": [ - { - "document_id": "7c9e6679-7425-40de-944b-e07fc1f90ae7", - "reason": "Exceeds 10 page limit", - "details": {"page_count": 15, "limit": 10} - } - ] -} -``` - ---- - -## 11) Ingestion Pipeline - -### Queue Configuration -```python -# Queue names -QUEUE_EXTRACT = 'ingest_extract' -QUEUE_INDEX = 'ingest_index' - -# Worker allocation -WORKERS = { - 'ingest_extract': 5, # I/O bound (PDF parsing) - 'ingest_index': 3 # API rate-limit bound (OpenAI) -} - -# Job settings -DEFAULT_JOB_TIMEOUT = 600 # 10 minutes -MAX_RETRIES = 3 -``` - -### Job 1: Extract Text (`ingest_extract`) - -**Input:** `document_id` - -**Steps:** -```python -def ingest_extract(document_id: str): - try: - # 1. Load document metadata - doc = db.get_document(document_id) - - # 2. Download PDF from Supabase Storage - pdf_bytes = supabase.storage.download(doc.storage_path) - - # 3. Extract text using Unstructured - from unstructured.partition.pdf import partition_pdf - elements = partition_pdf( - file=BytesIO(pdf_bytes), - strategy="fast" # No OCR - ) - - # 4. Group by page - pages = {} - for element in elements: - page_num = element.metadata.page_number - if page_num not in pages: - pages[page_num] = [] - pages[page_num].append(element.text) - - # 5. Validate page count - if len(pages) > 10: - raise ValueError(f"Document has {len(pages)} pages, limit is 10") - - if len(pages) == 0: - raise ValueError("No text extracted - document may be scanned/OCR required") - - # 6. Insert document_pages - for page_num, texts in sorted(pages.items()): - content = "\n".join(texts) - db.insert_document_page( - workspace_id=doc.workspace_id, - document_id=document_id, - page_number=page_num, - content=content - ) - - # 7. Update document status - db.update_document( - document_id, - status='indexing', - page_count=len(pages) - ) - - # 8. Enqueue indexing job - queue.enqueue(QUEUE_INDEX, 'ingest_index', document_id=document_id) - - except Exception as e: - logger.error(f"Extract failed for {document_id}: {e}") - db.update_document( - document_id, - status='failed', - error_message=str(e) - ) - raise # Let RQ handle retry logic -``` - -### Job 2: Index Document (`ingest_index`) - -**Input:** `document_id` - -**Steps:** -```python -def ingest_index(document_id: str): - try: - # 1. Load document and pages - doc = db.get_document(document_id) - pages = db.get_document_pages(document_id) - - # 2. Chunk each page - all_chunks = [] - chunk_index = 0 - - for page in pages: - page_chunks = chunk_text( - text=page.content, - max_tokens=500, - overlap_tokens=100 - ) - - for chunk_text in page_chunks: - all_chunks.append({ - 'chunk_index': chunk_index, - 'page_start': page.page_number, - 'page_end': page.page_number, - 'content': chunk_text, - 'content_hash': hashlib.sha256(chunk_text.encode()).hexdigest() - }) - chunk_index += 1 - - # 3. Estimate embedding token cost - estimated_tokens = sum( - int(len(chunk['content']) / 4 * 1.1) - for chunk in all_chunks - ) - - # 4. Check token budget - usage = db.get_daily_usage(doc.workspace_id) - if (usage.tokens_used + usage.tokens_reserved + estimated_tokens) > 100_000: - raise BudgetExceededError( - f"Insufficient budget for embeddings: {estimated_tokens} tokens needed" - ) - - # 5. Insert chunks (idempotent via content_hash) - chunk_ids = [] - for chunk_data in all_chunks: - chunk_id = db.upsert_chunk( - workspace_id=doc.workspace_id, - document_id=document_id, - **chunk_data - ) - chunk_ids.append(chunk_id) - - # 6. Generate embeddings - texts = [c['content'] for c in all_chunks] - embeddings = openai.embeddings.create( - model='text-embedding-3-small', - input=texts - ) - - actual_tokens = embeddings.usage.total_tokens - - # 7. Insert embeddings - for chunk_id, embedding_data in zip(chunk_ids, embeddings.data): - db.insert_chunk_embedding( - chunk_id=chunk_id, - workspace_id=doc.workspace_id, - document_id=document_id, - embedding=embedding_data.embedding, - embedding_model='text-embedding-3-small' - ) - - # 8. Deduct token usage - db.increment_token_usage(doc.workspace_id, actual_tokens) - - # 9. Mark document ready - db.update_document(document_id, status='ready') - - except BudgetExceededError as e: - logger.warning(f"Budget exceeded for {document_id}: {e}") - db.update_document( - document_id, - status='failed', - error_message=str(e) - ) - # Don't retry - budget issue - - except Exception as e: - logger.error(f"Index failed for {document_id}: {e}") - db.update_document( - document_id, - status='failed', - error_message=str(e) - ) - raise # Let RQ handle retry logic -``` - ---- - -## 12) Query System (RAG) - -### Endpoint: POST /query - -**Request:** -```json -{ - "question": "What is the refund policy for defective products?", - "document_ids": [ - "550e8400-e29b-41d4-a716-446655440000", - "7c9e6679-7425-40de-944b-e07fc1f90ae7" - ] -} -``` - -**Validation:** -```python -# 1. Required fields -if not request.document_ids: - return error("document_ids required") - -# 2. Document count limit -if len(request.document_ids) > 10: - return error("Maximum 10 documents per query") - -# 3. Query length limit -if len(request.question) > 500: - return error("Question exceeds 500 character limit") - -# 4. Verify all documents are ready -docs = db.get_documents(request.document_ids, workspace_id) -not_ready = [d for d in docs if d.status != 'ready'] -if not_ready: - return error(f"Documents not ready: {[d.id for d in not_ready]}") - -# 5. Check total page count -total_pages = sum(d.page_count for d in docs) -if total_pages > 50: - return error(f"Total pages ({total_pages}) exceeds limit of 50") -``` - -**Processing Flow:** -```python -def process_query(question: str, document_ids: List[str], workspace_id: str): - start_time = time.time() - - # === STEP 1: Generate query embedding === - query_embedding_start = time.time() - - query_embed_response = openai.embeddings.create( - model='text-embedding-3-small', - input=question - ) - query_embedding = query_embed_response.data[0].embedding - embedding_tokens = query_embed_response.usage.total_tokens - - retrieval_latency = int((time.time() - query_embedding_start) * 1000) - - # === STEP 2: Reserve tokens for LLM call === - PROMPT_TEMPLATE_TOKENS = 200 - MAX_OUTPUT_TOKENS = 2000 - - # Will retrieve chunks - estimate worst case - estimated_chunk_tokens = 5 * 600 # 5 chunks * max 600 tokens each - estimated_input = estimated_chunk_tokens + PROMPT_TEMPLATE_TOKENS + len(question) // 4 - estimated_total = embedding_tokens + estimated_input + MAX_OUTPUT_TOKENS - - # Atomic reservation - try: - reserve_tokens(workspace_id, estimated_total) - except BudgetExceededError as e: - # Log query attempt even though it failed - log_query( - workspace_id=workspace_id, - query_text=question, - error_message=str(e), - total_tokens_used=embedding_tokens - ) - raise - - try: - # === STEP 3: Retrieve chunks === - chunks = db.execute(""" - SELECT c.id, c.content, c.page_start, c.page_end, - c.document_id, d.filename, - ce.embedding <=> %s::vector AS similarity - FROM chunks c - JOIN chunk_embeddings ce ON ce.chunk_id = c.id - JOIN documents d ON d.id = c.document_id - WHERE c.workspace_id = %s - AND c.document_id = ANY(%s) - ORDER BY ce.embedding <=> %s::vector - LIMIT 5 - """, [query_embedding, workspace_id, document_ids, query_embedding]) - - if not chunks: - raise InsufficientContextError("No relevant content found in selected documents") - - # === STEP 4: Build LLM prompt === - context = build_context(chunks) - prompt = build_grounded_prompt(question, context) - - # === STEP 5: Call LLM === - llm_start = time.time() - - response = openai.chat.completions.create( - model='gpt-4o-mini', - messages=[ - {"role": "system", "content": SYSTEM_PROMPT}, - {"role": "user", "content": prompt} - ], - max_tokens=MAX_OUTPUT_TOKENS, - temperature=0.1 - ) - - llm_latency = int((time.time() - llm_start) * 1000) - - answer = response.choices[0].message.content - actual_input_tokens = response.usage.prompt_tokens - actual_output_tokens = response.usage.completion_tokens - actual_total = actual_input_tokens + actual_output_tokens - - # === STEP 6: Release reservation, deduct actual usage === - release_reservation_and_charge( - workspace_id, - reserved=estimated_total, - actual=embedding_tokens + actual_total - ) - - # === STEP 7: Extract citations === - citations = extract_citations(answer, chunks) - - # === STEP 8: Log query === - total_latency = int((time.time() - start_time) * 1000) - log_query( - workspace_id=workspace_id, - user_id=current_user_id, - query_text=question, - documents_searched=document_ids, - retrieved_chunk_ids=[c.id for c in chunks], - chunk_scores=[c.similarity for c in chunks], - answer_text=answer, - retrieval_latency_ms=retrieval_latency, - llm_latency_ms=llm_latency, - total_latency_ms=total_latency, - embedding_tokens_used=embedding_tokens, - llm_input_tokens=actual_input_tokens, - llm_output_tokens=actual_output_tokens, - total_tokens_used=embedding_tokens + actual_total - ) - - return { - "answer": answer, - "citations": citations, - "token_usage": { - "embedding": embedding_tokens, - "input": actual_input_tokens, - "output": actual_output_tokens, - "total": embedding_tokens + actual_total - } - } - - except Exception as e: - # Release reservation on any error - release_reservation(workspace_id, estimated_total) - raise -``` - -### System Prompt (Strict Grounding) -```python -SYSTEM_PROMPT = """You are a helpful assistant that answers questions based ONLY on the provided document context. - -CRITICAL RULES: -1. Answer ONLY using information from the context provided -2. If the context does not contain enough information, respond: "The provided documents do not contain sufficient information to answer this question." -3. ALWAYS cite your sources using this format: [Document: {filename}, Page {page_number}] -4. Include direct quotes when possible to support your answer -5. Do NOT use external knowledge or make assumptions -6. Do NOT speculate or infer beyond what is explicitly stated - -Your response should be: -- Accurate and grounded in the context -- Properly cited with document and page references -- Clear about limitations when information is insufficient -""" -``` - -### User Prompt Template -```python -def build_grounded_prompt(question: str, chunks: List[Chunk]) -> str: - context_parts = [] - - for i, chunk in enumerate(chunks, 1): - context_parts.append(f""" -[Document: {chunk.filename}, Page {chunk.page_start}] -{chunk.content} -""") - - context = "\n\n".join(context_parts) - - return f"""Context from documents: -{context} - -Question: {question} - -Answer (cite sources):""" -``` - -### Response Format -```json -{ - "answer": "According to the refund policy, defective products can be returned within 30 days of purchase for a full refund. The product must be in its original packaging with proof of purchase.", - "citations": [ - { - "document_name": "refund_policy.pdf", - "page": 3, - "snippet": "Defective products can be returned within 30 days of purchase for a full refund" - } - ], - "token_usage": { - "embedding": 45, - "input": 1250, - "output": 87, - "total": 1382 - } -} -``` - -### Insufficient Context Response -```json -{ - "answer": "The provided documents do not contain sufficient information to answer this question about extended warranty coverage.", - "citations": [], - "token_usage": { - "embedding": 42, - "input": 850, - "output": 35, - "total": 927 - } -} -``` - ---- - -## 13) API Endpoints - -### Authentication -All endpoints require: -```http -Authorization: Bearer {supabase_jwt_token} -``` - -JWT payload must include: -- `sub`: user_id -- `workspace_id`: extracted from user's workspace membership - -### Workspace Management - -#### Create Workspace -```http -POST /workspaces -Content-Type: application/json - -{ - "name": "My Company Workspace" -} -``` - -**Response (201 Created):** -```json -{ - "workspace_id": "550e8400-e29b-41d4-a716-446655440000", - "name": "My Company Workspace", - "owner_id": "user_uuid", - "created_at": "2024-02-13T10:30:00Z" -} -``` - -#### Get Current Workspace -```http -GET /workspaces/me -``` - -**Response (200 OK):** -```json -{ - "workspace": { - "id": "550e8400-e29b-41d4-a716-446655440000", - "name": "My Company Workspace", - "owner_id": "user_uuid", - "created_at": "2024-02-13T10:30:00Z" - }, - "stats": { - "document_count": 42, - "document_limit": 100, - "documents_by_status": { - "ready": 38, - "indexing": 3, - "failed": 1 - } - }, - "usage_today": { - "tokens_used": 45200, - "tokens_reserved": 800, - "limit": 100000, - "remaining": 54000, - "resets_at": "2024-02-14T00:00:00Z" - } -} -``` - -### Document Management - -#### Prepare Upload -```http -POST /documents/upload-prepare -Content-Type: application/json - -{ - "files": [ - {"filename": "policy.pdf", "size_bytes": 1234567} - ] -} -``` - -**Response (200 OK):** -```json -{ - "uploads": [ - { - "document_id": "doc_uuid", - "filename": "policy.pdf", - "upload_url": "https://...", - "storage_path": "workspaces/{workspace_id}/{document_id}.pdf", - "expires_at": "2024-02-13T15:30:00Z" - } - ] -} -``` - -#### Complete Upload -```http -POST /documents/upload-complete -Content-Type: application/json - -{ - "documents": [ - { - "document_id": "doc_uuid", - "filename": "policy.pdf", - "file_size_bytes": 1234567, - "page_count": 8, - "file_hash_sha256": "a3c5e8...", - "storage_path": "workspaces/{workspace_id}/{document_id}.pdf" - } - ] -} -``` - -**Response (200 OK):** -```json -{ - "successful": [ - { - "document_id": "doc_uuid", - "status": "uploaded", - "message": "Queued for processing" - } - ], - "failed": [] -} -``` - -#### List Documents -```http -GET /documents?page=1&page_size=20&status=ready +```text +MultiDoc-RAG/ ``` -**Response (200 OK):** -```json -{ - "documents": [ - { - "id": "doc_uuid", - "filename": "policy.pdf", - "file_size_bytes": 1234567, - "page_count": 8, - "status": "ready", - "created_at": "2024-02-13T10:00:00Z" - } - ], - "pagination": { - "page": 1, - "page_size": 20, - "total_count": 42, - "total_pages": 3 - } -} -``` - -#### Get Document -```http -GET /documents/{document_id} -``` - -**Response (200 OK):** -```json -{ - "id": "doc_uuid", - "filename": "policy.pdf", - "file_size_bytes": 1234567, - "page_count": 8, - "file_hash_sha256": "a3c5e8...", - "status": "ready", - "error_message": null, - "created_at": "2024-02-13T10:00:00Z", - "updated_at": "2024-02-13T10:05:00Z" -} -``` - -#### Delete Document -```http -DELETE /documents/{document_id} -``` - -**Response (204 No Content)** - -**Server Logic:** -```python -# Cascade delete (handled by FK constraints): -# - document_pages -# - chunks -# - chunk_embeddings -# Delete storage object -supabase.storage.delete(doc.storage_path) - -# Delete document row -db.delete_document(document_id) -``` - -### Query - -#### Ask Question -```http -POST /query -Content-Type: application/json - -{ - "question": "What is the refund policy?", - "document_ids": ["doc_uuid_1", "doc_uuid_2"] -} -``` - -**Response (200 OK):** -```json -{ - "answer": "...", - "citations": [...], - "token_usage": { - "embedding": 45, - "input": 1250, - "output": 87, - "total": 1382 - } -} -``` +## Package Naming Example -### Usage - -#### Get Today's Usage -```http -GET /usage/today -``` - -**Response (200 OK):** ```json { - "date": "2024-02-13", - "tokens_used": 45200, - "tokens_reserved": 800, - "limit": 100000, - "remaining": 54000, - "resets_at": "2024-02-14T00:00:00Z" + "name": "multidoc-rag-client" } ``` - -#### Get Usage Breakdown -```http -GET /usage/breakdown?days=7 -``` - -**Response (200 OK):** -```json -{ - "period": { - "start": "2024-02-07", - "end": "2024-02-13" - }, - "by_date": [ - { - "date": "2024-02-13", - "tokens_used": 45200, - "queries": 128, - "documents_indexed": 5 - } - ], - "by_operation": { - "embeddings_ingestion": 12500, - "embeddings_query": 5800, - "llm_input": 18900, - "llm_output": 8000 - }, - "top_documents": [ - { - "document_id": "doc_uuid", - "filename": "policy.pdf", - "tokens_used": 8500 - } - ] -} -``` - ---- - -## 14) Worker Configuration - -### Queue Setup (Redis) -```python -from redis import Redis -from rq import Queue - -redis_conn = Redis( - host=os.getenv('REDIS_HOST'), - port=6379, - db=0, - decode_responses=False -) - -queue_extract = Queue('ingest_extract', connection=redis_conn) -queue_index = Queue('ingest_index', connection=redis_conn) -``` - -### Worker Processes -```bash -# Start extract workers (5 processes) -rq worker ingest_extract --burst & -rq worker ingest_extract --burst & -rq worker ingest_extract --burst & -rq worker ingest_extract --burst & -rq worker ingest_extract --burst & - -# Start index workers (3 processes) -rq worker ingest_index --burst & -rq worker ingest_index --burst & -rq worker ingest_index --burst & -``` - -### Worker Configuration -```python -# worker_config.py -WORKER_CONFIG = { - 'ingest_extract': { - 'count': 5, - 'timeout': 600, # 10 minutes - 'max_retries': 3, - 'retry_delays': [10, 60, 300] # 10s, 1m, 5m - }, - 'ingest_index': { - 'count': 3, - 'timeout': 600, - 'max_retries': 3, - 'retry_delays': [10, 60, 300] - } -} -``` - -### Job Monitoring -```python -# Get queue status -def get_queue_status(queue_name: str) -> dict: - queue = Queue(queue_name, connection=redis_conn) - - return { - 'name': queue_name, - 'queued': queue.count, - 'started': len(queue.started_job_registry), - 'finished': len(queue.finished_job_registry), - 'failed': len(queue.failed_job_registry), - 'deferred': len(queue.deferred_job_registry) - } -``` - ---- - -## 15) Error Handling & Retries - -### Retry Policy - -#### Transient Errors (Retry) -- Network timeouts -- OpenAI rate limits (429) -- Supabase temporary unavailability -- Redis connection errors - -**Strategy:** -```python -MAX_RETRIES = 3 -RETRY_DELAYS = [10, 60, 300] # seconds: 10s, 1m, 5m - -# Exponential backoff with jitter -import random -def get_retry_delay(attempt: int) -> int: - base_delay = RETRY_DELAYS[min(attempt, len(RETRY_DELAYS) - 1)] - jitter = random.uniform(0, base_delay * 0.1) - return base_delay + jitter -``` - -#### Permanent Errors (No Retry) -- Invalid PDF format -- Document exceeds limits (pages, size) -- Budget exceeded (for ingestion) -- Extraction yields no text - -**Handling:** -```python -class PermanentError(Exception): - """Errors that should not be retried""" - pass - -class BudgetExceededError(PermanentError): - pass - -class InvalidDocumentError(PermanentError): - pass - -# In worker job -try: - process_document() -except PermanentError as e: - # Mark as failed immediately, don't retry - update_status('failed', str(e)) - return # Don't raise - prevents RQ retry -except Exception as e: - # Let RQ handle retry - raise -``` - -### Error Messages (User-Facing) - -**Document Processing Errors:** -```python -ERROR_MESSAGES = { - 'page_limit': 'Document exceeds 10 page limit (found {page_count} pages)', - 'extraction_failed': 'Unable to extract text. Ensure PDF contains selectable text (not scanned images)', - 'budget_exceeded': 'Insufficient token budget to index this document. Estimated: {estimated} tokens, Available: {available}', - 'invalid_format': 'Invalid PDF format or corrupted file', - 'timeout': 'Processing timed out after multiple retries. Please try re-uploading.', -} -``` - -### Dead Letter Queue -```python -# After max retries, move to DLQ for manual investigation -def handle_max_retries_exceeded(job): - dlq = Queue('dead_letter_queue', connection=redis_conn) - dlq.enqueue('log_failed_job', job_id=job.id, error=job.exc_info) - - # Update document status - db.update_document( - job.kwargs['document_id'], - status='failed', - error_message='Max retries exceeded. Support has been notified.' - ) -``` - ---- - -## 16) Monitoring & Health Checks - -### Health Check Endpoint -```http -GET /health -``` - -**Response (200 OK):** -```json -{ - "status": "healthy", - "timestamp": "2024-02-13T10:30:00Z", - "components": { - "database": { - "status": "healthy", - "latency_ms": 12 - }, - "redis": { - "status": "healthy", - "latency_ms": 3 - }, - "storage": { - "status": "healthy", - "latency_ms": 45 - }, - "workers": { - "ingest_extract": { - "status": "healthy", - "queued": 15, - "processing": 3, - "failed_last_hour": 2 - }, - "ingest_index": { - "status": "warning", - "queued": 42, - "processing": 3, - "failed_last_hour": 8 - } - } - } -} -``` - -### Metrics to Track - -#### Queue Metrics -```python -# Monitor every minute -metrics = { - 'queue_depth': queue.count, - 'jobs_processing': len(queue.started_job_registry), - 'jobs_failed_1h': count_failed_last_hour(), - 'avg_job_duration': calculate_avg_duration(), -} - -# Alerts -if metrics['queue_depth'] > 100: - alert('Queue depth high', severity='warning') - -if metrics['jobs_failed_1h'] > 10: - alert('High failure rate', severity='critical') -``` - -#### Token Usage Metrics -```python -# Track per workspace -metrics = { - 'tokens_used_today': get_usage(workspace_id), - 'tokens_reserved': get_reserved(workspace_id), - 'percent_used': (tokens_used / 100000) * 100, -} - -# Alerts (per workspace) -if metrics['percent_used'] > 90: - notify_workspace_owner('Approaching daily token limit') -``` - -#### Worker Health -```python -# Check worker heartbeat -def check_worker_health(): - workers = Worker.all(connection=redis_conn) - - for worker in workers: - last_heartbeat = worker.last_heartbeat - - if (datetime.now() - last_heartbeat).seconds > 300: - alert(f'Worker {worker.name} not responding', severity='critical') -``` - -### Alerting Rules - -| Condition | Severity | Action | -|-----------|----------|--------| -| Queue depth > 100 for 5+ min | Warning | Scale workers | -| Failed jobs > 10/hour | Critical | Investigate | -| Worker no heartbeat 5+ min | Critical | Restart worker | -| Workspace at 90% budget | Info | Notify user | -| Workspace at 100% budget | Warning | Notify user | -| Database latency > 500ms | Warning | Check DB load | - ---- - -## 17) Security & Rate Limiting - -### Authentication -```python -from fastapi import Depends, HTTPException, Header -from jose import jwt, JWTError - -async def get_current_user(authorization: str = Header(...)): - """Extract user_id from Supabase JWT""" - try: - token = authorization.replace('Bearer ', '') - payload = jwt.decode( - token, - SUPABASE_JWT_SECRET, - algorithms=['HS256'] - ) - user_id = payload['sub'] - return user_id - except JWTError: - raise HTTPException(status_code=401, detail='Invalid token') - -async def get_workspace_id(user_id: str = Depends(get_current_user)): - """Get user's workspace""" - workspace = db.get_user_workspace(user_id) - if not workspace: - raise HTTPException(status_code=404, detail='No workspace found') - return workspace.id -``` - -### Rate Limiting (Redis) -```python -from fastapi import Request -from redis import Redis - -redis_client = Redis(...) - -RATE_LIMITS = { - 'query': (100, 60), # 100 requests per 60 seconds - 'upload-prepare': (10, 60), # 10 requests per 60 seconds - 'upload-complete': (20, 60), # 20 requests per 60 seconds -} - -async def rate_limit(request: Request, operation: str, workspace_id: str): - limit, window = RATE_LIMITS[operation] - key = f'ratelimit:{workspace_id}:{operation}' - - current = redis_client.incr(key) - - if current == 1: - redis_client.expire(key, window) - - if current > limit: - raise HTTPException( - status_code=429, - detail=f'Rate limit exceeded. Max {limit} requests per {window}s', - headers={'Retry-After': str(window)} - ) -``` - -### Workspace Isolation (Critical) -```python -# EVERY database query MUST include workspace_id filter -# BAD - Vulnerable to unauthorized access -def get_document(document_id: str): - return db.query("SELECT * FROM documents WHERE id = ?", [document_id]) - -# GOOD - Enforces workspace isolation -def get_document(document_id: str, workspace_id: str): - return db.query(""" - SELECT * FROM documents - WHERE id = ? AND workspace_id = ? - """, [document_id, workspace_id]) -``` - -### Input Validation -```python -from pydantic import BaseModel, Field, validator - -class QueryRequest(BaseModel): - question: str = Field(..., max_length=500) - document_ids: List[str] = Field(..., min_items=1, max_items=10) - - @validator('question') - def question_not_empty(cls, v): - if not v.strip(): - raise ValueError('Question cannot be empty') - return v - - @validator('document_ids') - def valid_uuids(cls, v): - for doc_id in v: - try: - uuid.UUID(doc_id) - except ValueError: - raise ValueError(f'Invalid document ID: {doc_id}') - return v -``` - ---- - -## 18) Maintenance & Operations - -### Background Jobs - -#### Daily Token Reset -```python -# Runs at 00:00:05 UTC daily -def reset_daily_budgets(): - """Reset all workspace token budgets""" - db.execute(""" - INSERT INTO workspace_daily_usage (workspace_id, date, tokens_used, tokens_reserved) - SELECT id, CURRENT_DATE, 0, 0 - FROM workspaces - ON CONFLICT (workspace_id, date) DO NOTHING - """) -``` - -#### Cleanup Old Query Logs -```python -# Runs daily at 02:00 UTC -def cleanup_old_logs(): - """Delete query logs older than 90 days""" - deleted = db.execute(""" - DELETE FROM query_logs - WHERE created_at < NOW() - INTERVAL '90 days' - """) - logger.info(f"Deleted {deleted.rowcount} old query logs") -``` - -#### Release Stale Reservations -```python -# Runs every 5 minutes -def cleanup_stale_reservations(): - """Release token reservations older than 10 minutes""" - db.execute(""" - UPDATE workspace_daily_usage - SET tokens_reserved = 0, - updated_at = NOW() - WHERE tokens_reserved > 0 - AND updated_at < NOW() - INTERVAL '10 minutes' - """) -``` - -#### Reindex Embeddings (Manual) -```python -# For maintenance/upgrades -def reindex_embeddings_batch(batch_size=100): - """Reindex with HNSW optimization""" - db.execute("VACUUM ANALYZE chunk_embeddings") - db.execute(""" - REINDEX INDEX CONCURRENTLY idx_chunk_embeddings_vector - """) -``` - -### Database Maintenance -```sql --- Run weekly -VACUUM ANALYZE documents; -VACUUM ANALYZE chunks; -VACUUM ANALYZE chunk_embeddings; -VACUUM ANALYZE workspace_daily_usage; - --- Monitor index bloat -SELECT - schemaname, - tablename, - indexname, - pg_size_pretty(pg_relation_size(indexrelid)) AS index_size -FROM pg_stat_user_indexes -WHERE schemaname = 'public' -ORDER BY pg_relation_size(indexrelid) DESC; -``` - -### Backup Strategy -```bash -# Daily automated backups (Supabase handles this) -# Point-in-time recovery available - -# Manual backup for critical migrations -pg_dump -h supabase-host -U postgres -d database_name > backup.sql - -# Backup storage bucket -supabase storage backup --bucket document-storage -``` - -### Deployment Checklist - -**Pre-deployment:** -- [ ] Run database migrations -- [ ] Update environment variables -- [ ] Test worker connectivity -- [ ] Verify OpenAI API keys -- [ ] Check Supabase quotas - -**Post-deployment:** -- [ ] Verify health endpoint -- [ ] Check worker queue status -- [ ] Test end-to-end upload flow -- [ ] Monitor error rates (15 min) -- [ ] Verify token budget tracking - ---- - -## Appendix A: Token Estimation Reference - -### Embedding Tokens -```python -# text-embedding-3-small -def estimate_embedding_tokens(text: str) -> int: - # Conservative estimate: 1 token per 4 characters, +10% overhead - return int(len(text) / 4 * 1.1) - -# Example -text = "This is a sample chunk of text from a PDF document." -tokens = estimate_embedding_tokens(text) # ~15 tokens -``` - -### LLM Tokens -```python -# gpt-4o-mini -def estimate_llm_tokens(text: str) -> int: - # More accurate: use tiktoken - import tiktoken - encoder = tiktoken.encoding_for_model('gpt-4o-mini') - return len(encoder.encode(text)) - -# Conservative fallback -def estimate_llm_tokens_fallback(text: str) -> int: - return int(len(text) / 4) -``` - ---- - -## Appendix B: Example Queries - -### Create Workspace -```bash -curl -X POST https://api.example.com/workspaces \ - -H "Authorization: Bearer $TOKEN" \ - -H "Content-Type: application/json" \ - -d '{"name": "My Workspace"}' -``` - -### Upload Documents -```bash -# 1. Prepare -response=$(curl -X POST https://api.example.com/documents/upload-prepare \ - -H "Authorization: Bearer $TOKEN" \ - -H "Content-Type: application/json" \ - -d '{"files": [{"filename": "doc.pdf", "size_bytes": 123456}]}') - -upload_url=$(echo $response | jq -r '.uploads[0].upload_url') -doc_id=$(echo $response | jq -r '.uploads[0].document_id') - -# 2. Upload to Supabase -curl -X PUT "$upload_url" \ - -H "Content-Type: application/pdf" \ - --data-binary @doc.pdf - -# 3. Complete -curl -X POST https://api.example.com/documents/upload-complete \ - -H "Authorization: Bearer $TOKEN" \ - -H "Content-Type: application/json" \ - -d "{\"documents\": [{\"document_id\": \"$doc_id\", ...}]}" -``` - -### Query -```bash -curl -X POST https://api.example.com/query \ - -H "Authorization: Bearer $TOKEN" \ - -H "Content-Type: application/json" \ - -d '{ - "question": "What is the refund policy?", - "document_ids": ["doc-uuid-1", "doc-uuid-2"] - }' -``` - ---- - -## Appendix C: Migration Path (Future) - -### v1 → v2 Potential Enhancements -- Multi-user workspaces with roles -- OCR support for scanned PDFs -- Document summaries -- Batch upload orchestrator -- Custom embedding models -- Hybrid search (keyword + semantic) -- Document versioning -- Export to various formats - -**Migration Considerations:** -- Add `embedding_version` to support model upgrades -- Add `processing_version` to track chunking strategy changes -- Implement background re-processing jobs for upgrades - ---- - -**END OF SPECIFICATION** - -This architecture is locked for v1 implementation. Any changes require architectural review and version increment. diff --git a/README.md b/README.md index 7d2a0b6..1fdc19d 100644 --- a/README.md +++ b/README.md @@ -1,588 +1,119 @@ -# Enterprise RAG +# MultiDoc-RAG -Enterprise RAG is a workspace-scoped document intelligence platform for uploading PDFs, extracting text, indexing chunks with embeddings, and answering questions with grounded citations. +MultiDoc-RAG is a workspace-scoped PDF RAG platform for uploading many documents, processing them asynchronously, and querying indexed content with grounded citations. -The repository is organized as a monorepo with three application surfaces: -- `server`: FastAPI API and core RAG orchestration -- `worker`: Redis/RQ background jobs for extraction, indexing, and maintenance -- `client`: React + Vite frontend with Supabase authentication +The repository is a monorepo with three runtime surfaces: +- `client`: React + Vite frontend +- `server`: FastAPI API and orchestration layer +- `worker`: Redis/RQ ingestion workers -This README is written for the current codebase. It explains what the system does today, how the services interact, and how the architecture is intended to scale like an enterprise application. +## Current State -## Table of Contents +The codebase has moved beyond the original single-file ingestion baseline. The current implementation includes: +- batch upload preparation and completion +- ingestion runs with grouped status tracking +- extraction and indexing on separate Redis/RQ queues +- document retry and reindex flows +- ingestion queue, health, and reconciliation endpoints +- multi-document query support in the backend API +- ingestion timing visibility in the frontend observability view +- workspace-scoped token budget tracking with reservation and cleanup -- [Why This Exists](#why-this-exists) -- [What The System Does](#what-the-system-does) -- [System Architecture](#system-architecture) -- [How It Works End to End](#how-it-works-end-to-end) -- [Repository Layout](#repository-layout) -- [API Surface](#api-surface) -- [Data Model](#data-model) -- [Limits and Controls](#limits-and-controls) -- [Local Development](#local-development) -- [Environment Variables](#environment-variables) -- [Operational Notes](#operational-notes) -- [Current Status](#current-status) -- [Roadmap](#roadmap) +The main remaining product gap is the chat UX: the backend can query across multiple selected documents, but the current chat page still centers on one active document at a time. -## Why This Exists +## Recent Updates -Typical RAG demos stop at a single script that embeds documents and sends a prompt to an LLM. That is not enough for a production system. +Recent merged work that should now be reflected in the docs: -This project is built around the concerns that matter in enterprise environments: -- strict workspace isolation -- authenticated access with bearer tokens -- asynchronous ingestion so uploads do not block API requests -- token budget enforcement with reservation and commit semantics -- query logging and observability -- repeatable local development with Docker Compose -- clean separation between API, workers, storage, and UI +- `08f452d` on 2026-06-08: improved multi-document ingestion workflow with batch prepare/complete, ingestion runs, stronger upload UX, and worker callback coverage +- `9fd2a85` on 2026-06-08: hardened ingestion workflow, added reconciliation, load-test tooling, and multi-document query readiness in the API/retrieval layers +- `0409162` on 2026-06-08: added ingestion timing visibility to upload and observability dashboards +- `bffd3ed`, `b0d4fd2`, `d7ca2f1` on 2026-06-08: tightened security around load-test artifacts, frontend dependencies, and Trivy CI maintenance -## What The System Does - -At a high level, the platform supports this flow: -1. A user signs in with Supabase Auth. -2. The user creates a workspace. -3. The client requests a signed upload URL from the API. -4. A PDF is uploaded to Supabase Storage. -5. The API confirms the upload and enqueues background jobs. -6. Workers extract page text, split it into chunks, and generate embeddings. -7. The document becomes queryable. -8. The user asks a question against a document. -9. The API embeds the question, retrieves the most relevant chunks, calls the LLM with grounded context, and returns an answer with citations. -10. Usage, latency, and errors are recorded for observability. - -## System Architecture - -### Logical View - -```mermaid -flowchart LR - User[User] --> Client[React Client] - Client --> Auth[Supabase Auth] - Client -->|JWT| API[FastAPI API] - - API --> DB[(PostgreSQL + pgvector)] - API --> Redis[(Redis)] - API --> Storage[Supabase Storage] - API --> OpenAI[OpenAI API] - - API -->|enqueue| ExtractQ[RQ ingest_extract] - API -->|enqueue| IndexQ[RQ ingest_index] - - ExtractQ --> ExtractWorker[Extraction Worker] - IndexQ --> IndexWorker[Indexing Worker] - - ExtractWorker --> Storage - ExtractWorker --> DB - ExtractWorker --> Redis - - IndexWorker --> DB - IndexWorker --> OpenAI - - API --> Client -``` - -### Runtime Topology +## Architecture ```text -+--------------------+ +---------------------+ -| React Client | | Supabase | -| Vite app | | Auth + Storage | -+---------+----------+ +----------+----------+ - | ^ - | JWT / signed upload flow | - v | -+---------+-------------------------------------------+ -| FastAPI Server | -| - auth validation | -| - workspace-scoped APIs | -| - query orchestration | -| - token budget checks | -+---------+----------------------+----------------------+ - | | - | SQL | enqueue jobs - v v -+---------+----------+ +-------+----------------------+ -| PostgreSQL | | Redis / RQ | -| pgvector | | rate limiting + job queues | -+---------+----------+ +-------+----------------------+ - ^ | - | v - | +-------+----------------------+ - | | Worker Processes | - | | - extract PDF text | - | | - chunk pages | - | | - create embeddings | - | | - cleanup reservations | - | +------------------------------+ - | - +------------ OpenAI embeddings / chat model -``` - -### Enterprise Design Characteristics - -- API and worker responsibilities are separated. -- Document ingestion is asynchronous and queue-backed. -- Every major operation is scoped by `workspace_id`. -- Token usage is tracked centrally by day. -- Redis is used both for rate limiting and job execution. -- Vector retrieval stays in Postgres with `pgvector` instead of introducing another data store. -- The frontend is a separate deployable artifact. - -## How It Works End to End - -### 1. Authentication and Workspace Resolution - -The client authenticates with Supabase and sends the bearer token to the API. The server validates the token and derives the current user. Workspace-scoped endpoints then resolve the user's workspace before accessing documents or usage records. - -Primary files: -- `client/src/lib/supabase.ts` -- `server/app/api/deps.py` -- `server/app/core/auth.py` -- `server/app/api/workspaces.py` - -### 2. Upload and Ingestion Pipeline - -The upload pipeline is designed so the API never has to receive the PDF bytes directly. - -```mermaid -sequenceDiagram - participant C as Client - participant A as API - participant S as Supabase Storage - participant R as Redis/RQ - participant W1 as Extract Worker - participant W2 as Index Worker - participant D as PostgreSQL - - C->>A: POST /documents/upload-prepare - A->>D: create placeholder document row - A-->>C: signed upload URL + storage path - C->>S: upload PDF directly - C->>A: POST /documents/upload-complete - A->>R: enqueue extract job - R->>W1: ingest_extract - W1->>S: download PDF - W1->>D: write document_pages - W1->>R: enqueue ingest_index - R->>W2: ingest_index - W2->>D: write chunks - W2->>D: write chunk_embeddings - W2->>D: mark document ready/indexed -``` - -What happens in practice: -- `upload-prepare` validates file size, content type, workspace limits, and idempotency. -- `upload-prepare-batch` can prepare many small PDFs as one ingestion run and returns per-file results. -- The API stores a placeholder document record and returns a signed storage URL. -- `upload-complete` and `upload-complete-batch` confirm uploaded objects and enqueue extraction jobs. -- `ingest_extract` downloads the PDF and writes extracted page text into `document_pages`. -- `ingest_index` chunks page text, generates embeddings in batches, stores vectors, and marks the document ready. - -Primary files: -- `server/app/api/documents.py` -- `server/app/core/storage.py` -- `worker/jobs/ingest_extract.py` -- `worker/jobs/ingest_index.py` - -### 3. Query Pipeline - -The query flow is grounded retrieval, not free-form generation. - -```mermaid -sequenceDiagram - participant C as Client - participant A as API - participant D as PostgreSQL/pgvector - participant O as OpenAI - - C->>A: POST /query or POST /query/stream - A->>A: validate workspace, document, limits - A->>O: embed question - A->>D: retrieve top-k chunks by vector similarity - A->>A: reserve token budget - A->>O: generate grounded answer - A->>A: commit actual usage and release remainder - A->>D: write query log - A-->>C: answer + citations + usage +React client + -> Supabase Auth for session tokens + -> FastAPI server for workspace, document, query, usage, and chat APIs + -> Supabase Storage for direct PDF uploads + +FastAPI server + -> Postgres/pgvector for metadata, pages, chunks, embeddings, usage, and logs + -> Redis/RQ for ingestion queues and rate limiting + -> OpenAI for embeddings and grounded answer generation + +Workers + -> ingest_extract: download PDF, validate, extract page text + -> ingest_index: chunk pages, batch embeddings, persist vectors + -> maintenance/reconciliation helpers for stale reservations and stuck ingestion state ``` -What the server does: -- embeds the question with `text-embedding-3-small` -- retrieves top chunks from `chunk_embeddings` and `chunks` -- builds a grounded prompt using retrieved content -- reserves the estimated token budget before the LLM call -- commits actual usage after the response returns -- logs citations, latency, and token usage +## End-to-End Flow -Primary files: -- `server/app/api/query.py` -- `server/app/api/query_stream.py` -- `server/app/core/retrieval.py` -- `server/app/core/embeddings.py` -- `server/app/core/llm.py` -- `server/app/core/token_budget.py` +1. The user signs in with Supabase Auth. +2. The server resolves the user workspace from the bearer token. +3. The client prepares one file or a batch with `/documents/upload-prepare` or `/documents/upload-prepare-batch`. +4. PDFs are uploaded directly to Supabase Storage using signed URLs. +5. The client confirms upload completion with `/documents/upload-complete` or `/documents/upload-complete-batch`. +6. The API enqueues extraction jobs on `ingest_extract`. +7. Extraction writes `document_pages`, applies page/content validation, and enqueues indexing. +8. Indexing builds page-bounded chunks, batches embeddings, writes vectors, and marks documents `ready` or `indexed`. +9. The client queries via `/query` or `/query/stream`. +10. The server embeds the question, retrieves top chunks across one or more selected documents, reserves tokens, calls the LLM, and records usage/query logs. -### 4. Usage and Observability +## Important Limits In Code Today -The system exposes both real-time usage and an observability summary. +- max file size: `10 MB` +- max PDF pages: `10` +- max documents per workspace: `100` +- max bulk upload files per run: `50` +- max query question length: `500` chars +- max selected documents per query: `10` +- max total pages across selected query documents: `100` +- embedding batch size: `32` -Current observability coverage includes: -- daily token usage and remaining budget -- total query count -- 24-hour query volume and error rate -- latency statistics -- document status summary -- top queried documents -- recent query failures - -Primary files: -- `server/app/api/usage.py` -- `server/app/db/models.py` -- `worker/jobs/maintenance.py` +These values come from the current code in [server/app/config.py](/D:/Desktop/projects/MultiDoc-RAG/server/app/config.py:1) and [.env.example](/D:/Desktop/projects/MultiDoc-RAG/.env.example:1). ## Repository Layout ```text -enterprise-rag/ -├── client/ # React + Vite frontend -├── server/ # FastAPI API, core logic, DB layer -├── worker/ # Redis/RQ workers and maintenance jobs -├── scripts/ # DB bootstrap and utility scripts -├── infrastructure/ # Infrastructure placeholders -├── docker-compose.yml # Full local stack -├── docker-compose.prod.yml # Production-style compose file -├── AGENTS.md # Architecture contract and implementation notes -└── README.md +MultiDoc-RAG/ +|-- client/ +|-- server/ +|-- worker/ +|-- docs/ +|-- scripts/ +|-- artifacts/ +|-- docker-compose.yml +|-- AGENTS.md +`-- README.md ``` -### Important Directories - -- `server/app/api`: REST and streaming endpoints -- `server/app/core`: auth, retrieval, embeddings, prompts, token budget -- `server/app/db`: SQLAlchemy models and DB session setup -- `server/app/schemas`: request and response models -- `worker/jobs`: extraction, indexing, maintenance jobs -- `client/src/pages`: authenticated and public application pages -- `client/src/components`: UI modules for upload, chat, usage, and layout +## Primary Docs -## API Surface - -### Health and Auth - -- `GET /health` -- `GET /auth/me` - -### Workspace - -- `POST /workspaces` -- `GET /workspaces/me` - -### Documents - -- `GET /documents` -- `GET /documents/{document_id}` -- `GET /documents/{document_id}/pages/{page_number}` -- `POST /documents/upload-prepare` -- `POST /documents/upload-prepare-batch` -- `POST /documents/upload-complete` -- `POST /documents/upload-complete-batch` -- `GET /documents/ingestion-runs/{run_id}` -- `GET /documents/ingestion-queues` -- `POST /documents/{document_id}/retry` -- `POST /documents/{document_id}/reindex` -- `DELETE /documents/{document_id}` - -### Query and Retrieval - -- `POST /query` -- `POST /query/stream` -- `GET /citations/{chunk_id}` -- `GET /queries` -- `GET /queries/{query_id}` - -### Chat Sessions - -- `POST /chats/sessions` -- `PATCH /chats/sessions/{session_id}` -- `GET /chats/sessions` -- `GET /chats/sessions/{session_id}` - -### Usage and Observability - -- `GET /usage/today` -- `GET /usage/observability` - -## Data Model - -Core tables in the current implementation: - -- `workspaces`: tenant root for all user content -- `ingestion_runs`: batch upload and ingestion progress grouping -- `documents`: uploaded PDF metadata and pipeline status -- `document_pages`: extracted page text -- `chunks`: page-bounded text chunks used for retrieval -- `chunk_embeddings`: vector embeddings stored in `pgvector` -- `workspace_daily_usage`: daily token accounting with reserved and used buckets -- `query_logs`: query history, citations, latency, and token metrics -- `chat_sessions`: persisted chat metadata and messages - -### Status Lifecycle - -Document lifecycle in the current codebase: - -```text -pending_upload/uploading -> uploaded -> extracting -> indexing -> ready/indexed - \-> failed -``` - -## Limits and Controls - -Current enforced limits from the application config and rate limiter: - -- `1` workspace per user -- up to `100` documents per workspace -- maximum file size: `10 MB` -- maximum PDF page count: `10` -- maximum files per backend batch upload: `50` -- supported upload type: `application/pdf` -- maximum query length: `500` characters -- retrieval depth: `top_k = 5` -- LLM max output tokens: `2000` -- daily token limit: `100000` tokens per workspace -- upload prepare rate limit: `10` requests per minute per workspace -- upload complete rate limit: `20` requests per minute per workspace -- query rate limit: `100` requests per minute per workspace - -Bulk ingestion endpoints have separate rate limits from single-file endpoints: -- batch upload prepare: `5` requests per minute per workspace -- batch upload complete: `10` requests per minute per workspace - -### Token Budget Model - -The token budget is managed with reservation semantics so concurrent requests do not overspend the daily allowance. - -Flow: -1. Estimate query embedding + prompt + max output cost. -2. Reserve the estimated tokens. -3. Execute the LLM call. -4. Commit actual tokens used. -5. Release any unused reservation. -6. Periodically clean stale reservations. - -This logic is implemented in `server/app/core/token_budget.py` and `worker/jobs/maintenance.py`. +- [docs/project-context.md](/D:/Desktop/projects/MultiDoc-RAG/docs/project-context.md:1): current implemented architecture and remaining gaps +- [docs/ingestion-workflow-evaluation.md](/D:/Desktop/projects/MultiDoc-RAG/docs/ingestion-workflow-evaluation.md:1): ingestion evaluation summary and caveats +- [docs/ingestion-load-testing.md](/D:/Desktop/projects/MultiDoc-RAG/docs/ingestion-load-testing.md:1): repeatable ingestion benchmark usage +- [docs/task/document-ingestion-limits-and-failure-handling.md](/D:/Desktop/projects/MultiDoc-RAG/docs/task/document-ingestion-limits-and-failure-handling.md:1) +- [docs/task/ingestion-pipeline-scale-and-reliability.md](/D:/Desktop/projects/MultiDoc-RAG/docs/task/ingestion-pipeline-scale-and-reliability.md:1) +- [docs/task/report-ingestion-workflow.md](/D:/Desktop/projects/MultiDoc-RAG/docs/task/report-ingestion-workflow.md:1) +- [docs/task/multi-document-ingestion-ux.md](/D:/Desktop/projects/MultiDoc-RAG/docs/task/multi-document-ingestion-ux.md:1) +- [docs/task/ingestion-follow-up-hardening-and-validation.md](/D:/Desktop/projects/MultiDoc-RAG/docs/task/ingestion-follow-up-hardening-and-validation.md:1) ## Local Development -### Prerequisites - -- Docker and Docker Compose -- Node.js 20+ if running the client outside Docker -- Python 3.11 if running the API or worker outside Docker -- A Supabase project -- An OpenAI API key for embeddings and answer generation - -### Quick Start With Docker Compose - -1. Create your environment file. - -```bash -cp .env.example .env -``` - -2. Fill in at least these values: - -```bash -SUPABASE_URL= -SUPABASE_SERVICE_ROLE_KEY= -SUPABASE_ANON_KEY= -SUPABASE_JWT_SECRET= -OPENAI_API_KEY= -DATABASE_URL=postgresql://postgres:postgres@localhost:5432/enterprise_rag -REDIS_URL=redis://localhost:6379/0 -``` - -3. Start the stack. - ```bash docker-compose up --build ``` -4. Open the services: +Useful services: - client: `http://localhost:5173` -- api: `http://localhost:8000` -- rq dashboard: `http://localhost:9181` - -### Useful Commands - -```bash -# start everything -docker-compose up - -# rebuild and start -docker-compose up --build - -# stop services -docker-compose down - -# stop and remove volumes -docker-compose down -v - -# run DB migrations from the server container -docker-compose exec server alembic upgrade head - -# view server logs -docker-compose logs -f server - -# view worker logs -docker-compose logs -f worker-extract -docker-compose logs -f worker-index -``` - -### Run Services Individually - -Server: - -```bash -cd server -python -m venv .venv -source .venv/bin/activate -pip install -r requirements.txt -uvicorn app.main:app --reload --host 0.0.0.0 --port 8000 -``` - -Client: - -```bash -cd client -npm install -npm run dev -- --host 0.0.0.0 -``` - -Worker: - -```bash -cd worker -python -m venv .venv -source .venv/bin/activate -pip install -r requirements.txt -QUEUE_NAME=ingest_extract python worker.py -``` - -## Environment Variables - -Root `.env.example` is the primary template for local development. - -### Required Core Variables - -```bash -SUPABASE_URL=https://your-project.supabase.co -SUPABASE_SERVICE_ROLE_KEY=your-service-role-key -SUPABASE_ANON_KEY=your-anon-key -SUPABASE_JWT_SECRET=your-jwt-secret -SUPABASE_STORAGE_BUCKET=documents - -OPENAI_API_KEY=sk-... - -DATABASE_URL=postgresql://postgres:postgres@localhost:5432/enterprise_rag -REDIS_URL=redis://localhost:6379/0 -``` - -### Useful Application Variables - -```bash -ENVIRONMENT=development -API_HOST=0.0.0.0 -API_PORT=8000 -DAILY_TOKEN_LIMIT=100000 -RESERVATION_TTL_SECONDS=600 -LOG_EACH_QUERY=false -EMBEDDING_MODEL=text-embedding-3-small -MAX_FILE_SIZE_BYTES=10485760 -MAX_PDF_PAGE_COUNT=10 -MIN_EXTRACTED_TEXT_CHARS=1 -MAX_BULK_UPLOAD_FILES=50 -EMBEDDING_BATCH_SIZE=32 -OPENAI_EMBEDDING_TIMEOUT_SECONDS=300 -INGEST_EXTRACT_JOB_TIMEOUT_SECONDS=900 -INGEST_INDEX_JOB_TIMEOUT_SECONDS=1800 -VITE_API_URL=http://localhost:8000 -VITE_SUPABASE_URL=https://your-project.supabase.co -VITE_SUPABASE_ANON_KEY=your-anon-key -``` - -## Operational Notes - -### Workspace Isolation - -The system is designed around `workspace_id` as the isolation boundary. Document access, usage tracking, retrieval, and query logs are all scoped to a workspace. - -### Storage Strategy - -PDF binaries live in Supabase Storage. Extracted text, chunks, metadata, and embeddings live in Postgres. - -### Vector Search Strategy - -Embeddings are stored in `chunk_embeddings.embedding` using `pgvector`. Retrieval uses cosine distance and returns the most relevant chunk candidates for a single document. - -### Failure Recovery - -- failed documents can be retried with `POST /documents/{document_id}/retry` -- already processed documents can be reindexed with `POST /documents/{document_id}/reindex` -- stale token reservations can be cleared by the maintenance job -- RQ extraction/indexing jobs use explicit timeouts and a failure callback so timed-out jobs mark documents `failed` instead of leaving them stuck in processing -- document deletion removes metadata first, then attempts storage cleanup - -### Frontend Application Areas - -Current routed pages in the client: -- `/login` -- `/signup` -- `/workspace` -- `/app/upload` -- `/app/chat` -- `/app/observability` -- `/app/workspace` - -## Current Status - -The repository is beyond a scaffold. These capabilities are already present in code: - -- JWT-backed auth integration with Supabase -- one-workspace-per-user model -- signed upload preparation and upload completion -- background extraction and indexing jobs -- page text persistence -- chunk persistence and vector embedding storage -- grounded query endpoint -- streaming query endpoint using SSE -- citation source retrieval -- query history APIs -- chat session APIs -- usage and observability endpoints -- Docker-based local runtime - -Known gaps or areas still being hardened: -- not every table in the architecture contract is represented yet in SQLAlchemy models -- `server/app/core/chunking.py` remains a placeholder while worker-side chunking is active -- production deployment still needs full operational hardening, secrets handling, and CI maturity -- test coverage is still light for end-to-end ingestion and retrieval - -## Roadmap - -Near-term improvements that fit the current architecture: - -1. Move chunking into a shared core module so API and workers use one implementation. -2. Expand integration tests around upload, extraction, indexing, and query behavior. -3. Add stronger metrics, worker lifecycle hooks, and scheduled maintenance execution. -4. Harden migration coverage for all current tables and status transitions. -5. Expand query scope from single-document search to selected multi-document search where needed. -6. Improve production deployment docs and CI/CD validation. +- server: `http://localhost:8000` +- RQ dashboard: `http://localhost:9181` -## Related Docs +## Module Readmes -- `AGENTS.md`: architecture contract and implementation guidance -- `server/README.md`: server-specific notes -- `worker/README.md`: worker-specific notes -- `client/README.md`: client-specific notes +- [client/README.md](/D:/Desktop/projects/MultiDoc-RAG/client/README.md:1) +- [server/README.md](/D:/Desktop/projects/MultiDoc-RAG/server/README.md:1) +- [worker/README.md](/D:/Desktop/projects/MultiDoc-RAG/worker/README.md:1) diff --git a/client/README.md b/client/README.md index b221f8b..772702a 100644 --- a/client/README.md +++ b/client/README.md @@ -1,146 +1,49 @@ -# Client - -`client/` is the React frontend for Enterprise RAG. It handles authentication with Supabase, workspace onboarding, document upload UX, document-aware chat, and observability views. - -## Responsibilities - -- sign in and sign up users with Supabase Auth -- keep the current session and bearer token in client state -- route users through workspace creation before entering the app shell -- upload PDFs through the API + Supabase signed URL flow -- display ingestion progress and document status -- run streaming grounded chat against the selected document -- show citations, sources, usage, and observability metrics - -## Tech Stack - -- React 18 -- TypeScript -- Vite -- React Router -- Supabase JS -- Tailwind CSS utilities -- Axios and fetch-based API calls - -## App Structure - -```text -client/ -├── src/ -│ ├── App.tsx # top-level routes -│ ├── context/AuthContext.tsx # session lifecycle and auth actions -│ ├── lib/api.ts # typed API client -│ ├── lib/supabase.ts # Supabase client -│ ├── pages/ -│ │ ├── Login.tsx -│ │ ├── Signup.tsx -│ │ ├── WorkspaceGate.tsx -│ │ ├── UploadPage.tsx -│ │ ├── ChatPage.tsx -│ │ ├── ObservabilityPage.tsx -│ │ └── WorkspaceInfoPage.tsx -│ ├── components/ -│ │ ├── layout/ -│ │ ├── upload/ -│ │ ├── chat/ -│ │ └── documents/ -│ └── styles/ -├── package.json -└── vite.config.ts -``` - -## Route Map - -Public routes: -- `/login` -- `/signup` - -Authenticated routes: -- `/workspace` -- `/app/upload` -- `/app/chat` -- `/app/observability` -- `/app/workspace` - -Top-level flow in `src/App.tsx`: -1. unauthenticated users are redirected to `/login` -2. authenticated users are redirected to `/workspace` -3. workspace setup routes the user into `/app/*` -4. protected app routes render inside the shared app shell - -## UX Flows +# MultiDoc-RAG Client -### Authentication +`client/` is the React frontend for MultiDoc-RAG. It handles Supabase auth, workspace onboarding, multi-file upload flows, document monitoring, single-document chat, and observability dashboards. -`AuthContext` is the session source of truth. +## What It Does Today -Current behavior: -- reads the initial Supabase session on load -- subscribes to auth state changes -- exposes `signIn`, `signUp`, and `signOut` -- registers a global unauthorized handler so API `401` responses log the user out and redirect to `/login` +- signs users in and out with Supabase +- gates the app on workspace creation +- prepares and completes batch uploads through signed URLs +- shows ingestion-run progress, queue-aware document states, and retry actions +- displays usage and observability metrics +- streams grounded chat responses for the current active document -Primary files: -- `src/context/AuthContext.tsx` -- `src/lib/supabase.ts` -- `src/lib/api.ts` +## Recent Frontend Updates -### Document Upload +- `08f452d` on 2026-06-08: upload flow now supports up to 50 files per run, accepted/rejected file summaries, grouped processing states, and retry-oriented ingestion UX +- `0409162` on 2026-06-08: observability and upload views now surface ingestion timing +- `b0d4fd2` on 2026-06-08: frontend dependency vulnerabilities were addressed -Upload UX is centered in `UploadPage` and `components/upload/UploadPanel`. +## Important Current Constraint -Flow: -1. select and review up to 50 PDFs as one upload run -2. request `POST /documents/upload-prepare-batch` -3. upload accepted PDFs directly to their signed URLs -4. notify the backend with `POST /documents/upload-complete-batch` -5. poll documents, queue status, and the active ingestion run while processing is active -6. separate active, failed, and ready documents for recovery and review +The upload UX is multi-document aware, but the chat UX is still single-document oriented. The backend query API already accepts `document_ids`, but [client/src/pages/ChatPage.tsx](/D:/Desktop/projects/MultiDoc-RAG/client/src/pages/ChatPage.tsx:1) still submits one active `document_id`. -Current behavior: -- shows selected files before upload starts -- shows accepted/rejected/processing/ready/failed counts for the current run -- shows live document status grouped by workflow state -- supports search, status filtering, and sorting for larger document sets -- supports retrying failed documents individually or in a retryable batch -- refreshes every 4 seconds while documents are processing -- treats `ready` and `indexed` as successful ingest states +## Main Routes -### Chat and Streaming Query - -`ChatPage` drives the main RAG interaction. - -Current behavior: -- binds the active document from the app shell context -- streams answer deltas from `POST /query/stream` -- loads citations and source text on demand -- persists and restores draft transcript state locally -- stores chat sessions with `/chats/sessions` -- updates usage metrics in the app shell after responses - -Primary files: -- `src/pages/ChatPage.tsx` -- `src/components/chat/*` -- `src/lib/api.ts` - -### Observability +Public: +- `/login` +- `/signup` -`ObservabilityPage` renders a workspace-level operational summary using `GET /usage/observability`. +Authenticated: +- `/workspace` +- `/app/upload` +- `/app/chat` +- `/app/observability` +- `/app/workspace` -Displayed metrics: -- total queries -- 24-hour queries and errors -- error rate -- average latency -- p95 latency -- tokens used and remaining -- document pipeline health -- top queried documents -- recent query errors +## Key Files -## API Contracts Used +- [src/context/AuthContext.tsx](/D:/Desktop/projects/MultiDoc-RAG/client/src/context/AuthContext.tsx:1) +- [src/lib/api.ts](/D:/Desktop/projects/MultiDoc-RAG/client/src/lib/api.ts:1) +- [src/pages/UploadPage.tsx](/D:/Desktop/projects/MultiDoc-RAG/client/src/pages/UploadPage.tsx:1) +- [src/components/upload/UploadPanel.tsx](/D:/Desktop/projects/MultiDoc-RAG/client/src/components/upload/UploadPanel.tsx:1) +- [src/pages/ChatPage.tsx](/D:/Desktop/projects/MultiDoc-RAG/client/src/pages/ChatPage.tsx:1) +- [src/pages/ObservabilityPage.tsx](/D:/Desktop/projects/MultiDoc-RAG/client/src/pages/ObservabilityPage.tsx:1) -The client currently uses these backend APIs: +## APIs Used - `GET /auth/me` - `POST /workspaces` @@ -148,14 +51,15 @@ The client currently uses these backend APIs: - `GET /documents` - `GET /documents/{document_id}` - `GET /documents/{document_id}/pages/{page_number}` +- `GET /documents/ingestion-runs/{run_id}` +- `GET /documents/ingestion-queues` - `POST /documents/upload-prepare` - `POST /documents/upload-complete` - `POST /documents/upload-prepare-batch` - `POST /documents/upload-complete-batch` -- `GET /documents/ingestion-runs/{run_id}` -- `GET /documents/ingestion-queues` -- `DELETE /documents/{document_id}` - `POST /documents/{document_id}/retry` +- `POST /documents/{document_id}/reindex` +- `DELETE /documents/{document_id}` - `POST /query/stream` - `GET /citations/{chunk_id}` - `GET /queries` @@ -167,20 +71,12 @@ The client currently uses these backend APIs: - `GET /usage/today` - `GET /usage/observability` -All authenticated requests send: - -```http -Authorization: Bearer -``` - ## Environment -Create `client/.env` with: - ```bash VITE_API_URL=http://localhost:8000 VITE_SUPABASE_URL=https://your-project.supabase.co -VITE_SUPABASE_ANON_KEY=your-public-anon-key +VITE_SUPABASE_ANON_KEY=your-anon-key ``` ## Run @@ -190,33 +86,3 @@ cd client npm install npm run dev -- --host 0.0.0.0 ``` - -Build for production: - -```bash -npm run build -``` - -Preview the built app: - -```bash -npm run preview -``` - -## Development Commands - -```bash -cd client -npm run dev -npm run build -npm run lint -npm run test -``` - -## Notes for Contributors - -- `src/lib/api.ts` is the contract layer; keep request and response types aligned with the backend -- auth redirects are centralized through the unauthorized handler, so avoid duplicating logout-on-401 logic elsewhere -- chat is document-scoped in the current UI model -- upload and polling behavior assume asynchronous backend processing states -- preserve the existing visual language unless the task is explicitly a redesign diff --git a/client/package-lock.json b/client/package-lock.json index 22f8980..e229887 100644 --- a/client/package-lock.json +++ b/client/package-lock.json @@ -13,6 +13,7 @@ "axios": "^1.17.0", "clsx": "^2.1.0", "lucide-react": "^0.309.0", + "postcss": "^8.4.33", "react": "^18.2.0", "react-dom": "^18.2.0", "react-router-dom": "^6.21.3", @@ -25,7 +26,6 @@ "@vitejs/plugin-react": "^4.2.1", "autoprefixer": "^10.4.17", "eslint": "^8.56.0", - "postcss": "^8.4.33", "typescript": "^5.3.3", "vite": "^5.0.11", "vitest": "^1.2.0" @@ -2689,16 +2689,16 @@ } }, "node_modules/form-data": { - "version": "4.0.5", - "resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.5.tgz", - "integrity": "sha512-8RipRLol37bNs2bhoV67fiTEvdTrbMUYcFTiy3+wuuOnUog2QBHCZWXDRijWQfAkhBj2Uf5UnVaiWwA5vdd82w==", + "version": "4.0.6", + "resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.6.tgz", + "integrity": "sha512-vKatAh4SlVfgbv+YtmhiRjhEMJsYpsG1Y2rMQtR+SVSbytsSD1YGzDIcrAJmdFec88u/+VoGmxnl+80gL1tRCQ==", "license": "MIT", "dependencies": { "asynckit": "^0.4.0", "combined-stream": "^1.0.8", "es-set-tostringtag": "^2.1.0", - "hasown": "^2.0.2", - "mime-types": "^2.1.12" + "hasown": "^2.0.4", + "mime-types": "^2.1.35" }, "engines": { "node": ">= 6" @@ -2925,9 +2925,9 @@ } }, "node_modules/hasown": { - "version": "2.0.2", - "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz", - "integrity": "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==", + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.4.tgz", + "integrity": "sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==", "license": "MIT", "dependencies": { "function-bind": "^1.1.2" diff --git a/docs/ingestion-load-testing.md b/docs/ingestion-load-testing.md index ba27c1c..569b3b3 100644 --- a/docs/ingestion-load-testing.md +++ b/docs/ingestion-load-testing.md @@ -1,17 +1,24 @@ # Ingestion Load Testing -Use `scripts/ingestion_load_test.py` to run repeatable ingestion benchmarks through the real API, Supabase upload URLs, Redis queues, and workers. +Use `scripts/ingestion_load_test.py` to run repeatable ingestion benchmarks against the real MultiDoc-RAG API, signed upload flow, Redis queues, and workers. + +## What This Covers + +- batch upload prepare and complete +- polling ingestion runs to terminal state +- queue snapshots during processing +- per-run JSON artifacts for comparison across revisions ## Prerequisites -- Server and workers are running with Docker Compose. -- Supabase schema has been updated with `scripts/schema.supabase.sql`. -- You have a valid app bearer token for a user with a workspace. +- Docker Compose services are running +- schema changes have been applied with `scripts/schema.local.sql` or `scripts/schema.supabase.sql` +- you have a valid bearer token for a user with a workspace ## PowerShell Examples ```powershell -cd H:\varun\MultiDoc-RAG\MultiDoc-RAG +cd D:\Desktop\projects\MultiDoc-RAG $env:RAG_BEARER_TOKEN = "" python scripts\ingestion_load_test.py --count 10 --scenario valid python scripts\ingestion_load_test.py --count 25 --scenario valid @@ -19,16 +26,23 @@ python scripts\ingestion_load_test.py --count 50 --scenario valid python scripts\ingestion_load_test.py --count 10 --scenario mixed ``` -The script writes comparable JSON artifacts to: +## Output + +Artifacts are written to: ```text artifacts/ingestion-load/ ``` -Each artifact includes prepare/complete responses, final ingestion-run state, current document list, total wall-clock duration, prepare/upload duration, and queue snapshots captured while polling. +Each artifact includes: +- prepare and complete responses +- ingestion-run state +- document snapshots +- queue snapshots collected while polling +- total wall-clock duration +- upload and prepare timing -## Notes +## Recent Related Changes -- The generated PDFs are one-page text PDFs with deterministic names. -- The mixed scenario replaces the final generated file with an invalid text file to confirm isolated failures. -- Use `--timeout-seconds` and `--poll-seconds` to tune long runs. +- `9fd2a85` on 2026-06-08: added load-test tooling and supporting ingestion hardening +- `bffd3ed` on 2026-06-08: fixed security scan findings for generated load-test artifacts diff --git a/docs/ingestion-workflow-evaluation.md b/docs/ingestion-workflow-evaluation.md index 1eb0414..5ea6eee 100644 --- a/docs/ingestion-workflow-evaluation.md +++ b/docs/ingestion-workflow-evaluation.md @@ -2,321 +2,34 @@ Date: 2026-06-05 -## Executive Summary +## Context -The ingestion workflow is working for the current small local test set. Batch upload, extraction, indexing, expected validation failures, retry, queue visibility, and final document states were all exercised through the running Docker Compose stack. +This evaluation captured the ingestion workflow before the 2026-06-08 hardening pass landed. Read it together with these later updates: +- `08f452d`: multi-document ingestion workflow improvements +- `9fd2a85`: reconciliation, load testing, and multi-document query readiness +- `0409162`: ingestion timing in dashboards -The current system is ready for small PDF ingestion and manual validation. It is not yet proven for larger-scale ingestion such as 25, 50, or 100 PDFs in one run. The biggest remaining weaknesses are observability and run lifecycle accuracy, not the basic document pipeline. +## Summary -Main strengths: +The evaluation showed that the ingestion pipeline already worked for small local batches and correctly handled valid files, page-limit failures, and scanned/image-only failures. The main weaknesses at that point were stale run state, noisy failure reporting, and limited timing visibility. -- Valid PDFs move to `ready`. -- Page-limit failures and scanned/image-only PDFs are rejected with clear document-level messages. -- Extraction and indexing are handled asynchronously by separate RQ queues. -- Current queue depth was visible and drained to zero after the batch. -- Embedding calls are batched through `EMBEDDING_BATCH_SIZE`. -- Timeout/failure callback handling now exists so killed indexing jobs should not leave documents stuck forever. +## What Was Validated -Main weaknesses: +- valid PDFs could move to `ready` +- page-limit failures were rejected +- scanned/image-only PDFs were rejected +- extraction and indexing ran on separate queues +- retry/recovery worked for previously stuck indexing cases -- Historical ingestion runs can remain `processing` or `preparing` even after their documents are no longer active. -- Expected business-rule rejections still appear in RQ failed registries, which makes operational failure counts noisy. -- There is no durable per-stage timing for queue wait, extraction time, embedding time, and DB write time. -- Large-batch behavior was not tested in this pass. -- Token-budget exhaustion behavior during concurrent ingestion was not tested in this pass. +## What Changed After This Evaluation -## Test Setup +Since this report was written, the codebase added: +- ingestion reconciliation for stale active documents and runs +- richer batch-upload and ingestion-run behavior +- multi-document query contract support in the backend +- ingestion timing surfaced in frontend dashboards +- repeatable load-testing tooling under `scripts/ingestion_load_test.py` -Environment: +## Remaining Use -- Docker Compose local stack on Windows. -- Client: `http://localhost:5173`. -- API: `http://localhost:8000`. -- RQ dashboard: `http://localhost:9181`. -- Postgres with pgvector: `rag-postgres`. -- Redis: `rag-redis`. -- Extraction workers: 5 `worker-extract` containers. -- Index workers: 3 `worker-index` containers. -- OpenAI calls: real embedding calls, not mocked. -- Supabase: used for auth and raw PDF storage. - -Current configured limits observed from docs/config: - -- Maximum PDF page count: `10`. -- Maximum file size: `10 MB`. -- Maximum backend batch upload size: `50`. -- Embedding batch size: `32`. -- Extract job timeout: `900` seconds. -- Index job timeout: `1800` seconds. -- OpenAI embedding request timeout: `300` seconds. - -Documents tested in the latest real run: - -| File | Expected Result | Actual Result | -| --- | --- | --- | -| `terms-and-conditions-template.pdf` | valid, under 10 pages | `ready`, 2 pages | -| `About_ChatGPT_Pro_tiers_OpenAI_Help_Center.pdf` | valid, under 10 pages | `ready`, 4 pages | -| `Terms-and-Conditions-Free-Template_PDF-2.pdf` | too many pages | `failed`, page-limit failure | -| `Screenshot_22_.pdf` | image-only/scanned | `failed`, unsupported-content failure | - -## Workflow Validation - -### Upload Preparation - -Batch upload preparation worked. The frontend sent `POST /documents/upload-prepare-batch` and the server returned `201 Created`. The batch run was created in `ingestion_runs`. - -The frontend/backend status mismatch found earlier was fixed: the backend returns prepared batch items, and the frontend now treats prepared items as uploadable instead of leaving documents at `pending_upload`. - -### Upload Completion - -Upload completion worked for the latest real run. Documents moved past `pending_upload`; no document remained stuck in upload state after the validated batch. - -### Extraction - -Extraction behaved correctly: - -- Two valid PDFs produced page rows. -- One 28-page PDF failed because it exceeded the 10-page limit. -- One scanned/image-only PDF failed because no extractable text was found. -- The latest database state had 6 extracted `document_pages` rows, matching the two valid PDFs. - -### Indexing - -Indexing behaved correctly after the timeout/failure handling work and retry: - -- The two valid PDFs became `ready`. -- The latest database state had 6 `chunks` rows and 6 `chunk_embeddings` rows. -- Index workers logged successful index jobs for the two valid documents. -- No latest document remained stuck in `indexing`. - -### Final Document State - -Latest document summary from Postgres: - -| Metric | Value | -| --- | ---: | -| Total latest documents | 4 | -| Ready | 2 | -| Failed | 2 | -| In progress | 0 | -| Extracted pages | 6 | -| Chunks | 6 | -| Embeddings | 6 | -| Extract queue depth after run | 0 | -| Index queue depth after run | 0 | - -The latest document-level states are correct. - -### Retry and Recovery - -Retry/recovery was exercised after a previous indexing worker timeout left a document in `indexing`. The stuck document was marked failed and retried, then reached `ready`. - -The code now also wires explicit RQ job timeouts and an `on_failure` callback that marks active documents failed when an RQ job is killed or times out. A forced worker-kill test after this callback was added was not performed in this pass. - -### Reindex - -The reindex endpoint exists and the indexing code rebuilds chunks/embeddings idempotently. A full user-facing reindex scenario was not executed in this evaluation pass. - -## Performance Findings - -This was a small functional/performance check, not a high-scale benchmark. - -Measured latest run: - -| File | Final Status | Time From Row Creation To Final State | -| --- | --- | ---: | -| `Screenshot_22_.pdf` | failed | 7.88 seconds | -| `Terms-and-Conditions-Free-Template_PDF-2.pdf` | failed | 8.17 seconds | -| `terms-and-conditions-template.pdf` | ready | 13.88 seconds | -| `About_ChatGPT_Pro_tiers_OpenAI_Help_Center.pdf` | ready | 14.25 seconds | - -Observed worker timing from logs: - -- Valid extraction jobs started around `19:29:00` to `19:29:01` UTC. -- Valid extraction jobs finished around `19:29:04` to `19:29:05` UTC. -- Index jobs started around `19:29:04` to `19:29:05` UTC. -- Index jobs finished around `19:29:10` UTC. -- Queue wait was small for this run because workers were already idle. - -Current bottleneck for small valid PDFs appears to be indexing/embedding, not extraction. That conclusion is limited because only two valid PDFs were indexed in the latest run. - -Not measured: - -- 10, 25, 50, or 100 PDF batch throughput. -- Average queue wait under sustained backlog. -- Worker utilization. -- DB query/write count. -- OpenAI request latency by batch. -- Throughput while queries are being served at the same time. - -## Reliability Findings - -What passed: - -- Valid documents reached `ready`. -- Invalid documents reached `failed`. -- Failure messages were specific and visible in the UI. -- Expected failures were isolated to the individual documents. -- Queues drained to zero after the run. -- The latest run had no documents stuck in `pending_upload`, `uploaded`, `extracting`, or `indexing`. - -Issues found: - -- Earlier, an index worker was killed after the default RQ timeout and the document remained `indexing`. This exposed a real failure-state bug. The code now has explicit job timeouts and a failure callback, but a deliberate post-fix kill test is still needed. -- RQ failed registries contain expected validation failures. At inspection time Redis had `11` failed extract jobs and `7` failed index jobs from current/prior runs. These counts are noisy because they mix expected document rejections with infrastructure failures. -- Historical ingestion runs are not always terminal in the database. At inspection time `ingestion_runs` had statuses: `partial = 1`, `processing = 5`, `preparing = 4`. Some of these are older runs from earlier failed attempts, but this shows run lifecycle cleanup/status derivation still needs hardening. - -## Queue and Worker Behavior - -Current worker topology is stronger than the original baseline: - -- 5 extraction workers listen on `ingest_extract`. -- 3 indexing workers listen on `ingest_index`. -- Each document still uses one extract job and one index job. -- Valid documents can extract and index in parallel when workers are available. - -Observed state after the latest run: - -- `rq:queue:ingest_extract` length: `0`. -- `rq:queue:ingest_index` length: `0`. -- Workers remained up and continued cleaning registries. - -Remaining gaps: - -- There is no backpressure policy beyond queueing. -- There is no persistent queue-depth history. -- There is no clear separation between expected document rejection and worker/system failure in RQ failure monitoring. -- There is no automatic stale-run reconciliation job. - -## Database Efficiency - -Improvements now present: - -- Page rows are inserted in a batch-style execution. -- Embedding vectors are inserted in batches. -- Indexing clears and rebuilds chunks/embeddings for idempotent retry/reindex behavior. - -Remaining concerns: - -- Work is still organized one document per extract job and one document per index job. -- There is no recorded DB write count per ingestion run. -- There is no stage timing stored alongside rows. -- There is no evidence yet for lock contention or write amplification under load because larger runs were not tested. - -## Embedding Efficiency - -The implementation now uses `EMBEDDING_BATCH_SIZE`, configured as `32`, so it no longer embeds one chunk per OpenAI call by design. - -In the latest run there were only 6 total chunks, so each document fit within a small number of embedding calls. This validates correctness but does not prove high-throughput embedding behavior. - -Embedding is still the most likely throughput bottleneck for larger text-heavy PDFs because it depends on external OpenAI latency, quota, and rate limits. - -## Token-Budget Safety - -What exists: - -- Indexing reserves token budget before embedding batches. -- Successful embedding batches commit usage. -- Failure paths attempt to release outstanding reservations. -- A maintenance job exists for stale reservation cleanup. - -What was not tested: - -- Low-budget ingestion failure. -- Concurrent uploads competing for the same workspace budget. -- Reservation cleanup after forced worker termination. -- Stale reservation detection after long-running or killed jobs. - -This area needs a dedicated failure test before the ingestion system can be considered robust under concurrency. - -## Observability Findings - -What is working: - -- The UI shows ready, failed, and in-progress document groups. -- The API exposes `GET /documents/ingestion-queues`. -- The API exposes `GET /documents/ingestion-runs/{run_id}`. -- Document-level error messages are clear. -- Docker logs are useful for worker-level diagnosis. - -What is weak: - -- Run status history is not fully reliable for older runs. -- Expected validation failures create RQ failure noise. -- There is no per-stage timing model. -- There is no queue-depth trend, worker utilization view, or OpenAI latency breakdown. -- There is no simple operator command/API that says "show stuck ingestion work". - -## Multi-Document Readiness - -The ingestion side now supports multiple uploaded documents and preserves workspace/document metadata in pages, chunks, and embeddings. - -This helps future multi-document retrieval because the data model already carries: - -- `workspace_id` -- `document_id` -- page rows -- chunk rows -- embedding rows - -The main remaining multi-document gap is on the query side, not ingestion. Retrieval and chat are still effectively scoped to one selected document at a time. - -## What Is Missing - -Important missing pieces: - -- Larger load tests with 10, 25, and 50 small PDFs. -- Forced worker-kill test after the new failure callback. -- Low token-budget ingestion test. -- Reindex test from the UI/API. -- Automated run-status reconciliation. -- Cleaner RQ failure semantics for expected document rejection. -- Durable stage timing fields or metrics. -- Queue-depth metrics over time. -- Integration tests for batch upload, partial success, retry, and stuck-job recovery. - -## Recommended Next Improvements - -Highest impact for reliability: - -1. Add a reconciliation job that marks stale `uploaded`, `extracting`, and `indexing` documents failed after a configured timeout. -2. Update ingestion-run status from document states consistently, or derive it dynamically instead of trusting stale stored statuses. -3. Separate expected validation failures from infrastructure failures so RQ failed registries are actionable. -4. Add an automated test for RQ worker timeout/kill recovery. - -Highest impact for throughput: - -1. Add per-stage timing metrics for upload completion, queue wait, extraction, indexing, embedding, and DB writes. -2. Run repeatable 10/25/50 PDF load tests with small text PDFs. -3. Record embedding batch counts and OpenAI latency per document. -4. Consider bulk DB insert optimization only after measuring real DB bottlenecks. - -Highest impact for observability: - -1. Add a `GET /documents/ingestion-health` endpoint with stuck counts, queue depths, failed registry counts, and oldest active document age. -2. Add run-level timing and final status summary fields. -3. Show infrastructure failures separately from validation failures in the UI. -4. Add an operator command or script to inspect stuck documents and stale reservations. - -Future multi-document retrieval work: - -1. Add selected-document or workspace-wide retrieval APIs. -2. Update retrieval filters to search across multiple document IDs. -3. Update the chat UI so a user can select multiple ready documents. -4. Preserve citation quality by returning document title, page, and chunk metadata for every source. - -## Recommended Next Tasks - -Concrete next task files to create: - -1. `docs/task/ingestion-observability-and-stuck-job-recovery.md` -2. `docs/task/ingestion-load-test-suite.md` -3. `docs/task/token-budget-ingestion-failure-tests.md` -4. `docs/task/multi-document-retrieval-api.md` -5. `docs/task/multi-document-chat-ui.md` - -## Final Assessment - -Task 4 validates that the improved ingestion workflow works for the current real local test set. The latest batch ended in correct document states, queues drained, and valid PDFs became queryable. - -The pipeline should not yet be called large-scale ready. The next engineering focus should be observability, stale-state recovery, and repeatable load testing before adding more ingestion features. +This document is still useful as a baseline audit, especially for understanding the problems that the June 8 changes were meant to solve. It should not be read as the latest state of the ingestion system by itself. diff --git a/docs/project-context.md b/docs/project-context.md index d017140..4988964 100644 --- a/docs/project-context.md +++ b/docs/project-context.md @@ -2,166 +2,130 @@ ## Purpose -This repository implements a workspace-scoped PDF RAG platform with three main parts: - +MultiDoc-RAG is a workspace-scoped PDF RAG system with three main surfaces: - `client`: React + Vite frontend -- `server`: FastAPI API server -- `worker`: Redis/RQ background workers +- `server`: FastAPI API +- `worker`: Redis/RQ ingestion workers -The system lets a user sign in with Supabase, create a workspace, upload PDFs, process them asynchronously, and ask grounded questions against indexed content with citations. +The system lets a user sign in with Supabase, upload one or many PDFs, process them asynchronously, and query indexed content with grounded citations. ## What Is Implemented ### 1. Authentication and Workspace Scoping -- The frontend uses Supabase Auth for login and signup. -- The API expects a bearer token and validates it against Supabase. -- Each authenticated user is resolved to a single workspace in v1-style flow. -- Workspace-scoped endpoints derive `workspace_id` on the server and use it to isolate data access. +- Supabase Auth is the user identity source. +- The API validates bearer tokens and resolves the current workspace server-side. +- Data access is consistently scoped by `workspace_id`. Primary files: - - `client/src/context/AuthContext.tsx` -- `client/src/lib/supabase.ts` - `server/app/api/deps.py` - `server/app/core/auth.py` - `server/app/api/workspaces.py` -### 2. Upload and Document Lifecycle - -- The client requests an upload URL from the API. -- The API creates a placeholder document record and returns a signed Supabase Storage upload URL. -- The client uploads the PDF directly to storage instead of sending file bytes through the API. -- The client then calls `upload-complete`. -- The API verifies the uploaded object and enqueues extraction. -- Documents can later be deleted, retried after failure, or reindexed. +### 2. Multi-Document Upload and Ingestion Runs -Current document statuses used by the code include: - -- `pending_upload` -- `uploaded` -- `extracting` -- `indexing` -- `ready` -- `indexed` -- `failed` +- The API supports both single-file and batch upload flows. +- `upload-prepare-batch` creates an `ingestion_run_id` and returns per-file prepare results. +- `upload-complete-batch` confirms uploaded objects and enqueues extraction jobs. +- The frontend upload flow now treats a multi-file upload as one run with accepted/rejected/processing/ready/failed summaries. Primary files: - -- `client/src/lib/api.ts` +- `client/src/components/upload/UploadPanel.tsx` - `client/src/pages/UploadPage.tsx` - `server/app/api/documents.py` -- `server/app/core/storage.py` +- `server/app/schemas/documents.py` ### 3. Background Ingestion -- Extraction and indexing are handled by Redis-backed RQ workers. -- `ingest_extract` downloads the PDF from Supabase Storage, extracts page text, and writes `document_pages`. -- `ingest_index` reads page text, chunks it per page, creates embeddings, stores chunks and vectors, and marks the document queryable. -- The worker process can listen to one or more named queues from environment configuration. +- `ingest_extract` downloads PDFs from Supabase Storage, validates file/page/content rules, writes `document_pages`, and enqueues indexing. +- `ingest_index` chunks page text, batches embeddings, writes chunks/vectors, commits token usage, and marks documents queryable. +- Worker failure callbacks and ingestion reconciliation reduce the chance of documents staying stuck forever. Primary files: - -- `worker/worker.py` - `worker/jobs/ingest_extract.py` - `worker/jobs/ingest_index.py` +- `worker/jobs/ingest_callbacks.py` +- `server/app/core/ingestion_reconciliation.py` ### 4. Retrieval and Querying -- The API supports both standard query and streaming query flows. -- A query is embedded with OpenAI embeddings. -- Retrieval uses pgvector similarity search over stored chunk embeddings. -- The answer prompt is strict-grounded: it is supposed to use only retrieved context and include citations. -- The API returns answer text, citations, and updated usage state. +- The API supports standard query and streaming query flows. +- Query payloads can use legacy `document_id`, new `document_ids`, or both. +- Retrieval can search across a selected set of ready documents. +- Query logs store the searched document set. Important current limitation: - -- The implemented query flow is effectively single-document per request. -- The main query API accepts one `document_id`. -- Retrieval filters to one document at a time. -- The frontend chat flow also centers on one active document at a time. - -This means the repository supports many uploaded documents, but not true cross-document querying yet. +- the backend is multi-document capable, but the main chat UX still centers on one active document Primary files: - +- `server/app/schemas/query.py` - `server/app/api/query.py` - `server/app/api/query_stream.py` -- `server/app/core/embeddings.py` - `server/app/core/retrieval.py` -- `server/app/core/llm.py` -- `server/app/core/prompts.py` - `client/src/pages/ChatPage.tsx` ### 5. Token Budget Control -- The server tracks daily token usage per workspace. -- Query requests estimate token usage before the LLM call. -- Tokens are reserved first, then committed after actual usage is known. -- Embedding work in indexing also reserves and commits token usage. -- The usage state includes `used`, `reserved`, `remaining`, and reset timing. +- Query-time LLM usage is reserved before execution and committed after actual usage is known. +- Indexing reserves and commits embedding usage during batch embedding work. +- Stale reservation cleanup exists for stranded reservations. Primary files: - - `server/app/core/token_budget.py` -- `server/app/api/query.py` - `worker/jobs/ingest_index.py` -- `server/app/api/usage.py` +- `worker/jobs/maintenance.py` -### 6. Observability and History +### 6. Observability and Ingestion Health -- The system exposes usage and observability endpoints. -- Query logs are used to show totals, recent errors, latency stats, and top documents. -- The project also includes chat session APIs for saving and restoring conversations tied to a document/workspace context. +- Usage and query observability endpoints are implemented. +- The documents API now exposes queue counts, ingestion health, run status, and reconciliation endpoints. +- The frontend observability page shows query metrics plus average extraction/index/total ingestion timing based on document timing fields. Primary files: - - `server/app/api/usage.py` -- `server/app/api/queries.py` -- `server/app/api/chats.py` +- `server/app/api/documents.py` - `client/src/pages/ObservabilityPage.tsx` -## How It Works End to End +### 7. Load-Testing Support -1. The user signs in through Supabase Auth. -2. The API resolves the user to a workspace. -3. The client calls `upload-prepare`. -4. The API creates a document placeholder and returns a signed upload URL. -5. The client uploads the PDF to Supabase Storage. -6. The client calls `upload-complete`. -7. The API enqueues extraction on Redis/RQ. -8. The extraction worker downloads the PDF, extracts text, and writes page rows. -9. The indexing worker chunks page text, generates embeddings, and stores vectors in Postgres/pgvector. -10. The document becomes queryable. -11. The user selects an indexed document and asks a question. -12. The API embeds the question, retrieves top chunks, reserves token budget, calls the LLM, and returns a grounded answer with citations. -13. Usage and query metadata are recorded for later inspection. +- The repository includes `scripts/ingestion_load_test.py`. +- Load-test artifacts are stored under `artifacts/ingestion-load/`. -## Current Architecture Notes +Primary files: +- `scripts/ingestion_load_test.py` +- `docs/ingestion-load-testing.md` -- Storage is split across Supabase Storage for raw PDFs and Postgres for metadata, pages, chunks, embeddings, and usage data. -- Redis is used for background jobs and rate limiting behavior. -- OpenAI is used for embeddings and final answer generation. -- The codebase includes some schema-compatibility fallbacks, which suggests the schema and implementation have evolved over time. -- Some parts of the system are broader than the original architecture spec, including chat session APIs and observability views. +## Recent Shipped Changes -## Current Gaps Relative to a True Multi-Document RAG +Recent merged work reflected in the codebase: +- `08f452d` on 2026-06-08: multi-document ingestion workflow improvements +- `9fd2a85` on 2026-06-08: ingestion hardening and multi-document query readiness +- `0409162` on 2026-06-08: ingestion timing in dashboards +- `f3ad7db` on 2026-06-06: token budget lock-contention fix for streaming queries -These are the most important gaps to keep in mind for future tasks: +## Important Limits In Code Today -- Querying is still one-document-at-a-time. -- Retrieval does not yet search across a selected set of documents. -- The UI does not yet provide a multi-document query selection flow. -- Some status handling is adaptive because the code supports more than one schema/state shape. -- There are still TODOs and implementation rough edges, so this should be treated as a strong working baseline rather than a fully finished platform. +- max file size: `10 MB` +- max PDF pages: `10` +- max documents per workspace: `100` +- max bulk upload files: `50` +- max question length: `500` +- max selected documents per query: `10` +- max total pages across query documents: `100` -## Purpose of `docs/task` +Source of truth: +- `server/app/config.py` +- `.env.example` + +## Current Gaps -The `docs/task` folder is intended to hold focused markdown files for future improvements. Each file should describe one concrete task, such as: +The main gaps that still matter: +- chat UX is still single-document even though backend query supports multiple documents +- observability is stronger than before but still lacks persistent queue-depth history and deeper worker-level metrics +- docs and schema-history still contain traces of older status models and pre-batch assumptions +- the repository should still be treated as a strong working baseline, not a fully finished production system + +## Purpose of `docs/task` -- multi-document retrieval -- multi-document query UI -- schema cleanup -- ingestion performance improvements -- better observability -- test coverage expansion +The task docs should be read as implementation guidance and follow-up planning, not as a description of the original untouched baseline. Several backend ingestion tasks are now partially or mostly implemented and should be read with that updated context in mind. diff --git a/docs/task/document-ingestion-limits-and-failure-handling.md b/docs/task/document-ingestion-limits-and-failure-handling.md index 4252d5b..6c82984 100644 --- a/docs/task/document-ingestion-limits-and-failure-handling.md +++ b/docs/task/document-ingestion-limits-and-failure-handling.md @@ -2,344 +2,45 @@ ## Status -Drafted from current codebase and aligned with the ingestion improvement work. +Mostly implemented, with some follow-up cleanup still useful. -## Goal +## What Has Already Been Done -Define and implement clear ingestion requirements for documents, and make ingestion failure handling predictable, visible, and recoverable. - -This task is about two things: - -1. enforcing document limits clearly -2. handling ingestion failures properly from validation through retry - -## Why This Task Exists - -The project already has some limits and failure paths, but they are not yet treated as a fully defined product and engineering workflow. - -Right now the system has partial support for: - -- file size checks -- content type checks -- document status updates -- retry and reindex actions -- worker-side failure marking - -But the limits and failure rules still need to be made more explicit, more consistent, and easier to operate. - -## Scope - -This task should cover: - -- document upload requirements -- validation rules -- user-safe rejection behavior -- ingestion-time failure behavior -- status transitions for failures -- error classification -- retry behavior -- observability of failures - -This task is closely related to: - -- `docs/task/ingestion-pipeline-scale-and-reliability.md` -- `docs/task/report-ingestion-workflow.md` -- `docs/task/multi-document-ingestion-ux.md` - -## Current Baseline - -Based on the current code and project context, the intended constraints are small PDFs only. - -The main working assumptions are: - -- PDF only -- text-based PDFs only -- small files -- limited page count - -The codebase already contains some existing limits, including: - -- max file size validation -- allowed content type validation -- workspace document count limit -- query length limit +The current code now enforces and communicates most of the important ingestion rules: +- PDF-only uploads +- max file size checks in API and worker paths +- max page-count enforcement during extraction +- unsupported scanned/image-only PDF rejection +- structured failure-message helpers +- retryability classification +- retry and reindex flows +- stale-state reconciliation for stuck ingestion work Primary files: - - `server/app/config.py` - `server/app/api/documents.py` -- `server/app/schemas/documents.py` +- `server/app/core/ingestion_policy.py` - `worker/jobs/ingest_extract.py` - `worker/jobs/ingest_index.py` -## Requirements To Make Explicit - -The implementation should define and enforce clear ingestion requirements. - -### File Requirements - -At minimum, the system should have explicit rules for: - -- allowed file type: PDF only -- maximum file size -- maximum page count -- supported PDF type: text-based only -- invalid filename handling - -Recommended baseline for this project: +## Current Limits - maximum file size: `10 MB` -- maximum page count: `10 pages` - -If a different value is chosen, it should still be documented clearly and enforced consistently. - -### Validation Timing - -The system should define where each rule is enforced. - -Examples: - -- before upload begins -- at upload-prepare -- at upload-complete -- during extraction -- after extraction if document shape is invalid - -The goal is to reject early where possible, but still re-check critical constraints after upload so the system is safe against mismatches and bad clients. - -## What Is Missing Today - -### 1. Limits are not fully centralized as product rules - -Some constraints exist, but they are not yet fully presented as a clean ingestion policy. - -Missing areas: - -- one clear source of truth for file size and page count -- consistent enforcement across API and workers -- consistent error messages -- clearer documentation for agents and developers - -### 2. Page-count handling needs to be explicit - -If the product rule is “small PDFs only,” then page-count enforcement should be unambiguous. - -Missing areas: - -- guaranteed page-count validation path -- clear behavior when page count exceeds limit -- clear status and error message when rejection happens after extraction starts - -### 3. Failure states need stronger definition - -The system marks documents as failed, but failure behavior still needs to be defined more rigorously. - -Missing areas: - -- consistent failure categories -- clearer distinction between validation failure and processing failure -- clearer distinction between retryable and non-retryable failures -- stronger rules for partial progress cleanup - -### 4. Error messages need to be more actionable - -Users and operators need clearer reasons for failure. - -Missing areas: - -- “file too large” -- “page count exceeded” -- “unsupported PDF type” -- “text extraction failed” -- “embedding budget exceeded” -- “storage object missing” - -These should be structured and consistent, not just raw exception text where avoidable. - -### 5. Retry behavior needs stronger rules - -Retries should be allowed only when safe and meaningful. - -Missing areas: - -- which failures can be retried directly -- which failures require reupload -- which failures should stay terminal -- how partial extracted/chunked data is cleaned up before retry - -## Desired End State - -After this task is implemented, the ingestion system should behave like this: - -- document requirements are clearly defined -- invalid files are rejected as early as possible -- critical limits are revalidated during ingestion where necessary -- failures are categorized clearly -- document statuses reflect the true stage and outcome -- retries are safe and predictable -- operators and future agents can understand why a document failed - -## Failure Categories To Introduce - -The implementation should define failure categories or equivalent structured error types. - -Suggested categories: - -- validation failure -- upload/storage failure -- extraction failure -- page-limit failure -- unsupported-content failure -- indexing failure -- budget failure -- transient infrastructure failure - -These do not have to be stored exactly with these names, but the system should have a structured way to reason about them. - -## Recommended Handling Rules - -### A. Validation Failures - -Examples: - -- wrong content type -- file too large -- invalid filename -- too many documents in workspace - -Expected behavior: - -- reject before ingestion starts -- do not enqueue workers -- return clear API errors - -### B. Upload Verification Failures - -Examples: - -- object missing from storage -- bucket/path mismatch - -Expected behavior: - -- do not start extraction -- leave a clear document state or reject completion safely -- avoid creating stuck documents - -### C. Page Limit Failures - -Examples: - -- uploaded PDF exceeds max page count - -Expected behavior: - -- detect during extraction or immediately after page inspection -- mark document failed with a clear message -- avoid indexing -- make retry behavior clear - -### D. Unsupported Content Failures - -Examples: - -- scanned PDF with no extractable text -- effectively empty extracted text - -Expected behavior: - -- mark as failed with a clear reason -- do not continue indexing if the content is unusable -- tell the user the document type is not supported - -### E. Indexing Failures - -Examples: - -- chunk write failure -- embedding API failure -- vector insert failure - -Expected behavior: - -- mark document failed -- release reserved budget correctly -- preserve enough information to retry safely - -### F. Budget Failures - -Examples: - -- insufficient token budget for embeddings - -Expected behavior: - -- fail safely -- leave no stranded reservations -- return or store a clear failure reason -- support retry later if budget resets - -## Retry Policy Requirements - -The task should define a retry policy. - -Suggested expectations: - -- retry allowed for transient infrastructure failures -- retry allowed for indexing failures when cleanup is safe -- retry not useful for file-size or page-limit failures without changing the document -- retry not useful for unsupported scanned PDFs unless product support changes - -The UI can later reflect these distinctions, but the backend behavior should be defined first. - -## Data and Status Considerations - -The implementation should review whether the current document status model is sufficient. - -Important questions: - -- is one generic `failed` status enough -- should failure type be stored separately -- should retryability be derivable from failure type -- should error messages be structured in addition to human-readable text - -This task does not require a specific schema design, but it should leave the system easier to reason about. - -## Acceptance Criteria - -This task should be considered complete only when most or all of the following are true: - -- file-size rules are explicit and enforced consistently -- page-count rules are explicit and enforced consistently -- unsupported document types are rejected clearly -- ingestion failures produce meaningful failure states -- retry behavior is safe and defined -- budget-related failures are handled cleanly -- failure reasons are useful for both users and operators -- documentation reflects the final rules - -## Suggested Validation - -The implementation should be tested with cases like: - -- valid PDF within size and page limits -- PDF larger than allowed size -- PDF with more than allowed pages -- scanned or non-extractable PDF -- missing uploaded object -- embedding failure -- low token-budget ingestion -- retry of retryable failure -- attempted retry of non-retryable failure +- maximum page count: `10` +- supported content type: `application/pdf` +- supported content mode: text-based PDFs only -## Risks To Watch +## Remaining Work -- validating too late and wasting worker capacity -- weak error messages that hide the real cause -- allowing unsafe retries that create inconsistent state -- failing to release reserved tokens on error paths -- mixing user-facing errors and raw internal exceptions +- keep docs and schema references aligned with the actual runtime status model +- keep error wording consistent across API responses, worker failures, and frontend messaging +- validate that operator views clearly separate expected rejections from infrastructure failures +- re-run larger ingestion tests to confirm failure handling remains predictable under load -## Summary +## Expected Behavior -This task makes document ingestion requirements explicit and failure handling dependable. The goal is to make the system strict about what it accepts, clear about why a document failed, and safe when retries or recovery actions are needed. +- reject invalid files as early as possible +- revalidate critical limits during ingestion +- avoid starting downstream work for known-invalid inputs +- mark terminal failures clearly and safely +- allow retry only where the failure is meaningfully retryable diff --git a/docs/task/ingestion-follow-up-hardening-and-validation.md b/docs/task/ingestion-follow-up-hardening-and-validation.md index bcfa46f..d55f807 100644 --- a/docs/task/ingestion-follow-up-hardening-and-validation.md +++ b/docs/task/ingestion-follow-up-hardening-and-validation.md @@ -2,293 +2,27 @@ ## Status -Derived from the current ingestion workflow audit and the existing evaluation report. +This is now a follow-up task list, not a pure design draft. Several items from the original list have already landed. -## Goal +## Already Implemented -Capture the important ingestion gaps that are still not implemented or still weak, then define: +- repeatable ingestion load-test tooling +- ingestion timing in the frontend dashboard +- stale document and ingestion-run reconciliation +- backend support for multi-document query contracts +- broader ingestion API and retrieval test coverage -1. how each gap should be implemented -2. what should be tested after implementation +## Still Worth Doing -This file is meant to turn the audit into concrete follow-up work. +1. run larger repeatable benchmarks and keep the results comparable over time +2. add deeper operator observability such as queue-depth history and worker-level timing/latency logging +3. finish any remaining schema/runtime status cleanup +4. extend the frontend chat/query UX to take advantage of multi-document backend support +5. keep expanding automated tests around edge-case failure and recovery flows -## Why This Task Exists +## Why This Task Still Exists -The repository already has a working ingestion pipeline for small PDFs, including: - -- batch upload prepare and complete -- ingestion runs -- extraction and indexing workers -- structured failure messages -- retry and reindex flows -- queue visibility -- token-budget reservation and cleanup - -But the system is still missing several important pieces before the ingestion path can be considered robust, measurable, and ready for higher-volume usage. - -## Scope - -This task covers the remaining weak points found in the audit: - -1. no repeatable load and throughput validation -2. weak ingestion observability -3. incomplete stale-state and run lifecycle recovery -4. schema and runtime status mismatch -5. query path still single-document -6. thin automated ingestion test coverage - -## Recommended Order - -1. Fix schema and status-model mismatch -2. Add stale document and run reconciliation -3. Add ingestion observability and operator health visibility -4. Add automated ingestion integration tests -5. Add repeatable load-test tooling -6. Extend retrieval and querying toward multi-document support - -The first three items are backend safety and operability work. They should happen before calling the ingestion system stable at larger scale. - -## Follow-Up Items - -### 1. Repeatable Load and Throughput Validation Is Missing - -#### Current Weakness - -The code supports batch ingestion, but there is no repeatable load-test harness for: - -- 10 PDFs -- 25 PDFs -- 50 PDFs -- 100 PDFs - -There is also no durable way to compare one ingestion revision against another. - -#### How To Implement - -- Add a dedicated load-test task doc and script set under `scripts/` for ingestion benchmarking. -- Create a repeatable test dataset of small text PDFs with known page counts and known expected outcomes. -- Add a driver that can: - - create or reuse a workspace - - prepare and complete uploads in bulk - - poll ingestion runs until terminal - - capture final document states - - measure total wall-clock run time - - measure per-document completion time -- Persist results to a simple JSON or markdown artifact so runs can be compared over time. -- Include scenarios with: - - all-valid batches - - mixed valid and invalid documents - - queue backlog conditions - -#### What To Test After Implementation - -- 10 valid small PDFs complete successfully. -- 25 valid small PDFs complete successfully. -- 50 valid small PDFs complete successfully. -- Mixed valid and invalid batches still complete with isolated failures. -- The load-test artifact records: - - total duration - - per-document duration - - success count - - failure count - - queue depth snapshots if available -- Re-running the same scenario produces comparable output structure. - -### 2. Ingestion Observability Is Still Weak - -#### Current Weakness - -The system exposes queue counts and document/run status, but it still lacks: - -- stage timing -- queue-depth history -- worker utilization visibility -- operator-facing ingestion health summary -- clear separation between expected document rejection and infrastructure failure - -#### How To Implement - -- Add stage timestamps or timing fields for key transitions: - - upload completed - - extract job enqueued - - extraction started - - extraction finished - - index job enqueued - - indexing started - - indexing finished -- Add an operator endpoint such as `GET /documents/ingestion-health` that returns: - - current queue counts - - failed registry counts - - oldest active document age - - stale active document count - - active ingestion run count - - stale reservation count if practical -- Add structured worker logs for: - - queue wait start/end - - extraction duration - - indexing duration - - embedding batch counts - - embedding API latency if practical -- Track expected validation failures separately from infrastructure failures in logs or API summaries. - -#### What To Test After Implementation - -- A successful document exposes complete stage timing. -- A failed validation document is visible as a document failure but not mislabeled as infrastructure failure. -- The health endpoint reports queue and stale-state summaries correctly during active ingestion. -- Worker logs contain enough information to distinguish: - - queue delay - - extraction time - - indexing time - - embedding time -- The UI or operator flow can identify whether a slowdown is caused by queue wait or processing. - -### 3. Stale-State and Run Lifecycle Recovery Is Incomplete - -#### Current Weakness - -There is failure callback coverage for killed jobs, but there is still no general reconciliation process for: - -- documents stuck in `uploaded` -- documents stuck in `extracting` -- documents stuck in `indexing` -- ingestion runs left in `preparing` or `processing` after work is effectively done - -#### How To Implement - -- Add a scheduled reconciliation job for ingestion state. -- Define configurable stale thresholds for: - - `uploaded` - - `extracting` - - `indexing` -- Reconciliation should: - - mark stale active documents as `failed` - - attach a structured transient-infrastructure error message - - refresh or recompute affected ingestion run statuses -- Consider deriving ingestion run status dynamically from current document states instead of trusting persisted run status alone. -- Ensure reconciliation is idempotent and safe to run frequently. - -#### What To Test After Implementation - -- A document intentionally left in `uploaded` beyond threshold is marked `failed`. -- A document intentionally left in `extracting` beyond threshold is marked `failed`. -- A document intentionally left in `indexing` beyond threshold is marked `failed`. -- A run with only terminal document states is reconciled to `completed`, `partial`, or `failed` correctly. -- Re-running reconciliation does not corrupt already-terminal rows. -- Retry still works correctly after reconciliation marks a document failed. - -### 4. Schema and Runtime Status Model Do Not Fully Match - -#### Current Weakness - -The code uses a richer runtime model including statuses like: - -- `uploading` -- `extracting` -- `indexed` - -But the SQL schema files still define a narrower `documents.status` check constraint. The code currently works around this mismatch defensively, which is fragile. - -#### How To Implement - -- Update `scripts/schema.local.sql` and `scripts/schema.supabase.sql` so the document status constraint matches the actual runtime statuses used by the application and workers. -- Review migrations or schema upgrade steps so existing environments can be brought to the same status model safely. -- Remove compatibility branches only after the schema is consistently aligned across environments. -- Ensure frontend status types, backend schemas, workers, and DB constraints all use the same status vocabulary. - -#### What To Test After Implementation - -- Fresh schema setup accepts every runtime status the code uses. -- Existing environments can migrate without data loss. -- Worker transitions no longer rely on fallback behavior for missing statuses. -- Upload, extract, index, retry, and reindex transitions all succeed against the aligned schema. -- Status values shown in the UI match the persisted DB states exactly. - -### 5. Retrieval and Querying Are Still Single-Document - -#### Current Weakness - -The ingestion side now supports many documents per workspace, but the query side still accepts one `document_id` and retrieves chunks from only one document at a time. - -This is not an ingestion breakage, but it is a real readiness gap for the intended multi-document system. - -#### How To Implement - -- Add a new query contract that accepts multiple selected document IDs. -- Enforce explicit query-side limits for: - - max selected documents - - max total pages across selected documents -- Update retrieval SQL to search across a selected document set instead of one document. -- Preserve citation quality by returning document metadata together with page and chunk references. -- Update query logs and observability so they record all searched documents, not one. - -#### What To Test After Implementation - -- A query across two ready documents returns citations from both when relevant. -- Query limits reject oversized document selections safely. -- Workspace isolation still holds for multi-document queries. -- Query logs record the selected document set accurately. -- Existing single-document flows continue to work. - -### 6. Automated Ingestion Test Coverage Is Too Thin - -#### Current Weakness - -The repository has helper-level tests for ingestion policy, ingestion run status derivation, and token-budget primitives, but it does not have strong automated coverage for: - -- batch upload orchestration -- partial success batches -- retry flows -- reindex flows -- stuck-job recovery -- low-budget ingestion failure -- queue failure callback behavior - -#### How To Implement - -- Add integration-style tests around the documents API and worker jobs. -- Use fixtures or mocks for: - - Supabase storage interactions - - OpenAI embeddings - - Redis/RQ enqueue behavior where full worker execution is not needed -- Add job-level tests for: - - extraction success - - page-limit failure - - unsupported-content failure - - indexing success - - budget failure - - cleanup after failure -- Add API tests for: - - batch prepare - - batch complete - - retry allowed vs retry denied - - reindex with and without existing pages - -#### What To Test After Implementation - -- Batch prepare returns mixed accepted and rejected results correctly. -- Batch complete enqueues extraction correctly and fails safely on missing objects. -- Retry only works for retryable failures. -- Reindex chooses extract or index path correctly based on existing page rows. -- Index budget exhaustion leaves no stranded reservations. -- Failure callback marks timed-out or killed jobs as failed. -- Cleanup removes stale chunks, embeddings, and pages when expected. - -## Acceptance Criteria - -This task should be considered complete only when: - -- the remaining ingestion weak points are each turned into explicit implementation work -- every item has a clear post-implementation validation plan -- the schema matches the runtime status model -- stale-state recovery exists for documents and runs -- ingestion observability is materially better -- automated ingestion coverage is broader than helper-only unit tests -- repeatable load validation exists for larger batches - -## Summary - -The ingestion pipeline is no longer missing its basic architecture. The important remaining work is hardening, observability, lifecycle recovery, automated validation, and true multi-document readiness. - -This task captures that follow-up work in a way that is concrete enough to implement and test. +The hardest remaining questions are no longer “can the pipeline work?” but: +- how well it behaves under sustained load +- how clearly operators can diagnose slowdowns or failures +- how much of the backend multi-document capability is exposed cleanly in the UI diff --git a/docs/task/ingestion-pipeline-scale-and-reliability.md b/docs/task/ingestion-pipeline-scale-and-reliability.md index 773859a..386e535 100644 --- a/docs/task/ingestion-pipeline-scale-and-reliability.md +++ b/docs/task/ingestion-pipeline-scale-and-reliability.md @@ -2,332 +2,50 @@ ## Status -Drafted from current codebase and confirmed product direction. +Partially implemented. The main backend ingestion expansion landed across `08f452d` and `9fd2a85`, but this area still has follow-up work. -## Goal - -Enhance the ingestion pipeline end to end so the backend can ingest large numbers of small PDFs reliably and with better throughput. - -Primary focus: - -1. Performance -2. Reliability -3. User experience later, mainly through frontend follow-up tasks - -This task is backend-first, but it must be designed so future multi-document querying fits naturally on top of the ingestion and retrieval model. - -## Scope Confirmed - -- Focus on backend only for this task -- Target documents are small PDFs -- Expected per-document limits: - - up to 10 pages - - up to 10 MB -- Multi-document query is not the main deliverable of this task, but the design must prepare for it -- Frontend improvements can come later in separate tasks - -## Why This Task Exists - -The current project already supports: - -- workspace-scoped uploads -- asynchronous extraction and indexing -- token-budget tracking -- grounded query and streaming query -- retry and reindex flows - -But the current ingestion path is still shaped around single-document operations and moderate load. If the system needs to ingest `N` documents smoothly and predictably, the backend needs a stronger bulk-ingestion design, better throughput, clearer failure handling, and better operational visibility. - -## Current Workflow - -### Upload and Ingestion Today - -1. The client calls `POST /documents/upload-prepare` for one file. -2. The API creates one placeholder document row and returns one signed upload URL. -3. The client uploads one PDF directly to Supabase Storage. -4. The client calls `POST /documents/upload-complete` for that one file. -5. The API verifies storage state and enqueues one extraction job. -6. `ingest_extract` downloads the PDF, extracts text, stores page rows, and enqueues indexing. -7. `ingest_index` reads `document_pages`, chunks text page by page, creates embeddings, writes chunk rows and vector rows, then marks the document ready. - -Primary files: - -- `server/app/api/documents.py` -- `worker/jobs/ingest_extract.py` -- `worker/jobs/ingest_index.py` -- `worker/worker.py` - -### Querying Today - -1. The client selects one active document. -2. The client sends one `document_id` and one question. -3. The API embeds the question. -4. Retrieval searches chunks for that one document only. -5. The API calls the LLM with grounded context and returns citations. - -Primary files: - -- `server/app/schemas/query.py` -- `server/app/api/query.py` -- `server/app/core/retrieval.py` - -## What Is Missing Today - -### 1. No bulk ingestion contract - -The backend currently works one document at a time: - -- one `upload-prepare` request per file -- one `upload-complete` request per file -- one extract job per file -- one index job per file - -This is workable for small manual usage, but not ideal for ingesting large sets of documents efficiently. +## What Has Already Been Done -### 2. Throughput is limited by per-document overhead +The codebase now includes: +- batch upload prepare and complete endpoints +- ingestion runs and grouped status tracking +- embedding batching +- richer failure classification helpers +- queue status and ingestion health APIs +- stale ingestion reconciliation +- worker callback coverage for failed/timed-out jobs -The current flow repeats a lot of work per file: - -- repeated API validation -- repeated enqueue calls -- repeated storage verification -- repeated status transitions - -This overhead becomes expensive when `N` grows. - -### 3. Indexing is still very sequential - -The current indexing job does a lot of row-by-row and chunk-by-chunk work: - -- chunk rows are inserted one at a time -- embeddings are created one chunk at a time -- embedding usage is reserved and committed one chunk at a time -- vector rows are inserted one at a time - -This hurts throughput and increases DB and API round-trips. - -### 4. Failure handling is document-level, not ingestion-run-level - -The system can mark a document failed and retry it, which is good. But it does not yet have a strong concept of a large ingestion run: - -- no batch identity -- no run summary -- no grouped progress reporting -- no aggregated failure view -- no clear backpressure behavior when the queue grows - -### 5. Observability is query-heavy, not ingestion-heavy - -The project has useful usage and query observability, but ingestion-specific visibility is weaker: - -- no stage-level throughput metrics -- no queue-depth reporting in the API -- no ingestion latency histograms by stage -- no batch-level progress state - -### 6. Rate limits are tuned for ordinary usage, not bulk ingestion - -Current defaults: - -- `upload-prepare`: 10 per minute -- `upload-complete`: 20 per minute - -These values are too restrictive for large ingestion workflows unless a more deliberate bulk ingestion design is introduced. - -### 7. Retrieval is not yet multi-document - -This is not strictly an ingestion problem, but it matters for design: +## Goal -- the query schema accepts one `document_id` -- retrieval filters to one document -- the frontend centers around one active document +Continue improving the ingestion pipeline so MultiDoc-RAG can ingest larger numbers of small PDFs with better throughput, safer failure handling, and clearer operational visibility. -Any ingestion improvements should preserve or improve the metadata model needed for future retrieval across many documents in one workspace. +## Remaining Focus Areas -## Desired End State +1. higher-volume throughput validation under realistic load +2. deeper ingestion observability beyond current dashboard timing +3. clearer separation between expected validation failures and infrastructure failures in operator views +4. remaining status/schema cleanup and consistency work +5. any worker or DB bottlenecks discovered through real load testing -After this task is implemented, the backend should: +## Constraints In Code Today -- ingest many small PDFs more efficiently -- handle queue load more predictably -- reduce unnecessary per-document overhead -- make failures easier to understand and recover from -- expose better ingestion observability -- preserve workspace isolation and token-budget safety -- prepare the data and API design for future multi-document retrieval +- max file size: `10 MB` +- max pages per PDF: `10` +- max bulk upload files per run: `50` +- extraction timeout: `900s` +- indexing timeout: `1800s` +- embedding batch size: `32` ## Non-Goals -These are not the main goals of this task: - -- frontend bulk-upload UX +- redesigning the whole client - OCR support -- support for larger PDFs beyond current constraints -- replacing pgvector or changing core storage providers -- full multi-document chat UX - -## Implementation Direction - -The agent implementing this task should evaluate and likely combine the following improvements. - -### A. Add a bulk-ingestion backend contract - -Introduce a backend path that supports many documents as one logical ingestion operation. - -Possible directions: - -- batch upload-prepare endpoint -- batch upload-complete endpoint -- ingestion-run record or batch record -- per-run progress and error summary - -The exact shape can vary, but the system should stop treating high-volume ingestion as unrelated single-file actions. - -### B. Reduce enqueue and orchestration overhead - -The orchestration layer should be more efficient under load. - -Possible directions: - -- batch enqueueing for extraction/index jobs -- explicit job metadata for grouping and tracing -- clearer separation between user-facing completion and background scheduling -- queue policies for fair processing and backpressure - -### C. Improve extract and index worker throughput - -The workers should be reviewed for throughput bottlenecks. - -Likely improvements: - -- more efficient DB writes for `document_pages` -- more efficient inserts for `chunks` -- more efficient inserts for `chunk_embeddings` -- fewer transactions per document where safe -- better concurrency configuration across extract and index queues - -### D. Improve embedding efficiency - -Indexing currently performs one embedding call per chunk. - -The implementation should consider: - -- batching embeddings where the provider and limits allow -- reducing repeated client setup cost -- lowering DB round-trips around usage accounting where safe -- preserving idempotency and failure recovery - -### E. Strengthen ingestion reliability - -Bulk ingestion needs stronger operational safety. - -Important areas: - -- idempotent reruns -- safe retry behavior -- better handling of partially completed work -- queue recovery after worker failure -- clearer status transitions -- better distinction between transient and permanent failures - -### F. Make token-budget behavior predictable for ingestion - -Embedding many documents can consume budget quickly. - -The implementation must define: - -- when ingestion should fail early -- whether work is reserved per document, per chunk group, or per batch -- how to avoid leaving stranded reservations -- how to surface budget-related failures clearly - -### G. Prepare for multi-document retrieval - -This task should not stop at ingestion-only thinking. - -Even if multi-document query is implemented later, this task should preserve a clear path toward: - -- querying a selected set of document IDs -- retrieving top chunks across those documents -- keeping citations tied to document and page metadata -- avoiding ingestion decisions that assume only one active document will ever be queried - -## Recommended Phases - -### Phase 1: Reliability and Pipeline Cleanup - -Focus on making the existing ingestion flow safer before optimizing aggressively. - -Targets: - -- clarify document statuses and transitions -- strengthen retry and idempotency behavior -- improve error capture and logging -- ensure stale reservations are cleaned reliably -- make worker execution behavior clearer and more observable - -### Phase 2: Throughput Improvements - -Focus on reducing overhead and increasing throughput for `N` documents. - -Targets: - -- batch-oriented upload orchestration -- reduced DB round-trips -- more efficient chunk and embedding writes -- better worker concurrency and queue strategy -- improved rate-limit strategy for bulk ingestion paths - -### Phase 3: Multi-Document Retrieval Readiness - -Focus on preparing the backend for true cross-document querying. - -Targets: - -- query contract that can evolve from one `document_id` to many -- retrieval paths that can search across selected document sets -- metadata and observability that remain useful across multi-document contexts - -## Acceptance Criteria - -The task should be considered complete only when the implementation provides most or all of the following: - -- The backend supports a clear high-volume ingestion strategy, not just repeated single-file flows -- Ingestion throughput is materially improved for many small PDFs -- Failures are visible per document and, if introduced, per ingestion run or batch -- Worker and queue behavior are more predictable under load -- Token-budget handling remains safe under concurrent ingestion -- The design does not block future multi-document retrieval -- Documentation is updated to explain the new ingestion model - -## Suggested Validation - -The implementation should be validated with backend-focused tests and measurement such as: - -- ingesting dozens to hundreds of small PDFs -- mixed success and failure cases -- retry scenarios -- queue backlog scenarios -- low token-budget scenarios -- concurrent ingestion plus querying - -Useful outputs: - -- throughput before vs after -- average extraction time per document -- average indexing time per document -- queue wait time -- failure rate -- recovery behavior - -## Risks to Watch - -- making batching more complex than needed -- breaking idempotency -- overspending token budget during indexing -- adding bulk APIs without good observability -- optimizing for ingestion in a way that makes future multi-document retrieval harder +- changing core storage providers +- replacing pgvector -## Summary +## Next Useful Work -This task is the foundation for turning the current asynchronous document pipeline into a backend that can ingest large numbers of small PDFs reliably and efficiently. The current code already has the right building blocks, but it still operates mostly as a single-document workflow repeated many times. The work here is to turn that into a deliberate high-volume ingestion architecture while keeping future multi-document retrieval in view. +- add durable queue-depth history or operator metrics +- benchmark 10/25/50 document runs and compare revisions +- tighten ingestion-run lifecycle semantics where edge cases remain +- improve cross-surface status vocabulary so frontend, backend, worker, and schema stay aligned diff --git a/docs/task/multi-document-ingestion-ux.md b/docs/task/multi-document-ingestion-ux.md index 3b50c00..5de985a 100644 --- a/docs/task/multi-document-ingestion-ux.md +++ b/docs/task/multi-document-ingestion-ux.md @@ -2,285 +2,32 @@ ## Status -Drafted from current codebase and aligned with the existing backend-focused ingestion tasks. +Partially implemented. The upload workflow improved substantially in `08f452d`, but this task is not fully complete. -## Goal - -Improve the user experience for ingesting multiple documents so users can upload, monitor, and recover large document sets with less manual effort and less confusion. - -This task is focused on UX and frontend behavior, but it should assume the backend ingestion pipeline is also being improved through: - -- `docs/task/ingestion-pipeline-scale-and-reliability.md` -- `docs/task/report-ingestion-workflow.md` - -## Why This Task Exists - -The current project supports document upload and asynchronous processing, but the user experience is still mostly shaped around one document at a time. +## What Has Already Been Done -That is acceptable for light usage, but weak for real multi-document workflows where users want to: - -- upload many PDFs together -- understand what is processing -- see what succeeded and what failed -- retry only failed items -- know when ingestion is complete -- avoid repeatedly checking each document manually - -The backend can become more scalable, but users still need a workflow that makes large ingestion runs understandable and manageable. - -## Current UX Baseline - -Today the UX is centered on: - -- uploading documents through the upload page -- viewing a document list -- auto-refreshing status while processing is active -- deleting a document -- selecting a ready document for chat +The current upload UX already includes: +- multi-file selection +- pre-upload accepted and rejected file handling +- batch prepare and complete integration +- ingestion-run awareness +- live processing summaries +- status grouping for active, ready, and failed documents +- retry affordances for failed documents +- search, sort, and filter support for larger document sets Primary files: - - `client/src/pages/UploadPage.tsx` -- `client/src/components/upload/UploadPanel` -- `client/src/components/layout/AppShell.tsx` -- `client/src/components/sidebar/DocumentSidebar` +- `client/src/components/upload/UploadPanel.tsx` - `client/src/lib/api.ts` -## What Is Missing Today - -### 1. Multi-document ingestion does not feel like one workflow - -Even if users upload multiple files, the experience still feels like repeated single-document actions instead of one coordinated ingestion session. - -Missing pieces: - -- no clear “upload run” mental model -- no grouped progress summary -- no batch success/failure view -- no simple completion state for a multi-file upload session - -### 2. Weak progress visibility - -The current UI shows document-level status, but the user still has to interpret a lot manually. - -Missing pieces: - -- progress summary for all files in the current upload session -- counts by status such as queued, extracting, indexing, ready, failed -- clear indication of remaining work -- clearer stage-based language - -### 3. Failure handling is too manual - -Users need a better path when some documents fail. - -Missing pieces: - -- clear failed-items section -- batch retry for failed documents -- better error surfacing -- easier distinction between temporary and permanent failures - -### 4. No ingestion-oriented information architecture - -The upload page is functional, but it does not yet behave like a control center for a larger ingestion operation. - -Missing pieces: - -- a top-level ingestion summary -- a strong distinction between “processing” and “ready” -- filters for large document sets -- more actionable states - -### 5. Limited feedback during long-running ingestion - -When many documents are processing, users need reassurance and clarity. - -Missing pieces: - -- meaningful progress indicators -- estimated state of completion -- better empty/loading/error states -- confirmation when the ingestion run is effectively done - -## Desired End State - -After this task is implemented, the multi-document ingestion UX should: - -- feel like one coherent workflow -- make bulk upload easy to understand -- surface progress clearly across the whole set -- make failures easy to identify and recover from -- reduce the amount of manual checking users need to do -- scale to large document lists without becoming confusing - -## Non-Goals - -This task is not mainly about: - -- backend queue or worker implementation -- low-level ingestion throughput improvements -- query UX for multi-document chat -- OCR support -- redesigning the whole application shell - -## UX Requirements - -The implementation should aim to provide most or all of the following. - -### A. Multi-file upload flow - -The upload surface should support a clear bulk-ingestion experience. - -Possible requirements: - -- select many PDFs in one action -- show the selected files before upload starts -- validate file limits before submission -- show which files are accepted vs rejected before processing begins - -### B. Upload session summary +## What Is Still Missing -The page should communicate the current ingestion session as one thing. +- a true multi-document query/chat selection flow +- clearer run-history or batch-history UX beyond the current active upload experience +- stronger operator-facing ingestion health presentation in the frontend +- any additional UX adjustments discovered after larger real load tests -Useful elements: - -- total files selected -- total accepted -- total rejected -- total processing -- total completed -- total failed - -### C. Better document-state presentation - -Statuses should be understandable to non-technical users. - -The UI should make it easy to distinguish: - -- waiting to upload -- uploading -- queued -- extracting -- indexing -- ready -- failed - -If backend statuses are more technical, the frontend should present cleaner labels and explanations. - -### D. Better progress tracking - -The UI should reduce guesswork. - -Possible requirements: - -- grouped counts by state -- progress banner or summary block -- live refresh behavior while ingestion is active -- clear indication when processing has fully completed - -### E. Failure recovery UX - -The user should be able to recover from failures without hunting through the interface. - -Possible requirements: - -- dedicated failed documents section or filter -- retry failed documents individually -- retry all failed documents -- visible error message previews - -### F. Filtering and organization for larger sets - -Once many documents exist, the list becomes harder to use. - -Possible requirements: - -- filter by status -- search by filename -- sort by newest, oldest, status, or recent activity -- separate “active ingestion” and “ready library” views - -### G. Clear completion state - -Users should know when they are done waiting. - -Possible requirements: - -- “all documents processed” signal -- summary of final outcomes -- next-step prompt such as “documents are ready for querying” - -## Recommended UX Structure - -The implementation can vary, but a strong version of this page likely has these layers: - -1. Upload area -2. Current ingestion summary -3. Active processing section -4. Failed items section -5. Ready documents section -6. Controls for filtering and retrying - -This keeps the page oriented around workflow instead of a flat undifferentiated table. - -## Relationship With Backend Tasks - -This task should be coordinated with backend ingestion work. - -If the backend adds concepts like: - -- batch upload endpoints -- ingestion runs -- grouped progress metadata -- richer status information - -Then the frontend should use that directly instead of simulating everything client-side. - -If the backend does not yet expose run-level concepts, the frontend may need an interim approach using document-level aggregation, but that should be treated as a temporary UX layer. - -## Accessibility and Usability Expectations - -The UX should also account for: - -- clear visual hierarchy -- readable status labels -- keyboard-usable controls -- visible error messages -- understandable loading states -- responsive behavior on smaller screens - -## Acceptance Criteria - -This task should be considered complete only when the UI provides a clearly better workflow for multi-document ingestion, including most or all of the following: - -- users can upload multiple documents in a way that feels like one workflow -- users can see overall ingestion progress without inspecting each document manually -- failed documents are easy to identify -- retry behavior is easy to discover and use -- ready documents are clearly separated from in-progress documents -- the experience remains understandable as document count grows - -## Suggested Validation - -The implementation should be validated with scenarios like: - -- uploading 5 documents -- uploading 20 documents -- uploading 50 small documents -- mixed success and failure outcomes -- retrying only failed documents -- leaving and returning to the page during processing -- viewing the workflow on smaller screens - -## Risks To Watch - -- building a polished UI on top of weak backend status semantics -- showing misleading progress if true backend progress is not available -- overloading the page with too much status detail -- making bulk controls without clear safeguards -- treating large ingestion as just a bigger version of single-file upload - -## Summary +## Goal -This task upgrades the ingestion experience from a basic upload page into a workflow that supports real multi-document use. The goal is not only to let users send many files, but to help them understand what is happening, what failed, what finished, and what to do next. +Continue evolving the frontend so multi-document ingestion feels like one coherent workflow from selection through recovery, while staying aligned with the backend ingestion-run model that now exists. diff --git a/docs/task/report-ingestion-workflow.md b/docs/task/report-ingestion-workflow.md index 70ba462..e123548 100644 --- a/docs/task/report-ingestion-workflow.md +++ b/docs/task/report-ingestion-workflow.md @@ -1,299 +1,53 @@ -# Report Task: Ingestion Workflow Evaluation +# Task: Ingestion Workflow Evaluation -## Purpose - -This file defines a follow-up reporting task for the ingestion improvement work described in [docs/task/ingestion-pipeline-scale-and-reliability.md](D:/Desktop/projects/MultiDoc-RAG/docs/task/ingestion-pipeline-scale-and-reliability.md:1). -This file defines a follow-up reporting task for the ingestion improvement work described in `docs/task/ingestion-pipeline-scale-and-reliability.md`. - -The purpose of this task is not to add features directly. The purpose is to test, inspect, and evaluate the ingestion workflow properly after implementation work, then produce a grounded report that explains: - -- what is working well -- what is still missing -- what is still unreliable -- what is still too slow -- which areas need more improvement -- what should be changed next - -This report is performance-first and reliability-first. - -## Why This Report Is Needed - -The ingestion pipeline task is broad and likely to be implemented in phases. After changes are made, it will be easy to assume the system is “good enough” without properly testing: - -- high document counts -- queue pressure -- worker failures -- retry behavior -- token-budget edge cases -- DB write amplification -- partial success scenarios - -This report task exists so an agent can evaluate the real behavior of the backend and produce a structured assessment instead of relying on assumptions. - -## What This Report Should Evaluate +## Status -The report should cover the ingestion workflow end to end: +Evaluation baseline exists in `docs/ingestion-workflow-evaluation.md`, but it predates the 2026-06-08 hardening work. Future report work should measure the current codebase, not restate the old baseline. -1. upload preparation -2. upload completion -3. queue enqueue behavior -4. extraction worker behavior -5. indexing worker behavior -6. embedding throughput -7. DB write patterns -8. token-budget behavior during ingestion -9. retry and reindex behavior -10. failure recovery -11. observability -12. readiness for future multi-document retrieval - -## Current Baseline To Compare Against +## Purpose -The current codebase baseline is: +This task is for testing and evaluating the ingestion workflow after implementation changes, then writing a grounded report about: +- what is working +- what is still weak +- where throughput breaks down +- whether ingestion is operationally understandable +- what should be improved next -- one-file-at-a-time upload prepare -- one-file-at-a-time upload complete -- one extract job per document -- one index job per document -- chunk insertion one row at a time -- embedding generation one chunk at a time -- vector insertion one row at a time -- query flow still scoped to one document at a time +## Current Baseline -Primary reference files: +The current codebase is no longer single-file only. It now includes: +- batch upload prepare and complete +- ingestion runs +- queue and ingestion health endpoints +- reconciliation for stale active ingestion state +- embedding batching +- backend support for multi-document query contracts +Relevant files: - `docs/project-context.md` -- `docs/task/ingestion-pipeline-scale-and-reliability.md` - `server/app/api/documents.py` - `worker/jobs/ingest_extract.py` - `worker/jobs/ingest_index.py` - -## Main Evaluation Questions - -The report should answer these questions clearly. - -### 1. Throughput - -- Can the backend ingest `N` small PDFs without obvious bottlenecks? -- At what point does throughput degrade noticeably? -- Which stage is the bottleneck: - - upload orchestration - - extraction - - indexing - - embeddings - - database writes - - queue waiting - -### 2. Reliability - -- Do documents consistently reach a correct final state? -- Are failures isolated cleanly to individual documents or runs? -- Are retries safe and idempotent? -- Are partial failures recoverable without manual cleanup? - -### 3. Queue and Worker Behavior - -- Are extract and index queues balanced well? -- Do workers sit idle in one stage while another stage backs up? -- Is there backpressure behavior when the queue grows large? -- Do worker crashes leave jobs, documents, or reservations in bad states? - -### 4. Database Efficiency - -- Are inserts and updates still too chatty? -- Are there unnecessary transactions? -- Is indexing doing avoidable row-by-row work? -- Are there signs of lock contention or high write amplification? - -### 5. Embedding Efficiency - -- Are embeddings still created in too many small calls? -- Is batching being used effectively if implemented? -- Is embedding throughput the dominant bottleneck? -- Does usage accounting around embeddings create extra overhead? - -### 6. Token-Budget Safety - -- Does ingestion stop safely when budget is insufficient? -- Are reservations released correctly on failure paths? -- Are there stale or stranded reservations? -- Is the budget model predictable during concurrent ingestion? - -### 7. Observability - -- Can an operator understand where time is being spent? -- Are ingestion failures visible enough? -- Is there enough information to diagnose queue delay vs processing delay? -- Is there enough run-level or batch-level visibility if bulk ingestion was added? - -### 8. Multi-Document Readiness - -- Did ingestion improvements preserve clean metadata for future multi-document retrieval? -- Is the storage and chunk model still easy to extend for cross-document query? -- Did any optimization choices make future multi-document retrieval harder? - -## Expected Report Structure - -When this task is executed, the generated report should contain sections like these: - -### 1. Executive Summary - -- short summary of system health -- biggest strengths -- biggest weaknesses -- whether the ingestion pipeline is ready for larger-scale use - -### 2. Test Setup - -- environment used -- worker counts -- DB and Redis setup -- document count used in tests -- document size/page profile -- whether OpenAI calls were real or mocked - -### 3. Workflow Validation - -Explain whether each stage behaves correctly: - -- upload prepare -- upload complete -- extraction -- indexing -- final document state -- retry -- reindex - -### 4. Performance Findings - -Include: - -- throughput numbers -- average per-document time -- queue wait time -- stage-by-stage bottlenecks -- comparison against previous baseline if available - -### 5. Reliability Findings - -Include: - -- failure modes found -- inconsistent state transitions -- stuck jobs -- stranded reservations -- duplicate processing risk -- retry safety issues - -### 6. What Is Missing - -List the important gaps still remaining after the ingestion task implementation. - -Examples: - -- no strong batch run visibility -- no queue depth reporting -- indexing still too sequential -- weak retry semantics -- insufficient metrics - -### 7. What Needs Improvement Next - -This section should be actionable. - -It should rank improvements by impact: - -1. highest impact for throughput -2. highest impact for reliability -3. operational/observability fixes -4. design work that helps future multi-document retrieval - -### 8. Recommended Next Tasks - -Each recommended next task should be concrete and narrowly scoped. - -Examples: - -- batch embedding writes -- ingestion run tracking -- queue depth metrics endpoint -- worker crash recovery hardening -- token-budget cleanup improvements - -## Suggested Test Scenarios - -The agent running this report task should try to cover the following. - -### Load Scenarios - -- 10 small PDFs -- 25 small PDFs -- 50 small PDFs -- 100 small PDFs -- larger counts if practical - -### Failure Scenarios - -- one corrupted PDF in the middle of a batch -- storage verification failure -- worker interruption during extraction -- worker interruption during indexing -- low token-budget case -- retry after failure -- reindex after successful ingestion - -### Concurrency Scenarios - -- many documents uploaded together -- extraction backlog greater than indexing backlog -- indexing backlog greater than extraction backlog -- ingestion running while queries are also being served - -## Metrics To Capture - -At minimum, the report should try to capture: - -- total documents ingested -- total time for full ingestion run -- average extraction time per document -- average indexing time per document -- average queue wait time -- documents failed -- documents retried successfully -- token-budget failures -- stuck or long-running jobs - -If deeper instrumentation exists, also capture: - -- DB write counts or query counts -- embedding batch size -- queue depth over time -- worker utilization - -## Red Flags To Watch For - -The report should explicitly call out these kinds of problems if found: - -- documents stuck in `uploaded`, `extracting`, or `indexing` -- duplicate chunk or embedding generation -- retries causing inconsistent state -- stale token reservations -- queue starvation between extract and index phases -- ingestion speed collapsing sharply with moderate load -- too much dependence on manual operator cleanup -- improvements that help ingestion but block future multi-document retrieval - -## Output Requirement - -The final report produced by this task should be written in plain, concrete terms. It should not be a vague summary. It should identify: - -- what was tested -- what passed -- what failed -- where time is going -- what is still weak -- what should be done next - -## Summary - -This task exists to turn ingestion work into measurable engineering feedback. The earlier task improves the pipeline. This report task verifies whether that work actually solved the performance and reliability problems, identifies what is still weak, and creates a grounded path for the next round of backend improvements. +- `server/app/core/ingestion_reconciliation.py` + +## Evaluation Questions + +Focus the report on: +1. throughput under 10, 25, and 50 document runs +2. queue balance between extraction and indexing +3. correctness of ingestion-run terminal states +4. retry/reindex behavior after failures and reconciliation +5. token-budget behavior during larger ingestion runs +6. usefulness of timing and health data for operators +7. whether the remaining frontend and observability gaps block larger-scale use + +## Deliverable + +Produce a short report with: +- executive summary +- environment and worker setup +- scenario matrix +- performance findings +- reliability findings +- remaining gaps +- ranked next improvements diff --git a/server/README.md b/server/README.md index be98c11..aa127cf 100644 --- a/server/README.md +++ b/server/README.md @@ -1,39 +1,25 @@ -# Server - -`server/` is the FastAPI backend for Enterprise RAG. It owns authentication, workspace resolution, document APIs, grounded query orchestration, token budget enforcement, and observability endpoints. - -## Responsibilities - -- validate Supabase JWTs -- resolve the caller's workspace -- manage document upload preparation and upload completion -- enqueue extraction and indexing jobs -- execute grounded RAG queries and SSE streaming queries -- track daily token usage with reserve/commit/release semantics -- expose query history, citations, chat sessions, and observability data - -## Module Layout - -```text -server/ -├── app/ -│ ├── main.py # FastAPI app and router registration -│ ├── config.py # environment settings and UTC helpers -│ ├── api/ # route handlers -│ ├── core/ # auth, retrieval, llm, embeddings, token budget -│ ├── db/ # SQLAlchemy session, models, repositories -│ ├── schemas/ # request and response schemas -│ ├── storage/ # Supabase storage integration -│ └── utils/ # rate limiting and logging helpers -├── migrations/ # Alembic environment and revisions -├── tests/ -├── requirements.txt -└── pyproject.toml -``` +# MultiDoc-RAG Server -## API Surface +`server/` is the FastAPI backend for MultiDoc-RAG. It owns auth validation, workspace resolution, document orchestration, query execution, token budgets, chat/session APIs, and observability. + +## What Is Implemented + +- Supabase JWT validation and workspace-scoped access +- single-file and batch upload prepare/complete APIs +- ingestion-run tracking, queue inspection, ingestion health, and reconciliation endpoints +- retry and reindex flows +- grounded query and streaming query APIs +- backend support for querying across multiple selected documents +- daily token reservation, commit, release, and usage reporting +- query history, citation lookup, and chat session APIs -Routes registered in `app/main.py`: +## Recent Backend Updates + +- `08f452d` on 2026-06-08: batch upload contracts, ingestion-run APIs, upload policy helpers, and stronger worker failure callbacks +- `9fd2a85` on 2026-06-08: stale ingestion reconciliation, multi-document query contract support, retrieval updates, and broader ingestion tests +- `f3ad7db` on 2026-06-06: token budget lock-contention fix for streaming queries and Supabase setup cleanup + +## API Surface - `GET /health` - `GET /auth/me` @@ -42,8 +28,14 @@ Routes registered in `app/main.py`: - `GET /documents` - `GET /documents/{document_id}` - `GET /documents/{document_id}/pages/{page_number}` +- `GET /documents/ingestion-queues` +- `GET /documents/ingestion-health` +- `POST /documents/ingestion-reconcile` +- `GET /documents/ingestion-runs/{run_id}` - `POST /documents/upload-prepare` - `POST /documents/upload-complete` +- `POST /documents/upload-prepare-batch` +- `POST /documents/upload-complete-batch` - `POST /documents/{document_id}/retry` - `POST /documents/{document_id}/reindex` - `DELETE /documents/{document_id}` @@ -59,184 +51,46 @@ Routes registered in `app/main.py`: - `GET /usage/today` - `GET /usage/observability` -## Request Flow - -### Auth and Workspace Resolution - -1. Client sends `Authorization: Bearer `. -2. `app/api/deps.py` validates bearer presence. -3. `app/core/auth.py` validates the JWT with Supabase. -4. `get_workspace_id()` resolves the current user's workspace. -5. Workspace-scoped routes operate only within that resolved `workspace_id`. - -Primary files: -- `app/api/deps.py` -- `app/core/auth.py` -- `app/api/workspaces.py` - -### Document Upload Lifecycle - -```text -upload-prepare -> upload-complete -> enqueue extract -> extract pages - -> enqueue index -> create chunks/embeddings -> ready -``` - -What the API does: -- validates file size and content type -- enforces document count limits -- creates placeholder document records -- issues signed upload URLs through Supabase Storage -- verifies uploaded object existence -- enqueues RQ jobs for extraction and indexing -- supports retry, reindex, and delete flows - -Primary files: -- `app/api/documents.py` -- `app/core/storage.py` - -### Query Lifecycle - -```text -question -> embed question -> retrieve top-k chunks -> reserve tokens - -> call LLM -> commit actual usage -> return answer + citations -``` - -What happens in code: -- question embedding comes from `app/core/embeddings.py` -- vector retrieval comes from `app/core/retrieval.py` -- grounded prompting and LLM calls come from `app/core/llm.py` and `app/core/prompts.py` -- token reservation and usage accounting come from `app/core/token_budget.py` -- query metadata is logged into `query_logs` when enabled - -Primary files: -- `app/api/query.py` -- `app/api/query_stream.py` -- `app/core/retrieval.py` -- `app/core/llm.py` -- `app/core/token_budget.py` - -## Core Subsystems - -### Token Budget - -The server enforces a daily workspace token limit using `workspace_daily_usage`. - -Operations: -- `reserve_tokens()` -- `release_tokens()` -- `commit_usage()` -- `get_budget_status()` - -Behavior: -- reservations are acquired before LLM work -- actual usage is committed after the model returns -- unused reserved tokens are released -- stale reservations are cleaned separately by the worker maintenance job - -Primary file: -- `app/core/token_budget.py` +## Query Model Today -### Retrieval +The server accepts both legacy `document_id` and new `document_ids` payloads. Query validation and retrieval can span up to `10` selected documents and up to `100` total pages. -The retrieval layer reads vectors from `chunk_embeddings` and chunk/page text from Postgres. +Relevant files: +- [app/schemas/query.py](/D:/Desktop/projects/MultiDoc-RAG/server/app/schemas/query.py:1) +- [app/api/query.py](/D:/Desktop/projects/MultiDoc-RAG/server/app/api/query.py:1) +- [app/api/query_stream.py](/D:/Desktop/projects/MultiDoc-RAG/server/app/api/query_stream.py:1) +- [app/core/retrieval.py](/D:/Desktop/projects/MultiDoc-RAG/server/app/core/retrieval.py:1) -Current behavior: -- query embedding is turned into a `pgvector` literal -- cosine distance query returns top-k chunks for one document -- each result includes chunk text, page text, score, and token count +## Ingestion Model Today -Primary file: -- `app/core/retrieval.py` +- file validation begins in the API +- upload completion verifies the storage object and enqueues extract jobs +- extraction revalidates file shape and page/content limits +- indexing batches embeddings and writes chunks/vectors idempotently +- reconciliation can fail stale active documents and refresh ingestion runs -### Rate Limiting +Relevant files: +- [app/api/documents.py](/D:/Desktop/projects/MultiDoc-RAG/server/app/api/documents.py:1) +- [app/core/ingestion_policy.py](/D:/Desktop/projects/MultiDoc-RAG/server/app/core/ingestion_policy.py:1) +- [app/core/ingestion_reconciliation.py](/D:/Desktop/projects/MultiDoc-RAG/server/app/core/ingestion_reconciliation.py:1) +- [app/core/ingestion_runs.py](/D:/Desktop/projects/MultiDoc-RAG/server/app/core/ingestion_runs.py:1) -Redis-backed per-workspace rate limiting is enforced in the API. +## Environment Highlights -Current limits: -- queries: `100` requests / `60s` -- upload prepare: `10` requests / `60s` -- upload complete and retry/reindex mutations: `20` requests / `60s` - -Primary files: -- `app/core/rate_limit.py` -- `app/utils/rate_limit.py` - -## Data Model - -The server depends on these core tables: - -- `workspaces` -- `documents` -- `document_pages` -- `chunks` -- `chunk_embeddings` -- `workspace_daily_usage` -- `query_logs` -- `chat_sessions` - -Current SQLAlchemy models are defined in `app/db/models.py`. Full schema scripts live under `../scripts/`. - -## Environment - -Minimal server environment: - -```bash -SUPABASE_URL=https://your-project.supabase.co -SUPABASE_SERVICE_ROLE_KEY=your-service-role-key -SUPABASE_KEY=your-service-role-key -SUPABASE_STORAGE_BUCKET=documents - -DATABASE_URL=postgresql://postgres:postgres@localhost:5432/enterprise_rag -REDIS_URL=redis://localhost:6379/0 -OPENAI_API_KEY=sk-... - -ENVIRONMENT=development -API_HOST=0.0.0.0 -API_PORT=8000 -DAILY_TOKEN_LIMIT=100000 -RESERVATION_TTL_SECONDS=600 -LLM_MODEL=gpt-4o-mini -EMBEDDING_MODEL=text-embedding-3-small -``` +Important settings in [app/config.py](/D:/Desktop/projects/MultiDoc-RAG/server/app/config.py:1): +- `MAX_FILE_SIZE_BYTES=10485760` +- `MAX_PDF_PAGE_COUNT=10` +- `MAX_BULK_UPLOAD_FILES=50` +- `MAX_QUERY_DOCUMENTS=10` +- `MAX_QUERY_TOTAL_PAGES=100` +- `EMBEDDING_BATCH_SIZE=32` +- `INGEST_EXTRACT_JOB_TIMEOUT_SECONDS=900` +- `INGEST_INDEX_JOB_TIMEOUT_SECONDS=1800` ## Run -### Local - ```bash cd server -python -m venv .venv -source .venv/bin/activate pip install -r requirements.txt uvicorn app.main:app --reload --host 0.0.0.0 --port 8000 ``` - -### Docker Compose - -```bash -docker-compose up server -``` - -Health check: - -```bash -curl http://localhost:8000/health -``` - -## Development Commands - -```bash -cd server -pytest -ruff check . -black --check . -mypy app -``` - -## Notes for Contributors - -- keep every data access path workspace-scoped -- do not bypass the token budget layer for query-time model usage -- document status transitions matter because workers and UI depend on them -- `app/core/chunking.py` is still a placeholder; worker-side chunking is currently the active path -- if you add endpoints, register them in `app/main.py` and add corresponding schemas diff --git a/server/requirements.txt b/server/requirements.txt index 05feaf4..f1ccf56 100644 --- a/server/requirements.txt +++ b/server/requirements.txt @@ -31,7 +31,7 @@ pillow==12.2.0 # Auth python-jose[cryptography]==3.4.0 -python-multipart==0.0.27 +python-multipart==0.0.30 # Text Processing tiktoken==0.5.2 diff --git a/worker/README.md b/worker/README.md index 5aa2718..7c45ae4 100644 --- a/worker/README.md +++ b/worker/README.md @@ -1,179 +1,56 @@ -# Worker +# MultiDoc-RAG Worker -`worker/` runs asynchronous background jobs for Enterprise RAG. It is responsible for the long-running work that should not happen in the request path: PDF extraction, chunk indexing, embedding generation, and maintenance cleanup. +`worker/` runs the asynchronous ingestion pipeline for MultiDoc-RAG. It handles PDF extraction, chunk indexing, embedding generation, and maintenance-style recovery tasks that do not belong on the request path. -## Responsibilities +## What It Does Today -- consume RQ jobs from Redis -- extract page text from uploaded PDFs -- persist `document_pages` -- split page text into chunks -- generate embeddings for chunks -- mark documents as `ready` or `failed` -- clean stale token reservations +- consumes `ingest_extract` and `ingest_index` RQ jobs +- downloads uploaded PDFs from Supabase Storage +- validates size, page-count, and extractable-text requirements during extraction +- writes `document_pages` +- chunks page text and batches embeddings +- writes `chunks` and `chunk_embeddings` +- marks documents ready or failed +- releases stale token reservations and supports ingestion recovery helpers -## Module Layout +## Recent Worker Updates -```text -worker/ -├── jobs/ -│ ├── ingest_extract.py # PDF download and page extraction -│ ├── ingest_index.py # chunking and embedding generation -│ └── maintenance.py # stale reservation cleanup -├── shared/ # shared package placeholder -├── worker.py # RQ worker entrypoint -├── requirements.txt -└── tests/ -``` - -## Queue Model +- `08f452d` on 2026-06-08: stronger extraction/index status handling, idempotent run metadata, and explicit failure callbacks +- `9fd2a85` on 2026-06-08: refreshed ingestion-run state during worker progress, better failure-path handling, and tests for ingestion run recovery -The worker process reads one or more queue names from `QUEUE_NAME` or the first CLI argument. +## Queues -Current queue names: - `ingest_extract` - `ingest_index` -Entrypoint behavior in `worker.py`: -- connect to Redis -- create `Queue` objects for the configured names -- start an RQ `Worker` - -Example: - -```bash -cd worker -QUEUE_NAME=ingest_extract REDIS_URL=redis://localhost:6379/0 python worker.py -``` - -## Job Flows - -### `jobs/ingest_extract.py` - -Purpose: -- download the uploaded PDF from Supabase Storage -- read it with `pypdf` -- extract per-page text -- replace any previous `document_pages` rows for idempotency -- update the document status to `indexing` -- enqueue `ingest_index` - -Current behavior: -- extraction is page-based -- temporary PDF file is written under `/tmp` -- PDFs larger than `MAX_FILE_SIZE_BYTES` or over `MAX_PDF_PAGE_COUNT` are marked `failed` -- scanned/image-only PDFs with no extractable text are marked `failed` -- failures mark the document as `failed` -- if the schema does not include `extracting`, the code falls back to `indexing` -- batch ingestion jobs carry optional `ingestion_run_id` metadata for grouped tracing +The worker entrypoint in [worker.py](/D:/Desktop/projects/MultiDoc-RAG/worker/worker.py:1) can listen to one or both queues through `QUEUE_NAME`. -### `jobs/ingest_index.py` +## Job Modules -Purpose: -- load extracted page text -- split each page into chunks -- insert chunk rows -- reserve token budget per embedding batch -- call OpenAI embeddings in batches -- insert vectors into `chunk_embeddings` in batches -- commit token usage -- mark the document as `ready` or `indexed` +- [jobs/ingest_extract.py](/D:/Desktop/projects/MultiDoc-RAG/worker/jobs/ingest_extract.py:1) + - downloads PDFs + - enforces `10 MB` and `10 page` limits during extraction + - rejects scanned/image-only PDFs with no usable text + - writes page rows and enqueues indexing -Current chunking behavior: -- page-bounded chunks -- target chunk size: `500` tokens -- overlap: `100` tokens -- embedding batch size is controlled by `EMBEDDING_BATCH_SIZE` -- `tiktoken` when available, character approximation fallback otherwise +- [jobs/ingest_index.py](/D:/Desktop/projects/MultiDoc-RAG/worker/jobs/ingest_index.py:1) + - reads extracted pages + - creates page-bounded chunks + - batches embeddings with `EMBEDDING_BATCH_SIZE` + - persists vectors and token usage + - rebuilds chunk/vector rows safely for retry and reindex flows -Failure behavior: -- budget exhaustion marks the document `failed` -- any outstanding reservations are released on failure -- chunk rows are rebuilt idempotently for the document during reindex +- [jobs/maintenance.py](/D:/Desktop/projects/MultiDoc-RAG/worker/jobs/maintenance.py:1) + - clears stale token reservations -### `jobs/maintenance.py` - -Purpose: -- clear stale rows in `workspace_daily_usage` where tokens remain reserved beyond TTL - -Current behavior: -- supports PostgreSQL and SQLite variants -- reads `DATABASE_URL` and `RESERVATION_TTL_SECONDS` -- returns affected row count - -Example: - -```bash -cd worker -python -c "from jobs.maintenance import cleanup_stale_reservations; print(cleanup_stale_reservations())" -``` - -## Runtime Flow - -```text -API upload-complete - -> enqueue ingest_extract - -> extract pages from PDF - -> enqueue ingest_index - -> chunk pages - -> embed chunks - -> mark document ready -``` - -## Environment - -Minimal worker environment: - -```bash -DATABASE_URL=postgresql://postgres:postgres@localhost:5432/enterprise_rag -REDIS_URL=redis://localhost:6379/0 -SUPABASE_URL=https://your-project.supabase.co -SUPABASE_SERVICE_ROLE_KEY=your-service-role-key -SUPABASE_KEY=your-service-role-key -OPENAI_API_KEY=sk-... -QUEUE_NAME=ingest_extract -RESERVATION_TTL_SECONDS=600 -EMBEDDING_MODEL=text-embedding-3-small -MAX_FILE_SIZE_BYTES=10485760 -MAX_PDF_PAGE_COUNT=10 -MIN_EXTRACTED_TEXT_CHARS=1 -EMBEDDING_BATCH_SIZE=32 -``` +- [jobs/ingest_callbacks.py](/D:/Desktop/projects/MultiDoc-RAG/worker/jobs/ingest_callbacks.py:1) + - marks active ingestion jobs failed when RQ kills or times out a job ## Run -### Local - ```bash cd worker -python -m venv .venv -source .venv/bin/activate pip install -r requirements.txt QUEUE_NAME=ingest_extract python worker.py -``` - -Run the index queue: - -```bash QUEUE_NAME=ingest_index python worker.py ``` - -Run both queues in one process: - -```bash -QUEUE_NAME=ingest_extract,ingest_index python worker.py -``` - -### Docker Compose - -```bash -docker-compose up worker-extract worker-index -``` - -## Development Notes - -- worker jobs are stateful with respect to document status transitions; keep them explicit -- release reserved tokens on every failure path -- chunking currently lives in `jobs/ingest_index.py`, not in shared core yet -- `worker/shared/` exists for future code consolidation, but the active shared path in local compose is the mounted server app code -- if you add new jobs, ensure the API enqueues them by import path and that the queue name is wired in Compose diff --git a/worker/requirements.txt b/worker/requirements.txt index 4116499..62cfc5e 100644 --- a/worker/requirements.txt +++ b/worker/requirements.txt @@ -30,7 +30,7 @@ pillow==12.2.0 # Auth python-jose[cryptography]==3.4.0 -python-multipart==0.0.27 +python-multipart==0.0.30 # Text Processing tiktoken==0.5.2