contextray

LLM context understanding - illuminate the real contributors to your LLM's response in production systems.

Contextray is a diagnostic platform designed to help developers understand and optimize LLM interactions. It provides deep visibility into how specific parts of a prompt influence the model's output and identifies "bloat" in complex LLM execution traces.

🚀 Key Features

• Context Analysis (SHAP): Automatically calculate the contribution of each token or segment in your prompt to the final answer using SHAP (Shapley Additive Explanations).
• Structured JSON Analysis: Analyze JSON-formatted prompts with specialized masking modes—either treating the entire structure as text or selectively masking only the leaf values to preserve JSON validity during inference.
• Execution Bloat Analysis: Visualize execution traces (e.g., from Langfuse) to identify which nodes are contributing most to token usage and where the most significant "bloat" occurs in your agentic flows.
• Organization & Project Management: Manage multiple teams and projects with a structured hierarchy.
• Admin Dashboard: Comprehensive monitoring of system health, Celery worker status, and job success rates.
• SSO Integration: Secure access via Keycloak / OIDC.

🏗 Architecture

graph TD
    subgraph Frontend
        Vite[Vite + React TS]
        ReactFlow[React Flow]
    end

    subgraph Backend
        FastAPI[FastAPI / Python]
        CeleryWorker[Celery Worker]
        SHAP[SHAP Engine]
        Bloat[Bloat Analyzer]
    end

    subgraph Infrastructure
        Postgres[(PostgreSQL)]
        RabbitMQ[RabbitMQ]
        Keycloak[Keycloak SSO]
    end

    subgraph AI_Services
        vLLM[vLLM Inference]
        Infinity[Embeddings]
    end

    Vite <-->|REST API| FastAPI
    FastAPI <--> Postgres
    FastAPI -->|Task Queue| RabbitMQ
    RabbitMQ <--> CeleryWorker
    CeleryWorker <--> Postgres
    CeleryWorker --> SHAP
    CeleryWorker --> Bloat
    SHAP <-->|Inference| vLLM
    SHAP <--> |Embed| Infinity
    Vite <-->|Auth| Keycloak
    FastAPI <-->|Auth Validation| Keycloak

Loading

Component Breakdown

• Frontend: A modern React application built with Vite and Tailwind CSS. It uses React Flow for visualizing complex execution graphs.
• Backend API: A FastAPI-powered service that manages project metadata, user organizations, and orchestrates analysis jobs.
• Celery Worker: Handles long-running asynchronous tasks, including SHAP calculation and trace bloat analysis.
• vLLM & Infinity: AI infrastructure for local model inference and embedding generation.
• Keycloak: Centralized Identity and Access Management (IAM) for secure authentication.

🛠 Getting Started

Prerequisites

• Docker and Docker Compose
• Python 3.12+ (managed by uv recommended)
• Node.js & pnpm

1. Run the Infrastructure

Start the DB, RabbitMQ, Keycloak, and AI engines.

Note

By default, the docker-compose.yml expects an NVIDIA GPU with the NVIDIA Container Toolkit installed.

For CPU-only environments: Remove the deploy: resources: section from the vllm and infinity services in docker-compose.yml before running.

docker compose up -d

2. Backend Setup

cd backend
uv sync
uv run uvicorn app.main:app --reload --port 5000

To run the background worker:

uv run celery -A app.worker.celery_app worker --loglevel=info --pool=solo

3. Frontend Setup

cd frontend-vite
pnpm i
pnpm run dev

🔐 Authentication & Administration

Default system user: testuser / password.

The application uses OIDC with Keycloak.

Admin Access

To access the Admin Panel, your user account must have the is_admin flag set to true in the database. You can manually promote a user via SQL:

UPDATE users SET is_admin = true WHERE username = 'testuser';

🛠 Helper Scripts

The scripts/ directory contains utility scripts for maintenance:

• gen-certs.sh: Generates the necessary SSL certificates for secure RabbitMQ communication.
• download_images.sh: Downloads and saves Docker images, so you can download it to a flash drive and keep in your pocket for a sudden contextray itch. It's a real emergency!