Data Quality & Drift Detection Platform

A production-grade automated data quality monitoring system that detects schema changes, distribution drift, anomalies, and data quality issues in real-time.

🎯 Problem Statement

Data quality issues break ML pipelines and corrupt analytics. This platform catches data problems before they cause damage—detecting schema changes, distribution shifts, null spikes, and anomalies automatically.

🏗️ Architecture

┌─────────────────┐
│  Upload Data    │
│  (CSV/Parquet)  │
└────────┬────────┘
         │
         ▼
┌──────────────────────────────────────┐
│  Backend (FastAPI)                   │
│  ├─ Schema Validator                 │
│  ├─ Drift Detector (KS/Chi-Squared)  │
│  ├─ Anomaly Detector                 │
│  ├─ Completeness Checker             │
│  └─ Alert Manager                    │
└────────┬─────────────────────────────┘
         │
         ▼
┌──────────────────────────────────────┐
│  Database (SQLite)                   │
│  ├─ Results                          │
│  ├─ Alerts                           │
│  └─ Metrics History                  │
└────────┬─────────────────────────────┘
         │
         ▼
┌──────────────────────────────────────┐
│  Frontend (React + Tailwind)         │
│  ├─ Home (upload, run check, results)│
│  ├─ Alerts                           │
│  ├─ Reports                          │
│  └─ Settings                         │
└──────────────────────────────────────┘

⚡ Quick Start

Option 1: Local Development (Recommended for Development)

Prerequisites:

• Python 3.11+
• pip

Setup:

1. Clone the repo:

   git clone https://github.com/neethika12/data-quality-platform.git
   cd data-quality-platform

2. Create virtual environment:

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies:

   pip install -r requirements.txt

4. Start Backend (Terminal 1):

   python -m uvicorn backend.main:app --reload --host 0.0.0.0 --port 8000

   API docs available at: http://localhost:8000/docs

5. Start Frontend (Terminal 2):

   cd frontend-react
   npm install
   npm run dev

   Frontend available at: http://localhost:3000

Option 2: Docker (Recommended for Production)

Prerequisites:

• Docker
• Docker Compose

Run:

docker-compose up

Then:

• Backend API: http://localhost:8000/docs
• Frontend: http://localhost:3000

📊 Features

Backend Services

Service	Purpose	Methods
Schema Validator	Detect schema changes	Type changes, new/missing columns, versioning
Drift Detector	Statistical drift detection	Kolmogorov-Smirnov (numeric), Chi-Squared (categorical)
Anomaly Detector	Find data anomalies	Null rates, outliers (IQR/Z-score), domain rules
Completeness Checker	Monitor data freshness	Record counts, null percentages, update latency
Alert Manager	Generate & manage alerts	Severity levels, deduplication, acknowledgment

API Endpoints

Method	Endpoint	Purpose
POST	/api/datasets/upload	Upload CSV/Parquet file
GET	/api/datasets	List all datasets
POST	/api/datasets/{id}/analyze	Run full quality check
GET	/api/datasets/{id}/latest-result	Get latest analysis
GET	/api/alerts	Get all alerts
POST	/api/alerts/{id}/acknowledge	Mark alert as seen
GET	/api/health	System health check

Frontend Pages

1. Home - Upload a file, run a check, and see the quality score, plain-English breakdown, and an expandable technical view (schema changes, drift scores, per-column anomalies)
2. Alerts - Alert log, filtering, acknowledgment
3. Reports - Download a text/JSON quality report
4. Settings - View thresholds and alert configuration

🚀 Example Usage

1. Upload Dataset

• Go to Home → Upload a CSV/Excel/Parquet file
• System establishes a baseline (schema, distributions, a raw data sample for drift comparison)

2. Run a Check

• Click "Run Check" on Home
• Backend performs checks (30 sec for 100k rows)

3. View Results

• See the quality score and plain-English summary right on Home
• Expand "Show detailed technical breakdown" for drift scores, schema changes, and per-column stats
• Review Alerts for actionable issues

📈 Sample Output

Quality Score Metrics

Overall Quality Score: 0.92/1.0

Completeness:  95%
Drift Score:   0.23 (INFO)
Anomalies:     12 detected
Schema Changes: 0

Drift Detection Example

Feature Age:
- Drift Score: 0.68 (WARNING)
- P-Value: 0.001
- Test: Kolmogorov-Smirnov
- Interpretation: Distribution significantly changed

Alerts Generated

🔴 CRITICAL: 2 records (2%)
   - Negative prices in purchase_amount
   - Record count drop (10% below baseline)

🟡 WARNING: 5 features
   - High null rate in email (8%)
   - Distribution drift in age (p=0.001)

🔵 INFO: 3 records
   - New column: user_segment added
   - Data updated 2 hours ago (OK)

🧪 Testing

Run unit tests:

pytest tests/ -v

Test coverage:

• Schema validation: 100%
• Drift detection: 95%
• Anomaly detection: 90%

📁 Project Structure

data-quality-platform/
├── backend/
│   ├── main.py                 # FastAPI app
│   ├── config.py               # Settings
│   ├── models.py               # Pydantic schemas
│   ├── database.py             # SQLite ORM
│   ├── routes/                 # API endpoints
│   └── services/               # Quality check logic
├── frontend-react/
│   ├── src/pages/               # Home, Alerts, Reports, Configuration
│   └── src/components/          # Layout, Tour, Card, Alert, etc.
├── tests/                      # Unit tests
├── requirements.txt            # Dependencies
├── docker-compose.yml          # Container setup
└── README.md                   # This file

⚙️ Configuration

Edit backend/config.py to adjust:

# Drift thresholds
DRIFT_WARNING_THRESHOLD = 0.3        # 0-1 scale
DRIFT_CRITICAL_THRESHOLD = 0.7

# Null rate thresholds
NULL_RATE_WARNING = 0.05             # 5%
NULL_RATE_CRITICAL = 0.10            # 10%

# Freshness thresholds
FRESHNESS_WARNING_HOURS = 24
FRESHNESS_CRITICAL_HOURS = 48

# Record drop alert
RECORD_DROP_WARNING_PERCENT = 0.10   # 10%

🔄 Data Flow

1. Upload → CSV/Parquet file → Backend validates & stores
2. Baseline → First upload establishes baseline (schema, stats)
3. Analysis → Run quality checks on new data
4. Detection → Compare against baseline, detect changes
5. Alerts → Generate alerts by severity
6. Visualization → Home page shows results

📊 Key Metrics

Metric	Description	Normal Range
Quality Score	Overall data quality	0.80-1.00
Completeness	Non-null records	>0.95
Drift Score	Distribution change	0.00-0.30
Anomaly Score	Unusual records	0.00-0.05
Null Rate	Missing values per column	<5%

🛠️ Troubleshooting

Backend won't start

# Check if port 8000 is in use
lsof -i :8000
# Kill process
kill -9 <PID>

Frontend can't connect to backend

• Ensure backend is running on localhost:8000
• Check CORS settings in backend/main.py

Database errors

# Remove old database
rm data_quality.db
# Restart backend (will recreate DB)

📈 Performance Benchmarks

Analysis Time Complexity

• Schema Validation: O(columns) - ~2ms
• Drift Detection: O(features × n log n) - dominated by quantile calc
• Anomaly Detection: O(features × n) - linear scan
• Overall: O(n log n) - scales with data size

Measured Performance (Local Machine: M1 Mac)

Dataset Size	Analysis Time	Memory	Notes
15 rows (sample)	<100ms	5MB	Included sample data
1,000 rows	500ms	10MB	~5 features
10,000 rows	2s	30MB	~10 features
100,000 rows	15s	150MB	Statistical bottleneck
1,000,000 rows	120s	1.2GB	Quantile calculations

Breakdown (100k rows, 10 features)

File read (CSV):        2s   ██░░░░░░░░░░░░░░
Schema validation:      200ms  █░░░░░░░░░░░░░░░
Drift detection:        6s   ████░░░░░░░░░░░░
Anomaly detection:      3s   ██░░░░░░░░░░░░░░
Alert generation:       1s   █░░░░░░░░░░░░░░░
Database storage:       1s   █░░░░░░░░░░░░░░░
──────────────────────────────
Total:                  13s   

Scalability

• Horizontal: Multiple workers via task queue (future)
• Vertical: Optimized quantile calculations, vectorized NumPy operations
• Storage: SQLite <1GB for 1000+ analyses; migrate to PostgreSQL for enterprise

🚀 Next Steps

Phase 2 (Coming Soon)

• Email alerts
• Scheduled monitoring jobs
• Comparison mode (before/after)
• PDF report generation
• Multi-user support

Advanced Features

• Custom rule engine UI
• Drift trend analysis (time series)
• Impact scoring (which columns matter most)
• Automated remediation suggestions

📝 License

MIT

Questions? Check the API docs at http://localhost:8000/docs or GitHub issues.