API Security Monitoring System

A comprehensive real-time API security monitoring system powered by machine learning. Uses CNN and BiLSTM neural networks for threat detection and anomaly analysis.

Overview

This system provides:

• Real-time Threat Detection: Analyzes API requests using CNN models for immediate threat classification
• Temporal Pattern Detection: Uses BiLSTM models to identify attack patterns across multiple requests
• Security Dashboard: Modern web interface for monitoring and alert management
• Detailed Analytics: Time-series analysis and threat trend visualization
• Security Logs: Comprehensive logging of all analyzed requests with filtering and export
• Alert Management: Active threat alerts with recommendations and acknowledgement tracking

Architecture

Frontend

• Next.js 15 with React for the user interface
• Recharts for data visualization
• shadcn/ui components for consistent design
• Real-time updates via API polling

Backend

• FastAPI for high-performance API endpoints
• CNN Model: Analyzes individual request features for threat probability
• BiLSTM Model: Identifies temporal attack patterns in request sequences
• Feature Extraction: 52 CICIDS2017 network features from API traffic
• PostgreSQL: Persistent storage for logs and alerts (optional)

Machine Learning

• 52 CICIDS2017 Features: Flow-based, timing, protocol, and flag-based features
• CNN Model: Request-level threat classification (0-1 probability)
• BiLSTM Model: Temporal anomaly detection across request sequences
• Combined Scoring: 60% CNN + 40% BiLSTM for final threat assessment

Quick Start

Prerequisites

• Node.js 18+
• Python 3.8+
• (Optional) PostgreSQL for persistent storage

1. Installation

# Install Node.js dependencies
npm install

# Create Python virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install Python dependencies
pip install -r scripts/requirements.txt

# Initialize project (creates demo models)
python3 scripts/init.py

2. Configuration

Create a .env.local file in the project root:

# Backend URL (for development)
FASTAPI_URL=http://localhost:8000

# Optional: PostgreSQL connection for production
# DATABASE_URL=postgresql://user:password@localhost:5432/api_security_db

3. Start the Application

Terminal 1 - Start Backend Server:

python3 scripts/run_backend.py

The backend will start on http://localhost:8000

• API documentation: http://localhost:8000/docs
• Health check: http://localhost:8000/api/health

Terminal 2 - Start Frontend:

npm run dev

The dashboard will be available at http://localhost:3000

Project Structure

├── app/
│   ├── page.tsx                 # Main dashboard
│   ├── analytics/page.tsx        # Analytics dashboard
│   ├── logs/page.tsx             # Security logs viewer
│   ├── alerts/page.tsx           # Active alerts management
│   └── api/proxy/[...route]/     # API proxy to FastAPI
├── components/
│   ├── navigation.tsx            # Navigation bar
│   └── dashboard/                # Dashboard components
│       ├── statistics-card.tsx
│       ├── threat-chart.tsx
│       └── alerts-list.tsx
├── scripts/
│   ├── api_backend.py            # FastAPI application
│   ├── models.py                 # Pydantic models
│   ├── feature_extractor.py      # Feature extraction
│   ├── create_models.py          # Model generation
│   ├── run_backend.py            # Backend server launcher
│   ├── init.py                   # Project initialization
│   ├── requirements.txt          # Python dependencies
│   └── 01_create_schema.sql      # Database schema
├── public/models/                # Pre-trained models
│   ├── cnn_api_security_model.pkl
│   ├── bilstm_api_security_model.pkl
│   ├── scaler.pkl
│   └── model_metadata.json
└── README.md

Features

1. Real-time Dashboard

• Key Metrics: Total requests, threats detected, average threat score
• Threat Distribution: Visual breakdown by threat level
• Model Comparison: CNN vs BiLSTM score visualization
• Recent Alerts: Latest security incidents with details

2. Analytics

• Threat Trends: 24-hour time-series of threat scores
• Detection Rate: Threats detected per hour
• Model Performance: Detailed metrics for both models
• Historical Data: Hourly statistics for trend analysis

3. Security Logs

• Comprehensive Logging: All analyzed requests recorded
• Filtering: By threat level, source IP, and API path
• Export: Download logs as CSV for external analysis
• Pagination: Efficient browsing of large log sets

4. Alert Management

• Active Alerts: Only unacknowledged threats displayed
• Threat Details: Full request and analysis information
• Recommendations: Suggested actions for each threat
• Acknowledgement: Mark threats as reviewed

API Endpoints

Analysis

• POST /api/analyze - Analyze a single request
• POST /api/analyze-batch - Batch analyze multiple requests

Data Retrieval

• GET /api/logs - Fetch security logs with filtering
• GET /api/stats - Real-time statistics and trends
• GET /api/alerts - Get unacknowledged alerts

Alert Management

• POST /api/alerts/{id}/acknowledge - Mark alert as acknowledged
• GET /api/health - Health check endpoint

Threat Levels

Threats are classified by combined score:

• CRITICAL (0.8-1.0): Immediate action required, block the request
• HIGH (0.6-0.8): Investigate and monitor closely
• MEDIUM (0.4-0.6): Monitor for pattern confirmation
• LOW (0-0.4): Normal traffic, allow

Feature Extraction

The system extracts 52 CICIDS2017 features from API requests:

Feature Categories

1. Packet Length (12 features): Forward/backward packet statistics
2. Header Length (12 features): Forward/backward header information
3. Packet Count (6 features): Flow statistics and rates
4. TCP Flags (8 features): FIN, SYN, RST, PSH counts
5. Timing (9 features): Flow duration and inter-arrival times
6. Response (3 features): Status code, response time, content length

Model Performance

The demo models simulate threat detection with:

• CNN Model: Analyzes individual request characteristics
• BiLSTM Model: Detects temporal patterns across request sequences
• Combined Scoring: Weighted ensemble for robust detection

Note: Demo models are for demonstration. For production, train models with real network traffic data.

Development

Adding Custom Models

1. Train your own CNN/BiLSTM models
2. Save with pickle: pickle.dump(model, open('path/to/model.pkl', 'wb'))
3. Place in public/models/ directory
4. Restart the backend server

Extending Features

To add new features to the extractor:

1. Edit scripts/feature_extractor.py
2. Update feature count (currently 52)
3. Update models.py and api_backend.py
4. Retrain models with new feature set

Database Integration

To use PostgreSQL instead of in-memory storage:

1. Set DATABASE_URL in .env.local
2. Run migrations: psql -d your_db -f scripts/01_create_schema.sql
3. Update connection strings in api_backend.py

Troubleshooting

Backend won't start

# Check if port 8000 is in use
lsof -i :8000

# Install missing dependencies
pip install -r scripts/requirements.txt

# Check Python version
python3 --version  # Should be 3.8+

Frontend can't connect to backend

• Verify backend is running: curl http://localhost:8000/api/health
• Check FASTAPI_URL in .env.local
• Ensure CORS is properly configured

Models not loading

# Regenerate models
python3 scripts/create_models.py

Performance Tips

• Batch Analysis: Use /api/analyze-batch for multiple requests
• Filtering: Apply filters on logs to reduce data transfer
• Pagination: Load logs in pages (default 20 per page)
• Caching: Frontend caches stats and updates every 10 seconds

Security Considerations

• All traffic between frontend and backend should use HTTPS in production
• Implement authentication for the dashboard
• Store sensitive data (API credentials, models) securely
• Regularly update dependencies
• Monitor logs for suspicious activity in the monitoring system itself

Future Enhancements

• WebSocket support for real-time alerts
• Custom model training pipeline
• Integration with SIEM systems
• Multi-tenancy support
• Advanced anomaly detection algorithms
• Custom alert rules and workflows

License

MIT License - See LICENSE file for details

Support

For issues and questions:

1. Check the troubleshooting section above
2. Review the API documentation at http://localhost:8000/docs
3. Check FastAPI logs for detailed error messages
4. Review browser console for frontend errors

Contributing

Contributions are welcome! Please ensure:

• Code follows the existing style
• Features include appropriate tests
• Documentation is updated
• Commit messages are descriptive