Mini SIEM — Terminal-Based Security Information and Event Management System

A production-quality, terminal-first Security Information and Event Management (SIEM) application written in Python 3.12+. Built with SQLAlchemy 2.x, PostgreSQL (psycopg 3) / SQLite, PyYAML, Rich, and Pytest.

---

🌟 Overview

Mini SIEM collects security logs from raw files or live streams, parses them into normalized data models (SecurityEvent), detects threats using configurable sliding-window rules (DetectionEngine), manages alert lifecycles (AlertManager), stores events/alerts in SQL databases, and provides a styled interactive command-line interface with Rich terminal visualizer, incident investigation tools, event search, machine-readable JSON/CSV exports, and live log monitoring.

---

🏗️ Architecture

Raw Log Source (File / Stream / Live Tail Monitor)
                      ↓
     [ FileCollector / StreamCollector ]  (siem/collector)
                      ↓
         Raw Log Line / Event String
                      ↓
     [ AuthParser → WebParser → GenericParser ]  (siem/parser)
                      ↓
       SecurityEvent Model  (siem/models/event.py)
                      ↓
     [ SQL Storage ]  ↔  [ Event Bus ]  (siem/storage & siem/core)
                      ↓
     [ DetectionEngine & Rules ]  (siem/detection)
     ├── Standard Auth Rules (Failed Login, Brute Force, Port Scan, Suspicious Login)
     ├── Web Attack Rules (Path Traversal, Sensitive Files, Admin Probing, Error Bursts)
     ├── Auth Anomaly Rules (Credential Stuffing, Distributed Password Spray)
     ├── IOC Threat Engine (IP, Domain, URL matching against config/iocs.yaml)
     └── Event Correlation Layer (Recon + Web Attack, Multi-stage attacks)
                      ↓
       SecurityAlert Model  (siem/models/alert.py)
       (Enriched with Evidence, Recommendations, & Timestamps)
                      ↓
     [ AlertManager ]  (siem/alerts)
                      ↓
     [ Analytics & Rich CLI Operations Center ]  (siem/cli)
     ├── analyze / scan
     ├── investigate <alert_id>
     ├── search --ip / --user / --type / --severity
     ├── monitor <file_path> (continuous tailing)
     └── Formatted Exporters (--json, --csv)

---

🎯 Key Features

• Multi-Source Log Collectors: Stream and File log collectors with live tail monitor (monitor command), EOF handling, clean shutdown, and stats tracking.
• Extensible Parser Chain: Regular expressions and log structure detection (AuthParser, WebParser, fallback GenericParser).
• Normalized Event Model: Strongly-typed SecurityEvent model with timestamp normalization, IP/port/user validation, and severity categorization.
• 12+ Configurable Threat Detection Rules (config/rules.yaml):
  • Failed Login Rule: Detects N failed login attempts from an IP within time window.
  • Brute Force Rule: Detects rapid repeated failed logins per IP/user (CRITICAL severity).
  • Port Scan Rule: Detects connections to N unique destination ports from a single IP (HIGH severity).
  • Suspicious Login Rule: Detects successful logins immediately following multiple failed attempts (HIGH severity).
  • Path Traversal Rule: Detects directory traversal (/../../etc/passwd, %2e%2e/).
  • Sensitive File Rule: Detects probes for environment files (/.env, /.git/config, /config.php).
  • Admin Probing Rule: Detects unauthorized scans for admin interfaces (/wp-admin, /phpmyadmin).
  • HTTP Error Burst Rule: Detects N 4xx/5xx errors from a single source IP in time window.
  • IOC Detection Rule: Matches traffic against threat intelligence IPs, domains, and URLs (config/iocs.yaml).
  • Credential Stuffing Rule: Detects multiple accounts targeted by a single source IP.
  • Distributed Password Attack Rule: Detects single target user attacked across multiple source IPs.
  • Multi-Event Correlation Rule: Correlates network reconnaissance combined with web requests.
• Incident Investigation Tool (investigate <alert_id>): Displays full analyst incident card with WHAT happened, WHY it was detected, evidence, recommendations, and related host events.
• Dynamic Event Search (search --ip / --user / --type / --severity): Fast SQL search with composite database indexing.
• Machine-Readable Exports: --json and --csv export flags across analysis, alerts, stats, search, and rules.
• Dual Database Architecture: Fully compatible with PostgreSQL (postgresql+psycopg://) and SQLite (sqlite:///...).
• Rich Terminal UI: Cybersecurity Operational Center visualizer with processing metrics, alert severity distribution, threat highlights, and detection rules tables.

---

📁 Project Structure

Mini SIEM/
├── config/
│   ├── __init__.py
│   ├── settings.py
│   ├── rules.yaml
│   └── iocs.yaml
├── logs/
│   ├── sample_auth.log
│   ├── sample_web.log
│   └── sample_system.log
├── data/
│   └── mini_siem.db
├── scripts/
│   ├── generate_logs.py
│   └── seed_database.py
├── siem/
│   ├── __init__.py
│   ├── cli/
│   │   ├── __init__.py
│   │   ├── commands.py
│   │   └── display.py
│   ├── collector/
│   │   ├── __init__.py
│   │   ├── base.py
│   │   ├── file_collector.py
│   │   └── stream_collector.py
│   ├── parser/
│   │   ├── __init__.py
│   │   ├── base.py
│   │   ├── auth_parser.py
│   │   ├── web_parser.py
│   │   └── generic_parser.py
│   ├── models/
│   │   ├── __init__.py
│   │   ├── event.py
│   │   └── alert.py
│   ├── detection/
│   │   ├── __init__.py
│   │   ├── engine.py
│   │   ├── base_rule.py
│   │   └── rules/
│   │       ├── __init__.py
│   │       ├── brute_force.py
│   │       ├── port_scan.py
│   │       ├── failed_login.py
│   │       ├── suspicious_login.py
│   │       ├── web_rules.py
│   │       ├── ioc_rule.py
│   │       ├── auth_anomalies.py
│   │       └── correlation_rule.py
│   ├── storage/
│   │   ├── __init__.py
│   │   ├── database.py
│   │   ├── repositories.py
│   │   └── migrations.py
│   ├── alerts/
│   │   ├── __init__.py
│   │   └── manager.py
│   ├── analytics/
│   │   ├── __init__.py
│   │   ├── statistics.py
│   │   └── reports.py
│   └── core/
│       ├── __init__.py
│       ├── pipeline.py
│       ├── event_bus.py
│       └── logger.py
├── tests/
│   ├── __init__.py
│   ├── test_collectors.py
│   ├── test_parsers.py
│   ├── test_detection.py
│   ├── test_storage.py
│   ├── test_pipeline.py
│   ├── test_correctness_fixes.py
│   └── test_portfolio_features.py
├── .env.example
├── .gitignore
├── LICENSE
├── README.md
├── requirements.txt
└── pyproject.toml

---

⚙️ Installation

Prerequisites

• Python 3.12+

1. Clone & Install Dependencies

pip install -e ".[dev]"

2. Configure Environment Variables

Copy .env.example to .env:

cp .env.example .env

Default configuration uses SQLite: DATABASE_URL=sqlite:///data/mini_siem.db

To connect to PostgreSQL: DATABASE_URL=postgresql+psycopg://postgres:password@localhost:5432/mini_siem

---

🚀 Running the Terminal CLI

1. Generate Synthetic Sample Logs

Generate 1,000 realistic sample events containing brute force, IOC matches, credential stuffing, path traversal, sensitive file probes, and port scan signatures:

python scripts/generate_logs.py --count 1000

or via CLI:

python -m siem.cli.commands generate-sample-logs --count 1000

2. Analyze Log File

Analyze authentication log file:

python -m siem.cli.commands analyze logs/sample_auth.log

Output machine-readable JSON:

python -m siem.cli.commands analyze logs/sample_auth.log --json

3. Scan Log Directory

python -m siem.cli.commands scan logs/

4. Investigate Security Incidents

Investigate alert details by alert ID or short prefix:

python -m siem.cli.commands investigate 716c57ca

5. Search Security Events

Search database events by IP, username, type, or severity:

python -m siem.cli.commands search --ip 192.168.1.42
python -m siem.cli.commands search --user admin --type FAILED_LOGIN
python -m siem.cli.commands search --severity HIGH --csv search_results.csv

6. Live Log Monitoring

Continuously tail and analyze incoming log lines in real-time:

python -m siem.cli.commands monitor logs/sample_auth.log

7. Export Alerts & Database Stats

List historical security alerts:

python -m siem.cli.commands alerts
python -m siem.cli.commands alerts --json
python -m siem.cli.commands alerts --csv data/alerts.csv

Display stored database statistics:

python -m siem.cli.commands stats
python -m siem.cli.commands stats --json

View active detection rules and static configuration:

python -m siem.cli.commands rules
python -m siem.cli.commands rules --json

---

🧪 Running Automated Tests

Run complete test suite (51 passed tests):

pytest -v

Run test suite with code coverage (82% overall coverage):

pytest --cov=siem --cov-report=term-missing

---

🛡️ Security & Performance Considerations

• Sliding Time-Window Pruning: All stateful detection rules automatically prune expired events to prevent unbounded memory growth during continuous log monitoring.
• Safe Configuration Loading: Uses yaml.safe_load() exclusively to prevent arbitrary code execution vulnerabilities.
• SQL Injection Prevention: Built on SQLAlchemy 2.x ORM parametrized queries (select(), where()).
• Input Sanitization: Terminal output uses Rich escaping and handles UTF-8 replace fallbacks cleanly on Windows and POSIX systems.

---

🔮 Future Roadmap

• Native Windows Event Log (.evtx) binary ingestion parser plugin.
• Remote Syslog (UDP/TCP 514) listener collector daemon.
• External threat intelligence API connector (e.g., VirusTotal, AbuseIPDB).
• Slack/Webhook alert notification dispatcher plugin.