Skip to content

monitoring.md

Latest commit

 

History

History
376 lines (290 loc) · 11.9 KB

monitoring.md

File metadata and controls

376 lines (290 loc) · 11.9 KB

Monitoring & Observability

Production Features Overview

✅ Implemented Production Features

  • Health Checks & Monitoring
  • Alerting & Notifications
  • Backup & Recovery
  • Performance Testing
  • Security Scanning
  • Resource Management
  • High Availability
  • Observability

Comprehensive Monitoring Stack

Component URL Purpose Status
Prometheus http://localhost:9090 Metrics collection & alerting ✅
Grafana http://localhost:3000 Dashboards & visualization ✅
Jaeger http://localhost:16686 Distributed tracing ✅
Alertmanager http://localhost:9093 Alert management ✅

Key Metrics Monitored

Service Health Metrics

  • Service availability: Up/down status for all components
  • Response times: API endpoint latency measurements
  • Throughput: Requests per second across services
  • Error rates: 4xx/5xx response codes and application errors
  • Health check status: Individual service health endpoints

Performance Metrics

  • CPU utilization: Per-service and system-wide CPU usage
  • Memory consumption: Heap usage, garbage collection metrics
  • Disk I/O: Read/write operations and disk space usage
  • Network I/O: Bandwidth utilization and connection counts
  • Database performance: Query times, connection pool status

Business Metrics

  • Coupon redemptions: Rate and success of coupon usage
  • User activity: Login rates, session duration, active users
  • Event processing: Message queue throughput and latency
  • Cache performance: Hit/miss ratios, cache size, eviction rates

Security Metrics

  • Authentication failures: Failed login attempts and patterns
  • Suspicious activity: Unusual request patterns or access attempts
  • Token validation: JWT validation success/failure rates
  • Security alerts: Triggered security rules and violations

Health Checks

Comprehensive Health Check System

All services include comprehensive health checks accessible via dedicated endpoints:

# Run comprehensive health check
./scripts/production-health-check.sh

Health Check Features

  • ✅ Service availability validation: Verify all services are running and responsive
  • ✅ Database connectivity checks: Test connections to DynamoDB, Redis, PostgreSQL
  • ✅ Authentication system validation: Verify Keycloak connectivity and JWT validation
  • ✅ Resource usage monitoring: Check CPU, memory, and disk usage
  • ✅ Security configuration verification: Validate security settings and configurations

Individual Service Health Endpoints

Service Health Endpoint Response Format
Write Layer /actuator/health Spring Boot Actuator JSON
Public API /health Custom JSON health status
Coupon Service /actuator/health Spring Boot Actuator JSON
Web App /health Simple status response

Health Check Automation

# Automated health monitoring
while true; do
    ./scripts/production-health-check.sh
    sleep 30
done

Alerting & Incident Response

Alert Rules Configuration

Critical Alerts (Immediate Response Required)

  • Service downtime > 5 minutes
  • Database connectivity issues
  • Authentication system failures
  • High error rates > 5%
  • Resource exhaustion (CPU > 90%, Memory > 95%)

Warning Alerts (Review Within Hours)

  • High latency > 1 second (95th percentile)
  • Memory usage > 90%
  • Queue backlog growth
  • SSL certificate expiration (< 30 days)
  • Unusual traffic patterns

Info Alerts (Review Daily)

  • Performance degradation
  • Cache miss rate increases
  • Background job failures
  • Configuration changes

Alert Channels

Webhook Integration

Email Notifications

  • Configuration: Configurable for production deployment
  • Recipients: Operations team, on-call engineers
  • Escalation: Automatic escalation for critical alerts

Escalation Policies

  • Critical → Warning → Info
  • Immediate notification for critical issues
  • Batched notifications for warnings and info

Incident Response Procedures

# Check current active alerts
curl http://localhost:9093/api/v1/alerts

# View alert history in Grafana
# Navigate to: http://localhost:3000

# Silence specific alerts
curl -X POST http://localhost:9093/api/v1/silences \
  -d '{"matchers":[{"name":"alertname","value":"HighErrorRate"}]}'

Alert Investigation Playbook

  1. Receive Alert: Notification received via configured channels
  2. Initial Assessment: Check alert details and severity
  3. Service Status: Verify affected services using health checks
  4. Log Analysis: Review recent logs for error patterns
  5. Metrics Review: Check Grafana dashboards for trends
  6. Resolution: Apply fixes or escalate to development team
  7. Post-Incident: Update documentation and improve monitoring

Performance Testing & Baselines

Performance Testing Suite

# Run comprehensive performance tests
./scripts/performance-test.sh

# Run specific test types
./scripts/performance-test.sh read      # Read performance only
./scripts/performance-test.sh write     # Write performance only
./scripts/performance-test.sh concurrent # Load testing
./scripts/performance-test.sh stress    # Stress testing

Performance Baselines

Target Performance Metrics

  • Response Time: < 500ms (95th percentile)
  • Throughput: > 1000 req/sec
  • Error Rate: < 1%
  • Availability: > 99.9%

Load Testing Scenarios

Read Operations:

  • Test Load: 1000 requests
  • Concurrency: 10 concurrent users
  • Duration: Sustained load testing
  • Endpoints: All read APIs

Write Operations:

  • Test Load: 100 requests
  • Concurrency: 5 concurrent users
  • Duration: Burst and sustained testing
  • Endpoints: All write APIs

Concurrent Load:

  • Users: 20 concurrent users
  • Duration: 30-second sustained load
  • Mix: 80% read, 20% write operations
  • Ramp-up: Gradual increase to target load

Stress Testing:

  • Load Pattern: Gradual increase to breaking point
  • Monitoring: Resource utilization during stress
  • Recovery: System recovery after stress removal
  • Limits: Identify system capacity limits

Performance Monitoring Dashboard

Key Performance Indicators (KPIs)

  • Average Response Time: Real-time API response times
  • Request Rate: Requests per second by endpoint
  • Error Rate: Percentage of failed requests
  • Active Users: Current number of authenticated users
  • Queue Depth: Message queue backlog size

Performance Trends

  • Hourly Performance: Performance patterns throughout the day
  • Daily Performance: Day-over-day performance comparison
  • Weekly Trends: Long-term performance trend analysis
  • Capacity Planning: Growth trends and capacity requirements

Backup & Recovery

Automated Backup System

The comprehensive backup script covers all critical system components:

  • ✅ PostgreSQL databases (Keycloak data)
  • ✅ Redis cache snapshots
  • ✅ RabbitMQ configurations
  • ✅ DynamoDB tables
  • ✅ Application configurations
  • ✅ Docker volumes
# Create full system backup
./scripts/backup-restore.sh backup

# Restore from backup
./scripts/backup-restore.sh restore ./backups/2024-01-01_12-00-00

Backup Features

Automated Scheduling

  • Ready for cron integration
  • Configurable backup intervals
  • Retention policies
  • Storage management

Compression & Storage

  • Efficient storage with tar.gz compression
  • Backup manifest with metadata
  • Integrity verification
  • Multiple storage backends support

Point-in-time Recovery

  • Timestamped backups for precise recovery
  • Incremental backup support
  • Cross-service consistency coordination
  • Recovery validation procedures

Recovery Testing

# Test backup integrity
tar -tzf backups/backup-2024-01-01_12-00-00.tar.gz

# Validate backup manifest
jq . backups/2024-01-01_12-00-00/manifest.json

# Test restoration process
./scripts/test-restore.sh ./backups/2024-01-01_12-00-00

Disaster Recovery Procedures

  1. Assess Impact: Determine scope of data loss or corruption
  2. Select Recovery Point: Choose appropriate backup timestamp
  3. Stop Services: Safely shutdown affected services
  4. Restore Data: Execute restoration procedures
  5. Validate Integrity: Verify restored data consistency
  6. Restart Services: Bring services back online
  7. Monitor Recovery: Watch for issues post-recovery
  8. Document Incident: Update procedures based on experience

Security Scanning & Compliance

Security Scanning Suite

# Run comprehensive security scan
./scripts/security-scan.sh

# Run specific security checks
./scripts/security-scan.sh containers    # Container security
./scripts/security-scan.sh network      # Network security
./scripts/security-scan.sh auth         # Authentication
./scripts/security-scan.sh secrets      # Secret exposure

Security Features Implemented

Authentication & Authorization

  • ✅ JWT-based authentication
  • ✅ Keycloak identity management
  • ✅ Role-based access control
  • ✅ Session management

Network Security

  • ✅ Container network isolation
  • ✅ Port exposure validation
  • ✅ HAProxy load balancing
  • ⚠️ HTTPS configuration (ready for production certificates)

Data Protection

  • ✅ Database access controls
  • ✅ Secret scanning
  • ✅ Environment variable validation
  • ✅ Log sanitization

Security Compliance Monitoring

Continuous Security Monitoring

  • Real-time threat detection
  • Security event correlation
  • Compliance reporting
  • Vulnerability scanning

Security Metrics Dashboard

  • Authentication success/failure rates
  • Access control violations
  • Security alert frequency
  • Compliance status indicators

Observability Best Practices

Distributed Tracing

Trace Correlation

  • Request IDs: Unique identifiers for request tracking
  • Service Correlation: Track requests across service boundaries
  • Error Correlation: Link errors to specific request flows
  • Performance Analysis: Identify bottlenecks in request processing

Trace Analysis

  • Service Dependencies: Visualize service interaction patterns
  • Latency Analysis: Identify slow components in request chains
  • Error Propagation: Understand how errors spread through system
  • Capacity Planning: Use trace data for scaling decisions

Logging Strategy

Structured Logging

  • JSON Format: Machine-readable log entries
  • Consistent Fields: Standardized log fields across services
  • Log Levels: Appropriate use of DEBUG, INFO, WARN, ERROR
  • Context Information: Include relevant request/user context

Log Aggregation

  • Centralized Collection: All logs in single searchable location
  • Log Retention: Appropriate retention policies for different log types
  • Log Analysis: Tools for searching and analyzing log patterns
  • Alert Integration: Logs trigger appropriate alerts

Metrics Collection

Application Metrics

  • Business Metrics: Key performance indicators for business logic
  • Technical Metrics: System performance and health indicators
  • Custom Metrics: Application-specific measurements
  • Metric Aggregation: Appropriate time-based aggregation

Infrastructure Metrics

  • Host Metrics: CPU, memory, disk, network for all hosts
  • Container Metrics: Docker container resource usage
  • Service Metrics: Per-service performance indicators
  • Dependency Metrics: External service performance impact