Ahmed Baha Eddine Alimi - B23-SD-01 [Lab08 + Bonus Task]

[3llimi]

Lab 08 — Metrics & Monitoring with Prometheus

What was done:

• Deployed Prometheus v3.9.0 via Docker Compose on shared logging network with 15-day / 10GB TSDB retention and persistent named volume
• Configured Prometheus to scrape 4 targets: app-python:8000, loki:3100, grafana:3000, localhost:9090 — all confirmed UP
• Instrumented Python app with prometheus_client==0.23.1 — added http_requests_total (Counter), http_request_duration_seconds (Histogram), http_requests_in_progress (Gauge), and devops_info_endpoint_calls_total (Counter) with method, endpoint, status_code labels
• Exposed /metrics endpoint in FastAPI app; metrics recorded automatically via HTTP middleware with finally block ensuring gauge always decrements
• Added Prometheus data source to Grafana (http://prometheus:9090) and built "Prometheus Dashboard" with 6 panels covering the RED method: Request Rate, Error Rate, p95 Latency, Active Requests, Status Code Distribution, and Uptime
• Configured production hardening: health checks on all services, resource limits (Prometheus 1G/1CPU, app 256M/0.5CPU), data retention policies, persistent volumes verified after docker compose down/up
• Rebuilt and pushed updated app image to Docker Hub after adding metrics endpoint — old image was serving 404 on /metrics

Bonus:

• Extended roles/monitoring Ansible role with Prometheus config template (prometheus-config.yml.j2) using Jinja2 loop over prometheus_scrape_targets variable list
• Added Grafana datasource auto-provisioning via grafana-datasources.yml.j2 template — both Loki and Prometheus sources provisioned on deploy
• Added Grafana dashboard auto-provisioning — exported app-dashboard.json copied via Ansible files/ and mounted into Grafana provisioning directory
• All Prometheus variables parameterised in defaults/main.yml: version, port, retention, scrape interval, resource limits, and scrape targets
• Confirmed idempotency — second run produces changed=0