Loki, Promtail, Grafana, and Prometheus (PLG) Logging Stack

This repository contains a standalone, containerized logging and metric aggregation stack designed to collect, parse, and visualize logs and metrics from your applications.

Services & Ports

• Grafana: http://localhost:4000 (Visualization dashboard, mapped from internal port 3000)
• Loki: http://localhost:3100 (Log storage backend)
• Promtail: http://localhost:9080 (Log scraping & shipping agent)
• Prometheus: http://localhost:9095 (Performance metrics collector, mapped from internal port 9090)

---

Directory Structure

• docker-compose-logs.yml: Orchestrates Loki and Prometheus.
• docker-compose-promtail.yml: Orchestrates Promtail.
• docker-compose-grafana.yml: Orchestrates Grafana with persistent storage.
• loki-config.yml: Grafana Loki server and retention configurations.
• promtail-config.yml: Promtail log scraping definitions and processing pipeline.
• prometheus.yml: Prometheus metrics scraper targets.
• .env.example: Template for stack environment variables.

---

Getting Started

1. Create the Shared Docker Network

All application containers and logging containers must connect to the same external Docker network log-network to allow internal hostname resolution.

docker network create log-network

2. Configure Environment Variables

Create a local .env file from the template:

cp .env.example .env

Update the .env file with any customized basic auth credentials, root URLs, or storage backend settings (e.g. S3/GCS).

3. Start the Logging Stack

Run all services together using:

docker compose -f docker-compose-logs.yml \
               -f docker-compose-promtail.yml \
               -f docker-compose-grafana.yml \
               up -d

---

Integrating Your Application

This logging stack is completely framework-agnostic. Whether your application is built in Go, FastAPI, Flask, Django, or any other framework, you can integrate it by following these steps:

1. Write Application Logs to Disk: Configure your application to write logs to a file (e.g., app.log, django.log, or server.log) inside a directory (e.g., /app/logs).

2. Mount the Logs Directory to Host: In your application's docker-compose.yml, mount a host directory (e.g., ./logs) to the container logs folder and join the external network log-network:

   services:
     your-app:
       volumes:
         - ./logs:/app/logs
       networks:
         - log-network
   
   networks:
     log-network:
       external: true

3. Promtail Scraping: Promtail mounts the host ./logs directory to /var/log/app and is configured to scrape all *.log files under /var/log/app/. Any log file ending in .log written to ./logs/ on the host will automatically be scraped and shipped to Loki.

---

Log Rotation Integration

To prevent logs from filling up the disk, log rotation should be configured at the application level. Because this stack is framework-agnostic, you can configure rotation using the standard library or popular logging utilities for your platform:

• Python (FastAPI / Flask / Django): Use Python's built-in logging.handlers.RotatingFileHandler.
• Go: Use a library like lumberjack (e.g., github.com/natefinch/lumberjack).

Example Configuration (Python)

"handlers": {
    "file": {
        "level": "INFO",
        "class": "logging.handlers.RotatingFileHandler",
        "filename": "/app/logs/app.log",
        "maxBytes": 20 * 1024 * 1024,  # Roll over at 20 MB
        "backupCount": 3,              # Keep up to 3 backup files
        "formatter": "json",
    }
}

Example Configuration (Go with Lumberjack)

log.SetOutput(&lumberjack.Logger{
    Filename:   "/app/logs/app.log",
    MaxSize:    20, // megabytes before rollover
    MaxBackups: 3,  // keep up to 3 backups
})

Promtail Compatibility

Regardless of the language or framework used, Promtail handles rotated log files gracefully:

1. Ignoring Rotated Backups: When logs roll over, backup files are renamed (e.g. app.log.1, app.log.2). Because Promtail targets __path__: /var/log/app/*.log, these backup files are ignored by default, preventing duplicate lines in Loki.
2. Handling Compressed Backups: If your rotation logic is set up to compress old archives (e.g. app.log.1.gz), the Promtail pipeline automatically filters and drops them using a relabeling configuration:

   relabel_configs:
     - source_labels: [__path__]
       regex: '.*\.gz'
       action: drop

---

Stopping the Services

To tear down the containers:

docker compose -f docker-compose-logs.yml \
               -f docker-compose-promtail.yml \
               -f docker-compose-grafana.yml \
               down