Celery+redis+prom+grafana

.gitignore

@@ -8,8 +8,11 @@ plans
 .claude*
 CLAUDE.md
 temp*
-.next_features
-.envv
+!tempo
+!tempo/
+!**/tempo.yaml
+!**/tempo.yml
+.env*
 .infisical.json
 .task*
 task*

README.md

@@ -50,8 +50,12 @@ Licensed under the [GNU Affero General Public License v3.0](LICENSE).
   Mistral)
 - Celery + Redis task queue: receipt extraction, statement parsing, and report delivery all
   run as durable background tasks that survive process restarts
-- Prometheus metrics + Grafana dashboards for request latency, LLM latency, queue depth,
-  and rate-limit hits
+- Full observability stack: Prometheus metrics + Alertmanager (Slack alerts) + Grafana
+  dashboards + Loki log aggregation + Grafana Tempo distributed tracing (OpenTelemetry
+  auto-instrumented: FastAPI, SQLAlchemy, HTTPX, Redis, Celery; manual LLM spans)
+- Continuous Postgres backup via WAL-G to Cloudflare R2 (WAL archiving + daily base
+  backup, 7-day retention, point-in-time recovery)
+- PgBouncer connection pooling (session mode) between application and Postgres
 - Demo mode: public Bruce Wayne account, automatically reset every 30 minutes via Celery Beat
 - GDPR Article 17 compliance: full account erasure and selective data deletion endpoints
 
@@ -71,7 +75,9 @@ Licensed under the [GNU Affero General Public License v3.0](LICENSE).
 | Embeddings | Ollama (768-dim) + pgvector |
 | PDF generation | Puppeteer (headless Chromium via Next.js internal route) |
 | Email | Resend API |
-| Observability | Prometheus + Grafana (provisioned dashboards) + Flower (Celery monitor) |
+| Connection pooling | PgBouncer (session mode, scram-sha-256) |
+| Observability | Prometheus + Alertmanager (Slack) + Grafana + Loki + Promtail + Tempo (OTel traces) + Flower (Celery monitor) + Sentry |
+| Backup | WAL-G continuous backup to Cloudflare R2 (WAL archiving + daily base backup, 7-day retention, PITR) |
 | Secrets | Infisical (CLI-injected at runtime; no .env files in source) |
 | Containerization | Docker + Docker Compose |
 
@@ -169,13 +175,19 @@ configured as `ADMIN_EMAIL`. That account is auto-approved. All other accounts s
 |---|---|---|
 | Frontend | http://localhost:3000 | Next.js app |
 | Backend API | http://localhost:8000 | FastAPI; `/docs` only when `DEBUG=true` |
-| PostgreSQL | localhost:5432 | pgvector-enabled |
+| PostgreSQL | localhost:5432 | pgvector-enabled; custom image with WAL-G in prod |
+| PgBouncer | (internal only) | Connection pooler; prod only |
 | Redis | (internal only) | Task broker + cache; no host port exposed |
 | Celery worker | (no HTTP) | Processes receipt extraction and report tasks |
 | Celery Beat | (no HTTP) | Runs scheduled tasks (demo reset, recurring expenses) |
+| WAL-G backup | (no HTTP) | Daily base backup sidecar; prod only |
 | Flower | http://localhost:5555 | Celery task monitor (dev only) |
-| Prometheus | http://localhost:9090 | Metrics scraper |
+| Prometheus | http://localhost:9090 | Metrics scraper (SSH tunnel in prod) |
+| Alertmanager | http://localhost:9095 | Slack alert routing (SSH tunnel in prod) |
 | Grafana | http://localhost:3004 | Pre-provisioned dashboards; anonymous admin in dev |
+| Loki | (internal only) | Log aggregation; queried via Grafana |
+| Promtail | (internal only) | Docker log shipper to Loki |
+| Tempo | (internal only) | Distributed trace storage (OTel OTLP HTTP) |
 
 ---
 
@@ -268,16 +280,36 @@ Custom metrics exposed at `GET /metrics` (requires `Authorization: Bearer <METRI
 
 ### Grafana
 
-Pre-provisioned dashboards are in `grafana/provisioning/`. Grafana reads from the
-Prometheus datasource automatically. In dev, anonymous admin access is enabled. In
-production, set `GRAFANA_ADMIN_PASSWORD` and disable anonymous access.
+Pre-provisioned dashboards, datasources, and alert rules are in `grafana/provisioning/`
+(mounted read-only). Grafana starts with Prometheus, Loki, and Tempo datasources already
+configured. In dev, anonymous admin access is enabled. In production, set
+`GRAFANA_ADMIN_PASSWORD` and disable anonymous access.
 
-In production, Prometheus and Grafana are bound to `127.0.0.1` only. Access via SSH tunnel:
+Grafana Explore -> Loki: query logs across all containers with LogQL.
+Grafana Explore -> Tempo: search traces by service, duration, or status.
+Traces and logs are linked: a `traceID` field in structured logs becomes a clickable
+link to the Tempo trace.
+
+In production, all monitoring services are bound to `127.0.0.1` only. Access via SSH tunnel:
 
 ```bash
-ssh -L 9090:localhost:9090 -L 3004:localhost:3004 user@your-server
+ssh -L 9090:localhost:9090 -L 9095:localhost:9095 -L 3004:localhost:3004 user@your-server
 ```
 
+### Alertmanager
+
+Alert rules are in `prometheus/alert_rules.yml` (`BackendDown`, `HighErrorRate`).
+Alertmanager routes firing alerts to Slack via webhook. Set `SLACK_WEBHOOK_URL` in
+Infisical. The webhook URL is never stored in config files - it is injected at container
+startup via the entrypoint.
+
+### Distributed tracing (OpenTelemetry + Tempo)
+
+The backend and Celery worker auto-instrument FastAPI, SQLAlchemy, HTTPX, Redis, and
+Celery via the OpenTelemetry SDK. Spans are exported to Grafana Tempo over OTLP HTTP.
+Set `OTEL_ENDPOINT=http://tempo:4318` (already set in both compose files). Tracing is
+disabled when `OTEL_ENDPOINT` is empty.
+
 ---
 
 ## Demo mode
@@ -359,30 +391,46 @@ The prod stack includes a `cloudflared` sidecar that establishes a Cloudflare Tu
 Set `CLOUDFLARE_TUNNEL_TOKEN` in Infisical and configure your Cloudflare dashboard to
 route your domain to `http://frontend:3000`. No port 443 needs to be open on the host.
 
-### PostgreSQL backups
+### PostgreSQL backups (WAL-G + Cloudflare R2)
 
-A host-side systemd backup setup is in [`deploy/backup/`](deploy/backup/). Run once on
-the production host:
+Production uses WAL-G for continuous backup to Cloudflare R2.
+
+The production Postgres image (`postgres/Dockerfile`) starts with WAL archiving enabled:
 
-```bash
-sudo bash ./deploy/backup/install-spendhound-db-backup.sh
 ```
+-c wal_level=replica
+-c archive_mode=on
+-c archive_command=wal-g wal-push %p
+-c archive_timeout=60
+```
+
+Every WAL segment (up to 16 MB of changes, or at most 60 seconds of idle time) is
+compressed and uploaded to R2. A separate sidecar (`wal-g-backup`) runs a daily base
+backup at 02:00 UTC and prunes base backups older than 7 days.
 
-This installs a systemd timer that:
-- Runs daily at 03:15 UTC (`Persistent=true` catches missed runs)
-- Dumps the `db` container with `pg_dump`
-- Validates the archive with `pg_restore --list` and writes a SHA-256 checksum
-- Atomically renames the validated archive into place
-- Prunes old backups after the configured retention period
-- Uses `set -Eeuo pipefail`, `umask 077`, and `flock` to prevent overlapping runs
+Required Infisical secrets: `WALG_S3_PREFIX`, `WALG_R2_ENDPOINT`, `WALG_ACCESS_KEY_ID`,
+`WALG_SECRET_ACCESS_KEY`.
 
-Backups are stored in `/var/backups/spendhound`.
+Trigger the first backup manually after initial deploy (before the 02:00 UTC cron fires):
 
 ```bash
-sudo systemctl status spendhound-db-backup.timer --no-pager
-sudo journalctl -u spendhound-db-backup.service -n 50 --no-pager
+docker exec spendhound_walg_backup_prod /usr/local/bin/wal-g backup-push /var/lib/postgresql/data
+docker exec spendhound_walg_backup_prod /usr/local/bin/wal-g backup-list
 ```
 
+To restore (PITR):
+
+```bash
+# 1. Stop the stack and clear the data directory
+# 2. Fetch the most recent base backup
+docker exec spendhound_db_prod wal-g backup-fetch /var/lib/postgresql/data LATEST
+# 3. Create recovery.signal and set recovery_target_time in postgresql.conf
+# 4. Start Postgres - it replays WAL up to the target time
+```
+
+Maximum data loss is 60 seconds (set by `archive_timeout`). Retention: 7 daily base
+backups + all WAL since the oldest retained base backup.
+
 ### Single-worker constraint
 
 SpendHound is designed for `--workers 1`. The in-process `asyncio.Semaphore` that

alertmanager/alertmanager.yml

@@ -0,0 +1,40 @@
+global:
+  resolve_timeout: 5m
+
+route:
+  group_by: ['alertname', 'severity']
+  # Wait 30s after first alert fires before sending - collects related alerts into one message.
+  group_wait: 30s
+  # After the initial message, wait 5m before sending an update if new alerts join the group.
+  group_interval: 5m
+  # If an alert is still firing, re-notify every 4h so it doesn't silently stay broken.
+  repeat_interval: 4h
+  receiver: slack-alerts
+
+receivers:
+  - name: slack-alerts
+    slack_configs:
+      # Webhook URL is written to this file by the container entrypoint from $SLACK_WEBHOOK_URL.
+      # Same pattern as Prometheus reading the metrics bearer token from /run/metrics_token.
+      - api_url_file: /tmp/slack_webhook_url
+        channel: '#spendhound-alerts'
+        username: SpendHound
+        send_resolved: true
+        color: '{{ if eq .CommonLabels.severity "critical" }}danger{{ else }}warning{{ end }}'
+        title: '{{ if eq .Status "firing" }}[FIRING]{{ else }}[RESOLVED]{{ end }} {{ .CommonLabels.alertname }}'
+        text: |
+          {{ range .Alerts -}}
+          *{{ .Annotations.summary }}*
+          Severity: {{ .Labels.severity }}
+          {{ .Annotations.description }}
+          {{ if eq .Status "firing" }}Since: {{ .StartsAt.Format "2006-01-02 15:04 UTC" }}{{ else }}Resolved: {{ .EndsAt.Format "2006-01-02 15:04 UTC" }}{{ end }}
+          {{ end }}
+
+# Suppress downstream alerts when the root cause is already firing.
+# If BackendDown fires, HighErrorRate is a symptom - no need for two messages.
+inhibit_rules:
+  - source_match:
+      alertname: BackendDown
+    target_match:
+      alertname: HighErrorRate
+    equal: []

backend/app/celery_app.py

@@ -17,9 +17,31 @@
 
 from celery import Celery
 from celery.schedules import crontab
+from celery.signals import worker_init
 
 from app.config import settings
 
+if settings.sentry_dsn:
+    import sentry_sdk
+    from sentry_sdk.integrations.celery import CeleryIntegration
+
+    sentry_sdk.init(
+        dsn=settings.sentry_dsn,
+        environment=settings.sentry_environment,
+        traces_sample_rate=0.0,
+        send_default_pii=False,
+        integrations=[CeleryIntegration()],
+    )
+
+
+@worker_init.connect
+def _on_worker_init(**kwargs: object) -> None:
+    if settings.otel_endpoint:
+        from app.services.tracing import setup_tracing
+
+        setup_tracing(settings.otel_service_name, settings.otel_endpoint)
+
+
 celery_app = Celery(
     "spendhound",
     broker=settings.redis_url,
@@ -51,11 +73,17 @@
 )
 
 # Beat schedule — runs when `celery beat` is started (see docker-compose celery_beat service).
-# Wall-clock crontab (UTC :00 and :30) so the reset time is predictable and the
-# frontend can derive an accurate countdown without any server-side state.
 celery_app.conf.beat_schedule = {
+    # Wall-clock crontab (UTC :00 and :30) so the reset time is predictable and the
+    # frontend can derive an accurate countdown without any server-side state.
     "reset-demo-user-on-the-half-hour": {
         "task": "app.tasks.demo_tasks.reset_demo_user",
         "schedule": crontab(minute="0,30"),
     },
+    # 1st of each month at 00:20 UTC - matches the former systemd timer schedule.
+    # Fans out one deliver_monthly_report task per eligible user.
+    "dispatch-monthly-reports": {
+        "task": "app.tasks.report_tasks.dispatch_monthly_reports",
+        "schedule": crontab(hour=0, minute=20, day_of_month=1),
+    },
 }

backend/app/config.py

@@ -152,6 +152,22 @@ class Settings(BaseSettings):
         default="",
         description="Bearer token required to scrape /metrics. Empty = endpoint blocked. Set in Infisical for prod.",
     )
+    sentry_dsn: str = Field(
+        default="",
+        description="Sentry DSN for error tracking. Empty = Sentry disabled. Set in Infisical for prod.",
+    )
+    sentry_environment: str = Field(
+        default="production",
+        description="Sentry environment tag sent with every event (production / development).",
+    )
+    otel_endpoint: str = Field(
+        default="",
+        description="OTLP HTTP base URL for OpenTelemetry traces (e.g. http://tempo:4318). Empty = tracing disabled.",
+    )
+    otel_service_name: str = Field(
+        default="spendhound",
+        description="OTel service name shown in Grafana Tempo. Set per container: spendhound-api, spendhound-worker.",
+    )
 
     debug: bool = Field(default=False, description="Enable debug mode")
     cors_origins: list[str] = Field(

backend/app/main.py

@@ -21,6 +21,28 @@
 from app.services.cache import close_redis, get_celery_queue_depth, init_redis
 from app.services.metrics import RATE_LIMIT_HITS_TOTAL, RECEIPT_QUEUE_DEPTH, classify_limit_type
 
+if settings.sentry_dsn:
+    import sentry_sdk
+    from sentry_sdk.integrations.fastapi import FastApiIntegration
+    from sentry_sdk.integrations.starlette import StarletteIntegration
+
+    sentry_sdk.init(
+        dsn=settings.sentry_dsn,
+        environment=settings.sentry_environment,
+        traces_sample_rate=0.0,
+        send_default_pii=False,
+        integrations=[StarletteIntegration(), FastApiIntegration()],
+    )
+
+if settings.otel_endpoint:
+    from app.services.tracing import setup_tracing
+
+    setup_tracing(settings.otel_service_name, settings.otel_endpoint)
+    # FastAPI instrumentation is deferred to create_app() via instrument_fastapi()
+    # because the FastAPI app must exist before middleware can be attached.
+    # Calling FastAPIInstrumentor().instrument() here (before app creation) patches
+    # FastAPI.__init__ globally and conflicts with the Prometheus Instrumentator.
+
 logger = structlog.get_logger(__name__)
 
 
@@ -85,6 +107,11 @@ async def _rate_limit_handler(request: Request, exc: Exception) -> Response:
 
     app.add_exception_handler(RateLimitExceeded, _rate_limit_handler)
 
+    if settings.otel_endpoint:
+        from app.services.tracing import instrument_fastapi
+
+        instrument_fastapi(app)
+
     app.add_middleware(
         CORSMiddleware,
         allow_origins=settings.cors_origins,
@@ -93,6 +120,27 @@ async def _rate_limit_handler(request: Request, exc: Exception) -> Response:
         allow_headers=["*"],
     )
 
+    if settings.sentry_dsn:
+
+        @app.middleware("http")
+        async def _attach_sentry_user(request: Request, call_next):  # type: ignore[no-untyped-def]
+            auth = request.headers.get("authorization", "")
+            if auth.startswith("Bearer "):
+                try:
+                    from jose import jwt as _jwt
+
+                    payload = _jwt.decode(
+                        auth.removeprefix("Bearer "),
+                        settings.jwt_secret,
+                        algorithms=[settings.jwt_algorithm],
+                    )
+                    user_id = payload.get("sub")
+                    if user_id:
+                        sentry_sdk.set_user({"id": user_id})
+                except Exception:
+                    pass
+            return await call_next(request)
+
     # Starlette 0.40+ adds _IncludedRouter objects to app.routes when include_router()
     # is used. prometheus-fastapi-instrumentator unconditionally accesses route.path on
     # every route, but _IncludedRouter has no .path → AttributeError on every request.

backend/app/services/metrics.py

@@ -17,6 +17,7 @@
 import time
 from collections.abc import AsyncGenerator
 
+from opentelemetry import trace as _otel_trace
 from prometheus_client import Counter, Gauge, Histogram
 
 from app.services.llm.base import BaseLLMProvider, LLMConfig, Message
@@ -58,13 +59,22 @@ def __init__(self, inner: BaseLLMProvider, provider_name: str) -> None:
         self._provider_name = provider_name
 
     async def complete(self, messages: list[Message], config: LLMConfig | None = None) -> str:
-        start = time.monotonic()
-        try:
-            return await self._inner.complete(messages, config)
-        finally:
-            LLM_RESPONSE_SECONDS.labels(provider=self._provider_name).observe(
-                time.monotonic() - start
-            )
+        tracer = _otel_trace.get_tracer(__name__)
+        with tracer.start_as_current_span(
+            "llm.complete",
+            attributes={"gen_ai.system": self._provider_name},
+        ) as span:
+            start = time.monotonic()
+            try:
+                return await self._inner.complete(messages, config)
+            except Exception as exc:
+                span.record_exception(exc)
+                span.set_status(_otel_trace.Status(_otel_trace.StatusCode.ERROR))
+                raise
+            finally:
+                LLM_RESPONSE_SECONDS.labels(provider=self._provider_name).observe(
+                    time.monotonic() - start
+                )
 
     async def stream(
         self, messages: list[Message], config: LLMConfig | None = None